An architecture study of a due-diligence evidence system for counterparty risk.
The deliverable is the decision record — this README's code is evidence that the
decisions work, not the product. See docs/README.md for the problem and what the
data showed.
docs/README.md— the problem and what the data showeddocs/00-charter.md— design baseline and build sequencedocs/adr/— decisions and their justificationdocs/decision-journal.md— reasoning, including what was uncertaindocs/06-data-quality-findings.md— measured figures with provenance
The corpus is frozen. GLEIF, SEC and OFAC were captured on 2026-08-01; nothing
is scheduled, so every artefact is now overdue under collectors gaps and a missed
delta window is permanent data loss, not a delay. New captures are manual only.
Companies House PSC was captured once on 2026-08-02 — the day after the freeze — to feed the single-entity vertical slice. It sits on disk but is not part of a scheduled cadence.
What is built: four archival collectors that fetch, hash and manifest (GLEIF, SEC,
OFAC, Companies House), a classified error taxonomy, a run log, and a generated
status report. One exploration script (scripts/explore/) and one vertical-slice
extraction function (src/slice/gleif_entity.py).
What is not built: no assertion store, no database, no sanctions matcher, no memo
generator, no retrieval, no serving layer. There is no golden-case suite on disk
(eval/golden/ does not exist). This is a V0/V1 codebase at the end of its
data-engineering phase.
Sizes are bytes on disk under data/raw/, summed from the archived files (not the
publisher's declared size).
| Source | Format on disk | Contributes | Size on disk |
|---|---|---|---|
| GLEIF | 12 CSV-in-ZIP delta windows (lei2, rr, repex × IntraDay/LastDay/LastWeek/LastMonth) + publishes index | Legal entities (LEI), parent-child relationships, reporting exceptions | 49,337,756 B (47.1 MiB) |
| SEC | submissions.zip (JSON-in-ZIP) + company_tickers.json + company_tickers_exchange.json |
US filer registry (CIK), ticker→CIK crosswalk | 1,556,281,809 B (1,484.2 MiB) |
| OFAC (sanctions) | SDN.XML, CONSOLIDATED.XML + opensanctions-index.json (mirror transport) |
Sanctions designations and aliases; consolidated non-SDN list | 31,794,974 B (30.3 MiB) |
| Companies House | persons-with-significant-control-snapshot-*.zip (JSONL-in-ZIP) + listing page |
UK persons with significant control, including natural persons | 2,186,109,784 B (2,084.8 MiB) |
| Total | 3,823,524,323 B (≈3.6 GiB) |
Not captured: Companies House accounts (iXBRL filing images) and SEC
companyfacts (tagged financial facts). Neither is on disk; the Companies House
collector is configured for the PSC product only, and SEC companyfacts.zip was not
fetched.
uv sync # environment + the `collectors` console script
uv run collectors run --all --dry-run # what would be fetched, with publisher sizes
uv run collectors run --source gleif # capture one source; choices: gleif, sec, sanctions, companies_house
uv run collectors status # regenerate reports/collection-status.md
uv run collectors gaps # exits 3 when a capture is overdue
uv run pytest # failure-injection testsSOURCE_USER_AGENT must be a real contact address — SEC requires one and the
collector refuses to run with the placeholder (AUTH_FAILURE). Set it in
.claude/settings.local.json for your own runs; the project-level
.claude/settings.json keeps the placeholder for the team.
data/raw/ is gitignored (.gitignore line 2). Captures are not in the repository
and must be produced locally with the commands above.
Every architectural decision is recorded as an ADR before the code that implements
it, justified against measured data rather than intuition. Measurements are taken at
the version whose decision they change, and every quantity carries a provenance
marker (DERIVED / ASSUMED / MEASURED / EXTERNAL_CONSTRAINT). Findings are
superseded with a new entry, never overwritten. Where a source and its documentation
disagree, the data is right and the disagreement is itself a finding.