Document the release artifacts and version conventions that downstream tools depend on

[gaurav]

babel-explorer is a CLI that reads Babel's intermediate DuckDB/Parquet files to answer "why does Babel consider these two identifiers the same thing?" — a debugging tool for when a clique looks wrong. Writing its agent documentation this week turned up a set of things Babel produces and downstream code depends on, but that Babel does not document anywhere we could find. Each one below is currently encoded as an assumption in babel-explorer's source, verified empirically against real releases rather than read from a spec.

We are not asking for much writing — mostly for existing behaviour to be written down once so consumers stop reverse-engineering it.

What we could not find

Searched README.md, AGENTS.md and releases/ARTIFACTS.md.

1. The duckdb/*.parquet files are undocumented. releases/ARTIFACTS.md describes archived build reports but not duckdb/Concord.parquet, duckdb/Identifiers.parquet or duckdb/Metadata.parquet. babel-explorer reads all three. What we assume, from inspecting real files:

• Concord.parquet — one row per asserted cross-reference, columns filename, subj, pred, obj, where filename is the concord file the edge came from. That provenance column is the single most useful thing in the file for our purposes, and we would like to be able to rely on it.
• Identifiers.parquet — per-identifier records whose columns vary between releases, so we read them dynamically rather than by name.
• Metadata.parquet — small; we download it but do not currently interpret it. What is in it?

Even a short table of columns and their meanings, and a statement of which are stable, would let us delete a lot of defensive code.

2. Which files a release is guaranteed to publish. Release 2026jul22 publishes a 4.6 GB Concord.parquet with no Identifiers.parquet beside it. We had assumed "does this release have the Parquet files?" was one question; it is not, and finding that out cost a round of debugging. If the set is intentionally variable, saying so is enough. Public releases currently publish none of them, which is babel-explorer#16.

3. VERSION.txt. Undocumented, but we parse it: we expect a line matching Babel <release>, e.g. Babel 2026jul22, at the release root. We fall back to the final URL path segment when it is missing, for older trees. Is the format stable? Is it guaranteed present going forward?

4. Release naming and latest. We rely on releases being dated directories (2025dec11, 2026jul22) under a releases root, with latest/ resolving to the current one. None of that is written down. We treat a change in what latest resolves to as a cache-invalidation event, so the guarantees here matter to us more than they might look.

5. "Concord" is used but never defined. AGENTS.md refers to concord files repeatedly and README.md does not mention them at all. A one-paragraph definition — asserted cross-reference edges that feed clique construction, distinct from the compendia that come out — would help anyone reading the pipeline.

That definition matters to us specifically because of this line in your AGENTS.md:

Answer clique-membership questions from a finished build … never from the concords that fed it.

We think that is exactly right, and babel-explorer is deliberately on the other side of it: it reports the evidence Babel read, and defers to Node Normalization for the verdict. But a reader could easily take our tool for a clique oracle. Making the concord/clique boundary explicit upstream would help us keep that distinction honest downstream.

Documents in babel-explorer worth reading

For whoever picks this up — these show concretely what a consumer ends up assuming:

• AGENTS.md → "Domain context" (currently in PR #31) — our summary of Babel/clique/concord/conflation for a coding agent, with the parts we could not verify marked as such. The gaps flagged there are the shortest statement of what this issue is asking for.
• docs/Downloading.md — how we cache a release and detect that it changed. Most of it exists because the release layout and version semantics are informal.
• core/babel_xrefs.py — the CrossReference and IdentifierRecord dataclasses are our de facto schema for the two Parquet files.
• core/downloader.py — resolve_babel_version() is our VERSION.txt contract, including the fallback.
• tests/conftest.py — _download_or_skip() exists solely because file availability varies per release.

Related, but probably NodeNormalization's to answer

Noting for completeness rather than asking here: NodeNormalization's README does not document its /status endpoint or the babel_version it reports, nor the conflate / drug_chemical_conflate parameters. babel-explorer depends on babel_version to detect skew between the NodeNorm it queries and the Babel release it reads, and sends both conflation flags on every call. Happy to file that separately there if useful.