Add babel-explorer: a CLI for querying Babel cross-references via DuckDB and NodeNorm

[gaurav]

babel-explorer is a CLI for asking Babel why two identifiers are considered the same thing. It reads Babel's intermediate Parquet files through DuckDB and, optionally, enriches the results with labels from NodeNorm. BabelDownloader handles caching and freshness, BabelXRefs handles querying, NodeNorm handles labels, and cli.py wires them together with Click.

Supersedes #1, which GitHub closed and refused to reopen after this branch's history was rewritten.

Closes #12.

What's here

Three commands:

• xrefs — cross-references for one or more CURIEs. --recurse expands transitively through a single WITH RECURSIVE DuckDB query; --paths shows the shortest paths connecting the given CURIEs; --labels adds NodeNorm labels and Biolink types.
• ids — identifier records from Identifiers.parquet, with --labels.
• test-concord — compare a proposed concordance change against NodeNorm's current cliques.

xrefs and ids also take --format json|tsv|csv for machine-readable output. --paths is console-only, and both of its preconditions — a console format, and at least two CURIEs — are checked before anything is downloaded. Getting either wrong otherwise costs a multi-gigabyte Concord.parquet download and a full recursive query before the run is rejected, because --paths implies --recurse.

Failures from the two services this tool talks to are reported as errors rather than tracebacks. MissingBabelFileError explains that a release does not publish the DuckDB files; requests.RequestException reaching the top means NodeNorm, since the downloader handles its own network failures.

Configuring which Babel release to query

A Babel release is addressed as a releases directory plus a version, which is how both the public and internal trees are actually laid out — one subdirectory per release, plus a latest/ symlink:

Variable	CLI option	Default
BABEL_RELEASES_URL	--babel-releases-url	https://stars.renci.org/var/babel/
BABEL_VERSION	--babel-version	latest
—	--babel-url	(overrides both)

Pinning a release is a one-word change rather than a URL edit, which matters because pinning is the honest fix for a NodeNorm version mismatch. Precedence runs flag > environment variable > .env > built-in default, and only public URLs are committed.

--babel-url takes a complete URL for a tree that does not follow that layout. It is command-line only, with no envvar=, deliberately: two variables already feed the composed URL, and a third that silently outranked both would make "which release am I actually querying?" unanswerable from the environment alone. cli() warns if the pre-refactor BABEL_URL is still set, so a stale .env fails loudly instead of silently pointing somewhere else.

compose_babel_url lives in core/downloader.py rather than cli.py because tests/constants.py needs the identical composition and must not import Click to get it.

The committed template is env.default — visible in a plain ls, unlike a dotfile.

Babel version handling

The release is resolved from VERSION.txt, falling back to the final URL path segment — which under this scheme is exactly BABEL_VERSION, so a pinned release still resolves when VERSION.txt is unreachable, while latest yields None as before.

BABEL_LOCAL_DIR holds one release at a time. When it changes, only last_checked is cleared from the .meta sidecars, so the existing ETag path re-checks each file and re-downloads only what actually changed — the Parquet files are never deleted, and an unchanged file costs one HEAD rather than a fresh multi-gigabyte download. The .babel-version marker records the release the server resolved to rather than the one requested, so latest and an equivalent pinned version share a cache instead of thrashing it.

That marker is written after the cache catches up, not when the change is spotted. It claims "the local cache holds this release", which is only true once every cached file has been re-validated against it, so _write_version_marker_if_synced() stamps it once no .meta sidecar is still missing its last_checked. Writing it up front would leave a run interrupted between Concord.parquet and Identifiers.parquet with a marker naming the new release over a half-old cache, and the next run would see it match and skip the refresh entirely. The cost is that a cached file nobody asks for holds the marker back indefinitely, at one HEAD per run; that is the honest answer, since the file really is still from the previous release.

--check-download never suppresses re-checks within a release, not across one. _is_within_freshness() therefore tests for a missing last_checked before the float("inf") shortcut — reversed, never would hand back the previous release's Parquet with no network call at all, under a marker naming the new release, and nothing would ever notice.

--labels refuses to mix a NodeNorm built from one release with cross-references from another, since the result would be silently wrong rather than obviously wrong. --allow-version-mismatch overrides it; pinning BABEL_VERSION fixes it properly. --recurse never consults NodeNorm, so it does not trigger the check.

Partial downloads cannot corrupt the cache

A corrupt Parquet here is permanent: whatever lands on disk gets stamped with the correct remote ETag and then passes every later freshness check. Five routes to that are closed:

• A .tmp left by a killed process is discarded before each download rather than resumed. Resume is by byte offset, the only way to reach the download at all is that the remote bytes changed, and an orphaned .tmp carries no record of which version its bytes came from. Cleanup catches BaseException, so Ctrl-C leaves nothing resumable behind.
• In-run resumes send If-Range, so a file rebuilt mid-download restarts instead of splicing two versions together.
• HTTP 416 counts as "already complete" only once the local size matches the remote Content-Length — 416 is also what a server returns when the file shrank below the resume offset.
• A stream ending short of Content-Length raises IncompleteDownloadError and is retried, rather than being promoted as complete.
• A failed HEAD returns "unknown", not "unchanged", and no longer refreshes last_checked. One flaky HEAD could otherwise pin the previous release's Parquet as freshly validated for the whole freshness window.

.tmp files are deleted in two places on purpose; CLAUDE.md records which one is the safety guarantee and which is housekeeping, so neither gets removed later as redundant.

Querying

DuckDB connections are ephemeral and in-memory, but "in-memory" is not "touches no disk": a larger-than-memory query spills, and DuckDB's default temp_directory is .tmp in the current working directory. The recursive expansion materialises the whole Concord relation, so every connection goes through BabelXRefs._connect(), which points temp_directory at <BABEL_LOCAL_DIR>/duckdb-spill/ — the directory the user already chose to hold multi-gigabyte files, rather than wherever they happened to be standing.

What it deliberately does not do

• Resume a download across runs. A leftover .tmp cannot be proven to belong to the file being fetched, so it is discarded. Making it safe means persisting the validator alongside the .tmp; scoped in Resume an interrupted download across runs, safely #15.
• Enforce the no-private-URLs rule outside env.default. TestCommittedConfigTemplate guards the template, but the original leak came through a default value in cli.py. A tree-wide scan is Scan for committed secrets in CI: non-public Babel URLs and Google Sheet IDs #25.
• Reuse a DuckDB connection across queries. A known performance cost, deferred to Reuse a single DuckDB connection per BabelXRefs instance #13. (Batching NodeNorm lookups did ship — normalize_curies() collapses a clique into one request per 100 CURIEs, which is why this closes Batch NodeNorm lookups to reduce N round-trips when --labels is set #12.)

Known limitations at v0.1.0

This ships knowingly non-functional against its own defaults, so people can try the tool now rather than after the Babel side catches up. The CLI says so clearly rather than failing mid-download.

• Public Babel releases do not publish duckdb/Concord.parquet or duckdb/Identifiers.parquet. Publish the DuckDB Parquet files in public Babel releases #16 — must close before v1.0.0.
• NodeNorm dev reports Babel 2025sep1 against public 2025dec11, so --labels fails the skew check against the defaults. Update NodeNorm dev to a current Babel so --labels works out of the box #17
• CI therefore cannot run the 28 Parquet integration tests. CI cannot run the Parquet integration tests #18
• NodeNorm integration tests fail rather than skip when the API is unreachable. NodeNorm integration tests fail instead of skipping when the API is unreachable #19

Testing

261 unit tests and 13 NodeNorm integration tests pass. 28 Parquet-dependent integration tests skip without a Babel release publishing the files — that is the expected result, not a broken environment.

Verified end to end against a Translator releases tree: the composed URL resolves 2026jul22, downloads Concord.parquet, writes a correct .babel-version marker, and returns cross-references for MONDO:0004979. The version-marker sequencing was verified separately by simulating a 2025dec11 → 2026jul22 change under --check-download never: both Parquet files are re-fetched, and the marker flips only after the second one lands.

Note that not slow is not a promise of "small". Concord.parquet is 4.6 GB in 2026jul22 and its tests are not marked slow, because excluding them would leave the non-slow integration set covering nothing that touches real data. This matters for sizing #18.

Nothing is blocking this merge

Everything outstanding is tracked in #12, #13, #15, #16, #17, #18, #19 and #25. None of it makes what ships here wrong — #16 and #17 constrain what the defaults can do, and both are stated plainly in the README, CHANGELOG.md and the CLI's own error messages.

Review history — three review rounds and a history rewrite. Kept for anyone tracing why a particular line looks the way it does; the durable conclusions are in the code comments, CLAUDE.md and the sections above.

First review round found six defects, each fixed in its own commit with a regression test: --recurse triggered the NodeNorm version check it no longer needed; ids --labels silently dropped the NodeNorm label because Identifiers.parquet's own label column overwrote it; a version change forced a full re-download instead of an ETag re-check; a stale .tmp could splice two releases into one corrupt Parquet; the HTTP 416 fast path persisted the error response's headers as the file's metadata; and the integration skip probe did not normalise a slashless BABEL_URL, so it probed .../latestduckdb/... and silently skipped the entire integration suite.

Second review round found six more, all but one in the downloader's resume path — the five now described under "Partial downloads cannot corrupt the cache", plus record_to_dict dropping Identifiers.parquet's own label column whenever it was empty, because the omit-when-absent rule matched on a label name suffix rather than the three NodeNorm-derived field names. That made label present on some rows of a json/tsv/csv run and missing on others.

The first round's .tmp fix was narrower than it looked: it swept .tmp files only on a Babel version change, which does not cover a content change within one release or a rebuild in place. The second round replaced it with an unconditional discard before every download. Both deletes are kept, for different reasons, which is why CLAUDE.md spells out which is which.

Copilot review found four threads and one suppressed comment, all five real. Two were in the resume path and are folded into the rules above: a retry sent a bare Range whenever no validator was known, which is exactly the case where a splice cannot be detected afterwards; and a 416 whose HEAD carried no Content-Length was read as "already complete" when it is equally the answer for a file that shrank. Two existing tests had encoded the first behaviour by seeding a partial file with no validator — a state get_downloaded_file never produces — and were rewritten to reach the resume the way production does. The others: ten get_curie_xref.cache_clear() calls left over from an lru_cache that no longer exists, raising AttributeError in integration tests that skip and so never ran; BABEL_ALLOW_VERSION_MISMATCH missing from env.default, which survived because the test guarding that rule listed the settings by hand rather than reading them off the CLI; and this PR's own batched NodeNorm lookups still listed as future work.

Third review round found seven, in five commits. Two were the same defect from opposite ends and are the reason the version-marker rules above are stated so explicitly: _is_within_freshness() returned True on float("inf") before looking at last_checked, so the whole cross-release refresh was a no-op under --check-download never; and the marker was stamped before anything had been re-downloaded, so an interrupted refresh looked complete to the next run. Both let Concord.parquet and Identifiers.parquet be read together across two Babel builds, which is precisely what the marker exists to prevent. The other three code findings: xrefs --paths rejected a single CURIE only inside _print_paths, after the 4.6 GB download; NodeNorm's deliberately-uncaught HTTP errors reached the user as a stack trace; and DuckDB spilled into the working directory.

Two were documentation drifting from the code — CLAUDE.md described @functools.lru_cache in three places where none exists (disk caching by ETag, cached_property, and per-instance dicts respectively), and CHANGELOG.md carried a hard-coded test count that was already wrong, in a repo whose own CLAUDE.md says not to record them because they drift silently and then mislead.

History rewrite (2026-09-01). The internal releases URL had been the hardcoded default from the initial commit until it moved into .env, leaving it in 178 of 180 commits across six branches of a public repository. Every commit was rewritten to remove it, verified by four independent checks before anything was pushed, with every changed line confirmed to be that substitution and nothing else. GitHub then refused to reopen the affected PRs, because their original head commits no longer exist — hence #20 here, and #21-#24 replacing #4, #6, #7 and #11. The rewrite does not un-publish anything: GitHub retains pre-rewrite objects reachable by SHA, and the tree the URL pointed at still serves 200 unauthenticated. Both are being handled outside this repo.

Verifying the release caught a stale claim. Concord.parquet is 4.6 GB in 2026jul22, not the ~626 MB a fixture docstring had claimed, so pytest -m "integration and not slow" — documented in two places as avoiding 2 GB+ downloads — would in fact pull 4.6 GB. The docs were corrected rather than the marker, and hard byte figures were dropped from the fixture docstrings for the same reason the repo already gives for test counts: they drift silently and then mislead.

[Copilot]

Pull request overview

Introduces the initial Babel Explorer CLI for querying Babel Parquet data through DuckDB with optional NodeNorm enrichment.

Changes:

• Adds xrefs, ids, and test-concord commands with structured output.
• Implements release-aware downloading, caching, recursive queries, paths, and batched NodeNorm lookups.
• Adds packaging, CI, documentation, and comprehensive tests.

Reviewed changes

Copilot reviewed 21 out of 26 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
.github/workflows/ci.yml	Adds lint and test workflows.
.gitignore	Ignores generated data and local configuration.
.python-version	Selects Python 3.11.
CHANGELOG.md	Documents the initial release.
CLAUDE.md	Adds repository development guidance.
FUTURE.md	Lists tracked future work.
README.md	Documents setup, configuration, and usage.
env.default	Provides public endpoint defaults.
pyproject.toml	Configures packaging, dependencies, and tooling.
src/babel_explorer/__init__.py	Initializes the package.
src/babel_explorer/cli.py	Implements Click commands and configuration.
src/babel_explorer/core/__init__.py	Initializes the core package.
src/babel_explorer/core/babel_xrefs.py	Implements DuckDB querying and graph traversal.
src/babel_explorer/core/downloader.py	Implements downloads, caching, and release handling.
src/babel_explorer/core/nodenorm.py	Implements NodeNorm lookups and batching.
src/babel_explorer/formatting.py	Implements console and structured output.
tests/__init__.py	Initializes the test package.
tests/conftest.py	Adds shared integration fixtures.
tests/constants.py	Defines shared test configuration.
tests/data/valid_curies.txt	Supplies integration-test CURIEs.
tests/test_babel_xrefs.py	Tests querying and traversal.
tests/test_cli.py	Tests CLI behavior and configuration.
tests/test_downloader.py	Tests download and cache behavior.
tests/test_formatting.py	Tests output formatting.
tests/test_nodenorm.py	Tests NodeNorm integration and batching.
uv.lock	Locks project dependencies.

Suppressed comments (1)

src/babel_explorer/core/downloader.py:449

• When the HEAD response omits Content-Length, this condition falls through and treats the temporary file as complete without verifying its size. A 416 alone does not prove completeness, so an unavailable length must also discard the partial file and restart.

                        remote_length = head.headers.get("Content-Length")
                        if (
                            remote_length is not None
                            and int(remote_length) != resume_byte_pos
                        ):

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.