Skip absent test dependencies, and stop unit runs deleting the integration cache

[gaurav]

Running the full test suite against a Babel release that actually publishes the DuckDB Parquet files produced 4 failures and 4 errors, and not one of them was a defect in the code under test. Chasing those turned up a fifth problem that had been corrupting the runs themselves: any pytest invocation deleted the integration cache, including the -m "not integration" loop you run constantly while editing.

This PR makes the suite report absent dependencies honestly and stop sabotaging itself, so that a red run means a real problem again. It is tests and documentation only — no change to src/.

Closes #19.

What's here

A file being published is settled per file, not per session. A release can publish Concord.parquet without Identifiers.parquet — the current one does exactly that, serving a 4.6 GB Concord with no Identifiers beside it — so "does this release have the Parquet files?" is not one question. Every DuckDB file now goes through _download_or_skip(), which skips on the MissingBabelFileError the downloader already raises for a 404 on a duckdb/ path. shared_downloader is left answering only what is genuinely session-wide — is the server reachable — and probes the release root rather than a named file, so it cannot be misread as a publication check that forgot to check anything.

A unit-test run no longer deletes the integration cache. pytest_sessionfinish removed data/test unconditionally, and that path is fixed rather than per-run, so any session did it. A pytest -m "not integration" in one terminal silently destroyed a multi-gigabyte download in progress in another, and the integration run simply started over — which reads as a downloader defect rather than an unrelated pytest two windows away. The cleanup now runs only when the session actually selected integration tests. The unit suite never creates or reads that directory, so it has no business removing it.

An unreachable NodeNorm skips instead of failing. The nodenorm fixture only constructed a client, so a connection error surfaced inside each test body and failed it. CI runs integration-test on every push to main and on a weekly cron, so a RENCI outage turned the build red for reasons unrelated to any code change — the exact failure the Babel-side skip was written to avoid. The fixture now probes status once and skips on a RequestException.

NCIT:C55060 is removed from the test CURIEs. It has no rows at all in Concord.parquet as of Babel 2026jul22, so the four tests parametrized over it asserted len(results) > 0 against an empty list. NodeNorm still resolves it — as "Hypertension, CTCAE", a biolink:PhenotypicFeature with a clique of 2, from Babel 2025sep1 — so this is not a typo or a retired identifier, and something changed between those two releases. That question is #29, not this PR's: whether Babel was right to drop the cross-references is a question about Babel's data, and if a good one went for a bad reason it belongs upstream. valid_curies.txt keeps a two-line pointer so the short list explains itself.

Identifiers.parquet is checked for being a Parquet file, not for being large. The assertion was size > 2 GB, a figure picked against a release long since superseded. It would fail a legitimately smaller future release while passing a 2.5 GB truncated download. It now checks the PAR1 marker at both ends, which tests what actually goes wrong: a download cut short, or an error page saved under a .parquet name.

What it produces

Against a release that publishes the Parquet files, the full suite is 299 passed, 3 skipped, 0 failures, 0 errors (16m31s, dominated by the 4.6 GB download). It was 296 passed / 4 failed / 4 errors / 28 skipped before. The 25 tests that turned from skips into passes are the Parquet-dependent ones that had never actually run — they now exercise real data, including the DuckDB query paths and the recursive expansion. Against a release without the files, the suite still skips cleanly, as documented.

Nine new unit tests cover the three guards, because this whole class of bug is otherwise invisible: these code paths only execute during an integration run, and CI's integration job skips, so nothing would notice a guard being dropped. It is the same blind spot that hid ten stale get_curie_xref.cache_clear() calls until a review found them by reading rather than running. Each guard was confirmed non-vacuous by removing it and watching the test fail.

Four things that cost real time are now in CLAUDE.md: that a release can publish one DuckDB file without the other; that pytest_sessionfinish deletes data/test so a full run re-pays the download; that two concurrent integration runs still clash, with how to detect, avoid and recover; and that addopts carries -n auto, so a serial run needs -n0 rather than -p no:xdist.

What it deliberately does not do

• Diagnose NCIT:C55060. Tracked in NCIT:C55060 lost all its cross-references between Babel 2025sep1 and 2026jul22 #29, which covers both outcomes: picking a replacement CURIE if Babel was right to drop it, or taking it upstream if a genuinely good cross-reference went for a bad reason.
• Make two concurrent integration runs safe. Unavoidable while data/test is a shared fixed path. Documented instead, with the symptom spelled out, because it presents as a downloader bug.
• Cover an outage that begins mid-run. Both probes are single up-front checks. A service that dies between the probe and the call still fails the test, which is correct — the probes exist for the case that recurs.
• Give CI a release that publishes the Parquet files. Filed as CI never exercises the integration tests, because it has no Babel release that publishes the Parquet files #28, to be decided alongside CI cannot run the Parquet integration tests #18's sizing question, since Concord.parquet alone is a 4.6 GB download per run.
• Clean up DuckDB spill files. Filed as DuckDB spill files are never cleaned up after an interrupted query #26.

Nothing is blocking this merge

Everything outstanding is tracked in #18, #26, #28 and #29, and none of it makes what ships here wrong. #29 is the one worth reading before merging: it records that the test CURIE list is now two entries, both MONDO:, so the integration tests exercise no other namespace until a third is restored. That is reduced coverage, not a defect in this change.

How this landed — two things reached their final shape by correction rather than design. Kept for anyone tracing why a line looks the way it does; the durable conclusions are in the code comments and CLAUDE.md above.

The per-file skip arrived in two steps. The Identifiers.parquet skip was first added inline in the downloaded_identifiers fixture, duplicating the MissingBabelFileError handling that shared_downloader was doing with a HEAD probe. That left two mechanisms for one rule, which prompted collapsing both onto _download_or_skip() and reducing shared_downloader to a reachability check — and then re-pointing that check at the release root, since a probe naming Concord.parquet still looked like the publication check it had stopped being.

The cache deletion was misdiagnosed twice before the cause turned up. The symptom was a full run whose data/test shrank instead of growing and that never finished. It was blamed first on the new "a resume requires a validator" rule in the downloader — wrong, the server sends both an ETag and a Last-Modified, so that rule never fired — and then on two concurrent full suites, which was real but not the cause. The actual culprit was the unit suite: every pytest -m "not integration" run during the investigation deleted the download it was investigating. That is why pytest_sessionfinish now carries an explicit "do not remove this guard" comment naming the misdiagnosis, and why CLAUDE.md describes the symptom rather than just the rule — the failure mode is much easier to recognise than to reason out.