Validate NameRes ES: group its differences from Solr, and let a target declare it has no blocklist

[gaurav]

Validating the Elasticsearch-backed NameRes (namelookup-es.ci.transltr.io) turned up a
large, undifferentiated pile of failures. This branch turns it into named failure classes
with reproducers — filed as #131-#136 under the
NameRes ES validation, September 2026
milestone — and makes the changes to this repo that the investigation needed.

The findings

Running the same 2076 sheet rows against ci-es and against ci — the Solr-backed NameRes
on the same Babel data, which is what separates a backend difference from a data one:

Outcome	ES	CI (Solr)
expected CURIE is the top hit	58	127
rank 2-6 (reported as a silent xfail)	59	34
absent from the top 20	32	7
below rank 6	14	4
blocklisted CURIE returned	10	0

Three mechanisms account for nearly all of it — #131 (alphanumeric tokens are split, so
?string=BRCA1 and ?string=BRCA return byte-identical lists and NCBIGene:672 is
unreachable), #132 (longer names containing the query outrank the concept), #133 (candidates
tie on score and the canonical concept falls out of the window). #134 records the eight rows
where ES is already ahead of Solr, so they do not get "fixed" back.

What changed here

NameResHasBlocklist, a new per-target capability. namelookup-es implements no
blocklist, so all 313 blocklist-sheet entries and every negative-flagged sheet row failed
by construction — the same known thing, daily, drowning the findings that matter.
targets.ini already exists to record this kind of difference rather than infer it from the
hostname. --target ci-es tests/nameres/ goes from 164 failures to 58; --target ci is
unchanged.

Filtering is on the negative flag, not the "Blocklist Nov 2023" category: only 16 rows
carry the flag, 5 of them live in "Unit Tests", and row 175 (hypersexuality) is an ordinary
positive row sitting in the Blocklist category. Category would have been wrong both ways.

expected_rank is now recorded. test_label calls pytest.xfail() imperatively when
the expected CURIE is inside NameResXFailIfInTop but not first, so a demotion from rank 1
to rank 2 never reaches the failure count. That hid the single largest regression: 46 rows
demoted, against 32 failing outright. Recording the rank as a user property puts it in
--report-jsonl where a comparison can see it. NameResXFailIfInTop is deliberately not
raised — that would make every other environment noisier to fix one.

Two new assertion handlers. The vocabulary could not express either thing this
validation most needs to say. SearchByName is top-N with N fixed at 5 by targets.ini, so
it passes on both services for all 46 demoted rows; SearchByNameTopResult passes only on
rank 1. DoesNotSearchByName is the first negative NameRes assertion of any kind, and makes
a blocklisted term expressible.

Gotchas worth keeping

• The issues are prose and curl only, deliberately. An issue's assertions run against
  every --target with no way to scope them, and an open issue whose assertions all pass is
  a strict XPASS failure. An assertion that fails on ES and passes on Solr would turn
  prod, test, ci, dev and exp red daily. Scope an issue's assertions to particular targets #136 tracks the scoping that unblocks this.
• DoesNotSearchByName treats a CURIE it cannot normalize as a failure, not a pass.
  "We could not look it up" and "it was not returned" both end with the CURIE absent, so the
  other choice lets a typo'd blocklist assertion succeed forever while testing nothing.
  DoesNotResolve takes the opposite line because there the CURIE failing to resolve is
  the assertion.
• Two of the new tests catch BaseException rather than Exception. pytest's outcome
  exceptions derive from it, and catching Failed specifically would be worse than useless:
  a regressed skip would propagate and mark the test itself skipped, which reads as
  passing.
• A capability skip has to live in the test body — target_info is parametrized by the root
  conftest and does not exist in pytest_generate_tests. That is test_blocklist.py makes an HTTP request at collection time (before any target is selected) #91, now on the milestone.
• black was not run on the two nameres test files; it wanted to rewrite 236 and 101 lines
  and would have buried the change.

Before merging

• Settle the final form of the two new assertions. SearchByNameTopResult and
  DoesNotSearchByName are unit-tested against fakes only and have never run against a
  live service, because no issue can carry them until Scope an issue's assertions to particular targets #136 lands. Worth deciding now
  whether the shapes are right — in particular whether top-1 should instead be a
  parameter on SearchByName ([query, curie, rank]) rather than a separate assertion
  name, and whether DoesNotSearchByName should honour NameResXFailIfInTop as its N
  or take its own.

• Decide when NameResHasBlocklist = false comes back out, and schedule it. The
  flag is a stopgap, not the answer: it stops ci-es reporting the same 10 known
  failures daily, but it also means that target has no blocklist coverage, and nothing
  enforces its removal. The real fix is
  NCATSTranslator/Babel#738 —
  filter blocked synonyms in Babel so they are never written out, rather than having each
  consumer delete them from its own index — now decided and on Babel v1.19. If #738 lands
  well after this PR, this flag is the thing most likely to be forgotten, and ci-es
  silently keeps its exemption long after it has stopped needing one. Worth agreeing the
  trigger for removing it (Babel v1.19 shipping? the ci-es index rebuilt from it?) before
  this merges. Tracked in Decide the trigger for removing NameResHasBlocklist = false from [ci-es] #144, which lists the candidate triggers and what "done" means;
  note that the most accurate of them is itself blocked on NameRes ES: /status reports numDocs 0 and no babel_version #135, since NameRes ES
  /status cannot currently say which Babel release an index was built from.

Testing

• pytest -m unit — 217 passed (was 192 at the branch point).
• New tests/unit/test_target_capabilities.py: the checked-in targets.ini parses, only
  ci-es opts out, an absent key keeps its coverage, both skips fire, and expected_rank
  is recorded before the xfail. Each mutation-checked.
• pytest --target ci-es -n auto tests/nameres/ — 59 failed, 56 passed, 329 skipped. The
  failures are NameRes ES: alphanumeric tokens are split, so gene symbols never match #131-NameRes ES: candidates tie on score, pushing the canonical concept out of the results #133 and are the point.
• pytest --target ci -n auto tests/nameres/test_blocklist.py — 196 passed, 117 skipped,
  unchanged.