Validate NameRes ES: group its differences from Solr, and let a target declare it has no blocklist

.gitignore

@@ -1,3 +1,6 @@
+# Ignore worktrees
+/worktrees/
+
 # Ignore the root .env file.
 /.env
 

CLAUDE.md

@@ -82,6 +82,13 @@ The core of this project. Tests validate NodeNorm and NameRes services across mu
 
 **Target system:** `tests/targets.ini` defines endpoints for each environment (dev, prod, test, ci, exp, localhost). Tests use `target_info` fixture to get URLs. The `conftest.py` parametrizes tests across targets via `--target` CLI option; default is `dev`.
 
+It also declares per-target *capabilities* — `NodeNormBackend`, `NameResHasBlocklist` —
+so that a deployment which has not implemented something skips rather than failing the
+same way every day. Read them with `target_info.get`/`.getboolean` and default to the
+majority behaviour, so a target that forgets to declare one keeps its coverage instead of
+silently losing it. Such a skip has to live in the *test body*: `target_info` is
+parametrized by the root conftest and does not exist yet in `pytest_generate_tests`.
+
 **Google Sheet integration:** ~2000+ test cases are pulled from the shared Babel Validation
 Google Sheet. Its ID comes from the `BABEL_VALIDATION_SHEET_ID` environment variable (`.env`
 locally, a repository secret in Actions) and is deliberately not checked in.
@@ -276,4 +283,20 @@ When writing new tests:
   `FileLock`, rather than calling `requests.get` from a generate-tests hook. The same failure
   mode reappears if a cache's TTL can expire *during* a run, which is why
   `pytest_configure` deletes `gsheet_*.csv` before collection starts.
+- **A NameRes gsheet run's failure count understates regressions.** `test_label` calls
+  `pytest.xfail()` *imperatively* when the expected CURIE is inside the top
+  `NameResXFailIfInTop` (5) but not first, so a demotion from rank 1 to rank 2 is reported
+  as an `xfail` and never counted. Validating namelookup-es, that hid the largest
+  regression there was: 46 rows demoted, against 32 failing outright. To compare two
+  deployments, run both with `--report-jsonl` and classify each row rather than reading the
+  summary line — `passed`; `failed` with a msg starting `[XPASS(strict)]` is the sheet's
+  xfail passing; `wasxfail` with a msg starting `_pytest.outcomes.XFailed` is the
+  imperative rank-2-6 xfail; `wasxfail` otherwise is the sheet's own xfail; anything else
+  is a real failure. The rank itself is recorded as the `expected_rank` user property.
+- **Compare an -es target against `ci`, not against `dev` or `exp`.** `ci` and `ci-es` run
+  the same Babel data with different backends, which is what separates "the backend ranks
+  this differently" from "the data changed". `dev` and `exp` are newer Babel releases, so a
+  difference against them proves nothing. Note that NameRes ES `/status` reports
+  `babel_version: null`, so "same data" is currently an assumption the services cannot
+  confirm (issue #135).
 - Import shared classes from `src.babel_validation.*` (e.g. `from src.babel_validation.services.nodenorm import CachedNodeNorm`)

src/babel_validation/assertions/README.md

@@ -230,6 +230,52 @@ babel_tests:
 
 ---
 
+### SearchByNameTopResult
+
+**Applies to:** NameRes
+
+Each params_list must have exactly two elements: a search query string and an expected CURIE. The test passes only if the CURIE's normalized identifier is the very first result when NameRes looks up the search query. Use this rather than SearchByName when the point is that the concept must win, not merely appear: SearchByName accepts anything in the top N.
+
+**Parameters:** Each params_list: the **search query string** and the **expected CURIE**. The CURIE is normalized via NodeNorm before matching.
+
+**Wiki syntax:**
+```
+{{BabelTest|SearchByNameTopResult|water|CHEBI:15377}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  SearchByNameTopResult:
+    - [water, CHEBI:15377]
+    - [diabetes, MONDO:0005015]
+```
+
+---
+
+### DoesNotSearchByName
+
+**Applies to:** NameRes
+
+Each params_list must have exactly two elements: a search query string and a CURIE that must not be returned. The test passes if the CURIE's normalized identifier is absent from the top N results (default N=5) when NameRes looks up the search query. This is how a blocklisted term is asserted: the term is searchable, but the concept must not come back.
+
+**Parameters:** Each params_list: the **search query string** and the **CURIE that must not be returned**. The CURIE is normalized via NodeNorm before matching.
+
+**Wiki syntax:**
+```
+{{BabelTest|DoesNotSearchByName|mongoloid|HP:0000582}}
+```
+
+**YAML syntax:**
+```yaml
+babel_tests:
+  DoesNotSearchByName:
+    - [mongoloid, HP:0000582]
+    - [retard, HP:0006887]
+```
+
+---
+
 ## Special Assertions
 
 ### Needed

src/babel_validation/assertions/__init__.py

@@ -362,7 +362,9 @@ def test_params_list(self, params: ParamsList, nodenorm: NodeNormService,
     ResolvesHandler, DoesNotResolveHandler, ResolvesWithHandler,
     ResolvesWithTypeHandler, DoesNotResolveWithHandler, HasLabelHandler,
 )
-from src.babel_validation.assertions.nameres import SearchByNameHandler  # noqa: E402
+from src.babel_validation.assertions.nameres import (  # noqa: E402
+    SearchByNameHandler, SearchByNameTopResultHandler, DoesNotSearchByNameHandler,
+)
 from src.babel_validation.assertions.common import NeededHandler  # noqa: E402
 
 def _register(handlers: list[AssertionHandler]) -> dict[str, AssertionHandler]:
@@ -400,5 +402,7 @@ def _register(handlers: list[AssertionHandler]) -> dict[str, AssertionHandler]:
     HasLabelHandler(),
     ResolvesWithTypeHandler(),
     SearchByNameHandler(),
+    SearchByNameTopResultHandler(),
+    DoesNotSearchByNameHandler(),
     NeededHandler(),
 ])

src/babel_validation/assertions/nameres.py

@@ -40,21 +40,161 @@ def test_params_list(self, params: ParamsList, nodenorm: NodeNormService,
 
         expected_curie = expected_curie_result['id']['identifier']
         expected_curie_label = expected_curie_result['id'].get('label', '')
-        expected_curie_string = f"Expected CURIE {expected_curie_from_test}, normalized to {expected_curie} '{expected_curie_label}'"
+        expected_curie_string = (
+            f"Expected CURIE {expected_curie_from_test!r}, normalized to "
+            f"{expected_curie!r} {expected_curie_label!r}"
+        )
 
         results = nameres.lookup(search_query, autocomplete='false', limit=pass_if_found_in_top)
         if not results:
-            yield self.failed(f"No results found for '{search_query}' on NameRes {nameres} ({expected_curie_string})")
+            yield self.failed(
+                f"No results found for {search_query!r} on NameRes {nameres} ({expected_curie_string})")
             return
 
         curies = [result['curie'] for result in results]
         if expected_curie not in curies:
             logging.getLogger(__name__).debug(
-                "%s not found in top %d results for '%s' in NameRes %s: %s",
+                "%s not found in top %d results for %r in NameRes %s: %s",
                 expected_curie_string, pass_if_found_in_top, search_query, nameres,
                 json.dumps(results, indent=2, sort_keys=True)
             )
-            yield self.failed(f"{expected_curie_string} not found in top {pass_if_found_in_top} results for '{search_query}' in NameRes {nameres}")
+            yield self.failed(
+                f"{expected_curie_string} not found in top {pass_if_found_in_top} results for "
+                f"{search_query!r} in NameRes {nameres}")
             return
 
         yield self.passed(f"{expected_curie_string} found at index {curies.index(expected_curie) + 1} on NameRes {nameres}")
+
+
+class SearchByNameTopResultHandler(NameResTest):
+    """Test that a name search returns an expected CURIE as the *first* result in NameRes."""
+    NAME = "searchbynametopresult"
+    DESCRIPTION = (
+        "Each params_list must have exactly two elements: a search query string and an expected CURIE. "
+        "The test passes only if the CURIE's normalized identifier is the very first result when "
+        "NameRes looks up the search query. Use this rather than SearchByName when the point is that "
+        "the concept must win, not merely appear: SearchByName accepts anything in the top N."
+    )
+    PARAMETERS = (
+        "Each params_list: the **search query string** and the **expected CURIE**. "
+        "The CURIE is normalized via NodeNorm before matching."
+    )
+    WIKI_EXAMPLES = ["{{BabelTest|SearchByNameTopResult|water|CHEBI:15377}}"]
+    YAML_PARAMS = "    - [water, CHEBI:15377]\n    - [diabetes, MONDO:0005015]"
+
+    MIN_PARAMS = MAX_PARAMS = 2  # [search query, expected CURIE]
+
+    def curie_params(self, params: ParamsList) -> ParamsList:
+        # params[0] is a free-text search query; only the expected CURIE is a CURIE.
+        return params[1:2]
+
+    def test_params_list(self, params: ParamsList, nodenorm: NodeNormService,
+                         nameres: NameResService, pass_if_found_in_top: int = 5,
+                         label: str = "") -> Iterator[TestResult]:
+        [search_query, expected_curie_from_test] = params
+        expected_curie_result = nodenorm.normalize_curie(expected_curie_from_test)
+        if not expected_curie_result:
+            yield self.failed(f"Unable to normalize CURIE {expected_curie_from_test!r} in {label}")
+            return
+
+        expected_curie = expected_curie_result['id']['identifier']
+        expected_curie_label = expected_curie_result['id'].get('label', '')
+        expected_curie_string = (
+            f"Expected CURIE {expected_curie_from_test!r}, normalized to "
+            f"{expected_curie!r} {expected_curie_label!r}"
+        )
+
+        # Ask for the top N rather than just the top 1, so a failure can say how far
+        # down the expected CURIE actually landed. "at rank 4" tells you it is a
+        # ranking problem; "not in the top 5" tells you it is a retrieval one.
+        results = nameres.lookup(search_query, autocomplete='false', limit=pass_if_found_in_top)
+        if not results:
+            yield self.failed(
+                f"No results found for {search_query!r} on NameRes {nameres} ({expected_curie_string})")
+            return
+
+        curies = [result['curie'] for result in results]
+        if curies[0] == expected_curie:
+            yield self.passed(
+                f"{expected_curie_string} is the top result for {search_query!r} on NameRes {nameres}")
+            return
+
+        top = results[0]
+        if expected_curie in curies:
+            yield self.failed(
+                f"{expected_curie_string} is at rank {curies.index(expected_curie) + 1} for "
+                f"{search_query!r} on NameRes {nameres}, behind {top['curie']!r} "
+                f"{top.get('label', '')!r}"
+            )
+        else:
+            yield self.failed(
+                f"{expected_curie_string} is not in the top {pass_if_found_in_top} results for "
+                f"{search_query!r} on NameRes {nameres}; the top result is {top['curie']!r} "
+                f"{top.get('label', '')!r}"
+            )
+
+
+class DoesNotSearchByNameHandler(NameResTest):
+    """Test that a name search does *not* return a CURIE in the top-N results in NameRes."""
+    NAME = "doesnotsearchbyname"
+    DESCRIPTION = (
+        "Each params_list must have exactly two elements: a search query string and a CURIE that "
+        "must not be returned. The test passes if the CURIE's normalized identifier is absent from "
+        "the top N results (default N=5) when NameRes looks up the search query. This is how a "
+        "blocklisted term is asserted: the term is searchable, but the concept must not come back."
+    )
+    PARAMETERS = (
+        "Each params_list: the **search query string** and the **CURIE that must not be returned**. "
+        "The CURIE is normalized via NodeNorm before matching."
+    )
+    WIKI_EXAMPLES = ["{{BabelTest|DoesNotSearchByName|mongoloid|HP:0000582}}"]
+    YAML_PARAMS = "    - [mongoloid, HP:0000582]\n    - [retard, HP:0006887]"
+
+    MIN_PARAMS = MAX_PARAMS = 2  # [search query, CURIE that must not be returned]
+
+    def curie_params(self, params: ParamsList) -> ParamsList:
+        # params[0] is a free-text search query; only the rejected CURIE is a CURIE.
+        return params[1:2]
+
+    def test_params_list(self, params: ParamsList, nodenorm: NodeNormService,
+                         nameres: NameResService, pass_if_found_in_top: int = 5,
+                         label: str = "") -> Iterator[TestResult]:
+        [search_query, rejected_curie_from_test] = params
+
+        # A CURIE that will not normalize is a failure here, not a pass, even though
+        # "we could not look it up" and "it was not returned" both end with the CURIE
+        # absent from the results. A typo in a negative assertion would otherwise
+        # succeed forever while testing nothing, which is the one way a blocklist test
+        # can be worse than no test at all. DoesNotResolve takes the opposite line
+        # (VALIDATE_CURIES = False) because there the CURIE failing to resolve *is*
+        # the thing being asserted.
+        rejected_curie_result = nodenorm.normalize_curie(rejected_curie_from_test)
+        if not rejected_curie_result:
+            yield self.failed(f"Unable to normalize CURIE {rejected_curie_from_test!r} in {label}")
+            return
+
+        rejected_curie = rejected_curie_result['id']['identifier']
+        rejected_curie_label = rejected_curie_result['id'].get('label', '')
+        rejected_curie_string = (
+            f"Rejected CURIE {rejected_curie_from_test!r}, normalized to "
+            f"{rejected_curie!r} {rejected_curie_label!r}"
+        )
+
+        results = nameres.lookup(search_query, autocomplete='false', limit=pass_if_found_in_top)
+        curies = [result['curie'] for result in results]
+        if rejected_curie not in curies:
+            yield self.passed(
+                f"{rejected_curie_string} is absent from the top {pass_if_found_in_top} results "
+                f"for {search_query!r} on NameRes {nameres}")
+            return
+
+        index = curies.index(rejected_curie)
+        logging.getLogger(__name__).debug(
+            "%s was returned at rank %d for %r in NameRes %s: %s",
+            rejected_curie_string, index + 1, search_query, nameres,
+            json.dumps(results, indent=2, sort_keys=True)
+        )
+        yield self.failed(
+            f"{rejected_curie_string} was returned at rank {index + 1} for {search_query!r} "
+            f"on NameRes {nameres}: {results[index].get('label', '')!r}"
+        )

tests/nameres/test_blocklist.py

@@ -37,6 +37,17 @@ def test_check_blocklist_entry(target_info, blocklist_entry, categories_include)
     nameres_url = target_info['NameResURL']
     nameres_synonyms_url = nameres_url + 'synonyms'
 
+    # Some deployments have not implemented a blocklist at all (namelookup-es, at the
+    # time of writing). Every entry then fails identically, which drowns the sheet's
+    # real findings, so the target declares the capability in targets.ini instead.
+    # The skip is here rather than in pytest_generate_tests because target_info is
+    # parametrized by the root conftest and is not available at generation time.
+    if not target_info.getboolean('NameResHasBlocklist', True):
+        pytest.skip(
+            f"Skipping blocklist entry: {nameres_url} declares no blocklist "
+            f"(NameResHasBlocklist) in targets.ini."
+        )
+
     # If there is any test category provided, this test is not relevant and we can skip it.
     if categories_include:
         pytest.skip(f"Skipping blocklist entry as it is not part of any category and the category filter is set to include {categories_include}.")

tests/nameres/test_nameres_from_gsheet.py

@@ -50,6 +50,16 @@ def test_label(target_info, test_row, test_category, record_property):
     if not test_category(category):
         pytest.skip(f"Skipping category {category} because of the category filter.")
 
+    # A `negative` row asserts that a CURIE is *not* returned, which is what the
+    # blocklist is for. A NameRes without one fails every such row by construction,
+    # so this is a capability the target declares rather than a result worth
+    # reporting. Defaults to true: every deployment but namelookup-es has a blocklist.
+    if 'negative' in test_row.Flags and not target_info.getboolean('NameResHasBlocklist', True):
+        pytest.skip(
+            f"Skipping negative test row: {target_info['NameResURL']} declares no blocklist "
+            f"(NameResHasBlocklist) in targets.ini."
+        )
+
     source = test_row.Source
     source_url = test_row.SourceURL
     source_info = f"{source} ({source_url})"
@@ -152,6 +162,11 @@ def test_label(target_info, test_row, test_category, record_property):
             elif expected_id in all_curies:
                 expected_index = all_curies.index(expected_id)
 
+                # Record the rank even when we are about to xfail. A demotion from rank 1
+                # to rank 2 is the most common regression there is, and the imperative
+                # xfail below hides it from the failure count entirely.
+                record_property("expected_rank", expected_index + 1)
+
                 fail_message = f"{test_summary} returns {results[0]['curie']} ('{results[0]['label']}') as the " \
                     f"top result, but {expected_id} is at {expected_index} index."
                 if expected_index <= nameres_xfail_if_in_top:

tests/targets.ini

@@ -21,6 +21,13 @@
 [DEFAULT]
 NameResLimit = 20
 NameResXFailIfInTop = 5
+# Whether this target's NameRes applies the Translator blocklist. The Solr-backed
+# deployments do; namelookup-es does not implement one at all, so the sheet's
+# `negative`-flagged rows and the whole blocklist sheet would fail there by
+# construction rather than telling us anything. Default true: every deployment
+# except the one below has a blocklist, so a target that forgets to say so should
+# fail loudly rather than silently skip its blocklist coverage.
+NameResHasBlocklist = true
 # Which backend each target is expected to be talking to, checked against what
 # /status reports (deployments too old to report one are skipped). Set this when a
 # target is pointed at a different flavour, and set the OpenAPI path below to match.
@@ -73,6 +80,10 @@ NodeNormBackend = elasticsearch
 NodeNormOpenAPIPath = webapp/openapi.json
 NameResURL = https://namelookup-es.ci.transltr.io/
 NameResOpenAPIPath = webapp/openapi.json
+# namelookup-es has no blocklist. This is a capability it has not implemented yet,
+# not a bug we are tracking, so the blocklist tests skip here instead of reporting
+# ten failures that say the same known thing every day.
+NameResHasBlocklist = false
 
 [dev]
 NodeNormURL = https://nodenormalization-sri.renci.org/