Skip to content

NameRes ES: record where it already beats the Solr-backed NameRes #134

Description

@gaurav

The counterpart to the NameRes ES regressions filed alongside this: places where
namelookup-es.ci.transltr.io is better than the Solr-backed
name-lookup.ci.transltr.io on the same Babel data. Recording them so they are not lost
when the regressions get fixed, and so we do not "fix" ES back to the Solr behaviour.

Eight rows that ES gets right and Solr does not

All are marked "not expected to pass in NameRes" in the Babel Validation sheet, so they
report as strict XPASS against ES — the sheet is out of date for ES, not wrong.

They are one shape: a query for a term whose clique is led by a MeSH or UMLS identifier,
where Solr fails to return the clique and ES returns it first.

sheet row query expected, top hit on ES
327 Hydrogel MESH:D020136
339, 1346 Immunoconjugates MESH:D018796
340 Shiga Toxin 1 MESH:D022622
440 Immunosuppressive Agents MESH:D007166
341 Immunoglobulin Light Chains, Surrogate UMLS:C0253781
358 Immunoglobulin Km Allotypes UMLS:C0021016
390 Lysosomal-Associated Membrane Protein 1 UMLS:C0125271

Six more rows improve without reaching rank 1: Hormone receptor positive breast cancer
and ANAEROBIC PNEUMONITIS go from top-5 to rank 1; Hutchinson-Gilford progeria,
Neuromyelitis optica spectrum, ACUTE PAINFUL SHOULDER, type 2 diabetes, Humira and
Natalizumab go from failing to top-5.

Speed

The same 2076-row suite took 10s against ES and 38s against Solr, same machine, same
-n auto. Worth confirming that this holds under load rather than being a cold-cache
artefact, but it is a large difference.

Caveat on the comparison

ci and ci-es are assumed to share a Babel release, but we cannot currently verify it
from the services — see the /status issue filed alongside this one. If they turn out to
be on different Babel versions, some of the rows above are data differences rather than
NameRes differences.

Action

Once the ES regressions are resolved, these rows should be flipped to "expected to pass" in
the sheet rather than left as XPASS noise.


The NameRes ES validation set: #131 (alphanumeric tokens split) · #132 (ranking:
longer names win) · #133 (score ties push the concept out) · #134 (where ES is better) ·
#135 (/status reports zeros) · #136 (scoping assertions to targets, which is what stops
these being filed with live assertions).

Produced by pytest --target ci-es -n auto tests/nameres/test_nameres_from_gsheet.py
compared against the same run with --target ci. Over the same 2076 Babel Validation sheet
rows: 58 exact top-1 hits on ES against 127 on Solr.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions