The counterpart to the NameRes ES regressions filed alongside this: places where
namelookup-es.ci.transltr.io is better than the Solr-backed
name-lookup.ci.transltr.io on the same Babel data. Recording them so they are not lost
when the regressions get fixed, and so we do not "fix" ES back to the Solr behaviour.
Eight rows that ES gets right and Solr does not
All are marked "not expected to pass in NameRes" in the Babel Validation sheet, so they
report as strict XPASS against ES — the sheet is out of date for ES, not wrong.
They are one shape: a query for a term whose clique is led by a MeSH or UMLS identifier,
where Solr fails to return the clique and ES returns it first.
| sheet row |
query |
expected, top hit on ES |
| 327 |
Hydrogel |
MESH:D020136 |
| 339, 1346 |
Immunoconjugates |
MESH:D018796 |
| 340 |
Shiga Toxin 1 |
MESH:D022622 |
| 440 |
Immunosuppressive Agents |
MESH:D007166 |
| 341 |
Immunoglobulin Light Chains, Surrogate |
UMLS:C0253781 |
| 358 |
Immunoglobulin Km Allotypes |
UMLS:C0021016 |
| 390 |
Lysosomal-Associated Membrane Protein 1 |
UMLS:C0125271 |
Six more rows improve without reaching rank 1: Hormone receptor positive breast cancer
and ANAEROBIC PNEUMONITIS go from top-5 to rank 1; Hutchinson-Gilford progeria,
Neuromyelitis optica spectrum, ACUTE PAINFUL SHOULDER, type 2 diabetes, Humira and
Natalizumab go from failing to top-5.
Speed
The same 2076-row suite took 10s against ES and 38s against Solr, same machine, same
-n auto. Worth confirming that this holds under load rather than being a cold-cache
artefact, but it is a large difference.
Caveat on the comparison
ci and ci-es are assumed to share a Babel release, but we cannot currently verify it
from the services — see the /status issue filed alongside this one. If they turn out to
be on different Babel versions, some of the rows above are data differences rather than
NameRes differences.
Action
Once the ES regressions are resolved, these rows should be flipped to "expected to pass" in
the sheet rather than left as XPASS noise.
The NameRes ES validation set: #131 (alphanumeric tokens split) · #132 (ranking:
longer names win) · #133 (score ties push the concept out) · #134 (where ES is better) ·
#135 (/status reports zeros) · #136 (scoping assertions to targets, which is what stops
these being filed with live assertions).
Produced by pytest --target ci-es -n auto tests/nameres/test_nameres_from_gsheet.py
compared against the same run with --target ci. Over the same 2076 Babel Validation sheet
rows: 58 exact top-1 hits on ES against 127 on Solr.
The counterpart to the NameRes ES regressions filed alongside this: places where
namelookup-es.ci.transltr.iois better than the Solr-backedname-lookup.ci.transltr.ioon the same Babel data. Recording them so they are not lostwhen the regressions get fixed, and so we do not "fix" ES back to the Solr behaviour.
Eight rows that ES gets right and Solr does not
All are marked "not expected to pass in NameRes" in the Babel Validation sheet, so they
report as strict XPASS against ES — the sheet is out of date for ES, not wrong.
They are one shape: a query for a term whose clique is led by a MeSH or UMLS identifier,
where Solr fails to return the clique and ES returns it first.
MESH:D020136MESH:D018796MESH:D022622MESH:D007166UMLS:C0253781UMLS:C0021016UMLS:C0125271Six more rows improve without reaching rank 1:
Hormone receptor positive breast cancerand
ANAEROBIC PNEUMONITISgo from top-5 to rank 1;Hutchinson-Gilford progeria,Neuromyelitis optica spectrum,ACUTE PAINFUL SHOULDER,type 2 diabetes,HumiraandNatalizumabgo from failing to top-5.Speed
The same 2076-row suite took 10s against ES and 38s against Solr, same machine, same
-n auto. Worth confirming that this holds under load rather than being a cold-cacheartefact, but it is a large difference.
Caveat on the comparison
ciandci-esare assumed to share a Babel release, but we cannot currently verify itfrom the services — see the
/statusissue filed alongside this one. If they turn out tobe on different Babel versions, some of the rows above are data differences rather than
NameRes differences.
Action
Once the ES regressions are resolved, these rows should be flipped to "expected to pass" in
the sheet rather than left as XPASS noise.
The NameRes ES validation set: #131 (alphanumeric tokens split) · #132 (ranking:
longer names win) · #133 (score ties push the concept out) · #134 (where ES is better) ·
#135 (
/statusreports zeros) · #136 (scoping assertions to targets, which is what stopsthese being filed with live assertions).
Produced by
pytest --target ci-es -n auto tests/nameres/test_nameres_from_gsheet.pycompared against the same run with
--target ci. Over the same 2076 Babel Validation sheetrows: 58 exact top-1 hits on ES against 127 on Solr.