tests/nameres/test_nameres_from_gsheet.py::test_label now records the rank the expected
CURIE landed at as an expected_rank user property (#137), so it reaches report.json.
Nothing renders it.
Why it matters
test_label calls pytest.xfail() imperatively when the expected CURIE is inside
NameResXFailIfInTop (5) but not first. So a row that used to return the right answer at
rank 1 and now returns it at rank 2 is reported as an xfail — indistinguishable, on the
dashboard, from a row the sheet never expected to pass.
That is not a hypothetical gap. Validating namelookup-es, it hid the largest regression
there was: 46 rows demoted from rank 1 into ranks 2-6, against 32 failing outright. The
dashboard showed a service that had quietly halved its top-1 accuracy (58 exact hits against
Solr's 127, same 2076 rows) as slightly greener than before, because demotions moved rows out
of "failed" and into "xfailed".
What would fix it
Unclear, which is why this is an issue rather than a commit. Some options:
- A rank column on
/results/, populated where expected_rank is present. Cheap, but the
matrix is already dense and a mostly-empty column is a poor trade.
- Treat a change in
expected_rank between runs as interesting on /history/, alongside
the existing outcome changes. This is probably the real answer — a demotion is a change,
and the History page already exists to show changes.
- A distinct outcome bucket ("passed, but not first") rather than folding these into
xfailed. Most informative, most invasive: it touches build_results in
src/babel_validation/tools/generate_report.py, the report schema, the environment cards
and Results.vue.
Worth deciding whether the rank belongs in the aggregate at all, or only in the per-run
detail. Note that generate_report.py is the choke point for anything published, so a rank
needs to pass the same validation as every other value — it is an int, so this is easy, but
it should be range-checked rather than trusted.
Related
Found while working through the milestone this belongs to; the underlying finding is #132.
tests/nameres/test_nameres_from_gsheet.py::test_labelnow records the rank the expectedCURIE landed at as an
expected_rankuser property (#137), so it reachesreport.json.Nothing renders it.
Why it matters
test_labelcallspytest.xfail()imperatively when the expected CURIE is insideNameResXFailIfInTop(5) but not first. So a row that used to return the right answer atrank 1 and now returns it at rank 2 is reported as an
xfail— indistinguishable, on thedashboard, from a row the sheet never expected to pass.
That is not a hypothetical gap. Validating namelookup-es, it hid the largest regression
there was: 46 rows demoted from rank 1 into ranks 2-6, against 32 failing outright. The
dashboard showed a service that had quietly halved its top-1 accuracy (58 exact hits against
Solr's 127, same 2076 rows) as slightly greener than before, because demotions moved rows out
of "failed" and into "xfailed".
What would fix it
Unclear, which is why this is an issue rather than a commit. Some options:
/results/, populated whereexpected_rankis present. Cheap, but thematrix is already dense and a mostly-empty column is a poor trade.
expected_rankbetween runs as interesting on/history/, alongsidethe existing outcome changes. This is probably the real answer — a demotion is a change,
and the History page already exists to show changes.
xfailed. Most informative, most invasive: it touchesbuild_resultsinsrc/babel_validation/tools/generate_report.py, the report schema, the environment cardsand
Results.vue.Worth deciding whether the rank belongs in the aggregate at all, or only in the per-run
detail. Note that
generate_report.pyis the choke point for anything published, so a rankneeds to pass the same validation as every other value — it is an int, so this is easy, but
it should be range-checked rather than trusted.
Related
Found while working through the milestone this belongs to; the underlying finding is #132.