Skip to content

Commit e492a9c

Browse files
committed
Record honest 19/21 board in handoff; drop intermediate baselines
Documents that the old scoreboard was inflated by a too-loose eval matcher, why 8228476 is deliberately excluded from the superscript-row merge, and that poster 42 is a corpus bug: its ground truth carries 6 authors and 5 affiliations while the poster prints 3 and 2, with the extra names appearing only inside a References citation.
1 parent fe79689 commit e492a9c

1 file changed

Lines changed: 51 additions & 14 deletions

File tree

calibration/APPROACH_A_HANDOFF.md

Lines changed: 51 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -5,20 +5,57 @@ reading-order fix with no prior context. Read `TRACK_B_PLAN.md` in this folder
55
alongside this; that has the full Track B framing, this is the operational
66
runbook for approach A specifically.
77

8-
## STATUS: approach A complete (2026-07-16)
9-
10-
Both non-RTL targets now PASS end-to-end. Commit 1bc9ef1 added a top-band
11-
banner flatten to xy_cut (`_flatten_top_band`, constants TOP_BAND=0.22 and
12-
TOP_BAND_DOMINANCE=0.85, both sweepable; sweeps show plateaus 0.22-0.30 and
13-
0.80-0.90), fixing gasimova. Commit ca44006 rescued "Surname X1"-style bylines
14-
from the single-char junk filter in extract.py (isporeu2023's reading order
15-
was already correct; the byline block was being discarded as chart debris).
16-
Scoreboard: AFFIL 8/13 end-to-end (was 6/13), corpus w=0.976 rGlobal=0.835
17-
rField=0.729, zero per-poster regressions vs after_track_a.json, suite green.
18-
Snapshots: baselines/after_approach_a.json, baselines/try_byline_rescue.json.
19-
Remaining: 8228476 is RTL (approach D); 42, 4519718, 4560930, 6724771 are
20-
LOGIC-GAP (corrector-side, out of scope here). The sections below are the
21-
original runbook, kept for context.
8+
## STATUS (2026-07-16): approach A done, LOGIC-GAP tail done, 19/21
9+
10+
Board: **19/21 acceptable, 11/13 numbered end-to-end, 12/13 logic-OK**, corpus
11+
w=0.976 rGlobal=0.835 rField=0.729, suite green (260 passed). Reference
12+
snapshot: `baselines/try_orcidtail.json`.
13+
14+
Read this first if you are comparing to older numbers: **the eval matcher was
15+
too loose and the old scoreboard was inflated.** `_affil_match` accepted any
16+
assigned string that merely CONTAINED the ground-truth tokens, so a poster
17+
scored as passing while carrying junk ("Aydan Gasimova 1 FAIR Data Innovations
18+
Hub, ..."). It is now asymmetric (an assigned string may abbreviate GT, but may
19+
not add tokens GT lacks). Honest scoring cost 2 posters on the board before the
20+
fixes below earned them back on merit. Do not loosen it again to make a number
21+
go up. `calibration/diagnostics/` has no dump for this; the throwaway probe that
22+
found it simply printed every assigned affiliation verbatim and diffed against
23+
the previous commit. Do that after any corrector change — ROUGE and the
24+
token matcher both normalize punctuation away and will not show you junk.
25+
26+
What landed (each measured per-poster against `after_track_a.json`, no
27+
regressions):
28+
29+
- 1bc9ef1 `_flatten_top_band` in xy_cut (TOP_BAND=0.22, TOP_BAND_DOMINANCE=0.85,
30+
sweepable; plateaus 0.22-0.30 and 0.80-0.90).
31+
- ca44006 byline rescue from the single-char junk filter (isporeu2023's whole
32+
author line was being dropped as chart debris; its reading order was fine).
33+
- ab32c73 two eval-matcher artifacts (abbreviated legends, group authors).
34+
- 64e9bfe trailing legend entry bounded at its source line; hyphen-joined
35+
marker runs ("4-5-6") parse.
36+
- 86afd80 block split on a local-rhythm gap plus a size change (a legend set a
37+
point larger than the abstract under it fused into one block).
38+
- fe79689 line-bounded legend parse, superscript-row merge, ORCID tail, honest
39+
matcher.
40+
41+
The two remaining failures are NOT reading-order bugs:
42+
43+
- **8228476** — RTL (Hebrew). Still ORDER-GAP. This is approach D. Note the
44+
superscript-row merge is deliberately gated (SUPERSCRIPT_MIN_DIGITS=2) so it
45+
does not touch this poster: rejoining a marker row shifts the median line
46+
height `_lines_to_blocks` keys on and re-blocks the page, which cost this
47+
poster 0.059 rField for no gain (its banner got worse too, being RTL either
48+
way). Approach D should handle bidi first, then revisit.
49+
- **42** — NOT FIXABLE from the poster, and no code should try. Its GT lists 6
50+
authors and 5 distinct affiliations; the poster prints 3 authors and 2
51+
affiliations. Reddy, Dhillon and Tripathi appear only inside a References
52+
citation (the CarD-T preprint), and 3 of the 5 GT affiliations (Herbert
53+
Wertheim School of Public Health, Moores Cancer Center, Dept of Cellular and
54+
Molecular Medicine) appear nowhere in the poster text. The annotation was
55+
taken from the paper/deposit, not the poster. Raise it with the annotators;
56+
it is a corpus bug, not an extraction bug.
57+
58+
The sections below are the original runbook, kept for context.
2259

2360
## TL;DR
2461

0 commit comments

Comments
 (0)