Skip to content

Commit d053598

Browse files
committed
Record 20/21 and the 42 annotation fix in the handoff
Notes that 42 needed a corpus fix (pushed separately to posters-science-extraction-api main) before its real reading-order bug became visible, that re-running against an older corpus checkout will show 42 as LOGIC-GAP from the stale annotation, and that the same annotation error may have propagated via the annotator kit, whose zips were not rebuilt.
1 parent a5e83de commit d053598

1 file changed

Lines changed: 38 additions & 14 deletions

File tree

calibration/APPROACH_A_HANDOFF.md

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,13 @@ reading-order fix with no prior context. Read `TRACK_B_PLAN.md` in this folder
55
alongside this; that has the full Track B framing, this is the operational
66
runbook for approach A specifically.
77

8-
## STATUS (2026-07-16): approach A done, LOGIC-GAP tail done, 19/21
8+
## STATUS (2026-07-16): approach A done, LOGIC-GAP tail done, 20/21
99

10-
Board: **19/21 acceptable, 11/13 numbered end-to-end, 12/13 logic-OK**, corpus
11-
w=0.976 rGlobal=0.835 rField=0.729, suite green (260 passed). Reference
12-
snapshot: `baselines/try_orcidtail.json`.
10+
Board: **20/21 acceptable, 12/13 numbered end-to-end, 13/13 logic-OK**, corpus
11+
w=0.976 rGlobal=0.835 rField=0.735, suite green (260 passed). Reference
12+
snapshot: `baselines/try_42gen.json`. 13/13 logic-OK means that given ideal
13+
reading order the corrector is now correct on every numbered poster in the
14+
corpus; the single remaining failure (8228476) is reading order, not logic.
1315

1416
Read this first if you are comparing to older numbers: **the eval matcher was
1517
too loose and the old scoreboard was inflated.** `_affil_match` accepted any
@@ -37,26 +39,48 @@ regressions):
3739
point larger than the abstract under it fused into one block).
3840
- fe79689 line-bounded legend parse, superscript-row merge, ORCID tail, honest
3941
matcher.
40-
41-
The two remaining failures are NOT reading-order bugs:
42+
- a5e83de size-aware line clustering + baseline-nearest word mapping + curly
43+
apostrophe normalization (42 to PASS; also rField +0.101 on 5128504).
44+
45+
**Poster 42's annotation was fixed in the corpus repo**
46+
(`fairdataihub/posters-science-extraction-api`, commit 21d238d, pushed to
47+
main). Its `42.json` credited the CarD-T preprint's 6 authors and 5
48+
affiliations; the poster prints 3 and 2, which is what `42_raw.md` and
49+
`42_sub-json.json` already recorded and what the annotation guide asks for
50+
("creators - as shown on poster"; the full .json adds looked-up ORCID/ROR/DOI,
51+
not extra people). If you re-run against an older corpus checkout you will see
52+
42 as LOGIC-GAP again; that is the stale annotation, not a regression. NOTE:
53+
42 is one of the two reference examples in `gerard_annotation_kit/`, which the
54+
guide tells annotators to study as a template, so the same error may have been
55+
copied into posters annotated from it. The kit's unzipped
56+
`reference_examples/42/42.json` was corrected too, but the distributed .zip
57+
files were NOT rebuilt, and five stale copies remain under `extraction-beta-dev/`
58+
(left alone deliberately: they are historical experiment snapshots).
59+
60+
The one remaining failure is NOT a reading-order-logic gap in the corrector:
4261

4362
- **8228476** — RTL (Hebrew). Still ORDER-GAP. This is approach D. Note the
4463
superscript-row merge is deliberately gated (SUPERSCRIPT_MIN_DIGITS=2) so it
4564
does not touch this poster: rejoining a marker row shifts the median line
4665
height `_lines_to_blocks` keys on and re-blocks the page, which cost this
4766
poster 0.059 rField for no gain (its banner got worse too, being RTL either
4867
way). Approach D should handle bidi first, then revisit.
49-
- **42** — NOT FIXABLE from the poster, and no code should try. Its GT lists 6
50-
authors and 5 distinct affiliations; the poster prints 3 authors and 2
51-
affiliations. Reddy, Dhillon and Tripathi appear only inside a References
52-
citation (the CarD-T preprint), and 3 of the 5 GT affiliations (Herbert
53-
Wertheim School of Public Health, Moores Cancer Center, Dept of Cellular and
54-
Molecular Medicine) appear nowhere in the poster text. The annotation was
55-
taken from the paper/deposit, not the poster. Raise it with the annotators;
56-
it is a corpus bug, not an extraction bug.
5768

5869
The sections below are the original runbook, kept for context.
5970

71+
## Lesson worth keeping: a bad annotation hid a real bug
72+
73+
42 sat at LOGIC-GAP for the whole of Track B and was written off as
74+
corrector-side. It was really two faults stacked: an annotation that made the
75+
poster unwinnable, and underneath it a genuine reading-order bug. Because the
76+
GT was impossible, the bug was invisible - no amount of extraction work could
77+
have moved the poster, so nothing pointed at xy_cut. Fixing the annotation made
78+
it winnable, it still failed, and the failure was then diagnosable in minutes.
79+
80+
When a poster fails on IDEAL reading order, check the ground truth against the
81+
PDF before assuming the corrector is at fault. Cheap test: does every GT
82+
affiliation string appear anywhere in `_raw.md`? For 42, three of five did not.
83+
6084
## TL;DR
6185

6286
The affiliation corrector (in `poster2json/extract.py`) is now correct: given

0 commit comments

Comments
 (0)