Retract the identifier charge; find the real limit is resolution, and it is architectural

Two corrections to the first pass, one of them mine to own.

The identifier finding was wrong. I led with "LightOnOCR fabricates ORCIDs,
therefore disqualifying" without reading how the pipeline gets an ORCID.
It does not read them off the poster: enrich_creators_orcid queries the
ORCID API with given-names + family-name + affiliation-org-name and
attaches an id only on a single unambiguous hit, and ROR resolves
institutions the same way. A hallucinated ORCID in raw text is never
consulted, so counting them measured nothing about this system.

Re-measured on what the pipeline actually consumes -- the lookup keys --
the picture reverses: the VLM recovers author names slightly BETTER than
we do (0.978 vs 0.968) and affiliations worse (0.851 vs 0.915), where all
four of its extra losses are the one poster it failed outright. Its other
losses are the same abbreviation cases pdfplumber loses. And since
enrichment demands an exact unambiguous match, a corrupted key fails safe
-- no id -- rather than attaching the wrong person's. Coverage cost, not
misattribution. keys_check.py measures this; fidelity_check.py is kept
because "does an extractor invent exact strings" is still worth knowing,
with a note that it does not gate this pipeline.

The real limit is resolution, and it cannot be tuned. PixtralVisionConfig
image_size=1540 with patch_size=14 gives the vision tower a RoPE table of
110 patch positions per axis; a larger image indexes off the end and
asserts in modeling_pixtral.py:126. Two traps cost a sweep first:
PixtralImageProcessor silently resizes anything back to 1540, so
rendering larger only resamples twice and measurably hurts (10890106 w
0.942 -> 0.723), and the assert is async, so the first oversized page can
look like it succeeded while computing on out-of-bounds indices.

That ceiling is page-sized and posters are not. 1540px is ~132 DPI over
A4, which is what the model card assumes, but ~33 over a four-foot
poster: 17 of 20 posters get under 80 DPI and 17268692 gets 26. We are
asking it to read 8pt text at a quarter of its design resolution. The
three page-sized documents in the corpus show no VLM recall gap at all
(mean dW +0.012) while the 17 poster-sized ones average -0.048, though
r=+0.313 on n=20 is weak support and is reported as such.

Tiling confirms the diagnosis without fixing it. --tiles N OCRs an NxN
grid at 1540 each, multiplying effective DPI by N. On the biggest posters
recall improves exactly as predicted (17268692 w 0.916 -> 0.993, beating
pdfplumber; 42 0.897 -> 0.972), but it is not uniform and it destroys
reading order (rGlobal 0.788 -> 0.471) because concatenated quadrants chop
columns mid-flow. Not usable naively; the file records the three ways
forward, of which a banner-only crop is the narrowest and safest, since a
banner crop is naturally page-sized.

Author: jimnoneill

calibration/vlm/FINDINGS.md

@@ -1,104 +1,146 @@
-# Findings: LightOnOCR-2-1B vs pdfplumber + xy_cut (2026-07-16)
+# Findings: LightOnOCR-2-1B vs pdfplumber + xy_cut (2026-07-17)
 
-First run, 19 corpus posters + gasimova. Model `lightonai/LightOnOCR-2-1B`,
-200 DPI, longest side 1540px, bf16, greedy. **3.0 GB peak VRAM, ~60s/poster**
-on one GPU alongside the running ollama/vLLM services.
+19 corpus posters + gasimova. `lightonai/LightOnOCR-2-1B`, bf16, greedy,
+3.0 GB peak VRAM, ~60s/poster, on one GPU alongside the running ollama/vLLM
+services.
 
 ## Verdict
 
-**Not a drop-in replacement — it fabricates identifiers. But it reads poster
-structure markedly better than we do, and that is worth having.**
+**A serious candidate for structure, resolution-starved on posters, and not yet
+usable as a drop-in.** It reads banners far better than we do. It loses fine
+text because its vision tower is fixed at a page-sized 1540px and a poster is
+four feet. Its identifier hallucinations, which the first version of this file
+called disqualifying, are **irrelevant to this pipeline** -- see the retraction.
 
-Do not read the headline averages alone; they say the opposite of what matters.
+## RETRACTED: the identifier charge
 
-## It reads better
+The first version of this file led with "it fabricates identifiers -- 6 of 10
+ORCIDs wrong -- therefore disqualifying". **That was wrong, and it was wrong
+because I did not read the pipeline before measuring it.**
+
+poster2json never reads an ORCID off a poster. `orcid.enrich_creators_orcid`
+queries the ORCID API with `given-names` + `family-name` +
+`affiliation-org-name` and attaches an id only on a single unambiguous hit; ROR
+resolves institutions the same way. A hallucinated ORCID in the raw text is
+simply never consulted, so counting hallucinated ORCIDs measured nothing about
+this system. The lesson is the same one this corpus keeps teaching: **measure
+the thing the pipeline actually consumes.**
+
+What the pipeline consumes is the **lookup keys** -- author names and
+affiliation strings. Re-measured on those (`keys_check.py`):
+
+| extractor | authors found | affils found |
+|---|---|---|
+| pdfplumber | 90/93 = 0.968 | 43/47 = **0.915** |
+| LightOnOCR | 91/93 = **0.978** | 40/47 = 0.851 |
+
+The VLM is *slightly better* at recovering author names, and worse on
+affiliations — where **all four of its extra losses are isporeu2023**, the one
+poster it failed outright. Its other losses are the same abbreviation cases
+pdfplumber loses (VTT, CARL, STScI, Perdomo García, Münz-Manor). It actually
+recovered `the RECONS Team` and `University of Kent`, which pdfplumber missed.
+
+And because enrichment demands an exact unambiguous match (precision over
+coverage, 0.9.17), a corrupted key fails **safe**: no id attached, rather than
+the wrong person's id. So the real cost of VLM corruption is coverage, not
+misattribution. A far smaller charge than the one I filed.
+
+## It reads structure better
 
 | metric | pdfplumber + xy_cut | LightOnOCR-2-1B |
 |---|---|---|
 | `w` (word capture) | **0.976** | 0.936 |
 | `rGlobal` | **0.835** | 0.788 |
 | `rField` (length-normalized) | 0.741 | **0.765** |
 
-rField, the metric we treat as the headline, favours the VLM. Head-to-head it
-wins 12 of 19. On the **banner** — `authors+affiliations`, the field that cost
-this project Track A, Track B and approach A — it wins **15 of 19** (2 ties, 2
-losses), mean **+0.179**:
-
-    gasimova   0.742 -> 1.000     4607450   0.244 -> 0.909
-    4560930    0.600 -> 1.000     aysaekanger 0.429 -> 0.880
-    4446908    0.667 -> 1.000     10890106  0.600 -> 0.925
-    42         0.640 -> 0.913     AISec     0.696 -> 0.929
-
-It gets these right for free, from pixels, with no xy_cut, no
-`_flatten_top_band`, no superscript-row merge, no marker parsing. It also
-returns wrapped titles in one piece (10890106, which our block grouper still
-splits) and emits its own markdown headers.
-
-## It invents identifiers
-
-This is disqualifying for a metadata pipeline and ROUGE cannot see it. Exact
-strings checked against the human transcription (`fidelity_check.py`):
-
-| extractor | kind | recovered | missed | **invented** |
-|---|---|---|---|---|
-| pdfplumber | orcid | 9 | 1 | **0** |
-| pdfplumber | doi | 10 | 0 | **0** |
-| pdfplumber | email | 20 | 2 | 1* |
-| LightOnOCR | orcid | 4 | 6 | **6** |
-| LightOnOCR | doi | 7 | 3 | **2** |
-| LightOnOCR | email | 18 | 4 | 2 |
-
-\* not a fabrication: the poster prints `1aperdomo@iac.es` where the `1` is a
-superscript affiliation marker glued to the address. Our extractor correctly
-splits it; the checker counts the clean address as "not in the reference".
-
-**6 of 10 ORCIDs are wrong.** Actual corruptions:
-
-    DOI    10.1101/2024.08.13.24311948  ->  10.1105/2024.08.13.24311348
-    DOI    10.1007/978-3-031-02170-1    ->  10.1007/978-3-031-02701-1
-    email  joneilliii@sdsu.edu          ->  joneilliii@sdssu.edu
-    ORCID  0000-0002-2862-7302          ->  0000-0002-3982-7202
-
-These are not near-misses, they are different identifiers. A corrupted ORCID
-attributes a poster to another researcher; a corrupted DOI resolves to the
-wrong paper or nowhere. Silent, plausible, and worse than no value at all.
-A text-layer extractor cannot do this: it can only miss.
-
-The reason is structural, not a tuning problem. The VLM re-renders every glyph
-from pixels, so an identifier is a prediction. pdfplumber copies bytes the
-author embedded.
-
-## Other failure modes seen
-
-- **isporeu2023** is a genuine failure: it dropped 4 of 8 authors, mis-assigned
-  markers (Ciccarone 3 -> 2, Schlichting 4 -> 1), read "Delta Hat Ltd" as
-  "Delta et Ltd", and hallucinated 3 ORCIDs. It also never terminated: 6144
-  tokens truncated, and at 16384 it was STILL going (471s), emitting HTML
-  tables. Our pipeline scores 0.849 rField on this poster; the VLM 0.509.
-- **8228476** (RTL Hebrew) is worse under the VLM too (rField 0.503 vs 0.692),
-  so approach D is not solved by switching extractor.
-- Superscripts come back as LaTeX (`$^{1,2}$`). Harmless for raw-text scoring
-  (`_alpha()` reduces it to `12`, matching the reference's NFKD-normalized
-  `¹˒²`) but the affiliation corrector would need to read it.
+On the **banner** — `authors+affiliations`, the field that cost this project
+Track A, Track B and approach A — it wins **15 of 19** (2 ties, 2 losses), mean
+**+0.179**: gasimova 0.742 -> 1.000, 4607450 0.244 -> 0.909, 4560930 0.600 ->
+1.000. From pixels, with no xy_cut, no `_flatten_top_band`, no superscript-row
+merge, no marker parsing. It also returns wrapped titles in one piece.
+
+## Resolution: the ceiling is architectural, and posters fall off it
+
+**1540px is not a setting.** `PixtralVisionConfig.image_size = 1540`,
+`patch_size = 14`, so the vision tower's 2D RoPE table holds 110 patch
+positions per axis. Hand it a bigger image and it indexes off the end:
+
+    modeling_pixtral.py:126  freqs = self.inv_freq[position_ids]
+    CUDA error: device-side assert triggered
+
+Two traps here, both of which cost me a sweep:
+
+1. `PixtralImageProcessor` ships `size={"longest_edge": 1540}, do_resize=True`,
+   so it **silently rescales whatever you give it back to 1540**. Rendering
+   larger and feeding it in does not raise resolution — it only resamples
+   twice, and measurably hurts (10890106 `w` 0.942 -> 0.723 at "2048").
+   The knob is the processor's `size`, not the render.
+2. Raising the processor's `size` past 1540 asserts (above). The failure is
+   async, so the *first* oversized page may appear to succeed while computing
+   on out-of-bounds indices; do not trust it.
+
+**What 1540px means for a poster** (`eff_dpi.py`):
+
+| | effective DPI at 1540px |
+|---|---|
+| A4 (the model card's design point) | 132 |
+| poster mean | **48** |
+| 17268692 (60 x 44 in) | **26** |
+| **17 of 20 posters** | **under 80** |
+
+The model card's "200 DPI, longest dimension 1540px" is self-consistent for a
+page. A conference poster is 3-4 feet, so the same 1540px spreads to ~33 DPI.
+**We are asking it to read 8pt body text at a quarter of the resolution it was
+built for.**
+
+Supporting evidence, honestly weak (`dpi_corr.py`): on the 3 page-sized
+documents in the corpus (>=100 DPI) the VLM's recall gap **vanishes**
+(mean dW **+0.012**); on the 17 poster-sized ones it is **-0.048**.
+r(DPI, dW) = +0.313 — positive but weak, n=3 in the high group. Suggestive,
+not proven.
+
+## Tiling: confirms the diagnosis, does not yet fix it
+
+Tiling is the only lever left, so `--tiles N` renders an NxN grid (6% overlap)
+and OCRs each tile at 1540, multiplying effective DPI by N. At 2x2, on the
+biggest posters, **recall improves exactly as the theory predicts**:
+
+    17268692 (26 DPI)   w 0.916 -> 0.993   (beats pdfplumber's 0.986)
+    42       (32 DPI)   w 0.897 -> 0.972
+    15963941 (39 DPI)   w 0.945 -> 0.987   (beats pdfplumber's 0.958)
+
+But it is not uniform (4 of 8 gained, 4 lost: 4448680 0.923 -> 0.828, gasimova
+0.956 -> 0.906), and **it destroys reading order**: rGlobal 0.788 -> 0.471,
+rField 0.765 -> 0.547, because concatenating quadrants chops multi-column text
+mid-flow and duplicates the overlaps. Naive grid tiling is not usable.
 
 ## Where this points
 
-A hybrid is the obvious shape, and the numbers support it: **take structure
-from the VLM, take exact strings from the text layer.** The VLM is good at
-precisely what xy_cut finds hard (which text belongs to which line, in what
-order) and bad at precisely what pdfplumber gets for free (reproducing a string
-exactly). They fail in opposite directions.
-
-Concretely, worth testing next:
-
-1. VLM output as the reading-order source, then verify/replace every ORCID,
-   DOI and email against the PDF text layer — reject any identifier the text
-   layer does not contain verbatim. This bounds the fabrication to zero while
-   keeping the banner gains.
-2. Or narrower and safer: keep our pipeline, and use the VLM only for the
-   banner region, where it wins by +0.179 and where identifiers can be
-   cross-checked against a small, well-defined slice of text.
-3. Re-run with `--dpi 300` before concluding on recall; `w` is 0.936 vs our
-   0.976 and some of that gap may be resolution, not the model.
-
-Do NOT wire this into the pipeline on the strength of rField=0.765.
+The two extractors fail in opposite directions, and so do the two VLM modes:
+
+- pdfplumber has the text exactly but has to *infer* layout (all of xy_cut).
+- the VLM sees layout natively but has to *predict* every glyph, at 33 DPI.
+- a full-page VLM pass has the order but starves on detail; tiles have the
+  detail but lose the order.
+
+Worth trying next, in order:
+
+1. **Full-page pass for order + tiles for recall.** Use the full-page output as
+   the skeleton and tiles only to recover text the full pass missed. Keeps
+   rGlobal while capturing the +0.077 recall.
+2. **Layout-aware tiling** — tile on column boundaries (xy_cut already finds
+   them) instead of a blind grid, so no tile cuts a column mid-flow.
+3. **Banner-only VLM.** Narrowest and safest: our pipeline everywhere, the VLM
+   on the banner crop alone, where it wins +0.179 and where a crop is naturally
+   page-sized so resolution starvation disappears.
+4. Anything that consumes VLM text must read LaTeX superscripts (`$^{1,2}$`)
+   before the affiliation corrector can use it.
+
+## Other failure modes
+
+- **isporeu2023** fails outright: dropped 4 of 8 authors, mis-assigned markers
+  (Ciccarone 3->2, Schlichting 4->1), read "Delta Hat Ltd" as "Delta et Ltd",
+  and never terminated — 6144 tokens truncated, still going at 16384 (471s),
+  emitting HTML tables. Ours scores rField 0.849 there; the VLM 0.509.
+- **8228476** (RTL Hebrew) is worse under the VLM too (0.503 vs 0.692), so
+  approach D is not solved by switching extractor.

calibration/vlm/README.md

@@ -65,12 +65,27 @@ with plain transformers rather than vLLM precisely because vLLM pre-allocates a
 memory pool and would fight the running engines. **Do not kill those services to
 make room.** GPU 0 (the 4090) has only ~3.5 GB free and drives the display.
 
-## Rendering
-
-Per the model card: 200 DPI, longest side 1540px, aspect preserved. Both are
-flags on the runner (`--dpi`, `--longest`) since posters are unusually large
-(gasimova is 3312x3312pt) and resolution is the obvious thing to sweep if
-recall disappoints.
+## Rendering and the 1540px ceiling
+
+**Read this before trying to raise resolution.** `--longest` sets the render
+size AND the processor's `longest_edge` together, and 1540 is a hard ceiling:
+
+- `PixtralImageProcessor` ships `size={"longest_edge": 1540}, do_resize=True`,
+  so it silently rescales whatever you hand it back to 1540. Rendering bigger
+  without changing the processor buys nothing and costs a second resampling
+  pass — measurably (10890106 `w` 0.942 -> 0.723). The knob is the processor.
+- Raising the processor past 1540 asserts: `PixtralVisionConfig.image_size` is
+  1540 with `patch_size` 14, so the vision RoPE table holds 110 patch positions
+  per axis and a larger image indexes off the end
+  (`modeling_pixtral.py:126  freqs = self.inv_freq[position_ids]`). The CUDA
+  assert is **async**, so the first oversized page can look like it worked while
+  computing on out-of-bounds indices. Do not trust it.
+
+1540px is ~132 DPI over A4 — self-consistent with the model card — but a
+conference poster is 3-4 feet, so it lands at ~33 DPI (17 of our 20 posters get
+under 80; see `eff_dpi.py`). `--tiles N` is the only lever: it OCRs an NxN grid
+at 1540 each, multiplying effective DPI by N. It recovers recall on the largest
+posters but destroys reading order; see FINDINGS.md.
 
 `--max-new-tokens` defaults to 6144. A generation that stops exactly at the cap
 was truncated and has silently lost recall; the runner flags those in `run.json`
@@ -79,8 +94,13 @@ and in its output rather than letting them be scored as if complete.
 ## What to watch for
 
 - **Hallucination.** A VLM can produce fluent text that is not on the poster.
-  ROUGE against `_raw.md` rewards recall and will not punish invention hard
-  enough on its own. Read some outputs before believing a headline number.
+  ROUGE rewards recall and will not punish invention. Read some outputs before
+  believing a headline number. But measure it against what the pipeline
+  CONSUMES: poster2json looks ORCIDs up from name + affiliation and never reads
+  them off the poster, so hallucinated ORCIDs are moot and the lookup keys are
+  what matter (`keys_check.py`). `fidelity_check.py` still reports invented
+  exact strings, since that is worth knowing, but it does not gate this
+  pipeline. FINDINGS.md has the retraction.
 - **LaTeX.** LightOnOCR emits `$^{1,2}$` for superscript markers. Harmless for
   raw-text scoring (`_alpha()` strips it to `12`, and the reference's `¹˒²`
   NFKD-normalizes to the same), but it would need handling before the

calibration/vlm/compare_vlm.py

@@ -72,19 +72,28 @@ def main():
     ap = argparse.ArgumentParser()
     ap.add_argument("--save", default=None)
     ap.add_argument("--details", action="store_true")
+    ap.add_argument("--vlm-dir", default=None,
+                    help="dir of VLM .md files under calibration/vlm "
+                         "(default: out); use to compare a resolution sweep")
+    ap.add_argument("--only-vlm", action="store_true",
+                    help="skip the control (faster when sweeping the VLM)")
     args = ap.parse_args()
 
+    vlm_dir = (os.path.join(os.path.dirname(os.path.abspath(__file__)), args.vlm_dir)
+               if args.vlm_dir else OUT)
     rows = []
     for pid, pdf, rawp in items():
         with open(rawp, encoding="utf-8") as fh:
             ref = fh.read()
         row = {"id": pid}
-        vlm_path = os.path.join(OUT, f"{pid}.md")
+        vlm_path = os.path.join(vlm_dir, f"{pid}.md")
         if os.path.exists(vlm_path):
             with open(vlm_path, encoding="utf-8") as fh:
                 row["vlm"] = score(fh.read(), ref)
-        if pdf:
+        if pdf and not args.only_vlm:
             row["ctl"] = score(E.extract_text_with_pdfplumber(pdf) or "", ref)
+        if args.vlm_dir and "vlm" not in row:
+            continue
         rows.append(row)
 
     print(f"  {'poster':40s} {'--- pdfplumber + xy_cut ---':>28} "

calibration/vlm/dpi_corr.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Does the VLM's recall gap track effective DPI?
+
+Hypothesis: LightOnOCR-2 is not worse at reading than pdfplumber; it is
+resolution-starved on posters. Its vision tower is fixed at 1540px longest edge
+(image_size=1540 -- raising it indexes off the Pixtral RoPE table and asserts),
+which is ~132 DPI over A4 but ~33 DPI over a four-foot poster. If that is the
+cause, then (vlm_w - ctl_w) should rise with a poster's effective DPI, and the
+few page-sized posters should show no gap at all.
+"""
+import glob
+import json
+import os
+import sys
+
+import pypdfium2 as pdfium
+
+CMP = "/tmp/vlm_cmp.json"
+CORPUS = ("/home/joneill/Nextcloud/vaults/jmind/calmi2/poster_science/"
+          "json_schema/manual_poster_annotation")
+EXTRA = {"gasimova(oos)": "/storage/poster-work/gasimova.pdf"}
+LONGEST = 1540
+
+paths = {}
+for d in sorted(glob.glob(os.path.join(CORPUS, "*"))):
+    pdf = glob.glob(os.path.join(d, "*.pdf"))
+    if os.path.isdir(d) and pdf:
+        paths[os.path.basename(d)] = pdf[0]
+paths.update(EXTRA)
+
+rows = []
+for r in json.load(open(CMP, encoding="utf-8")):
+    pid = r["id"]
+    if pid not in paths or not (r.get("ctl") and r.get("vlm")):
+        continue
+    page = pdfium.PdfDocument(paths[pid])[0]
+    dpi = LONGEST / (max(page.get_width(), page.get_height()) / 72.0)
+    rows.append((dpi, pid, r["ctl"]["w"], r["vlm"]["w"],
+                 r["vlm"]["w"] - r["ctl"]["w"],
+                 r["vlm"]["r_field"] - r["ctl"]["r_field"]))
+
+rows.sort()
+print(f"{'poster':40s} {'DPI':>5} {'ctl w':>6} {'vlm w':>6} {'dW':>7} {'dField':>7}")
+for dpi, pid, cw, vw, dw, df in rows:
+    print(f"{pid:40s} {dpi:5.0f} {cw:6.3f} {vw:6.3f} {dw:+7.3f} {df:+7.3f}")
+
+
+def pearson(xs, ys):
+    n = len(xs)
+    mx, my = sum(xs) / n, sum(ys) / n
+    num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
+    dx = sum((x - mx) ** 2 for x in xs) ** 0.5
+    dy = sum((y - my) ** 2 for y in ys) ** 0.5
+    return num / (dx * dy) if dx and dy else float("nan")
+
+
+dpis = [r[0] for r in rows]
+dws = [r[4] for r in rows]
+print(f"\npearson r(effective DPI, vlm_w - ctl_w) = {pearson(dpis, dws):+.3f}  "
+      f"(n={len(rows)})")
+
+hi = [r for r in rows if r[0] >= 100]
+lo = [r for r in rows if r[0] < 60]
+for label, grp in (("page-sized (>=100 DPI)", hi), ("poster-sized (<60 DPI)", lo)):
+    if grp:
+        print(f"  {label:24s} n={len(grp)}  "
+              f"mean dW={sum(r[4] for r in grp) / len(grp):+.3f}  "
+              f"mean dField={sum(r[5] for r in grp) / len(grp):+.3f}")