fairdataihub
diff --git a/‎calibration/vlm/FINDINGS.md‎
Lines changed: 104 additions & 0 deletions b/‎calibration/vlm/FINDINGS.md‎
Lines changed: 104 additions & 0 deletions
diff --git a/‎calibration/vlm/README.md‎
Lines changed: 90 additions & 0 deletions b/‎calibration/vlm/README.md‎
Lines changed: 90 additions & 0 deletions
diff --git a/‎calibration/vlm/compare_vlm.py‎
Lines changed: 139 additions & 0 deletions b/‎calibration/vlm/compare_vlm.py‎
Lines changed: 139 additions & 0 deletions
@@ -0,0 +1,104 @@
+# Findings: LightOnOCR-2-1B vs pdfplumber + xy_cut (2026-07-16)
+
+First run, 19 corpus posters + gasimova. Model `lightonai/LightOnOCR-2-1B`,
+200 DPI, longest side 1540px, bf16, greedy. **3.0 GB peak VRAM, ~60s/poster**
+on one GPU alongside the running ollama/vLLM services.
+
+## Verdict
+
+**Not a drop-in replacement — it fabricates identifiers. But it reads poster
+structure markedly better than we do, and that is worth having.**
+
+Do not read the headline averages alone; they say the opposite of what matters.
+
+## It reads better
+
+| metric | pdfplumber + xy_cut | LightOnOCR-2-1B |
+|---|---|---|
+| `w` (word capture) | **0.976** | 0.936 |
+| `rGlobal` | **0.835** | 0.788 |
+| `rField` (length-normalized) | 0.741 | **0.765** |
+
+rField, the metric we treat as the headline, favours the VLM. Head-to-head it
+wins 12 of 19. On the **banner** — `authors+affiliations`, the field that cost
+this project Track A, Track B and approach A — it wins **15 of 19** (2 ties, 2
+losses), mean **+0.179**:
+
+    gasimova   0.742 -> 1.000     4607450   0.244 -> 0.909
+    4560930    0.600 -> 1.000     aysaekanger 0.429 -> 0.880
+    4446908    0.667 -> 1.000     10890106  0.600 -> 0.925
+    42         0.640 -> 0.913     AISec     0.696 -> 0.929
+
+It gets these right for free, from pixels, with no xy_cut, no
+`_flatten_top_band`, no superscript-row merge, no marker parsing. It also
+returns wrapped titles in one piece (10890106, which our block grouper still
+splits) and emits its own markdown headers.
+
+## It invents identifiers
+
+This is disqualifying for a metadata pipeline and ROUGE cannot see it. Exact
+strings checked against the human transcription (`fidelity_check.py`):
+
+| extractor | kind | recovered | missed | **invented** |
+|---|---|---|---|---|
+| pdfplumber | orcid | 9 | 1 | **0** |
+| pdfplumber | doi | 10 | 0 | **0** |
+| pdfplumber | email | 20 | 2 | 1* |
+| LightOnOCR | orcid | 4 | 6 | **6** |
+| LightOnOCR | doi | 7 | 3 | **2** |
+| LightOnOCR | email | 18 | 4 | 2 |
+
+\* not a fabrication: the poster prints `1aperdomo@iac.es` where the `1` is a
+superscript affiliation marker glued to the address. Our extractor correctly
+splits it; the checker counts the clean address as "not in the reference".
+
+**6 of 10 ORCIDs are wrong.** Actual corruptions:
+
+    DOI    10.1101/2024.08.13.24311948  ->  10.1105/2024.08.13.24311348
+    DOI    10.1007/978-3-031-02170-1    ->  10.1007/978-3-031-02701-1
+    email  joneilliii@sdsu.edu          ->  joneilliii@sdssu.edu
+    ORCID  0000-0002-2862-7302          ->  0000-0002-3982-7202
+
+These are not near-misses, they are different identifiers. A corrupted ORCID
+attributes a poster to another researcher; a corrupted DOI resolves to the
+wrong paper or nowhere. Silent, plausible, and worse than no value at all.
+A text-layer extractor cannot do this: it can only miss.
+
+The reason is structural, not a tuning problem. The VLM re-renders every glyph
+from pixels, so an identifier is a prediction. pdfplumber copies bytes the
+author embedded.
+
+## Other failure modes seen
+
+- **isporeu2023** is a genuine failure: it dropped 4 of 8 authors, mis-assigned
+  markers (Ciccarone 3 -> 2, Schlichting 4 -> 1), read "Delta Hat Ltd" as
+  "Delta et Ltd", and hallucinated 3 ORCIDs. It also never terminated: 6144
+  tokens truncated, and at 16384 it was STILL going (471s), emitting HTML
+  tables. Our pipeline scores 0.849 rField on this poster; the VLM 0.509.
+- **8228476** (RTL Hebrew) is worse under the VLM too (rField 0.503 vs 0.692),
+  so approach D is not solved by switching extractor.
+- Superscripts come back as LaTeX (`$^{1,2}$`). Harmless for raw-text scoring
+  (`_alpha()` reduces it to `12`, matching the reference's NFKD-normalized
+  `¹˒²`) but the affiliation corrector would need to read it.
+
+## Where this points
+
+A hybrid is the obvious shape, and the numbers support it: **take structure
+from the VLM, take exact strings from the text layer.** The VLM is good at
+precisely what xy_cut finds hard (which text belongs to which line, in what
+order) and bad at precisely what pdfplumber gets for free (reproducing a string
+exactly). They fail in opposite directions.
+
+Concretely, worth testing next:
+
+1. VLM output as the reading-order source, then verify/replace every ORCID,
+   DOI and email against the PDF text layer — reject any identifier the text
+   layer does not contain verbatim. This bounds the fabrication to zero while
+   keeping the banner gains.
+2. Or narrower and safer: keep our pipeline, and use the VLM only for the
+   banner region, where it wins by +0.179 and where identifiers can be
+   cross-checked against a small, well-defined slice of text.
+3. Re-run with `--dpi 300` before concluding on recall; `w` is 0.936 vs our
+   0.976 and some of that gap may be resolution, not the model.
+
+Do NOT wire this into the pipeline on the strength of rField=0.765.
@@ -0,0 +1,90 @@
+# LightOnOCR-2 as an extraction option: raw-text comparison
+
+Feature branch `feature/lightonocr-eval`. Asks one question and no others:
+**does an end-to-end VLM read a poster better than pdfplumber + xy_cut?**
+Raw text only. No affiliation corrector, no JSON, no LLM stage.
+
+Branched from `feature/xy-cut-calibration`, not `main`, deliberately: that
+branch holds the current best pipeline (rField 0.741 vs main's 0.727) and the
+harness whose metrics both sides are scored with. Comparing a new model against
+a stale control would flatter it.
+
+## What is being compared
+
+| | control | candidate |
+|---|---|---|
+| | `extract_text_with_pdfplumber` (pdfplumber + xy_cut) | `lightonai/LightOnOCR-2-1B` |
+| input | PDF text layer | page rendered to an image |
+| output | markdown-ish text | markdown |
+
+Both are scored by the same code against the same reference, the human
+`_raw.md` transcription. Neither side gets a metric of its own.
+
+- `w` — word capture vs `_raw.md`. Format-blind; the fairest single number for
+  "did it read the page".
+- `rGlobal` — whole-document ROUGE-L. Order-sensitive, format-blind.
+- `rField` — per-field ROUGE-L, length-normalized (title, authors+affiliations,
+  and each section count once each regardless of length). **Needs the output to
+  carry markdown headers** so it can be chunked into fields. Ours emits `## `;
+  LightOnOCR emits `#`/`##` of its own accord, so the comparison holds. A model
+  that emitted flat text would score badly here for reasons of format rather
+  than reading — so read `w` and `rGlobal` first.
+
+## Running it
+
+Two phases, two environments, on purpose.
+
+    # phase 1: generate (needs transformers v5)
+    CUDA_VISIBLE_DEVICES=1 PYTHONPATH=~/locr-libs \
+        ~/myenv/bin/python calibration/vlm/run_lightonocr.py
+
+    # phase 2: score (needs poster2json, i.e. plain ~/myenv)
+    ~/myenv/bin/python calibration/vlm/compare_vlm.py --details
+
+### The environment split matters
+
+`LightOnOcrForConditionalGeneration` / `LightOnOcrProcessor` landed in
+**transformers v5**. `~/myenv` is on **4.57.6** and is what poster2json, the
+calibration harness and the 8B validation all import. **Do not upgrade it in
+place.** Instead:
+
+    ~/myenv/bin/pip install --target ~/locr-libs "transformers>=5" pypdfium2
+
+`PYTHONPATH=~/locr-libs` shadows 4.57 for that one process and leaves every
+other consumer alone. Verified: transformers 5.14.1 + torch 2.8.0+cu128 (torch
+is reused from `~/myenv`, not reinstalled). A `--system-site-packages` venv does
+NOT work here, because `~/myenv` is itself a venv and the new venv inherits the
+base interpreter's packages, not `~/myenv`'s.
+
+### GPU etiquette
+
+hpcf's GPUs are usually busy: GPU 1 typically runs an ollama llama-server and
+two vLLM engines (~74 of 98 GB). LightOnOCR-2-1B is small — **2.5 GB peak, bf16**
+— so it coexists on GPU 1's headroom via `CUDA_VISIBLE_DEVICES=1`. It is loaded
+with plain transformers rather than vLLM precisely because vLLM pre-allocates a
+memory pool and would fight the running engines. **Do not kill those services to
+make room.** GPU 0 (the 4090) has only ~3.5 GB free and drives the display.
+
+## Rendering
+
+Per the model card: 200 DPI, longest side 1540px, aspect preserved. Both are
+flags on the runner (`--dpi`, `--longest`) since posters are unusually large
+(gasimova is 3312x3312pt) and resolution is the obvious thing to sweep if
+recall disappoints.
+
+`--max-new-tokens` defaults to 6144. A generation that stops exactly at the cap
+was truncated and has silently lost recall; the runner flags those in `run.json`
+and in its output rather than letting them be scored as if complete.
+
+## What to watch for
+
+- **Hallucination.** A VLM can produce fluent text that is not on the poster.
+  ROUGE against `_raw.md` rewards recall and will not punish invention hard
+  enough on its own. Read some outputs before believing a headline number.
+- **LaTeX.** LightOnOCR emits `$^{1,2}$` for superscript markers. Harmless for
+  raw-text scoring (`_alpha()` strips it to `12`, and the reference's `¹˒²`
+  NFKD-normalizes to the same), but it would need handling before the
+  affiliation corrector could consume VLM output. Out of scope here.
+- **The banner is the interesting part.** That is where our pipeline needed all
+  of Track A/B and approach A. Compare `authors+affiliations` per-field
+  (`--details`) as much as the corpus average.
@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""Score LightOnOCR-2's raw output against the same controls as our pipeline.
+
+Phase 2 of the VLM comparison. Runs under plain ~/myenv (it needs poster2json
+for the control) and reads the .md files phase 1 wrote.
+
+Both extractors are scored by the SAME code against the SAME reference: the
+human `_raw.md` transcription. Nothing here is VLM-specific, so neither side
+gets a metric of its own.
+
+    w        word capture vs _raw.md. Format-blind; the fairest single number
+             for "did it read the page".
+    rGlobal  whole-document ROUGE-L. Order-sensitive, format-blind.
+    rField   per-field ROUGE-L, length-normalized (title, authors+affiliations,
+             each section count once each). CAVEAT: this one needs the output
+             to carry markdown headers so it can be chunked into fields. Our
+             pipeline emits '## '; LightOnOCR emits '#'/'##' of its own accord.
+             If a future model emits flat text, its rField will be low for
+             reasons of format rather than reading, so read w and rGlobal first.
+"""
+import argparse
+import glob
+import json
+import os
+import statistics
+import sys
+
+REPO = "/home/joneill/Nextcloud/vaults/jmind/calmi2/poster_science/poster2json"
+sys.path.insert(0, REPO)
+sys.path.insert(0, os.path.join(REPO, "calibration"))
+from poster2json import extract as E   # noqa: E402
+E.log = lambda *a, **k: None
+import reading_order_eval as REV       # noqa: E402
+
+CORPUS = ("/home/joneill/Nextcloud/vaults/jmind/calmi2/poster_science/"
+          "json_schema/manual_poster_annotation")
+EXTRA = [("gasimova(oos)", "/storage/poster-work/gasimova.pdf",
+          "/storage/poster-work/gasimova_clean_raw.md")]
+OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "out")
+
+
+def score(gen, ref_md):
+    if not gen or not gen.strip():
+        return None
+    macro, fields = REV.field_scores(gen, ref_md)
+    return {
+        "w": round(len(REV._words(gen) & REV._words(ref_md))
+                   / max(len(REV._words(ref_md)), 1), 3),
+        "r_global": round(REV._rougeL(REV._alpha(ref_md), REV._alpha(gen)), 3),
+        "r_field": round(macro, 3),
+        "fields": fields,
+    }
+
+
+def items():
+    out = []
+    for d in sorted(glob.glob(os.path.join(CORPUS, "*"))):
+        if not os.path.isdir(d):
+            continue
+        pid = os.path.basename(d)
+        pdf = glob.glob(os.path.join(d, "*.pdf"))
+        raw = glob.glob(os.path.join(d, "*_raw.md"))
+        if raw:
+            out.append((pid, pdf[0] if pdf else "", raw[0]))
+    for pid, pdf, raw in EXTRA:
+        if os.path.exists(raw):
+            out.append((pid, pdf if os.path.exists(pdf) else "", raw))
+    return out
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--save", default=None)
+    ap.add_argument("--details", action="store_true")
+    args = ap.parse_args()
+
+    rows = []
+    for pid, pdf, rawp in items():
+        with open(rawp, encoding="utf-8") as fh:
+            ref = fh.read()
+        row = {"id": pid}
+        vlm_path = os.path.join(OUT, f"{pid}.md")
+        if os.path.exists(vlm_path):
+            with open(vlm_path, encoding="utf-8") as fh:
+                row["vlm"] = score(fh.read(), ref)
+        if pdf:
+            row["ctl"] = score(E.extract_text_with_pdfplumber(pdf) or "", ref)
+        rows.append(row)
+
+    print(f"  {'poster':40s} {'--- pdfplumber + xy_cut ---':>28} "
+          f"{'--- LightOnOCR-2-1B ---':>26}")
+    print(f"  {'':40s} {'w':>6} {'rGlob':>7} {'rField':>7} "
+          f"{'w':>6} {'rGlob':>7} {'rField':>7}  {'dField':>7}")
+    for r in rows:
+        c, v = r.get("ctl"), r.get("vlm")
+
+        def f(d, k):
+            return f"{d[k]:.3f}" if d else "   -  "
+        d = (f"{v['r_field'] - c['r_field']:+.3f}" if (c and v) else "   -  ")
+        print(f"  {r['id']:40s} {f(c, 'w'):>6} {f(c, 'r_global'):>7} "
+              f"{f(c, 'r_field'):>7} {f(v, 'w'):>6} {f(v, 'r_global'):>7} "
+              f"{f(v, 'r_field'):>7}  {d:>7}")
+
+    print("  " + "-" * 104)
+    for label, key in (("pdfplumber+xy_cut", "ctl"), ("LightOnOCR-2-1B  ", "vlm")):
+        got = [r[key] for r in rows if r.get(key) and "oos" not in r["id"]]
+        if got:
+            print(f"  {label}  n={len(got):2d}  "
+                  f"w={statistics.fmean(g['w'] for g in got):.3f}  "
+                  f"rGlobal={statistics.fmean(g['r_global'] for g in got):.3f}  "
+                  f"rField={statistics.fmean(g['r_field'] for g in got):.3f}")
+
+    both = [r for r in rows if r.get("ctl") and r.get("vlm") and "oos" not in r["id"]]
+    if both:
+        wins = sum(1 for r in both if r["vlm"]["r_field"] > r["ctl"]["r_field"])
+        print(f"  head-to-head on {len(both)} posters scored by both: "
+              f"VLM better on rField for {wins}, worse for {len(both) - wins}")
+
+    if args.details:
+        print("\nper-field (control -> vlm):")
+        for r in rows:
+            if not (r.get("ctl") and r.get("vlm")):
+                continue
+            print(f"  {r['id']}")
+            cf = {n: s for n, s, _ in r["ctl"]["fields"]}
+            for n, s, ln in r["vlm"]["fields"]:
+                base = cf.get(n)
+                d = f"{s - base:+.3f}" if base is not None else "  new"
+                print(f"      {base if base is None else f'{base:.3f}'} -> "
+                      f"{s:.3f}  {d}  ({ln:4d}w)  {n}")
+
+    if args.save:
+        with open(args.save, "w", encoding="utf-8") as fh:
+            json.dump(rows, fh, indent=2)
+        print(f"\nsaved {args.save}")
+
+
+if __name__ == "__main__":
+    main()