Skip to content

corpus-expansion.md

Latest commit

 

History

History
839 lines (679 loc) · 44.7 KB

corpus-expansion.md

File metadata and controls

839 lines (679 loc) · 44.7 KB
Raw

Corpus expansion: from 16 frozen fixtures to 40–60

The metric corpus is 16 PDFs frozen in testkit/fixtures/, pinned by SHA-256 in testkit/corpus_manifest.json. They are regression evidence, not market coverage (ROADMAP.md). Every number this repository reports describes those 16 files and nothing else, which is exactly the property that makes them useful and exactly the property that makes them narrow.

This document specifies how the corpus grows to roughly 40–60 frozen documents without putting a single existing number at risk.

1. What the 16 actually sample

producer documents share
ReportLab 01–05, r1_reportlab_report 6/16
Chromium/Skia c1–c8 8/16
fpdf2 f1_fpdf_brief 1/16
LibreOffice (word-processor dialect) l1_word_native 1/16
downloaded real-world document — 0/16

Two gaps follow directly from that table.

The word-processor dialect is one document. It is also the single most common shape of PDF that a user will hand this converter, and the one whose round trip back to DOCX matters most. One sample cannot distinguish a general rule from a fixture accident — and the project already made that mistake once, with l1's symbol-font list markers.

Nothing here was produced by a stranger. All 16 were generated by scripts in this repository, so every one of them inherits this repository's assumptions about margins, fonts, and structure. A producer we have never seen is the only honest test of a general rule.

Separately, the 16 over-sample designed stress. c3 (nested tables), c4 (CJK + RTL) and c5 (gradients, rotation, SVG) are 3 of 16 — 19% of the corpus spent on constructs the policy already classifies as non-blocking, while the blocking ordinary_digital tier carries the whole product claim on 13 documents that are mostly one prose layout with a table in it.

2. Where expansion fixtures live, and why not in testkit/fixtures/

Expansion fixtures live in testkit/fixtures_expansion/, pinned by a separate manifest testkit/corpus_expansion.json. The gate's manifest and fixture directory are not touched at all.

This is not tidiness. Three mechanisms currently make testkit/fixtures/ a closed set, and adding a file to it breaks all three:

  1. corpus_manifest.verify() rejects strangers. It lists testkit/fixtures/, and any .pdf there that is not in the manifest is reported as unexpected. runall.resolve_corpus() turns that into a gate problem. A new fixture in that directory is a red gate.

  2. runall.resolve_corpus() measures whatever the manifest names. It iterates manifest["documents"] and hands every entry to both lanes. Adding an entry silently enlarges the gated corpus, and gate.py then compares a 32-document run against a 16-document baseline.

  3. gdocs_quality_policy.json pins the manifest file's own SHA-256, and that pin is live:

    sha256(testkit/corpus_manifest.json) = cc4dd4c1c449ddabe85d7538454ed4a3d6163d762f68d2e59d97801e11c3c7da
gdocs_quality_policy.json "manifest"  = cc4dd4c1c449ddabe85d7538454ed4a3d6163d762f68d2e59d97801e11c3c7da

    gdocs_oracle._load_quality_policy() compares them. It used to return policy status mismatch with tiers = None on any difference, which silently stopped evaluating the Google Docs quality policy altogether — a one-byte edit to the manifest disabled the tiered assessment without failing anything, and the resulting findings: [] reads like "nothing wrong" rather than "nothing was checked".

    Fixed: a pin mismatch now raises QualificationError, naming both the hash the policy pins and the hash the manifest actually has. It refuses in preflight, before any Drive service is constructed, so a consented run cannot spend an upload collecting evidence nothing can grade; assess refuses the same way offline. Schema and candidate-profile mismatches still return mismatch without raising — those describe a file that is not this policy at all, and accusing it of the wrong pin would send the reader to the wrong file. _load_quality_policy additionally requires every manifest document to be assigned to a tier and unsupported.documents to stay empty, so even a correctly re-pinned manifest with new documents comes back malformed.

    gdocs_oracle._source_plan() also requires the set of .pdf files in the fixtures directory to match the manifest exactly, so the two cannot be desynchronised in either direction.

A sibling directory rather than testkit/fixtures/expansion/ is deliberate. The two os.listdir scans above are non-recursive today, so a subdirectory would survive — but it would survive by accident, and the first glob("**/*.pdf") anyone writes over the fixtures tree would quietly fold the expansion set into the gate. A sibling cannot be reached by a mistake of that shape.

The resulting separation

file pins consumed by gates
testkit/corpus_manifest.json the 16 metric inputs runall.py, gate.py, gdocs_oracle.py yes
testkit/corpus_expansion.json the expansion inputs expansion.py, corpus_manifest.py verify no
testkit/parity_policy.json acceptance for the gated 16 backend_parity.py; parity_expansion.py reads its margins and nothing else yes
testkit/expansion_parity_policy.json acceptance for the expansion corpus expansion_policy.py, via parity_expansion.py no

corpus_manifest.py verifies both corpora. It does so through two separate functions: verify() is untouched and still answers only for the 16, so runall.py — which calls verify(manifest) directly — cannot see an expansion problem. The CLI verify command calls both and reports them in separate sections. That is the whole coupling: identity is checked in one place, and the gate reads one half of it.

The two acceptance policies are separate for the same reason, and the readers enforce it in both directions. An expansion finding written into parity_policy.json would have been unreachable — backend_parity.adjudicate never sees a document outside the gated manifest, and parity_expansion.py refuses the waiver sections by design — while carrying corpus_manifest_sha256, a binding naming a manifest that does not contain the document. So expansion_parity_policy.json pins corpus_expansion.json by its own hash and keys its entries by full profile ID, because the expansion corpus is measured at two profiles and a finding at one says nothing about the other. expansion_policy.load() refuses a file carrying the gated policy's sections, and parity_expansion.load_margins() refuses a file carrying the expansion schema. Neither artifact can be read as the other by accident.

That policy annotates and never adjudicates: a ratified row keeps the verdict the measurement produced, with ratified: true beside it, and parity_expansion.py's exit code still reports only whether every document could be measured. What the reader does enforce is the policy's own integrity — a moved corpus, a malformed section, an entry naming an absent document, a floor breached, a finding that has stopped describing anything, or an entry no run measured are all errors, in the same class as a corpus that does not match its manifest.

3. Target producer mix

Target the midpoint, ~48 documents. Tranche 1 (this change) is 16 generated documents; the remainder is downloads, which need explicit approval per docs/corpus-download-candidates.md.

producer class target now (16) +tranche 1 to find
word-processor export (LibreOffice, Word) 12 1 7 5
browser print-to-PDF (Chromium, WebKit) 12 8 14 —
report generator (ReportLab, fpdf2, JasperReports, Crystal) 10 7 11 —
LaTeX-light (pdfTeX, XeTeX, LuaTeX) 8 0 0 8
other / unknown real-world producers 6 0 0 6
total 48 16 32 ~16

Two things this table admits.

LaTeX cannot be generated here. The canonical image has no TeX distribution, and adding one is a ~2 GB change to the environment of record for the sake of input files — which would then need a fresh image digest and a baseline migration. arXiv is full of permissively licensed pdfTeX output. The LaTeX tranche is therefore download-only and is deferred to tranche 2.

The browser and report-generator classes overshoot after tranche 1 (14 and 11 against targets of 12 and 10). That is fine and is not corrected by deleting fixtures: the targets are floors for coverage, and the surplus is in ordinary_digital, which is the tier the product claim rests on. If the ratio matters later, it is corrected by adding to the thin classes, never by unfreezing a document.

What tranche 1 covers that the 16 do not

Running headers and footers with real page-number fields; nested bulleted and numbered lists at three levels; borderless tables alongside bordered ones; footnote-shaped page-bottom apparatus; block quotes; inline hyperlinks in ordinary prose; four-level heading hierarchies; TOC pages with tab leaders; Latin-diacritic, Cyrillic and Greek text (all inside the pinned font set, so unlike c4_i18n this is ordinary, not stress); multi-page memos and reports where a per-page offset compounds.

The resume pair, added after tranche 1

x17_resume_twocol and x18_resume_twocol_tnr were added because the release bar names resume-class documents as the exemplar and the corpus contained no resume at all. That gap was found the way gaps usually are: the owner converted their own resume, opened it in Docs, and it was wrong in three separate ways at once. Their document is personal and is not in this repository and never will be; these two are the synthetic stand-in, with fictional employers and every URL on example.com.

The pair is a treatment and its control. Both carry the same three structures:

  • role/date pairs on one baseline — two per page. Inference used to split the pair and push the right-hand member across with an absolute indent, one of which reached 103% of the content width, i.e. past the right margin;
  • two populations of link anchor — four in the contact header, each of which is its own span, and four in prose, each a fragment of a longer span. The first four were always tagged and the second four were always dropped. One of the prose anchors covers 0.462 of its span, deliberately just under the 0.5 test, so that lowering the constant cannot be mistaken for fixing the cause;
  • a family the DOCX declares nowhere — x17 has a monospaced run and x18 does not, which is the only difference between them.

What this pair cannot reproduce. The real document's body font was Georgia, and Georgia is not installed in the measurement container — nor is Verdana; fc-match resolves both to DejaVu Serif. Of the installed families, the only one the mapper emits that python-docx's stock fontTable.xml omits is Courier New. So the pair holds the declaration mechanism (a family on runs that the document never declares) but not the severity of the real case, where the undeclared family was the body text rather than one line of it. That severity is measured only on the owner's own document, outside this repository, and the honest place to say so is here.

4. Tier classification rules

testkit/gdocs_quality_policy.json defines three tiers. Assignment has been ad hoc; these are the rules, written down so a new document classifies itself.

A document is ordinary_digital — the blocking tier — when all hold:

  • born-digital with a real text layer; extractable text covers essentially all visible glyphs (no scanned or outlined text);
  • reading order is single-column, or plain two-column with a full-width header;
  • every table is a rectangular grid of cells — no nesting, no cell that contains another table;
  • all text is left-to-right, and every script used is fully covered by the pinned font set in scripts/fonts.conf without fallback (Latin incl. diacritics, Cyrillic, Greek);
  • no rotated or transformed text, no text knocked out of a gradient or image, no vector graphic that carries meaning the text does not.

Size is not a tier criterion. The first version of this rule required ordinary_digital to be ≤ 20 pages and ≤ 2 MB, and tranche 2 broke it immediately: the real documents that arrived are 46, 80, 114 and 126 pages, and an 80-page Word-to-Acrobat government report is the most ordinary document imaginable. That rule conflated two unrelated axes — how hard a document is to convert, which is what a tier means, and how expensive it is to measure, which is a corpus budget. Length is not difficulty. The budget is stated separately in §6 and governs what we choose to freeze, never what tier a frozen document belongs to.

A document is designed_stress — tracked, non-blocking — when it is otherwise ordinary but deliberately exercises a construct on the known-hard list: nested tables, RTL or bidirectional text, CJK or any script requiring font fallback, rotated/transformed text, gradients or knockout text, meaning-bearing vector graphics, or true multi-column with column-spanning elements.

A document is unsupported — expected to be refused before qualification — when it has no usable text layer (pure scan), is encrypted or password protected, uses XFA or interactive form logic for data entry, or exceeds the converter's own declared limits: > 250 pages or > 250 MiB, which are MAX_PAGES_PER_DOCUMENT and MAX_BYTES_PER_DOCUMENT in exactdoc/batch.py.

Those two numbers are read from the product rather than invented here, and that is the point: the unsupported tier should mean "the converter says it refuses this", not "the corpus authors guessed it would be hard". The earlier > 200 pages, > 50 MB was a guess and disagreed with the shipping code.

Interactive form logic means fields a user fills in — text fields, checkboxes, radio groups. Navigation widgets are not form logic: y03_nist_fips197 carries 16 Button annotations emitted by LaTeX's hyperref, which are links, and it is ordinary_digital.

Two consequences worth stating plainly.

Tranche 1 is 100% ordinary_digital, by design. The scope was the ordinary variety the current 16 under-sample, and ordinary_digital is the only blocking tier — it is where a broader claim has to be earned. The stress and unsupported tiers are populated in tranche 2, where real downloaded documents (scans, LaTeX with unusual encodings) provide honest examples rather than manufactured ones. A stress case we invented to fail is worth less than one we found.

The tier is a property of the document, not of the result. A document does not move to designed_stress because it scored badly. Reclassification requires a rule above to have been misapplied, and shows up in the diff as a changed tier field with a reason.

5. Provenance schema

Every expansion entry carries a provenance object. The gate's manifest has no such field and does not gain one.

"x01_lo_memo_pageno.pdf": {
  "sha256": "…64 hex…",
  "bytes": 24096,
  "content": "…32 hex…",
  "src_pages": 3,
  "tier": "ordinary_digital",
  "dialect": "libreoffice",
  "why": "running header/footer with a real page-number field",
  "provenance": {
    "origin": "generated",
    "recipe": "testkit/gen_expansion.py::x01_lo_memo_pageno",
    "source_url": null,
    "license": "Apache-2.0",
    "acquired": "2026-08-04",
    "toolchain": "exactdoc-gate:dev / LibreOffice 24.2.7.2 / Chrome 149.0.7827.55 / Python 3.12.3"
  }
}
field required rule
origin always generated or downloaded; nothing else verifies
recipe when generated path::function naming the code that made it; source_url must be null
source_url when downloaded absolute https:// URL it came from; recipe must be null
license always non-empty SPDX identifier or a named public-domain statement. unknown is not acceptable — an unlicensed file does not enter the corpus
acquired always ISO YYYY-MM-DD the bytes were frozen or fetched
toolchain when generated the environment of record that produced it

origin is exclusive-or by construction: exactly one of recipe and source_url is non-null, and corpus_manifest.verify_expansion() rejects a document that satisfies both or neither. This is what keeps "we made this" and "we found this" from blurring after a year, which is the failure mode that makes a licence audit impossible.

6. Freezing procedure

Identical in spirit to the 16: generate in the canonical container, freeze the bytes, pin the hash, never regenerate in place.

docker run -d --name exactdoc-corpus -w /work exactdoc-gate:dev sleep infinity
docker cp <worktree> exactdoc-corpus:/work/repo
docker exec exactdoc-corpus bash -lc \
  'cd /work/repo && FONTCONFIG_FILE=/work/repo/scripts/fonts.conf bash scripts/bootstrap.sh --strict'

# 1. generate into a scratch directory (never into fixtures_expansion/)
docker exec exactdoc-corpus bash -lc \
  'cd /work/repo && FONTCONFIG_FILE=/work/repo/scripts/fonts.conf \
   uv run python testkit/gen_expansion.py /work/build'

# 2. seal: copy in, compute identity, merge the generator's provenance sidecar
docker exec exactdoc-corpus bash -lc \
  'cd /work/repo && uv run python testkit/corpus_manifest.py expansion-seal /work/build'

# 3. verify, then carry the bytes back out
docker exec exactdoc-corpus bash -lc \
  'cd /work/repo && uv run python testkit/corpus_manifest.py verify'
docker cp exactdoc-corpus:/work/repo/testkit/fixtures_expansion <worktree>/testkit/
docker cp exactdoc-corpus:/work/repo/testkit/corpus_expansion.json <worktree>/testkit/
docker rm -f exactdoc-corpus

gen_expansion.py writes expansion_provenance.json beside the PDFs: tier, dialect, why, licence, recipe and toolchain — everything except the hashes. expansion-seal computes the hashes. The split is the point: a human authors the claims, the tool computes the identity, and neither can forge the other.

Fixtures are not byte-reproducible — LibreOffice, Chromium and ReportLab all stamp a creation time and a document ID into every file. Re-running gen_expansion.py produces 16 different SHA-256s and the same documents. That is why the bytes are frozen and committed rather than rebuilt, and it is the same argument recorded in corpus_manifest.py for the original 16. The content fingerprint (page geometry + whitespace-normalised text) is what moves only when a document really changes.

Measuring the expansion set, at any time, without touching the gate:

uv run python testkit/expansion.py --lane product     # or raw, or both

It refuses to run on a corpus that does not match its manifest, prints per-tier summaries, writes testkit/expansion_out/, and exits 0 on any metric value. It has no baseline, makes no comparison, and imports nothing from gate.py.

7. Protocol: when expansion fixtures become gating

They do not become gating by being measured, by scoring well, or by being merged. There is no automatic promotion, no "after N green runs", and no threshold that trips. Promotion is one reviewable commit, and this is the protocol.

Preconditions, all of them:

  1. The expansion set has been measured in the canonical container on at least two separate commits, and the numbers are stable — no document moving more than the gate.py tolerance between runs on identical bytes.
  2. Every document proposed for promotion is ordinary_digital under §4, or is being promoted into an explicitly non-blocking tier.
  3. Any document that would enter the gate already failing has a recorded defect ID, exactly as gate.py requires of the existing known shortfalls. A new document is not permitted to arrive as an unexplained red.
  4. The Google Docs quality policy has been reviewed and ratified (review.status), because promotion changes the document set that policy binds. Ratification is a decision for the owner and is not part of this protocol.

The promotion commit, which must be a single commit and must contain all of:

  • the promoted entries moved from corpus_expansion.json into corpus_manifest.json, and the fixture bytes moved from testkit/fixtures_expansion/ into testkit/fixtures/ unchanged — a promotion that regenerates a document is not a promotion, it is a new document;
  • gate_baseline.json re-recorded in the canonical container, because the gate compares against per-document records and has none for the newcomers;
  • gdocs_quality_policy.json re-pinned: the manifest.sha256 updated to the new manifest bytes, and every promoted document assigned to a tier — otherwise _load_quality_policy refuses outright on the pin, or returns malformed on the tier assignment, and no quality result can be produced at all (§2);
  • parity_policy.json re-pinned for the same reason;
  • the provenance and tier of every promoted document preserved in the new manifest. Provenance does not get dropped on the way into the gate.

The rule underneath all of it is the one the project already paid to learn: a change to the corpus re-bases every recorded number, so it must land together with the re-recorded numbers, in one reviewable diff. A promotion spread over two commits leaves an interval in which every published figure describes a different corpus than the one on disk.

Expansion fixtures are non-gating until such a commit exists. Tranche 1 contains no such commit.

8. First measurement of tranche 1

Recorded here because it is the reason the corpus was expanded, and because a tranche that is merely frozen and never measured proves nothing.

testkit/expansion.py --lane product, run in the canonical container against the 16 sealed fixtures: 16/16 converted, rendered, scored, and page-exact. Medians across the tier:

metric median draft ordinary_digital bar
live_text_cov 0.998 ≥ 0.90
doc_recall 0.996 ≥ 0.90
word_recall 0.996 ≥ 0.90
mean_ssim 0.826 ≥ 0.70
dy_p50 12.2 pt ≤ 10.0
within2pt 0.188 not gated
raster_frac 0.002 not gated

This is not a policy verdict. gdocs_quality_policy.json describes the pdfium/gdocs/none/refine0@240dpi candidate measured through the Google Docs oracle; this run is the product lane through the LibreOffice proxy. Same metric names, different renderer, and no cloud call was made. Read against those thresholds anyway, as an indication, the split is stark:

  • text fidelity: zero findings. Every document clears coverage, document recall, word recall and page match. Whatever is wrong is not that the words are missing.
  • geometry: 23 findings across 12 documents — dx_p50 on 9, dy_p50 on 9, mean_ssim on 5.

The offenders are not random. Ranked by dx_p50: x02_lo_report_toc 103.3pt, x01_lo_memo_pageno 83.7pt, x07_chrome_memo_running 45.5pt, x08_chrome_print_default 43.1pt. Ranked by dy_p50: x11_chrome_toc_headings 74.7pt, x01 29.3pt, x08 27.7pt, x02 16.4pt, x07 15.5pt.

Every one of the worst documents carries a running header, a running footer, or browser-furnished page furniture — and those are precisely the constructs the frozen 16 do not sample. l1_word_native, the only word-processor document in the gated corpus, is a single page with no header and no footer, so no gated number has ever been able to move in response to this.

That is the expansion doing its job on the first run: it did not discover that the converter is bad at ordinary prose, it discovered that page furniture is the dominant unmodelled construct in ordinary documents, and that the gate is currently blind to it. Whether that becomes a defect ID, a layout rule, or a deliberate non-goal is a product decision and is not settled here. What is settled is that it is now measurable, repeatably, on committed bytes.

The corresponding raw-lane numbers were not collected in this pass; the two lanes are run together by default and only product was needed to establish the above. That is a gap in the record, not a claim about the raw lane.

9. Tranche 2: the first documents this project did not write

Twelve candidates were attempted under the §5 protocol. Six arrived, six failed, five were sealed and one was fetched and then deliberately discarded. The expansion corpus is now 21 fixtures; the gated 16 are untouched.

Licence verification, per document

Verified from the statement inside each document, which is first-party and needs no network call to re-check. Where a document is silent, that silence is recorded as silence rather than upgraded into a claim.

fixture licence basis evidence found in the document
y01_nist_sp80063b US Gov work, 17 U.S.C. §105 p4: "…is not subject to copyright in the United States. Attribution would, however, be appreciated by NIST."
y02_nist_sp800171r2 US Gov work p4: same NIST statement. A p6 notice disclaims third-party names as trademarks; it reserves no rights over the text
y03_nist_fips197 US Gov work none found — no copyright or trademark notice anywhere. Public domain by publisher identity (a NIST FIPS), not by a statement
y06_irs_1040_instructions US Gov work none found. Public domain by publisher identity (IRS instruction booklet)
y07_irs_f1040_form US Gov work none found. Public domain by publisher identity (official IRS form)

Three of five rest on publisher identity rather than an explicit grant. That is sound — a US Government work is uncopyrighted whether or not it says so, and agencies routinely omit the notice for exactly that reason — but it is weaker than a statement and is recorded as such in corpus_expansion.json rather than smoothed over.

The one that was thrown away

y05_gao_report.pdf was fetched successfully and is not in the corpus. Its GAO notice is the most explicit of the six: "This is a work of the U.S. government and is not subject to copyright protection… may be reproduced and distributed in its entirety without further permission." So it could have been kept. Three facts said not to:

  1. 286 pages exceeds MAX_PAGES_PER_DOCUMENT (250) in exactdoc/batch.py, so its only possible role was as an over-cap refusal input.
  2. 15.3 MB is 13× the entire existing corpus — the 16 gated fixtures and 16 tranche-1 fixtures together are about 1.1 MB. Permanently carrying that in git history to prove one ResourceLimitError is a bad trade, and the trade is what matters: any 251-page document tests the same branch.
  3. The same notice adds that the work "may contain copyrighted images or other material" needing separate permission. It has 176 images. Whole-file redistribution is covered, but it is the one document of the six whose redistribution carries a qualification at all.

Splitting or trimming it was never an option: a trimmed document is a document this project wrote, which is the exact property tranche 2 exists to avoid. The over-cap slot is refilled in the corrected plan by NIST SP 800-53r5 (~490 pages, ~5 MB) — same test, one third the weight, from a host that has already answered three times.

What the corpus gained

Five producer chains it had never contained, none of them ours:

fixture producer creator
y01 Adobe PDF Library 15.0 Acrobat PDFMaker 17 for Word
y02 Adobe PDF Library 20.13.106 Acrobat PDFMaker 20 for Word
y03 pdfTeX 3.14159265-2.6-1.40.21 (TeX Live 2020) LaTeX with hyperref
y06 Antenna House PDF Output Library 6.6.1437 AH XSL Formatter V6.6
y07 Adobe LiveCycle Designer 6.5 Designer 6.5

Two of these matter beyond variety. y01/y02 are genuine Word exports — the dialect the whole product targets, which the corpus previously sampled with one single-page document. And y03 closes the LaTeX gap that §3 said could only be closed by download: pdfTeX output, without touching arXiv's licence problem, because a NIST FIPS is a US Government work.

y03 also produced an artifact no generator here could fake: its text layer drops the fi ligature. "Specification" extracts as "Specifcation", "affine" as "affne", "field" as "feld". That is a real Type-1/pdfTeX extraction property, it is invisible to a human reading the PDF, and it will suppress word recall on every document from that toolchain.

Two rules this tranche corrected

The tier ceiling was wrong (§4). Requiring ordinary_digital to be ≤ 20 pages and ≤ 2 MB failed on contact: the real documents are 46, 80, 114 and 126 pages, and an 80-page Word-to-Acrobat government report is as ordinary as a document gets. Length is not difficulty, and the ceiling has been split out into a corpus-budget concern. The unsupported thresholds now read 250 pages and 250 MiB from exactdoc/batch.py rather than the invented "200 pages, 50 MB".

unsupported needed refusal semantics in expansion.py. A fixture the converter is expected to refuse would otherwise make the runner red for working correctly. A raised exception on an unsupported document is now recorded as refused and is a pass; the converse — an unsupported document that converts anyway — is flagged as a refusal-contract gap, which is the case actually worth knowing about.

Round-1 transport failures and their remedies

candidate failure remedy
D11a CRS 403 Forbidden Not retried, User-Agent not disguised. Getting past a bot filter by impersonating a browser is evasion, and a corpus is not worth it. Re-pointed at EveryCRSReport, which publishes an open access policy
D19a/D19b/D20a RFC 404 The path /rfc/rfcNNNN.pdf serves a native PDF only for RFCXML v3 documents (~2019+). RFCs 8259, 2119 and 793 predate it; their generated PDFs live at /rfc/pdfrfc/rfcNNNN.txt.pdf. Corrected, plus RFC 9110 added to exercise the native form
D26a ECB CERTIFICATE_VERIFY_FAILED A real bug in fetch_expansion.py: Windows populates chain intermediates lazily through CryptoAPI and Python does not drive it. Now prefers certifi's bundle, which resolves identically on every platform. The ECB URL was also an unverified guess and is dropped
D32a Wikimedia 429 A real gap: no backoff. Now honours Retry-After exactly once — no loop, no exponential hammering. If a host says no twice, the answer is no

Round 1 also established which hosts are worth planning around: nvlpubs.nist.gov answered 3/3 and irs.gov 2/2, so the corrected plan is weighted to them — 13 candidates, ~12.3 MB, still unfetched and still requiring --allow-download.

Measurement: the real documents fail, and they fail differently

testkit/expansion.py --lane product, canonical container, same caveat as §8 — this is the LibreOffice proxy, not the Google Docs oracle the policy describes.

fixture pages in → out live_text_cov doc_recall word_recall dy_p50 mean_ssim
y01_nist_sp80063b 80 → 158 0.906 0.923 0.131 88.8 0.314
y02_nist_sp800171r2 114 → 314 0.869 0.900 0.128 81.4 0.243
y03_nist_fips197 46 → 53 0.931 0.961 0.558 16.8 0.614
y07_irs_f1040_form 2 → 10 0.937 0.919 0.457 63.7 0.085
y06_irs_1040_instructions — crashed: UnrecognizedImageError

Three things, in order of how much they matter.

1. Pagination doubles. y01 produces 158 pages from 80; y02 produces 314 from 114. That is 1.98× and 2.75×, and it is the same defect tranche 1 saw at small scale finally showing its actual shape. Note which metrics move and which do not: doc_recall stays at 0.90–0.92, so the words are nearly all there, while word_recall collapses to 0.13, because word recall is position-aware and every word after the first overflow is on the wrong page. A converter that keeps 92% of the text and puts 87% of it on the wrong page is not 92% correct.

The gated 16 could not have found this. They are 1–7 pages, and a per-page overflow needs length to compound into a page-count error — which is exactly what c6_long was designed to catch at 7 pages and was too short to see.

2. The refusal contract is not enforced. y07_irs_f1040_form is a 199-widget fillable form, tiered unsupported, meaning the converter is expected to reject it. It converted: 2 pages became 10, at mean_ssim 0.085 — visually almost nothing in common with the input. The new refusal-semantics check in expansion.py flagged it as REFUSAL CONTRACT GAP. exactdoc/batch.py enforces page and byte ceilings but nothing rejects interactive form logic, so a user handing this converter a tax form gets ten pages of confident garbage rather than a refusal. That is a product decision to make, not a bug to quietly fix here, but it is now measurable and named.

3. One document crashes outright. y06_irs_1040_instructions, the Antenna House XSL-FO booklet, raises UnrecognizedImageError and produces no measurement at all. The runner exits 1 and calls it an infrastructure failure rather than a fidelity score, which is the honest reading: we do not know how well it converts, because it does not convert.

On the tier medians, which look fine and are not. The ordinary_digital median row barely moved between §8 and now — dy_p50 12.2 → 13.0, mean_ssim 0.826 → 0.803. That is an artefact of counting: 16 of the 19 documents in that tier are the small tranche-1 fixtures, so they outvote the three real ones. The median is doing what a median does and it is the wrong statistic for a corpus this lopsided. Read the per-document table, not the tier summary, until the real documents are no longer a minority — which is an argument for finishing tranche 2, not for adjusting the statistic.

None of this gates anything. No baseline describes these documents and gate.py has never seen them.

10. Round 3, and the close of acquisition

Thirteen candidates attempted, eight fetched and all eight sealed — the first round with no discards. The corpus is now 16 gated + 29 expansion = 45 documents, inside the 40–60 target ROADMAP.md set, and acquisition is closed. testkit/expansion_download_plan.json now carries an empty candidates array, so fetch_expansion.py refuses it even when handed --allow-download; the five that never arrived are kept as an annex in the same file.

Licence outcomes

fixture basis evidence in the document
y08_nist_sp80088r1 US Gov work p4: explicit NIST "not subject to copyright in the United States"
y09_nist_sp800207 US Gov work p3: same NIST statement
y10_nist_fips180 US Gov work none found — publisher identity (NIST FIPS)
y11_nist_sp80053r5 US Gov work p3: explicit NIST statement
y12_irs_pub15 US Gov work no notice over the publication; a partner's NCMEC trademark credit only
y13_irs_pub501 US Gov work same NCMEC credit; no rights asserted over the text
y14_irs_fw9_form US Gov work none found — publisher identity (IRS form)
y17_rfc9110 IETF Trust TLP — not public domain p1: "Copyright (c) 2022 IETF Trust… All rights reserved." p2 forbids modification and derivative works outside the IETF Standards Process

y17_rfc9110 is the only fixture in the entire corpus that is not public domain, and it is the only one carrying an active restriction. The restriction is on modification and derivative works; verbatim redistribution of an unmodified copy is what the TLP permits and is exactly what a frozen fixture is. Its provenance record says so explicitly, and says that it must never be edited, trimmed or re-rendered in place. That is a constraint the freezing procedure already imposes on every fixture, so it costs nothing — but it is now written down against the one document where breaking it would be a licence breach rather than merely bad method.

Tiers, assigned from measured geometry

The plan guesses tiers before anyone has seen the file. Column count is the criterion §4 turns on, and it was measured rather than eyeballed — by detecting text blocks that overlap vertically while being horizontally disjoint, which is what a real column does and what an indented sub-list never does. A plain left-edge histogram cannot tell those apart and initially mis-read NIST SP 800-171's indented requirement lists as a second column.

fixture measured layout tier note
y08, y09, y10, y17 single column ordinary_digital
y12_irs_pub15 2 columns on 100% of pages, 3 on 8% ordinary_digital corrected from the plan's designed_stress; §4 places plain two-column in ordinary
y13_irs_pub501 3+ columns on 92% of pages designed_stress genuine multi-column reading order
y11_nist_sp80053r5 single column unsupported 492 pages > the 250-page cap. Its only disqualification is length, which makes it a clean test of that one branch
y14_irs_fw9_form — unsupported 23 form widgets over 6 pages

The same measurement re-checked the already-sealed tranche-2 tiers and confirmed them: y06_irs_1040_instructions really is 3+ column on 68% of pages, so its designed_stress tier holds, and y01/y02/y03 really are single column.

Producer census, which was the point

Fourteen documents fetched across three rounds introduced producer chains the corpus had never contained, none of them written by this project:

producer chain fixtures
Acrobat PDFMaker for Word, gen. 11 / 17 / 20 y08, y01, y02 y09 y11
Microsoft Word 2010, direct — no Acrobat y10
pdfTeX (TeX Live 2020) y03
Antenna House PDF Output Library 6.6 / AH XSL Formatter y06, y12, y13
Adobe LiveCycle Designer 6.5 y07, y14
cairo 1.16.0 via xml2rfc 3.12.10 y17

y10 is worth singling out: producer and creator are both "Microsoft Word 2010", so it is Word's own PDF writer rather than Acrobat's. Against the five PDFMaker documents it separates "what Word's layout does" from "what Acrobat's writer does" — two things the corpus previously could not distinguish at all, because it had one word-processor document in total.

Measurement: the doubling is a producer property, not a length one

testkit/expansion.py --lane product, canonical container, measured at commit 0b4e289 — the unmodified tip, before the pagination fix being built concurrently. This is the "before" record; same §8 caveat, LibreOffice proxy rather than the Google Docs oracle.

fixture producer chain pages in → out ratio doc_recall word_recall mean_ssim
y09_nist_sp800207 Adobe PDF Library / PDFMaker 20 59 → 116 1.97× 0.956 0.155 0.312
y10_nist_fips180 Microsoft Word 2010 direct 36 → 40 1.11× 0.800 0.585 0.693
y17_rfc9110 cairo / xml2rfc 194 → 221 1.14× 0.978 0.346 0.561
y14_irs_fw9_form LiveCycle Designer 6 → 9 1.50× 0.990 0.213 0.268
y08_nist_sp80088r1 Adobe PDF Library 11.0 — crashed: UnrecognizedImageError
y12_irs_pub15 Antenna House — crashed: UnrecognizedImageError
y13_irs_pub501 Antenna House — crashed: UnrecognizedImageError

The headline question has a sharp answer, and it is not the one §9 assumed. Lining up every measured document by producer:

producer chain documents inflation
Acrobat PDFMaker for Word / Adobe PDF Library y01 1.98×, y02 2.75×, y09 1.97× ~2× and up
Microsoft Word 2010 direct y10 1.11× near-correct
pdfTeX y03 1.15× near-correct
cairo / xml2rfc y17 1.14× near-correct

Three PDFMaker documents all roughly double. Three non-PDFMaker documents all land within 15%. No overlap. §9 called this "pagination doubles on real documents" and attributed it to length; that was wrong. y17_rfc9110 is 194 pages — by far the longest document ever put through this converter — and inflates 1.14×. Length is not the variable. The variable is the producer, and specifically the Adobe PDF Library writer that Acrobat PDFMaker emits.

y10 is what makes the claim clean rather than suggestive. It is a Word document, from the same publisher, in the same series as y08/y09/y11, differing in exactly one respect: Word wrote the PDF itself instead of handing off to Acrobat. It does not double. So this is not "Word documents inflate" — it is the Adobe PDF Library output specifically.

The word/document recall split holds everywhere and is worth restating: y09 keeps 95.6% of the text (doc_recall) and places 15.5% of it (word_recall). The words survive; the pagination destroys their positions.

A second crash class. UnrecognizedImageError now accounts for four documents — y06, y08, y12, y13 — and the grouping is stark: all three Antenna House documents in the corpus fail, plus one Adobe PDF Library 11.0 document. Three of the seven round-3 documents produce no measurement at all, which is why the runner exits 1. That is an infrastructure failure, not a fidelity score: we do not know how well these convert, because they do not convert.

The refusal contract is absent, not leaky. y14 (23 widgets over 6 pages) converted, exactly as y07 (199 widgets over 2 pages) did — so the gap is general, not one document's accident. Reading the code explains why: MAX_PAGES_PER_DOCUMENT = 250 and the byte ceiling live only in exactdoc/batch.py, and neither expansion.py nor gdocs_oracle.prepare goes through it — both call exactdoc.convert.convert directly, which has no resource or capability check at all. The unsupported tier's expectation of "reject-before-qualification" is therefore unmet by construction on the qualification path. y11_nist_sp80053r5_overcap (492 pages) was deliberately not measured for that reason: it would not be refused, it would grind, and the code already answers the question.

11. Final census — acquisition closed

45 documents: 16 gated, 29 expansion. Inside the 40–60 target ROADMAP.md set. The gated 16 are byte-identical to what they were before any of this started; every number this repository has ever published still describes exactly the same inputs.

documents origin
gated metric corpus 16 generated in-repo
expansion tranche 1 16 generated in-repo
expansion tranche 2 13 downloaded
total 45

Expansion tiers: 24 ordinary_digital, 2 designed_stress, 3 unsupported.

Producer mix against the §3 target

producer class target achieved
word-processor export 12 13 LibreOffice ×7, Acrobat PDFMaker for Word ×5, Word 2010 direct ×1
browser print-to-PDF 12 14 Chromium/Skia
report generator 10 16 ReportLab ×9, fpdf2 ×2, Antenna House ×3, LiveCycle Designer ×2
LaTeX-light 8 1 pdfTeX (y03)
other / unknown real-world 6 1 cairo via xml2rfc (y17)

Two honest shortfalls. LaTeX is 1 of a target 8, because the only bulk source is arXiv and its default submission licence does not grant onward redistribution; the one LaTeX document held was reachable only because a NIST FIPS happens to be typeset with pdfTeX. "Other real-world" is 1 of 6, because the three RFC candidates and the CRS candidate never resolved to a working URL and Wikimedia rate-limited twice. Both are recorded in the annex in testkit/expansion_download_plan.json rather than quietly rounded up.

Against the two gaps §1 named at the start, the picture is different: word-processor coverage went from 1 document to 13, and the corpus went from 0 documents this project did not write to 13.

What the expansion actually bought

Four defects, none of which the gated 16 could express:

  1. Adobe PDF Library pagination inflation (~2×), producer-specific, proven by a same-publisher Word-direct control that does not inflate.
  2. UnrecognizedImageError on 4 of 13 real documents — every Antenna House document plus one Adobe, a third of the downloaded corpus producing no measurement at all.
  3. No refusal contract on the conversion path — resource and capability limits exist only in batch.py, which neither the qualification path nor any single-document caller goes through.
  4. fi-ligature loss in pdfTeX text extraction (§9), invisible to a reader and silently suppressing word recall.

All four are properties of the converter as of 0b4e289, all four are now measurable on committed bytes, and none of them gates anything yet. Promotion into the gate remains the deliberate commit described in §7 — and these numbers are the reason to be careful about it, not a reason to hurry.