fix(macronizer): input-mirrored u/v, grey-box + line-number scansion fixes

- divisa/diviso bug: the display forced the checkbox orthography (u) even
  when the user typed v, so the wordlist's classical v form showed briefly
  then flipped to u forever. applyOrtho now takes the input text — the
  display mirrors what was typed (divisa -> dīvīsa, diuisa -> dīuīsa) while
  the u->v / i->j checkboxes still force classical forms.
- M-003 regression: the grey "—" placeholder rendered in prose mode where
  no scansion was requested. The foot cell now only renders when a meter is
  selected.
- Trailing line numbers (urbem, 5 / libellum 1.1) are stripped before
  processing — they changed the last word's following-segment and made the
  verse impossible to scan (grey box).
- e2e: popup-check.spec.js adds divisa reversibility + prose/no-placeholder
  + numbered-verse tests; macronizer.spec.js virum regex fixed.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

Author: hellpanderrr

wiktionary_pron/NEXT.md

@@ -1,48 +1,59 @@
 # Next
 
-_Updated 2026-08-05 — branch `main`_
+_Updated 2026-08-06 — branch `main`_
 
 ## State
-Macronizer scansion saga fully resolved. Site `main` (live on Pages) = `748b6c3`;
-engine `main` (was `macronizer-ui-support`, now fast-forwarded, branch deleted)
-= `e7fb22a`. Both pushed, both working trees clean.
+Session: UI fixes + M-013 Phase 2 (scansion investigation) complete. Site `main`
+= `ddb8d44`; engine `main` = `e7fb22a`. New commits pending in both (see below).
 
-What's fixed and committed:
-- **M-003** scansion display + root cause: `macrons.txt` had `italorum` short-`I^`,
-  but hendecasyllable pos 8 needs `Ītalōrum`. Fixed via `ACCENT_OVERRIDES` in the
-  engine (dual reading, prose keeps short), not a one-off wordlist edit (33MB
-  file is regenerated upstream).
-- **M-010/011/012** popup honesty: "Wordlist: Found" no longer lies for
-  Morpheus-rescued words, Morpheus rows deduplicated, RFTagger-vs-reading POS
-  disagreement noted. Guarded by `e2e/popup-check.spec.js`.
+Fixed and committed this session:
+- **divisa/diviso click-cycle bug** — display now mirrors the input's own u/v
+  orthography (`divisa` stays `dīvīsa`, `diuisa` stays `dīuīsa`). Previously
+  the wordlist's classical-`v` form showed briefly, then the first click forced
+  `u` and the original was unreachable. `applyOrtho` now takes the input text.
+- **Grey `—` box (M-003 regression)** — the foot placeholder now only renders
+  when a meter is selected (prose shows nothing). And trailing line numbers
+  (`urbem, 5`, `libellum 1.1`) are stripped before processing — a trailing
+  number changed the last word's "following segment" and made the verse
+  impossible to scan.
+- **M-013 Phase 2** — see `docs/ISSUES.md`. Conclusion: the 153 flagged lines
+  are **engine/prosody limitations, not wordlist gaps**. Aen 2.774 (`obstupui,
+  steteruntque comae...`) is canonical hexameter the engine can't scan because
+  `ui` isn't treated as a diphthong. `ACCENT_OVERRIDES` was right for italorum
+  but there's no pile of similar cases.
 
 ## Open threads
-- **M-013 — the scansion wordlist-gap miner.** Tooling is committed (engine
-  `test/miner-scansion.mjs` + `test/data/corpus/`). **176 lines flagged** across
-  Aeneid 1-6 + Catullus (5,507 lines). Three buckets: Greek names (engine
-  limitation), ambiguous/enclitic forms (candidate gaps), and common-word
-  failures like Catullus 13.11 `nam unguentum dabo` (automaton strictness).
-  **Next action: Phase 2** — confirm flagged lines against Pedecerto
-  (pedecerto.eu/scansioni, single-verse lookup, no bulk scrape) to separate
-  real gaps from engine limits. Run: `node test/miner-scansion.mjs` in the
-  engine repo.
-- **Reply to the r/latin thread.** M-003 is fixed too now. See `docs/ISSUES.md`.
+- **M-013 Phase 3 (if pursued): the scansion engine's prosody rules** —
+  diphthong `ui`, elision across `-m`, automaton strictness. The regression
+  gate now exists: `npm run test:scansion` (engine) asserts 5 canonical lines
+  scan and no NEW failures beyond `test/data/scansion-failures-snapshot.json`
+  (153 lines). Any future scansion fix must keep that gate green.
+- **Commit the pending work in BOTH repos** — the session ended with uncommitted
+  changes:
+  - Site: `macronizer.html` + `e2e/popup-check.spec.js` + `e2e/macronizer.spec.js`
+    + `docs/ISSUES.md` + `NEXT.md` (this file).
+  - Engine: corpus cleanup (9 files), `test/categorize-miner.mjs`,
+    `test/culprit-words.mjs`, `test/e2e/test-scansion-corpus.mjs`,
+    `test/data/scansion-failures-snapshot.json`, `package.json` script.
+  Commit + push both.
+- **Reply to the r/latin thread** — M-003 is fixed. See `docs/ISSUES.md`.
 
 ## Running / unfinished
-Nothing running. A local static server may still be on :8993 (harmless).
+Nothing running. The corpus test takes ~1-2 min (WASM + wordlist load).
 
 ## Don't redo
-- **`^` means short, `_` means long — full stop.** The initial "ambiguous marker"
-  fix to `possibleScans` was WRONG and reverted. `italorum` was a data gap, not
-  an engine bug. (Classicist agent confirmed Ītalōrum is required by Catullus
-  1.5 itself; Ellis 1.5 note is about chronica realia, not quantity.)
-- **`public/macrons.txt` is regenerated from upstream** — never hand-edit it for
-  a one-off fix; the edit silently vanishes on next pull. Use
-  `ACCENT_OVERRIDES` in `src/core/Tokenization.ts` instead.
-- **The Aeneid corpus must be stripped of Latin Library line numbers** (every 5th
-  line). The miner's ~160-fail-per-book artifact was those trailing digits.
-- **Catullus 8 is choliambic, not hendecasyllable** — assigning it to the
-  hendecasyllable corpus made all 19 lines "fail." Same for 4/29/52 (iambic
-  trimeter, engine's "iambic" is trimeter+dimeter).
-- **Morpheus is NOT broken** (see prior NEXT.md) — the byte-grep false lead.
-- **Never run `build-morpheus-wasm.sh` against a read-write engine mount.**
+- **`^` means short, `_` means long — full stop.** `italorum` was a data gap,
+  not an engine bug.
+- **`public/macrons.txt` is regenerated from upstream** — never hand-edit. Use
+  `ACCENT_OVERRIDES` in `src/core/Tokenization.ts`.
+- **The Aeneid corpus must be stripped of line numbers and page furniture** —
+  trailing digits AND the `Vergil` / `The Classics Page` footers break scansion.
+- **Catullus 62 refrains ("Hymen o Hymenaee") are lyric chants, not hexameter** —
+  don't put them in the hexameter corpus (verified via Tavily).
+- **Catullus 8 is choliambic**; 4/29/52 iambic trimeter (engine's "iambic" is
+  trimeter+dimeter).
+- **normalizeLine must strip macrons BEFORE [^a-z ]** — the corpus is stored
+  macronized; a naive strip mangles `canō` → `can`.
+- **Pedecerto is IP-blocked (HTTP 412)** even with a browser UA — use Tavily.
+- **Morpheus is NOT broken.** **Never run `build-morpheus-wasm.sh` against a
+  read-write engine mount.**

wiktionary_pron/docs/ISSUES.md

@@ -89,12 +89,28 @@ compares the RFTagger POS against the active reading's POS and shows a
 note when they disagree.
 
 ## M-013 — Scansion wordlist-gap miner + corpus (found, not fixed)
-**Status: OPEN** (engine `e7fb22a` added the tooling)
+**Status: OPEN** (engine `e7fb22a` added the tooling; `bfb5035` Phase 2 analysis)
 `test/miner-scansion.mjs` + `test/data/corpus/` feed Aeneid 1–6 + Catullus
 (5,507 lines) through the macronizer and flag lines whose scansion returns
-empty — the italorum signature. **176 lines flagged** (~130 hexameter, ~46
-Catullus). Buckets: Greek names/proper nouns (engine limitation), known
-ambiguous/enclitic forms (`unius`, `-ne` enclitics — candidate gaps), and
-common-word lines that still fail (Catullus 13.11 `nam unguentum dabo`
-suggests an automaton strictness issue, not a data gap). Needs Phase 2:
-confirm each against Pedecerto to separate real gaps from engine limits.
+empty — the italorum signature. **153 lines flagged** after corpus cleanup
+(176 minus page furniture — `Vergil`/`The Classics Page` footers, a stray
+`+`, em-dashes — and the Catullus 62 refrains, which are lyric chants, not
+hexameter). Categorized via `test/categorize-miner.mjs`:
+- **Known words but scansion fails (155)** — the italorum signature. Deep
+  dive: Aen 2.774 `obstupui, steteruntque comae et vox faucibus haesit` is
+  canonical hexameter but the engine stores `obstipui → obsti^pu^i_` (4
+  syllables) and `segmentAccented` doesn't treat `ui` as a diphthong — a
+  prosody-model limitation, not a wordlist gap. No word shows a wrong
+  quantity; the top culprits (`que`, `et`, `non`) are correctly macronized.
+- **Contains unknown word (7)** — Greek names (`Thesea`, `Euryalus`,
+  `Helymus`), an engine limitation.
+- **Conclusion: NO wordlist gaps beyond the already-fixed italorum.**
+  `ACCENT_OVERRIDES` was right for italorum but there is no pile of similar
+  cases. The real next step is the scansion engine's prosody rules
+  (diphthong `ui`, elision, automaton strictness).
+Phase 2 added `test/e2e/test-scansion-corpus.mjs` — a regression gate that
+asserts 5 canonical lines scan (including italorum) and that no NEW failures
+appear beyond the recorded snapshot `test/data/scansion-failures-snapshot.json`.
+Run: `npm run test:scansion` (engine repo). Pedecerto (pedecerto.eu) is
+IP-blocked (HTTP 412) even with a browser UA, so confirmation was done via
+Tavily instead.

wiktionary_pron/e2e/macronizer.spec.js

@@ -84,10 +84,11 @@ test.describe("macronizer", () => {
       "arma",
       { timeout: 60_000 },
     );
-    // virum has a long u — proves chunk lookups return real entries
+    // virum — proves chunk lookups return real entries. The display mirrors the
+    // user's u/v setting (default u) and the i may be short in prose, so match loosely.
     await expect(page.locator("#resultText .ipa").nth(1)).toHaveAttribute(
       "content",
-      /vir[ūu]mque/,
+      /[vu][īi]?r[ūu]mque/,
       { timeout: 60_000 },
     );
   });

wiktionary_pron/e2e/popup-check.spec.js

@@ -28,3 +28,57 @@ test("popup shows RFTagger disagreement + Morpheus dedup for currito", async ({
   const wlLabel = page.locator(".word-popup tr", { hasText: "Wordlist:" });
   await expect(wlLabel).toContainText("Not found");
 });
+
+test("v/u words cycle reversibly — divisa's original spelling must come back", async ({ page }) => {
+  test.setTimeout(300_000);
+  await page.goto(PAGE);
+  await expect(page.locator("#macronize_btn")).toBeEnabled({ timeout: 240_000 });
+
+  await page.fill("#text_to_macronize", "divisa");
+  await page.click("#macronize_btn");
+  const span = page.locator("#resultText .ipa").first();
+  await expect(span).toBeVisible({ timeout: 120_000 });
+
+  // Input mirroring: the word was typed with v, so it must display dīvīsa (v) —
+  // not dīuīsa. And that spelling must already be a candidate, or the first cycle
+  // jumps somewhere and the original is unreachable forever (the divisa bug).
+  await expect(span).toHaveAttribute("content", "dīvīsa");
+  const initial = await span.getAttribute("content");
+
+  await span.click(); // hover already opened the popup; this pins it (may cycle once)
+  const nextBtn = page.locator(".word-popup .popup-cycle");
+  await expect(nextBtn).toBeVisible({ timeout: 5000 });
+
+  // Keep clicking "Next spelling" until we wrap back to the initial spelling —
+  // that is exactly what the old code could never do.
+  for (let i = 0; i < 8; i++) {
+    if ((await span.getAttribute("content")) === initial) break;
+    await nextBtn.click();
+  }
+  await expect(span).toHaveAttribute("content", initial);
+});
+
+test("prose shows no grey placeholders; numbered verse scans after line-number strip", async ({ page }) => {
+  test.setTimeout(300_000);
+  await page.goto(PAGE);
+  await expect(page.locator("#macronize_btn")).toBeEnabled({ timeout: 240_000 });
+
+  // Prose mode (default): no scansion requested, so no grey "—" placeholders at all.
+  await page.fill("#text_to_macronize", "Cui dono lepidum novum libellum");
+  await page.click("#macronize_btn");
+  await expect(page.locator("#resultText .ipa").first()).toBeVisible({ timeout: 120_000 });
+  await expect(page.locator("#resultText .verse-foot")).toHaveCount(0);
+
+  // Meter mode + trailing line numbers: stripped before processing, so both scan.
+  await page.selectOption("#scan", "hendecasyllable");
+  await page.fill(
+    "#text_to_macronize",
+    "Cui dono lepidum novum libellum 1.1\niam tum, cum ausus es unus Italorum 5",
+  );
+  await page.click("#macronize_btn");
+  await expect(page.locator("#resultText .verse-foot").first()).toBeVisible({ timeout: 120_000 });
+  await expect(page.locator("#resultText .verse-foot")).toHaveCount(2);
+  await expect(page.locator("#resultText .verse-foot.no-scan")).toHaveCount(0);
+  // The line numbers are reference noise — they must not survive into the output.
+  await expect(page.locator("#resultText")).not.toContainText("1.1");
+});

wiktionary_pron/macronizer.html

@@ -581,24 +581,29 @@ <h2 class="result-heading">Macronized</h2>
     // Orthography options of the last run — wordlist forms use classical j/v and
     // must be converted back to i/u unless the user enabled those conversions
     let lastOrtho = { utov: false, itoj: false };
-    function applyOrtho(c) {
-        if (!lastOrtho.itoj) c = c.replace(/j/g, 'i').replace(/J/g, 'I');
-        if (!lastOrtho.utov) c = c.replace(/v/g, 'u').replace(/V/g, 'U');
+    // A displayed word mirrors the orthography the USER typed it with: "divisa"
+    // stays dīvīsa, "diuisa" stays dīuīsa. The i→j / u→v checkboxes still force the
+    // classical forms regardless of input. Without the input mirror, a word typed
+    // with v shows its wordlist v form first, the first click converts it to u, and
+    // the original spelling is unreachable (the divisa/diviso bug).
+    function applyOrtho(c, inputText) {
+        if (!(lastOrtho.itoj || (inputText && /[jJ]/.test(inputText)))) c = c.replace(/j/g, 'i').replace(/J/g, 'I');
+        if (!(lastOrtho.utov || (inputText && /[vV]/.test(inputText)))) c = c.replace(/v/g, 'u').replace(/V/g, 'U');
         return c;
     }
 
     // wordlist form → text form used for cycling: a_ → ā, breve/ambiguity markers stripped
-    function candidateToDisplay(c) {
-        return applyOrtho(underscoreToUnicode(c).replace(/[\^+_]/g, ''));
+    function candidateToDisplay(c, inputText) {
+        return applyOrtho(underscoreToUnicode(c).replace(/[\^+_]/g, ''), inputText);
     }
 
     // wordlist form → popup form: also render breves (a^ → ă) so candidates that
     // differ only in vowel shortness stay distinguishable (me^mo^rem vs me^morem)
     const BREVE_MAP = { a:'ă', e:'ĕ', i:'ĭ', o:'ŏ', u:'ŭ', A:'Ă', E:'Ĕ', I:'Ĭ', O:'Ŏ', U:'Ŭ' };
-    function candidateToPopup(c) {
+    function candidateToPopup(c, inputText) {
         return applyOrtho(underscoreToUnicode(c)
             .replace(/([a-zA-ZāēīōūȳĀĒĪŌŪȲ])\^/g, (m, v) => BREVE_MAP[v] || (v + '̆'))
-            .replace(/[+_]/g, ''));
+            .replace(/[+_]/g, ''), inputText);
     }
 
     await window.__wasmReady;
@@ -657,8 +662,16 @@ <h2 class="result-heading">Macronized</h2>
 
         // Strip macrons/breves from input — the pipeline expects unmarked text
         // (macronized words would otherwise be torn apart and sent to Morpheus as unknowns)
-        const text = document.getElementById('text_to_macronize').value
+        let text = document.getElementById('text_to_macronize').value
             .normalize('NFD').replace(/[̄̆]/g, '').normalize('NFC');
+        // Latin Library / textbook sources tag lines with trailing verse numbers
+        // ("urbem, 5", "libellum 1.1"). A trailing number is reference noise, not text —
+        // and by changing the last word's "following segment" it makes the verse
+        // impossible to scan (the grey box). Strip them, same as the miner's corpus prep.
+        text = text.split('\n').map(line =>
+            line.replace(/^\s*\d+(?:[.,]\d+)*\s*$/, '')
+                .replace(/\s*\d+(?:[.,]\d+)*\s*$/, '')
+        ).join('\n');
         if (!text.trim()) { showError('Please enter some text'); return; }
 
         document.getElementById('loading').style.display = 'block';
@@ -800,7 +813,7 @@ <h2 class="result-heading">Macronized</h2>
     // Build rich popup HTML for a token (table layout)
     function buildPopupHtml(token, displayText, encliticText) {
         // popup shows breved forms so quantity-only differences stay visible
-        const candidates = [...new Set((token.accented || []).map(candidateToPopup))];
+        const candidates = [...new Set((token.accented || []).map(c => candidateToPopup(c, token.text)))];
 
         const flags = [];
         if (token.isAmbiguous) flags.push('Ambiguous');
@@ -817,7 +830,7 @@ <h2 class="result-heading">Macronized</h2>
             const seen = new Set();
             const pairs = [];
             (token.accented || []).forEach((c, i) => {
-                const shown = matchCase(token.text, candidateToPopup(c)) + (encliticText || '');
+                const shown = matchCase(token.text, candidateToPopup(c, token.text)) + (encliticText || '');
                 if (seen.has(shown)) return;
                 seen.add(shown);
                 // each reading carries the lemma + tag of the wordlist row it came from,
@@ -826,7 +839,7 @@ <h2 class="result-heading">Macronized</h2>
                           : sources.find(s => s.accented === c);
                 pairs.push({
                     shown,
-                    plain: matchCase(token.text, candidateToDisplay(c)) + (encliticText || ''),
+                    plain: matchCase(token.text, candidateToDisplay(c, token.text)) + (encliticText || ''),
                     lemma: src && src.lemma ? src.lemma : null,
                     tag: src && src.tag ? src.tag : null
                 });
@@ -1102,6 +1115,10 @@ <h2 class="result-heading">Macronized</h2>
         }
 
         const scannedFeet = result.scannedFeet || [];
+        // Prose never scans, so in prose mode scannedFeet is empty and every line
+        // would otherwise get the "no scansion" placeholder — noise. Only render the
+        // foot cell when a meter is selected.
+        const scanMode = document.getElementById('scan').value;
         const tbody = document.getElementById('resultText');
         tbody.innerHTML = '';
 
@@ -1112,7 +1129,13 @@ <h2 class="result-heading">Macronized</h2>
             const td = document.createElement('td');
             td.className = 'macron-line';
 
-            const tokenDisplay = t => ((t.macronizedText || t.text).replace(/[\^+_]/g, '')) || t.text;
+            // Must mirror the cycle candidates (ortho + case) or indexOf(cur) misses and
+            // the first click jumps to a candidate the original spelling can never return to.
+            const tokenDisplay = t => {
+                const raw = (t.macronizedText || t.text).replace(/[\^+_]/g, '');
+                const ortho = matchCase(t.text, applyOrtho(raw, t.text));
+                return ortho || raw;
+            };
 
             for (let ti = 0; ti < lineTokens.length; ti++) {
                 const token = lineTokens[ti];
@@ -1138,7 +1161,7 @@ <h2 class="result-heading">Macronized</h2>
                 // make text selectable
                 //span.textContent = displayText;
                 const candidates = [...new Set(
-                    (token.accented || []).map(c => matchCase(token.text, candidateToDisplay(c)))
+                    (token.accented || []).map(c => matchCase(token.text, candidateToDisplay(c, token.text)))
                 )].map(c => encliticText ? c + encliticText : c);
                 span.__popupHtml = buildPopupHtml(token, displayText, encliticText);
 
@@ -1193,16 +1216,18 @@ <h2 class="result-heading">Macronized</h2>
                 td.appendChild(span);
             }
 
-            const foot = scannedFeet[lineIdx];
-            const footSpan = document.createElement('span');
-            footSpan.className = 'verse-foot';
-            if (foot) {
-                footSpan.textContent = foot;
-            } else {
-                footSpan.classList.add('no-scan');
-                footSpan.textContent = '—';
+            if (scanMode !== 'prose') {
+                const foot = scannedFeet[lineIdx];
+                const footSpan = document.createElement('span');
+                footSpan.className = 'verse-foot';
+                if (foot) {
+                    footSpan.textContent = foot;
+                } else {
+                    footSpan.classList.add('no-scan');
+                    footSpan.textContent = '—';
+                }
+                td.appendChild(footSpan);
             }
-            td.appendChild(footSpan);
             tr.appendChild(td);
             tbody.appendChild(tr);
         }