|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## Repo layout & Deploy |
| 6 | + |
| 7 | +- **This directory is a subdirectory** of the git repo `hellpanderrr/hellpanderrr.github.io` (the repo root is the parent directory `F:\projects\wiktionary_pron`). Git paths are prefixed `wiktionary_pron/`. |
| 8 | +- **No build step.** Static site deployed on GitHub Pages from the `main` branch; live at https://hellpanderrr.github.io/wiktionary_pron/. |
| 9 | +- To preview: serve the repo root with any static file server (e.g. `python -m http.server` or VS Code Live Server). The Lua `require` shim fetches modules via relative paths like `../wiktionary_pron/lua_modules/...`, so serve from the parent directory. |
| 10 | + |
| 11 | +## Tests |
| 12 | + |
| 13 | +`npm install` once in this directory (`wiktionary_pron/`), then: |
| 14 | + |
| 15 | +```bash |
| 16 | +npm test # unit + IPA engine tests (Mocha, ~3s) |
| 17 | +npm run test:unit # pure JS helpers: sanitize, memoizeLocalStorage, V3/V4 lexicon decode |
| 18 | +npm run test:ipa # wasmoon Lua engine: exact-IPA tests + golden files (15 languages) |
| 19 | +npm run test:e2e # Playwright browser tests, excludes macronizer (~5 min: includes Russian lexicon load) |
| 20 | +npm run test:e2e:macronizer # macronizer smoke tests (~30s; covers first-visit and return-visit wordlist paths) |
| 21 | +npx playwright test -g "Latin" # run a single e2e test |
| 22 | +``` |
| 23 | + |
| 24 | +Golden files: `scripts/tests/golden/golden.json` holds expected IPA for ~50 word/language pairs. After an *intended* engine change (e.g. updating a Lua module from Wiktionary), regenerate with `cd scripts/tests && node golden/generate.js` and review the diff. Czech is Node-incompatible (module load fails under the test shim) — covered by e2e instead. |
| 25 | + |
| 26 | +`e2e/pending-features.spec.js` holds skipped acceptance tests for the stashed french-liaison and portuguese-support features — un-skip when applying those stashes. |
| 27 | + |
| 28 | +Notes: |
| 29 | +- `scripts/tests/setup.cjs` shims `localStorage` for Node (utils.js touches it at import time). Do **not** shim `document` there — wasmoon's Emscripten glue uses its presence for environment detection. |
| 30 | +- The e2e server serves the **parent** directory (repo root) because the Lua require shim fetches `../wiktionary_pron/lua_modules/...` relative to the page URL. |
| 31 | +- E2E tests must wait for `#lang` to be enabled before interacting — main.js top-level-awaits the wasmoon engine and attaches all listeners only after. |
| 32 | +- The browser Latin flow macronizes input before IPA (provinciarum → prōvinciārum), so e2e Latin expectations differ from the Node suite's. |
| 33 | +- CI: `.github/workflows/tests.yml` (repo root) runs both suites on push/PR. |
| 34 | + |
| 35 | +## Architecture |
| 36 | + |
| 37 | +### Entry flow |
| 38 | + |
| 39 | +1. `index.html` loads `scripts/main.js` (ES module) which imports everything else. |
| 40 | +2. On page load, `lua_init.js` initializes the **wasmoon** Lua 5.4 VM. It installs a custom Lua-side `require` shim: |
| 41 | + - Converts dot-separated paths (e.g. `ustring.charsets`) to slash-separated (`ustring/charsets`) |
| 42 | + - Fetches `.lua` files from `lua_modules/` over HTTP (or filesystem in tests) |
| 43 | + - Memoizes all requires via a Lua `memoize` wrapper to avoid redundant fetches |
| 44 | +3. `loadLanguage(code)` runs `require("<code>-pron_wasm")` inside Lua, which loads the Wiktionary pronunciation module and exposes it as `window[code + "_ipa"]`. |
| 45 | +4. When the user hits "Show transcription", `main.js` → `getIpa()` (memoized in localStorage) → `get_ipa_no_cache()` in `utils.js`. |
| 46 | + |
| 47 | +### IPA Router (`utils.js`) |
| 48 | + |
| 49 | +`get_ipa_no_cache(text, args)` is the central routing function. `args` is a semicolon-delimited string `"Language;Style;Form"` (e.g. `"Latin;Classical;Phonetic"`). |
| 50 | + |
| 51 | +The `ipaHandlers` object maps each language to a handler function that: |
| 52 | +- Receives `{ cleanText, lang, langStyle, langForm }` |
| 53 | +- Optionally consults a lexicon (`lookupInLexicon()`) |
| 54 | +- Calls the appropriate Lua-generated function (e.g. `window.la_ipa.convert_words(...)`) |
| 55 | +- Applies language-specific post-processing |
| 56 | + |
| 57 | +**Language handler groups:** |
| 58 | +- **Complex handlers** (Latin, Portuguese, Spanish, Greek, Armenian, Ukrainian, Russian, Italian) — bespoke logic per language |
| 59 | +- **Direct generation** (Belorussian, Bulgarian, Polish, Mongolian) — no lexicon fallback |
| 60 | +- **Lexicon lookup + generation** (German, French, Czech, Lithuanian, Icelandic) — try dictionary first, fall back to Lua rules |
| 61 | + |
| 62 | +### Lua Modules (`lua_modules/`) |
| 63 | + |
| 64 | +Two categories: |
| 65 | +- **Wiktionary modules** — verbatim from en.wiktionary.org (pronunciation modules like `la-pronunc` → inside `la-pron_wasm.lua` via require). Also MediaWiki compat layer: `mw.lua`, `mw-text.lua`, `mw-title.lua`, `ustring/`, `debug/`, etc. |
| 66 | +- **`*_wasm.lua` adapters** (16 languages) — thin wrappers that bridge the Wiktionary module's API to the interface expected by `loadLanguage()`. Each requires `mw`, the language's pronunciation module, and exports a function like `convert_words(...)` or `IPA(...)` that JavaScript calls via `window[code + "_ipa"].functionName(...)`. |
| 67 | + |
| 68 | +Example: `la-pron_wasm.lua`: |
| 69 | +```lua |
| 70 | +local m_IPA = require("IPA") |
| 71 | +local lang = require("languages").getByCode("la") |
| 72 | +-- exports convert_words(word, phonetic, eccl, vul) |
| 73 | +``` |
| 74 | + |
| 75 | +### Lexicons (`scripts/lexicon.js`) |
| 76 | + |
| 77 | +Some languages use dictionary lookup as a faster/more-authoritative source than Lua rules: |
| 78 | +- **German, Czech, French, Lithuanian, Ukrainian, Russian, Icelandic, Portuguese** |
| 79 | +- Stored as compressed `.zip` files in `utils/`, each containing a `lexicon.json` |
| 80 | +- Decompressed client-side via JSZip, loaded into `OptimizedV3Lexicon` (a `Map` wrapper) |
| 81 | +- **V3/V4 prefix compression**: entries stored as `[prefix_len, suffix, value]` triples — the key is reconstructed incrementally (`currentKey.substring(0, prefixLen) + suffix`). V4 format for RU/UK indexes the stressed vowel position instead of storing IPA. |
| 82 | +- Parsing yields to the browser via `setTimeout(0)` to keep UI responsive during large loads (500k+ entries for Russian). |
| 83 | +- **Chunked IndexedDB store** (`ChunkedLexicon`): decoded entries persist once as ~1000-word sorted range-chunk records. First visit parses the zip, serves from memory, and persists chunks in the background; return visits skip download+decode entirely and load only chunk *keys*. Lookups stay synchronous — `transcribe()` in `main.js` calls `lexicon.prefetch(words)` (async, pulls the needed chunks) before the sync `get()` calls run. Prefetch normalization must mirror `lookupInLexicon` (strip non-letters, retry lowercase). The zip filename in `LEXICON_LANGUAGES` acts as the version key — renaming the file invalidates stored chunks. |
| 84 | + |
| 85 | +### Caching Strategy |
| 86 | + |
| 87 | +| Data | Storage | TTL | Mechanism | |
| 88 | +|------|---------|-----|-----------| |
| 89 | +| IPA results | `localStorage` | 7 days | `memoizeLocalStorage()` in `utils.js` — wraps any function, supports background refresh near expiry | |
| 90 | +| Lexicon ZIPs | IndexedDB (localforage) | persistent | `fetchWithCache()` in `utils.js` — caches full HTTP responses | |
| 91 | +| TTS audio (Edge) | IndexedDB (raw) | persistent | `IndexedDBCache` class in `tts.js` — keyed by voice+rate+pitch+text hash | |
| 92 | + |
| 93 | +### TTS (`scripts/tts.js`) |
| 94 | + |
| 95 | +Two engines: |
| 96 | +- **Browser** (Web Speech API via EasySpeech wrapper) — fast, limited voices |
| 97 | +- **Edge** (`StreamingTTS` class) — higher quality via Microsoft Edge TTS API proxied through Cloudflare Workers. Caches audio blobs in IndexedDB. Auto-falls back through a pool of 6 worker endpoints with retry logic. |
| 98 | + |
| 99 | +### File-by-file overview |
| 100 | + |
| 101 | +| File | Role | |
| 102 | +|------|------| |
| 103 | +| `scripts/main.js` | UI controller — event handlers, DOM manipulation, transcription modes (default/line/column/sideBySide), pre-processing (liaison, macrons, stress marks), cycle-through-alternative-IPA clicking, dark mode, export triggers | |
| 104 | +| `scripts/lua_init.js` | Wasmoon engine init, custom Lua `require` shim, `loadLanguage()` | |
| 105 | +| `scripts/utils.js` | `get_ipa_no_cache()` router, `ipaHandlers`, `memoizeLocalStorage()`, `fetchWithCache()`, helpers (`sanitize`, `loadJs`, `loadFileFromZipOrPath`) | |
| 106 | +| `scripts/lexicon.js` | Lexicon download → decompress → parse V3/V4 → expose as Map-like interface | |
| 107 | +| `scripts/tts.js` | Dual-engine TTS (Browser + Edge StreamingTTS with Cloudflare Workers) | |
| 108 | +| `scripts/languages.js` | Language configs (styles, forms, langCode, ttsCode) | |
| 109 | +| `scripts/pdf_export.js` | Client-side PDF via pdf-lib (3 layout modes, fonts from `fonts/`) | |
| 110 | +| `scripts/csv_export.js` | Client-side CSV export | |
| 111 | +| `scripts/liaison.js` | French liaison marker insertion (nlp via fr-compromise) | |
| 112 | +| `scripts/macronizer.js` | Latin vowel-length dictionary lookup | |
| 113 | +| `scripts/dynamic_meta.js` | SEO meta tag updates per language | |
| 114 | +| `scripts/optimized_lexicon.js` | Alternative lexicon loader | |
| 115 | +| `scripts/lexicon_loader_worker.js` | WebWorker for lexicon parsing | |
| 116 | +| `lua_modules/*_wasm.lua` | Per-language adapter shims | |
| 117 | +| `lua_modules/` (rest) | Wiktionary Lua modules + MediaWiki compat layer | |
| 118 | +| `help/*.html` | Static help pages per language | |
| 119 | +| `css/style.css` | All custom styles (dark mode, popups, liaisons) | |
| 120 | + |
| 121 | +### Key patterns |
| 122 | + |
| 123 | +- **No bundler, all ES modules** loaded via `<script type="module">` in the browser. |
| 124 | +- **CDN dependencies** are loaded dynamically via `loadJs()` or static `<script>` tags: wasmoon, localforage, EasySpeech, JSZip, pdf-lib, fontkit, fr-compromise. |
| 125 | +- **Dark/light theme** toggled via `body.dark_mode` class — all components must support both. |
| 126 | +- **Async yielding** during long operations: `await wait(1)` / `await new Promise(r => setTimeout(r, 0))` to keep the UI thread responsive. |
| 127 | +- **Multiple IPA values** for a word are stored in `all_values` HTML attribute and cycled on click. |
| 128 | + |
| 129 | +## Self-correcting notes (mistakes made and fixed — don't repeat them) |
| 130 | + |
| 131 | +Each of these cost real debugging time in past sessions. Check this list before "fixing" related symptoms. |
| 132 | + |
| 133 | +**Testing traps** |
| 134 | +- **Never shim `document` in `scripts/tests/setup.cjs`** — wasmoon's Emscripten glue uses `typeof document` for environment detection and dies with "Invalid URL" under Node. Shim it per-test-file only in suites that don't load wasmoon (see `unit/lexicon_decode.test.js`). |
| 135 | +- **Lexicon test words must be letters only.** Every lookup path strips `[^\p{Letter}\p{Mark}-]` — synthetic words like `word000042` contain digits, get cleaned to `word`, and silently never match. Cost a failed "language isolation" test until spotted. |
| 136 | +- **Macronizer results have empty `textContent`** — words render via `<span class="ipa" content="...">` painted with CSS `attr(content)`. Assert the `content` attribute, not text. (Same idiom in the transcriber's line mode.) |
| 137 | +- **E2E must wait for `#lang` to be enabled before any interaction** — `main.js` top-level-awaits the wasmoon engine; clicking earlier hits elements with no listeners attached. Symptom: clear/dark-mode/persistence tests fail with stale values. |
| 138 | +- **Browser Latin ≠ Node Latin**: the browser flow macronizes input first (provinciarum → prōvinciārum), so e2e IPA expectations carry length marks that the Node suite's don't. |
| 139 | +- **RU/UK stress-transfer skips multi-form dictionary entries by design** — test it with a single-form word (голова → голова́), not вода (record "во́да, вода́" has a comma → skipped). |
| 140 | + |
| 141 | +**Code traps** |
| 142 | +- **Regex char classes with `-` between Unicode literals form ranges**: `[^\p{L}\p{M}'’-‿]` parsed `’-‿` as U+2019–U+203F and stripped ASCII hyphens from every word. Put `-` last in the class. (Was a live bug in `sanitize()` for years.) |
| 143 | +- **`#header > a > i` selects the HOME link's icon** — the dark-mode toggle on macronizer.html restyled the wrong button for this reason. Use `#dark_mode i`. |
| 144 | +- **Duplicated language lists in `main.js` drift**: `lang === "Lituanian"` (typo, missing h) appears in several copies of the multi-value language list — Lithuanian silently loses features in some modes. If touching those lists, extract one shared constant. |
| 145 | + |
| 146 | +**IndexedDB performance (measured in Chromium, 100k rows)** |
| 147 | +- Row-per-entry `put()` with a secondary index is the killer: ~58s/100k. `durability: 'relaxed'` changes nothing (already default). Grouping by unique key: 1.8×. **Packing ~1000 rows per record: 20×** (2.8s/100k). This is why both the macronizer wordlist and the app lexicons use sorted range-chunk records. |
| 148 | +- **Structured-cloning big objects is the persist wall time** — storing each chunk's payload as one JSON string (parse on read) cut the French lexicon persist ~3× to ~11s. |
| 149 | +- **Write the meta record last** so an interrupted persist reads as unpopulated. And **show the user that the background save is running** — invisible saves get interrupted by reloads, which looks like "caching never works" (that exact bug report happened). |
| 150 | + |
| 151 | +**Git traps (repo layout)** |
| 152 | +- `git checkout <branch> -- <file>` **stages** the restored file — follow with `git restore --staged` if you want it unstaged. |
| 153 | +- The git repo root is the parent directory; `.github/workflows/` and `.gitignore` live there, not here. |
| 154 | +- GitHub Pages deploys from `main` only — work on other branches is invisible in production until merged. |
| 155 | + |
| 156 | +## Work-in-progress state (as of 2026-07-27) |
| 157 | + |
| 158 | +Uncommitted local work was split into stashes on `main` (with a full backup on branch `wip-everything`): |
| 159 | + |
| 160 | +| Stash | Contents | |
| 161 | +|-------|----------| |
| 162 | +| `french-liaison` | Liaison preprocessing/tooltips in `main.js`, new `scripts/liaison.js`, liaison styles in `css/style.css`, checkbox in `index.html` | |
| 163 | +| `portuguese-support` | Portuguese in `main.js` dict/multi-value lists, PT lexicon fallback in `utils.js`, PT entry in `lexicon.js` | |
| 164 | +| `help-pages-and-fixes` | Stripped-down help pages (superseded by remote versions pulled later) + `pdf_export.js` local font-path fix | |
| 165 | +| `utils-scripts-and-tests` | Python/CJS lexicon build scripts and Lua verify tests in `utils/` | |
| 166 | + |
| 167 | +`wip-everything` also holds large generated lexicon data files in `utils/` that were never committed to `main`. Note: parts of `main.js`/`utils.js`/`liaison.js`/`lexicon.js` currently on `main` may not include these stashed features until the stashes are applied. |
| 168 | + |
| 169 | +### Update 2026-07-28 |
| 170 | + |
| 171 | +Branch `macronizer` (pushed to origin, **not merged to `main`** — GitHub Pages still serves the pre-macronizer site) now carries, on top of the stash situation above: |
| 172 | +- the macronizer page + WASM engine (`macronizer/dist/`, synced from the `latin-macronizer-wasm` repo's `macronizer-ui-support` branch — never hand-edit `dist/`) |
| 173 | +- the full test infrastructure (see Tests section) + CI workflow |
| 174 | +- range-chunk IndexedDB stores for both the macronizer wordlist (first visit 10min → ~10s) and the app lexicons (return visits skip parsing) |
| 175 | +- `e2e/pending-features.spec.js` still holds skipped acceptance tests for the stashes |
| 176 | + |
| 177 | +Applying the `french-liaison`/`portuguese-support` stashes onto this branch will conflict lightly in `main.js` (a prefetch block was added to `transcribe()`) and `lexicon.js` (chunk-store layer added) — resolve keeping both. |
0 commit comments