Skip to content

Commit 4dcfaa2

Browse files
Punjabi and Hindi, and rtl becomes a script (#53)
* The Punjabi and Hindi Bibles data/pan-fbi.jsonl and data/hin-fbi.jsonl, stamped pan-fbi-tok1 and hin-fbi-tok1, in kjv.jsonl's frozen shape. Free Bibles India's USFM editions of ਪਵਿੱਤਰ ਬਾਈਬਲ and पवित्र बाइबल, CC BY-SA 4.0 — the traditional Protestant Bible of each language, at KJV addresses, Textus Receptus in all twenty readings a critical text omits or brackets. One build script for two languages, which the other three corpora do not do. Same publisher, same conventions, same two verse splits; two files would be one file and a copy of it. THE PUNJABI IS NOT THE OBVIOUS TEXT. tfbf/Bible-Punjabi-Pavitr-Bible-1945 is public domain outright and was rejected on the evidence: eight whole books of it are a different modern translation spliced in — Titus, John, James, 1 Peter, 1-2 Thessalonians, 2 Peter, 1 Corinthians, 1,772 verses — plus ~217 scattered verses, Acts 8:37 among them. Every other check passes on that file. What gives it away is punctuation: the 1945 keyboarding types the danda as an ASCII "|" in 19,306 verses and the spliced books use a real U+0964. Its own STATUS.md says the files are not ready to be used in a real project. So check-indic.py carries that test as a standing claim — a sentence terminator over 1% of the corpus's must be used by 90% of its books — because the next Indian-language corpus offered here is likely to have been assembled the same way. Three things the checks caught in this build, none of which any other check would have: \q lines are POETRY, not skippable decoration, and skipping them ended Psalm 23:1 at "the LORD is my shepherd,"; \qa is Psalm 119's acrostic stanza heading, which kjv.jsonl already carries as a title token at every eighth verse; and Rev 12:18 merges FORWARD into 13:1 while 3 John 15 merges back into 14, so a build that appended both keeps every word and prints Revelation 13:1 with its first clause last. The text is never normalised. NFC decomposes the precomposed Gurmukhi nukta letters — they are on Unicode's composition exclusion list — so normalising rewrites letters. Both files are already stable under NFC and NFD, measured, and the check asserts it so the one line a later maintainer might add by habit fails the build. No Strong's codes and none invented, as with the Van Dyck. * Punjabi and Hindi are rows in the registry Lang::Pa reads pan-fbi.jsonl, Lang::Hi reads hin-fbi.jsonl. The interface is translated — 868 keys each, book names taken from the publishers' own \h fields and shortened to the forms a picker can hold. lexicon: None for both, as Arabic's is: neither corpus carries Strong's codes and none are invented. `rtl` IS NOW A SCRIPT. Every non-Latin question this app had answered, it had answered for Arabic — the only language that is neither Latin nor left to right — so "which faces can set this" and "which way does the page run" had one answer and were written as one condition. Gurmukhi and Devanagari split them: Punjabi reads left to right and no Latin face has a glyph of it. So LangSpec carries a Script, Font carries the script it can set, and offered_for compares the two instead of asking is_rtl(). Direction is derived from the script, which is the only thing that decides it. Without that, readerFace hands a Punjabi reader EB Garamond. Nothing errors — the text renders through per-glyph fallback at a Latin face's optical scale, in whatever the browser found, measured in the worker against something else again. FOUR THINGS THE SCRIPTS TURNED UP, none of which any existing test could see: - normalize_word DELETED EVERY VIRAMA. char::is_alphanumeric is false for U+094D, so the filter that strips punctuation stripped the join out of every conjunct: परमेश्वर indexed as परमेशवर. Applied to both sides, so no search broke — it merged अन्त with अनत and nothing said so. The exact mirror of the Arabic bug this file already records, where every mark carried Other_Alphabetic and the same filter kept a vowelling it should have dropped. - THE NUKTA FOLD HAS TO BE PER LETTER. Dropping it wholesale is Arabic's alef fold and looks right; counted over the corpora it merges ਸ with ਸ਼ (61.4% of Punjabi's nuktas) and ड with ड़ (74.5% of Hindi's). Those are native letters. Only the Perso-Arabic set — क ख ग ज फ, ਖ ਗ ਜ ਫ — is a dot readers leave off, and only it folds. - blank_out LEAKED THE WORD IT WAS HIDING. mask_word maps on is_alphanumeric and passes everything else through verbatim, so a masked परमेश्वर came out _____्__ with the virama drawn on a dotted circle, and a masked Arabic word kept all its tashkeel. Arabic had shipped with that. - is_indic_mark's two ranges are NOT the same shape. Devanagari's marks start at U+093A so its nukta at U+093C falls inside; Gurmukhi's matras start at U+0A3E and its nukta at U+0A3C sits two below. The range copied across from Devanagari deletes every Punjabi nukta and keeps every Hindi one. Fonts: Noto Serif Gurmukhi and Noto Serif Devanagari, OFL, subset with every layout feature kept — an Indic script is SHAPED, and subsetting away nukt/half/ blwf/pres leaves each conjunct as separate letters at a different advance than the shaped text, which is the measure-here-paint-there split at the scale of most words in the language. The danda at U+0964 is in both subsets: it lives outside either script's block and ends 27,354 Punjabi and 31,541 Hindi verses. The three script faces stay loaded unconditionally in the engine worker, and the cheap narrowing is wrong for a reason worth writing down: the pane strip lets a German reader open the Van Dyck beside the Luther, so the script this worker is asked to MEASURE is not a function of the language it paints chrome in. They cost 287 KB rather than 106 — Devanagari's unused wdth axis is pinned out first (237 KB → 130 KB) and the three now load in parallel rather than one after another. BIBLIOGRAPHY.md gains both CC BY-SA 4.0 rows with their attribution and their statement of modification, and closes two gaps the Arabic release left: the Van Dyck had no row and Amiri no mention. * TODO: Punjabi and Hindi are shipped, and the 1945 text is not the one * The language picker names each language twice An English reader's list read "Deutsch · Español · العربية · ਪੰਜਾਬੀ · हिन्दी" — right for the reader looking for their own language, useless to the person offering their phone to a Hindi speaker, who was looking at six scripts they cannot read with no way to tell which row was the one. This app is built to be handed over. It now reads "Hindi (हिन्दी)": the English name to scan, the endonym to confirm. English only, and that is the data rather than a decision — `exonym` on the registry row is the language's ENGLISH name, so there is nothing to put in front of the bracket for a German reader with the same problem. They keep the endonyms alone. A LangSpec carrying names per language would serve them too. No bracket where the two are the same word; "English (English)" is noise. The three language pickers in the e2e suite now select by VALUE — the option's language code — rather than by label. The label is exactly what this commit edits, so a test that spelled it broke on a copy change and said nothing useful when it did. * The picker names a language in its own words and the reader's "ਪੰਜਾਬੀ (Punjabi)" to an English reader, "English (Englisch)" to a German one. Either half alone serves one person and fails the other. The endonym alone is right for someone looking for their own language and useless to someone handing their phone to a Hindi speaker — six scripts they cannot read, no way to tell which row is the one. The reader's own name alone is the reverse: it puts "Punjabi" in Latin letters in front of the person being handed the phone. So both, and the ENDONYM LEADS, because the row belongs to the person being offered it and the bracket is the reader's note to themselves. The bracket is a catalogue lookup, `lang.<code>`, in all six catalogues — 36 strings. Not the registry's `exonym`: that column is the language's ENGLISH name, which is what the hymnal finder matches on, and it could only ever have served an English reader. No bracket when the two are the same word. Not a special case for English — it is the one comparison that silences the reader's own language in every catalogue, "Deutsch (Deutsch)" in German and "हिन्दी (हिन्दी)" in Hindi. * Two screens named a book from the frozen English table The reading plan's card read "Today: John 3" to a Hindi reader while the passage navigator, the search results and every other reference on the same screen read "यूहन्ना 3". `canon::display_name` is the table `refKey` is built from — right for storage, never right for a reader — and in an English test run it is indistinguishable from the correct call. The second site is worse for sitting six lines under two calls that get it right: copying a VERSE gave that reader "यूहन्ना 3:16" and copying the CHAPTER it is in gave them "John 3", from the same function. `copy_text` now takes `lang` instead of reading `i18n::active()` inside it. It renders text for one reader, so the language is an input — the same reason `VRef::display_in` exists beside `VRef::display` — and the hidden global was not merely untidy: `set_active` is process-wide, the test binary runs in parallel, and the first attempt at a German test broke two English ones running in another thread. The guard is a source scan: nothing outside `canon` and `book_name` may call `display_name`. A behavioural test would have to be written once per screen, and the failure here is a call that should not exist rather than an output that is wrong. It asserts it actually reached the files, because a walk that found nothing would pass silently forever. * Version 0.64.0 — Punjabi and Hindi
1 parent bf6ac00 commit 4dcfaa2

43 files changed

Lines changed: 66914 additions & 714 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

BIBLIOGRAPHY.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ module id `engKJV2006eb` itself no longer resolves there (checked 2026-07-29).
4242
| `data/strongs-de.json` | German Strong's dictionary: machine-translated definitions + Luther renderings derived from the tagged corpus | translation of `data/strongs.json` by `data-prep/strongs-lang/translate.py` (Claude, Batch API; labelled as machine-translated in the app) + renderings computed by `build-strongs.py` | **CC-BY-SA** — a derivative of openscriptures/strongs, share-alike carries over |
4343
| `data/rv1909.jsonl` (text + Strong's tags) | Reina-Valera 1909 at KJV addresses, tokenized (`rv1909-tok1`), with the source's own inline Strong's tags on each phrase's head word | eBible.org USFX edition via [seven1m/open-bibles](https://github.com/seven1m/open-bibles) (`spa-rv1909.usfx.xml`); built by `data-prep/rv1909/build-rv1909.py`, proved by `check-rv1909.py` | public domain (the 1909 revision; the 1960 is not) |
4444
| `data/strongs-es.json` | Spanish Strong's dictionary: machine-translated definitions + Reina-Valera renderings derived from the tagged corpus | translation of `data/strongs.json` by a Claude Sonnet subagent fleet, 2026-08-16 (same prompt and validation as `data-prep/strongs-lang/translate.py`, which remains the reproducible path; labelled as machine-translated in the app) + renderings computed by `build-strongs.py` from `data/rv1909.jsonl` | **CC-BY-SA** — a derivative of openscriptures/strongs, share-alike carries over |
45+
| `data/svd1865.jsonl` | Smith & Van Dyck 1865 at KJV addresses, tokenized (`svd1865-tok1`), fully vocalized; two split verses merged back to the KJV address | eBible.org USFM edition `arb-vd`; built by `data-prep/svd/build-svd.py`, proved by `check-svd.py` | public domain (the 1865 text; eBible's `copr.htm` and catalogue both say so) |
46+
| `data/pan-fbi.jsonl` | ਪਵਿੱਤਰ ਬਾਈਬਲ (Punjabi) at KJV addresses, tokenized (`pan-fbi-tok1`); two split verses merged back to the KJV address | [FreeBiblesIndia/Punjabi_Bible](https://github.com/FreeBiblesIndia/Punjabi_Bible), original work available at <http://freebiblesindia.in>; built by `data-prep/indic/build-indic.py`, proved by `check-indic.py`. **Modified**: the text is tokenized into the frozen `kjv.jsonl` shape, its two split verses are merged to the KJV address, and the publisher's footnotes, section headings and book introductions are not carried. | **CC BY-SA 4.0** — attribution above, share-alike carries to the tokenized corpus |
47+
| `data/hin-fbi.jsonl` | पवित्र बाइबल (Hindi) at KJV addresses, tokenized (`hin-fbi-tok1`); two split verses merged back to the KJV address | [FreeBiblesIndia/Hindi_Bible](https://github.com/FreeBiblesIndia/Hindi_Bible), original work available at <http://freebiblesindia.in>; built by `data-prep/indic/build-indic.py`, proved by `check-indic.py`. **Modified**: as above, and the source's inline cross-reference apparatus (`\bdit`, 2,513 spans) is dropped with the footnotes. | **CC BY-SA 4.0** — attribution above, share-alike carries to the tokenized corpus |
4548

4649
## Morphology
4750

@@ -101,3 +104,13 @@ offers four more faces, all bundled the same way and all OFL: **Literata**
101104
(Type Together, for Google Fonts), **Inter** (Rasmus Andersson), **Fira Code**
102105
(Nikita Prokopov et al.), and **Atkinson Hyperlegible** (Braille Institute of
103106
America — the low-vision face).
107+
108+
Three further faces are bundled for everyone and offered to nobody who cannot
109+
read them, because none of the seven above contains a single glyph of their
110+
scripts: **Amiri** (Khaled Hosny — the naskh face that carries the Van Dyck,
111+
and the one that positions its tashkeel properly), **Noto Serif Gurmukhi** and
112+
**Noto Serif Devanagari** (Google, for ਪਵਿੱਤਰ ਬਾਈਬਲ and पवित्र बाइबल). All
113+
three are OFL, and each licence travels with its source at
114+
`apps/web/fonts-src/OFL-*.txt`. Which face a reader is offered follows the
115+
SCRIPT of their language and not its direction — `core::font::Font::offered_for`
116+
is the one rule, and `core::i18n::Script` is what it reads.

Cargo.lock

Lines changed: 5 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ default-members = [
2121
# shells — About renders it as `engine ` + this number. The workflow refuses a tag
2222
# whose version disagrees with this line (and with apps/web/package.json), so a
2323
# release ships one number, not three.
24-
version = "0.63.1"
24+
version = "0.64.0"
2525
edition = "2021"
2626
license = "MIT"
2727
authors = ["Glendon Klassen <gjklassen@proton.me>"]

TODO.md

Lines changed: 97 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -16,15 +16,25 @@ than "TR or Majority Text": Acts 8:37 is a TR reading that the Byzantine majorit
1616
Masoretic is not a live question — every candidate here is a Protestant translation and
1717
they all use it.
1818

19+
## Shipped
20+
21+
**Punjabi and Hindi are in.** `data/pan-fbi.jsonl` and `data/hin-fbi.jsonl`,
22+
built by `data-prep/indic/build-indic.py`, proved by `check-indic.py`, both at
23+
KJV addresses with all twenty TR discriminators present. Rows in
24+
`crates/core/src/i18n.rs`, 868 catalogue keys each, Noto Serif Gurmukhi and
25+
Noto Serif Devanagari bundled, `apps/web/e2e/indic.spec.ts` green.
26+
27+
The Punjabi text is **not** the 1945 one this file used to name. See §1.
28+
1929
## Verified
2030

2131
`\v` markers counted in the downloaded USFM. KJV Acts: **40 verses in ch8, 1007 in the book.**
2232

2333
| Language | Text | Licence | Acts ch8 | Acts total | 8:37 |
2434
|---|---|---|---|---|---|
25-
| Punjabi | Pavitr Bible 1945 — `tfbf/punjabi_bible_1945` | PD / CC-BY-SA (conflict) | 40 | 1007 ||
26-
| Punjabi | `FreeBiblesIndia/Punjabi_Bible` | CC-BY-SA 4.0 | 40 | 1007 | |
27-
| Hindi | `FreeBiblesIndia/Hindi_Bible` | CC-BY-SA 4.0 | 40 | 1007 ||
35+
| Punjabi | **`FreeBiblesIndia/Punjabi_Bible` — SHIPPED** | CC-BY-SA 4.0 | 40 | 1007 ||
36+
| Punjabi | Pavitr Bible 1945 — `tfbf/Bible-Punjabi-Pavitr-Bible-1945` | PD / CC-BY-SA (conflict) | 40 | 1007 | ⚠ spliced |
37+
| Hindi | **`FreeBiblesIndia/Hindi_Bible` — SHIPPED** | CC-BY-SA 4.0 | 40 | 1007 ||
2838
| Tamil | `FreeBiblesIndia/Tamil_Bible` | CC-BY-SA 4.0 | 40 | 1007 ||
2939
| Gujarati | `FreeBiblesIndia/Gujarati_Bible` | CC-BY-SA 4.0 | 40 | 1007 ||
3040
| Urdu-Devanagari | `FreeBiblesIndia/Urdu_Devanagari_Bible` | CC-BY-SA 4.0 | 40 | 1007 ||
@@ -33,42 +43,66 @@ they all use it.
3343
| French | Ostervald — `open-bibles/fra-ostervald.osis.xml` | PD ||||
3444
| Chinese | CUV 1919 — `open-bibles/chi-cuv{,-simp}.usfx.xml` | PD ||||
3545

36-
TR marker sweep run against Punjabi 1945 and Hindi FBI — **all present in both**: Mt 17:21,
37-
Mt 18:11, Mt 23:14, Mk 16:20, Lk 17:36, Rom 16:24, Acts 8:37. Both are Textus Receptus.
38-
39-
## 0. The checker — first
40-
41-
Port `data-prep/rv1909/check-rv1909.py` into a general tool. Given a corpus, verify:
46+
**1 John 5:6–8 is renumbered in every Indian-language text found**, the two
47+
shipped ones included: the KJV's 5:6b sits at 5:7, the Comma Johanneum is
48+
absent, 5:8 realigns. Counts still match the KJV everywhere, so it is a content
49+
misalignment at one address rather than a structural one — a note or weave
50+
anchored at `1John 5:7` shows the Comma in English and "the Spirit is truth"
51+
here. `NumberingSpec` is the wrong shape for it (nothing is renumbered) and both
52+
registry rows say so.
4253

43-
- all 66 books, every chapter count and every last-verse number against the KJV;
44-
- the TR discriminators — Acts 8:37, Mt 17:21 / 18:11 / 23:14, Mk 7:16 / 9:44 / 9:46 /
45-
11:26 / 15:28 / 16:9–20, Lk 17:36 / 23:17, Jn 5:4 / 7:53–8:11, Acts 15:34 / 24:7 / 28:29,
46-
Rom 16:24, and 1 Jn 5:7 (the Comma — TR-only, informative rather than disqualifying).
54+
Two splits, in both texts and in opposite directions: 3 John 15 is the tail of
55+
the KJV's v14 and is appended; Rev 12:18 is the **head** of 13:1 and is
56+
prepended. At Rev 13:1a they read "he stood" where the TR has "I stood" — the
57+
one critical reading that shows through the twenty discriminators.
4758

48-
The table above is single-verse spot checks; nothing is confirmed across 66 books until
49-
this runs. One question already waiting for it: **both Malayalam texts are short in Acts**
50-
— 1004 and 1005 against 1007, with ch8 complete — so there are gaps elsewhere in the book.
51-
That is a hole at a valid KJV address, which weaves, the Treasury cross-references and the
52-
stock study set can all link into.
59+
## 0. The checker — done for these two
5360

54-
A textual finding counts only if a script produced it from a local file.
61+
`data-prep/indic/check-indic.py` verifies all 66 books, every chapter count and
62+
every last-verse number against the KJV, the twenty TR discriminators, that
63+
tokens reassemble, that no word begins inside a grapheme cluster, that NFC and
64+
NFD are both no-ops, and — with the source in hand — that every verse's letters
65+
are the source's.
5566

56-
## 1. Punjabi
67+
It also carries the **splice guard** (§1), and the Comma is treated as
68+
informative rather than disqualifying, as agreed.
5769

58-
- **Text:** `tfbf/punjabi_bible_1945`, USFM. Passes the full marker sweep.
59-
- **Settle the licence first.** The repo's `LICENSE.md` asserts public domain under Indian
60-
copyright law; the USFM headers carry CC-BY-SA 4.0. Either ships — they differ only in
61-
whether attribution and share-alike are owed.
62-
- `FreeBiblesIndia/Punjabi_Bible` is a revision of the same base, passes equally, and has an
63-
unambiguous CC-BY-SA 4.0. Choose on readability.
64-
- **Gurmukhi: do not Unicode-normalise.** Composition exclusions; normalising corrupts the
65-
text. Flagged by the source repo.
66-
- New font subset; shares nothing with the Devanagari family.
70+
Still open: **both Malayalam texts are short in Acts** — 1004 and 1005 against
71+
1007, with ch8 complete — so there are gaps elsewhere in the book. Run
72+
`check-indic.py` against them before either is considered; it is the same USFM
73+
shape and the script takes a language code.
6774

68-
## 2. Hindi
75+
A textual finding counts only if a script produced it from a local file.
6976

70-
- **Text:** `FreeBiblesIndia/Hindi_Bible`, USFM, CC-BY-SA 4.0. Passes the full marker sweep.
71-
- Devanagari — also buys Marathi and Urdu-Devanagari later.
77+
## 1. Punjabi — done, and not the text this file named
78+
79+
- **Shipped:** `FreeBiblesIndia/Punjabi_Bible`, CC-BY-SA 4.0. Uniform apparatus
80+
in every book, all five of the publisher's checking stages complete.
81+
- **`tfbf/Bible-Punjabi-Pavitr-Bible-1945` was rejected on the evidence.** Eight
82+
whole books of it are a different modern translation spliced in — Titus, John,
83+
James, 1 Peter, 1–2 Thessalonians, 2 Peter, 1 Corinthians, 1,772 verses — plus
84+
~217 scattered verses elsewhere. **Acts 8:37 is one of the splices**, so the
85+
verse the text was accepted for is the one verse in it that cannot be
86+
attributed to it. Its own `STATUS.md` says the files "are not ready to be used
87+
in a real project".
88+
The tell is punctuation: the 1945 keyboarding types the danda as an ASCII `|`
89+
in 19,306 verses and the spliced material uses a real `` U+0964. Every other
90+
check passes on that file, which is why `check-indic.py` now carries the test
91+
as a standing claim — **a sentence terminator over 1% of a corpus's must be
92+
used by 90% of its books.** The next Indian-language corpus offered here is
93+
likely to have been assembled the same way.
94+
- The PD-vs-CC-BY-SA conflict in that repo is moot now; the shipped text's
95+
licence is unambiguous.
96+
- **Gurmukhi is not normalised**, and the checker asserts it: NFC decomposes the
97+
precomposed nukta letters, so "normalising" rewrites letters.
98+
99+
## 2. Hindi — done
100+
101+
- **Shipped:** `FreeBiblesIndia/Hindi_Bible`, CC-BY-SA 4.0, the traditional
102+
पवित्र बाइबल with modernised spelling.
103+
- Devanagari also buys Marathi and Urdu-Devanagari later: the font, the search
104+
rules and `Script::Devanagari` are all per-script, not per-language, so a
105+
second Devanagari language is a registry row and a catalogue.
72106

73107
## 3. French
74108

@@ -108,28 +142,43 @@ A textual finding counts only if a script produced it from a local file.
108142

109143
## Cross-cutting
110144

111-
**Per script, not per language.** Only Devanagari amortises (Hindi, Marathi,
112-
Urdu-Devanagari, Nepali). Gurmukhi, Tamil, Gujarati, Malayalam and Chinese are each their
113-
own script and font. French adds nothing.
114-
115-
**Re-cost the Indic scripts against the Arabic release.** RTL, complex-script shaping and
116-
non-Latin font subsetting are no longer greenfield if Arabic shipped. Chinese is unaffected
117-
— its problem is line-breaking and segmentation, not shaping.
118-
119-
**Grapheme clusters.** Indic scripts break any code that treats one Rust `char` as one
120-
visible character. Audit the memorize drill first — blanking or revealing by "first letter"
121-
means first *cluster*, and getting it wrong renders broken glyphs rather than a hint. Then
122-
verse-preview truncation and search highlighting.
123-
124-
**No Strong's tags exist for any of these texts.** Every language here ships reader-only:
145+
**Per script, not per language** — and that is now a column rather than a
146+
convention. `core::i18n::Script` and `core::font::Font::script` are what the
147+
font picker and `readerFace` compare; direction is derived from the script.
148+
Adding a Devanagari language (Marathi, Urdu-Devanagari, Nepali) costs a row and
149+
a catalogue and no font work. Tamil, Gujarati, Malayalam and Chinese are each a
150+
new script, a new face and a new `Script` variant. French adds nothing.
151+
152+
**The Arabic release did not de-risk the Indic scripts as much as expected.**
153+
What ported was the subsetting and the bundle-for-everyone fallback. What did
154+
not: `rtl` was standing in for "script" everywhere, because Arabic was the only
155+
language where the two questions had one answer. Chinese will need the same
156+
audit — its problem is line-breaking and segmentation, not shaping.
157+
158+
**Grapheme clusters — audited, and two were real.** `blank_out` masked on
159+
`is_alphanumeric` and passed everything else through, so a masked Devanagari
160+
word kept its viramas and a masked Arabic one kept its tashkeel, hanging off the
161+
underscores; and `normalize_word` deleted every virama from the search index,
162+
because `is_alphanumeric` is false for one. Both fixed. `first_letters` was
163+
fine — it takes the first alphanumeric char, which is a base consonant.
164+
Underscore counts still run one per codepoint rather than per cluster, which
165+
overstates a masked word's length; exact needs UAX #29 and a dependency this
166+
crate does not take.
167+
168+
**No Strong's tags exist for any of these texts.** Punjabi and Hindi ship
169+
reader-only, as Arabic does — `lexicon: None` on both rows. Every language here
170+
ships reader-only:
125171
reading, search, notes, tags, weaves, memorization, plans and the reading map all work, and
126172
every word tap answers `study.noStrongs`. Word-level Greek alignment for Indian languages
127173
does exist — `tfbf/irv_ugnt_alignment`, CC-BY-SA — but it is aligned to the IRV, a different
128174
text. Unresolved for French too. The Arabic row set the precedent for refusing
129175
machine-guessed codes.
130176

131-
**i18n catalogue, ~600 keys per language.** `every_shipped_string_is_translated` blocks a
132-
partial catalogue, so it is all-or-nothing per language.
177+
**i18n catalogue, 868 keys per language.** `every_shipped_string_is_translated`
178+
blocks a partial catalogue, so it is all-or-nothing. Book names come free: the
179+
`\h` field of each USFM file is the publisher's own name for the book, though
180+
the long forms ("ਕੁਰਿੰਥੀਆਂ ਨੂੰ ਪਹਿਲੀ ਪੱਤ੍ਰੀ") need shortening to what a picker
181+
can hold.
133182

134183
**First-run prose.** Already gated: every language ships with the welcome and curious paths
135184
closed until someone inside that culture writes them, and the devotional booklet is offered

apps/web/e2e/arabic.spec.ts

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,8 @@ async function pick(page: Page, now: Record<string, string>, want: string): Prom
5555
const dialog = page.locator('[data-surface="settings"]');
5656
await expect(dialog).toBeVisible();
5757
await page.evaluate(() => ((globalThis as any).__beforeSwitch = true));
58-
await dialog.getByLabel(now["settings.language"], { exact: true }).selectOption({ label: want });
58+
// By VALUE, not label — see the same helper in language.spec.ts.
59+
await dialog.getByLabel(now["settings.language"], { exact: true }).selectOption(want);
5960
await page.waitForFunction(
6061
() => !(globalThis as any).__beforeSwitch && !!(globalThis as any).__plumbline,
6162
undefined,
@@ -89,7 +90,7 @@ async function boxes(page: Page): Promise<Box[]> {
8990

9091
test("the Arabic reader runs right to left, in the chrome and in the text", async ({ page }) => {
9192
await reader(page, EN);
92-
await pick(page, EN, "العربية");
93+
await pick(page, EN, "ar");
9394

9495
// 1. THE DOCUMENT MIRRORS. `dir` is what flips every logical margin, the
9596
// order of a flex row, and which side the menus open on.
@@ -163,7 +164,7 @@ test.describe("an Arabic device, cold", () => {
163164

164165
test("Arabic search finds a word the reader can actually type", async ({ page }) => {
165166
await reader(page, EN);
166-
await pick(page, EN, "العربية");
167+
await pick(page, EN, "ar");
167168

168169
// The Van Dyck prints "ٱلْبَدْءِ" — an alef wasla nobody has a key for, under
169170
// full vowelling nobody types. A reader searches for "البدء".
@@ -212,7 +213,7 @@ test("a language one face can render gets that face, not a font menu", async ({
212213
expect(enFonts, "a naskh face is offered to an English reader").not.toContain("Amiri");
213214
await page.keyboard.press("Escape");
214215

215-
await pick(page, EN, "العربية");
216+
await pick(page, EN, "ar");
216217
await page.getByLabel(AR["common.menu"]).click();
217218
await page.locator(".menu").getByRole("button", { name: AR["shell.settings"] }).click();
218219
dialog = page.locator('[data-surface="settings"]');
@@ -243,7 +244,7 @@ test("a language one face can render gets that face, not a font menu", async ({
243244
// nothing at all — the value was a token the filtered list doesn't carry
244245
// (maintainer's phone, 2026-08-28). `readerFace` resolves off-list tokens to
245246
// the language's default, and the select binds to the RESOLVED face.
246-
await pick(page, AR, "English");
247+
await pick(page, AR, "en");
247248
await page.getByLabel(EN["common.menu"]).click();
248249
await page.locator(".menu").getByRole("button", { name: EN["shell.settings"] }).click();
249250
dialog = page.locator('[data-surface="settings"]');

0 commit comments

Comments
 (0)