Commit 4dcfaa2
authored
Punjabi and Hindi, and rtl becomes a script (#53)
* The Punjabi and Hindi Bibles
data/pan-fbi.jsonl and data/hin-fbi.jsonl, stamped pan-fbi-tok1 and
hin-fbi-tok1, in kjv.jsonl's frozen shape. Free Bibles India's USFM editions
of ਪਵਿੱਤਰ ਬਾਈਬਲ and पवित्र बाइबल, CC BY-SA 4.0 — the traditional Protestant
Bible of each language, at KJV addresses, Textus Receptus in all twenty
readings a critical text omits or brackets.
One build script for two languages, which the other three corpora do not do.
Same publisher, same conventions, same two verse splits; two files would be
one file and a copy of it.
THE PUNJABI IS NOT THE OBVIOUS TEXT. tfbf/Bible-Punjabi-Pavitr-Bible-1945 is
public domain outright and was rejected on the evidence: eight whole books of
it are a different modern translation spliced in — Titus, John, James, 1 Peter,
1-2 Thessalonians, 2 Peter, 1 Corinthians, 1,772 verses — plus ~217 scattered
verses, Acts 8:37 among them. Every other check passes on that file. What gives
it away is punctuation: the 1945 keyboarding types the danda as an ASCII "|" in
19,306 verses and the spliced books use a real U+0964. Its own STATUS.md says
the files are not ready to be used in a real project.
So check-indic.py carries that test as a standing claim — a sentence terminator
over 1% of the corpus's must be used by 90% of its books — because the next
Indian-language corpus offered here is likely to have been assembled the same
way.
Three things the checks caught in this build, none of which any other check
would have: \q lines are POETRY, not skippable decoration, and skipping them
ended Psalm 23:1 at "the LORD is my shepherd,"; \qa is Psalm 119's acrostic
stanza heading, which kjv.jsonl already carries as a title token at every
eighth verse; and Rev 12:18 merges FORWARD into 13:1 while 3 John 15 merges
back into 14, so a build that appended both keeps every word and prints
Revelation 13:1 with its first clause last.
The text is never normalised. NFC decomposes the precomposed Gurmukhi nukta
letters — they are on Unicode's composition exclusion list — so normalising
rewrites letters. Both files are already stable under NFC and NFD, measured,
and the check asserts it so the one line a later maintainer might add by habit
fails the build.
No Strong's codes and none invented, as with the Van Dyck.
* Punjabi and Hindi are rows in the registry
Lang::Pa reads pan-fbi.jsonl, Lang::Hi reads hin-fbi.jsonl. The interface is
translated — 868 keys each, book names taken from the publishers' own \h fields
and shortened to the forms a picker can hold. lexicon: None for both, as
Arabic's is: neither corpus carries Strong's codes and none are invented.
`rtl` IS NOW A SCRIPT. Every non-Latin question this app had answered, it had
answered for Arabic — the only language that is neither Latin nor left to
right — so "which faces can set this" and "which way does the page run" had one
answer and were written as one condition. Gurmukhi and Devanagari split them:
Punjabi reads left to right and no Latin face has a glyph of it. So LangSpec
carries a Script, Font carries the script it can set, and offered_for compares
the two instead of asking is_rtl(). Direction is derived from the script, which
is the only thing that decides it.
Without that, readerFace hands a Punjabi reader EB Garamond. Nothing errors —
the text renders through per-glyph fallback at a Latin face's optical scale, in
whatever the browser found, measured in the worker against something else
again.
FOUR THINGS THE SCRIPTS TURNED UP, none of which any existing test could see:
- normalize_word DELETED EVERY VIRAMA. char::is_alphanumeric is false for
U+094D, so the filter that strips punctuation stripped the join out of
every conjunct: परमेश्वर indexed as परमेशवर. Applied to both sides, so no
search broke — it merged अन्त with अनत and nothing said so. The exact
mirror of the Arabic bug this file already records, where every mark
carried Other_Alphabetic and the same filter kept a vowelling it should
have dropped.
- THE NUKTA FOLD HAS TO BE PER LETTER. Dropping it wholesale is Arabic's alef
fold and looks right; counted over the corpora it merges ਸ with ਸ਼ (61.4% of
Punjabi's nuktas) and ड with ड़ (74.5% of Hindi's). Those are native
letters. Only the Perso-Arabic set — क ख ग ज फ, ਖ ਗ ਜ ਫ — is a dot readers
leave off, and only it folds.
- blank_out LEAKED THE WORD IT WAS HIDING. mask_word maps on is_alphanumeric
and passes everything else through verbatim, so a masked परमेश्वर came out
_____्__ with the virama drawn on a dotted circle, and a masked Arabic word
kept all its tashkeel. Arabic had shipped with that.
- is_indic_mark's two ranges are NOT the same shape. Devanagari's marks start
at U+093A so its nukta at U+093C falls inside; Gurmukhi's matras start at
U+0A3E and its nukta at U+0A3C sits two below. The range copied across from
Devanagari deletes every Punjabi nukta and keeps every Hindi one.
Fonts: Noto Serif Gurmukhi and Noto Serif Devanagari, OFL, subset with every
layout feature kept — an Indic script is SHAPED, and subsetting away nukt/half/
blwf/pres leaves each conjunct as separate letters at a different advance than
the shaped text, which is the measure-here-paint-there split at the scale of
most words in the language. The danda at U+0964 is in both subsets: it lives
outside either script's block and ends 27,354 Punjabi and 31,541 Hindi verses.
The three script faces stay loaded unconditionally in the engine worker, and
the cheap narrowing is wrong for a reason worth writing down: the pane strip
lets a German reader open the Van Dyck beside the Luther, so the script this
worker is asked to MEASURE is not a function of the language it paints chrome
in. They cost 287 KB rather than 106 — Devanagari's unused wdth axis is pinned
out first (237 KB → 130 KB) and the three now load in parallel rather than one
after another.
BIBLIOGRAPHY.md gains both CC BY-SA 4.0 rows with their attribution and their
statement of modification, and closes two gaps the Arabic release left: the Van
Dyck had no row and Amiri no mention.
* TODO: Punjabi and Hindi are shipped, and the 1945 text is not the one
* The language picker names each language twice
An English reader's list read "Deutsch · Español · العربية · ਪੰਜਾਬੀ · हिन्दी" —
right for the reader looking for their own language, useless to the person
offering their phone to a Hindi speaker, who was looking at six scripts they
cannot read with no way to tell which row was the one. This app is built to be
handed over. It now reads "Hindi (हिन्दी)": the English name to scan, the
endonym to confirm.
English only, and that is the data rather than a decision — `exonym` on the
registry row is the language's ENGLISH name, so there is nothing to put in
front of the bracket for a German reader with the same problem. They keep the
endonyms alone. A LangSpec carrying names per language would serve them too.
No bracket where the two are the same word; "English (English)" is noise.
The three language pickers in the e2e suite now select by VALUE — the option's
language code — rather than by label. The label is exactly what this commit
edits, so a test that spelled it broke on a copy change and said nothing useful
when it did.
* The picker names a language in its own words and the reader's
"ਪੰਜਾਬੀ (Punjabi)" to an English reader, "English (Englisch)" to a German one.
Either half alone serves one person and fails the other. The endonym alone is
right for someone looking for their own language and useless to someone handing
their phone to a Hindi speaker — six scripts they cannot read, no way to tell
which row is the one. The reader's own name alone is the reverse: it puts
"Punjabi" in Latin letters in front of the person being handed the phone. So
both, and the ENDONYM LEADS, because the row belongs to the person being
offered it and the bracket is the reader's note to themselves.
The bracket is a catalogue lookup, `lang.<code>`, in all six catalogues — 36
strings. Not the registry's `exonym`: that column is the language's ENGLISH
name, which is what the hymnal finder matches on, and it could only ever have
served an English reader.
No bracket when the two are the same word. Not a special case for English —
it is the one comparison that silences the reader's own language in every
catalogue, "Deutsch (Deutsch)" in German and "हिन्दी (हिन्दी)" in Hindi.
* Two screens named a book from the frozen English table
The reading plan's card read "Today: John 3" to a Hindi reader while the
passage navigator, the search results and every other reference on the same
screen read "यूहन्ना 3". `canon::display_name` is the table `refKey` is built
from — right for storage, never right for a reader — and in an English test run
it is indistinguishable from the correct call.
The second site is worse for sitting six lines under two calls that get it
right: copying a VERSE gave that reader "यूहन्ना 3:16" and copying the CHAPTER
it is in gave them "John 3", from the same function.
`copy_text` now takes `lang` instead of reading `i18n::active()` inside it. It
renders text for one reader, so the language is an input — the same reason
`VRef::display_in` exists beside `VRef::display` — and the hidden global was
not merely untidy: `set_active` is process-wide, the test binary runs in
parallel, and the first attempt at a German test broke two English ones running
in another thread.
The guard is a source scan: nothing outside `canon` and `book_name` may call
`display_name`. A behavioural test would have to be written once per screen,
and the failure here is a call that should not exist rather than an output that
is wrong. It asserts it actually reached the files, because a walk that found
nothing would pass silently forever.
* Version 0.64.0 — Punjabi and Hindi1 parent bf6ac00 commit 4dcfaa2
43 files changed
Lines changed: 66914 additions & 714 deletions
File tree
- apps/web
- e2e
- fonts-src
- public
- fonts
- scripts
- src
- engine
- lib
- shell
- crates
- core/src
- i18n
- ffi
- src
- tests
- data-prep
- indic
- data
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
45 | 48 | | |
46 | 49 | | |
47 | 50 | | |
| |||
101 | 104 | | |
102 | 105 | | |
103 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
19 | 29 | | |
20 | 30 | | |
21 | 31 | | |
22 | 32 | | |
23 | 33 | | |
24 | 34 | | |
25 | | - | |
26 | | - | |
27 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
28 | 38 | | |
29 | 39 | | |
30 | 40 | | |
| |||
33 | 43 | | |
34 | 44 | | |
35 | 45 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
42 | 53 | | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
47 | 58 | | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
| 59 | + | |
53 | 60 | | |
54 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
55 | 66 | | |
56 | | - | |
| 67 | + | |
| 68 | + | |
57 | 69 | | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
67 | 74 | | |
68 | | - | |
| 75 | + | |
69 | 76 | | |
70 | | - | |
71 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
72 | 106 | | |
73 | 107 | | |
74 | 108 | | |
| |||
108 | 142 | | |
109 | 143 | | |
110 | 144 | | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
125 | 171 | | |
126 | 172 | | |
127 | 173 | | |
128 | 174 | | |
129 | 175 | | |
130 | 176 | | |
131 | | - | |
132 | | - | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
133 | 182 | | |
134 | 183 | | |
135 | 184 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
| 59 | + | |
59 | 60 | | |
60 | 61 | | |
61 | 62 | | |
| |||
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
92 | | - | |
| 93 | + | |
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| |||
163 | 164 | | |
164 | 165 | | |
165 | 166 | | |
166 | | - | |
| 167 | + | |
167 | 168 | | |
168 | 169 | | |
169 | 170 | | |
| |||
212 | 213 | | |
213 | 214 | | |
214 | 215 | | |
215 | | - | |
| 216 | + | |
216 | 217 | | |
217 | 218 | | |
218 | 219 | | |
| |||
243 | 244 | | |
244 | 245 | | |
245 | 246 | | |
246 | | - | |
| 247 | + | |
247 | 248 | | |
248 | 249 | | |
249 | 250 | | |
| |||
0 commit comments