Skip to content

0.6.0 — Traditional Chinese (Taiwan) support - #59

Open
davadev wants to merge 7 commits into
chore/obsidian-lint-0.4.2from
feat/traditional-chinese-0.6.0
Open

0.6.0 — Traditional Chinese (Taiwan) support#59
davadev wants to merge 7 commits into
chore/obsidian-lint-0.4.2from
feat/traditional-chinese-0.6.0

Conversation

@davadev

@davadev davadev commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Closes #57.

Stacked on #58 (lint refresh) — retarget to main once that merges.

What a Traditional reader gets

台灣的天氣很熱 used to fall apart into 台|灣|的|天|氣|很|熱, because DictionaryService.surfaces() fed the tokenizer trie simplified headwords only — 77,778 traditional forms were invisible to segmentation. It now reads 台灣|的|天氣|很|熱, with 圖書館, 學習, 網路 and 軟體 all merging correctly.

Vocabulary was already script-agnostic (canonical 简体|pinyin key), so 學習 and 学习 have always been one record — what changed is that multi-character Traditional words are now actually recognised, so those records finally get used.

Everything was measured against the real 125,052-entry CC-CEDICT build, not assumed.

Parity gaps closed

The tokenizer was the obvious one. These were not:

  • Flashcards drew rec.simplified — a Taiwan learner was being drilled on Simplified characters. SRS is the core loop, so this was the worst of them.
  • Stats search matched r.simplified only, so typing 學習 found nothing.
  • Note-scoped stats compared r.simplified against noteSurfaces, which holds raw token surfaces and is therefore traditional in a traditional note — the list came back empty.
  • Flashcard pinyin read the record's snapshot, missing both the tone repair and the Taiwan reading.
  • AI stories were silently simplified-only; buildRepairPrompt hardcoded "the exact simplified-Chinese surface form".

Three pre-existing bugs found while verifying

All predate this release and affect every user:

  1. -iu tone marks on the wrong vowel. 六 lìu, 九 jǐu, 牛奶 níu nǎi, 休息 xīu xi, 秋天 qīu tiān, 丢 dīu — core HSK 1–3 vocabulary.
  2. Every ü syllable kept a raw tone digit. The downloader converted u: → ü after the tone converter, whose character class has no :, so 女 shipped as the literal nü3 and 绿 as lü4. 1,069 entries.
  3. Trie.insert walked code points while matchesFrom walks code units, so any word with a character outside the BMP was stored under a key the matcher could never reach.

4,788 of 125,052 entries displayed wrong pinyin. They are repaired at index time, so existing users get correct pinyin on next load with no dictionary re-download.

The constraint that shaped the implementation

makeKey() must keep using the simplified form and the Mainland reading, and must never depend on a setting — vocabulary keys, dictionary overrides, custom words and the sync mirror all hang off it.

  • The pinyin repair is key-safe: toneMarksToNumbers now absorbs a trailing digit as its syllable's tone, so nü3 and nǚ hash identically. Verified over all 125,052 entries: 4,788 repaired, 0 keys moved, 0 case damage. Without that change the same repair moves 1,066 keys and orphans those records.
  • The repair is deliberately not a round-trip through numbersToToneMarks, which lowercases and would corrupt 23,485 proper nouns (Qū → qū, 3D → 3d).
  • lookup() concatenates simplified-map entries first, always. An earlier draft made the order settings-dependent, which would have re-keyed records for the 66 surfaces that are both a simplified headword and someone else's traditional form (著, 乾, 宁, 於 …) every time the user toggled the switch. Measured across all 66: candidates[0] is unchanged.

Two decisions worth reviewing

The plugin never converts between scripts. 1,078 simplified headwords map to more than one traditional form, CC-CEDICT orders entries by codepoint rather than frequency, and frequencyRank is never populated — so taking the first candidate gives 发 → 發 (wrong for 头发), 干 → 乹 and 历 → 厤 (obsolete variants), 里 → 裏 (the Hong Kong form where Taiwan writes 裡). Flashcards show the form the learner actually encountered instead; the counterpart appears only when the mapping is unambiguous.

Traditional detection is one-directional. The symmetric test does not work: 台, 只, 后 and 里 are simplified headwords yet entirely normal in Taiwan writing, so a "looks simplified" score misfires — a pure Traditional paragraph scored 7 against 4, and 你好我是人今天 read as simplified. Presence of traditional-only markers gave 0 false positives on every simplified sample. The same asymmetry is why the story validator tests for the presence of traditional characters rather than the absence of simplified ones.

Invalidation

Four paths can change scriptVariant — settings tab, view toolbar, importSettings(), and a remote SettingsMirror apply. The last two replace settings wholesale and then only refresh views, so a script flip arriving that way would repaint fresh colours over a stale trie with no error anywhere. All four funnel through one applyScriptSideEffects() guard in saveSettingsSilently().

Honest limits (documented, not hidden)

  • Taiwan readings cover CC-CEDICT's ~500 idiosyncratic re-readings (垃圾 lè sè) and not the neutral-tone difference you actually hear most — 謝謝 xièxiè, 東西 dōngxī, 先生 xiānshēng. CC-CEDICT records no Taiwan reading for any of them.
  • HSK is a Mainland standard. A Taiwan learner works to TOCFL, so the colours, "Top HSK" label and story level are calibrated to a syllabus they do not use.
  • No Zhuyin/Bopomofo yet.
  • Upgrade synced devices together: the re-key can transiently duplicate ~1,066 words on a device still on 0.5.1. It self-heals on upgrade.

Verification

  • 398 tests, lint 0 errors / 0 warnings, coverage 91.96% lines / 80.11% branches.
  • check-release --tag 0.6.0-rc.1 --with-build: 40 passed, 0 failed.
  • Simplified segmentation verified byte-identical on every sample — defaults keep an upgrading user on exactly today's behaviour.

Needs device testing before promotion, in particular: that known-word counts are unchanged after the pinyin fix, and whether Traditional characters stay legible in the ruby row on mobile at the default font size.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A

davadev and others added 7 commits September 2, 2026 14:45
@davadev @claude
0.6.0 — fix pinyin tone placement, stranded ü digits, and Trie surrog…
6ad7e8a 
…ate handling

Three pre-existing correctness bugs found while verifying the Traditional
Chinese work. All predate this release and affect every user.

1. -iu tone marks landed on the wrong vowel. numbersToToneMarks ranked
   vowels a > o > e > i > u, but the rule for -iu/-ui is to mark the LAST
   vowel. Core HSK 1-3 vocabulary was wrong in the shipped data: 六 lìu,
   九 jǐu, 牛奶 níu nǎi, 休息 xīu xi, 秋天 qīu tiān, 丢 dīu.

2. Every ü syllable kept a raw tone digit. The downloader converted CC-CEDICT's
   "u:" to ü AFTER numbersToToneMarks, whose character class has no ":", so
   "nu:3" never matched and became the literal "nü3". 1,069 entries: 女 nü3,
   绿 lü4, 律 lü4.

3. Trie.insert walked code points while Trie.matchesFrom walks code units, so
   any word containing a character outside the BMP was stored under a key the
   matcher could never reach. 268 simplified headwords and 10 traditional-only
   forms never matched.

4,788 of 125,052 entries display wrong pinyin. repairPinyin() fixes them at
index time, so existing users get correct pinyin on next load with no
dictionary re-download; the downloader fix covers fresh downloads.

The repair is vocabulary-key-safe, which is the constraint that dictates the
implementation. toneMarksToNumbers now absorbs a trailing digit as its
syllable's tone, so "nü3" and "nǚ" hash identically. Verified against all
125,052 shipped entries: 4,788 repaired, 0 vocabulary keys moved, 0 case or
letter damage. Without that change the same repair would have moved 1,066
keys and orphaned those records. The repair is also deliberately NOT a
round-trip through numbersToToneMarks, which lowercases and would corrupt
23,485 proper nouns (Qū -> qū, 3D -> 3d).

Dictionary overrides are keyed by makeKey and, unlike WordRecords, are never
re-derived on load, so they get a one-shot re-key. It is a pure string
transform because onload() runs vocab.load() long before the dictionary is
read — forcing an early load would add a ~17 MB read to every startup.
Verified: 1,066 of 1,066 legacy keys map correctly, 0 false positives.

Also lands the groundwork for Taiwan readings: DictionaryEntry.pinyinTaiwan,
extracted at index time from the "Taiwan pr. [le4 se4]" gloss CC-CEDICT
already ships. 512 of 513 entries parse cleanly; the one skip is a prose note
with no bracketed reading. Costs 6 ms (repair) + 5 ms (extraction) at load,
against 41 ms just to JSON.parse the file.

346 tests pass; lint 0 errors, 0 warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A
@davadev @claude
0.6.0 — script/region settings, Traditional tokenization, and display…
18603f6 
… helpers

Makes the plugin able to read Traditional Chinese (#57). Defaults are
unchanged, so an upgrading Simplified user gets a bit-identical trie.

Settings: `scriptVariant` and `pronunciationRegion` as a top-level
"Script & region" group placed right after Display — a Traditional reader has
to find this before anything else works for them, so it must not sit behind an
Advanced sub-page. Both are top-level scalars, so the shallow DEFAULT_SETTINGS
spread in onload fills them for existing users with no migration and no
schemaVersion bump.

Tokenizer: DictionaryService.surfaces() gains an opt-in `includeTraditional`,
and it is a UNION rather than a swap — Traditional readers still meet
Simplified text, and mixed vaults have to keep working without a per-note
switch. Verified against the real 125k-entry dictionary: 台灣的天氣很熱 was
台|灣|的|天|氣|很|熱 and is now 台灣|的|天氣|很|熱; 圖書館, 學習, 網路 and
軟體 all merge correctly, and every simplified sample segments byte-identically.

lookup() now concatenates the traditional map after the simplified one instead
of `simplified ?? traditional`, recovering the sense that was silently dropped
for the 66 surfaces that are both a simplified headword and someone else's
traditional form (著, 乾, 宁, 於 …). The ordering is fixed and must never
depend on a setting: VocabularyStore.ensure() keys records off lookup()[0], so
a settings-dependent order would re-key those records on every toggle.
Measured across all 66: candidates[0] is unchanged.

Invalidation is centralised in applyScriptSideEffects(), called from
saveSettingsSilently(). Four paths can change the script — the settings tab,
the view toolbar, importSettings(), and a remote SettingsMirror apply — and
the last two replace settings wholesale and then only refresh views, so a
script flip arriving that way would repaint fresh colors over a stale trie
with no error anywhere. One guard off one remembered value covers all four.
VocabularyStore.clearSurfaceCache() is now public for the same reason:
DictionaryService.reload() has eight call sites and none could reach it.

displayForms.ts deliberately never converts between scripts. 1,078 simplified
headwords map to more than one traditional form, CC-CEDICT orders entries by
codepoint rather than frequency, and frequencyRank is never populated — so
taking the first candidate gives 发 -> 發 (wrong for 头发), 干 -> 乹 and
历 -> 厤 (obsolete variants), 里 -> 裏 (the Hong Kong form). It returns a form
the learner actually encountered instead, and offers the counterpart only when
the mapping is unambiguous.

scriptDetect.ts is one-directional by design. The symmetric test does not
work: 台, 只, 后 and 里 are simplified headwords yet normal in Taiwan writing,
so a "looks simplified" score misfires — a pure Traditional paragraph scored
7 against 4, and 你好我是人今天 read as simplified.

387 tests pass; lint 0 errors, 0 warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A
@davadev @claude
0.6.0 — Traditional parity in the reader, flashcards, stats, and AI
fdfa7fd 
Closes the places where a Traditional reader would have got a worse
experience than a Simplified one.

Reader: the word popup's headword is now the form the learner actually
tapped, and the counterpart script is shown as a meta row only when the
mapping is unambiguous. RubyWidget resolves pinyin through displayPinyin, so
the Taiwan reading reaches the page and a region flip correctly busts widget
equality.

Flashcards were the worst gap — StatsView drew rec.simplified, so a Taiwan
learner was drilled on Simplified characters. They now show the encountered
form, and read pinyin through the live dictionary rather than the record's
snapshot, which predates both the tone repair and the Taiwan field.

Stats: the search box matched r.simplified only, so typing 學習 found nothing;
it now matches every surface. The note-scope filter compared r.simplified
against noteSurfaces, which holds raw token surfaces and is therefore
traditional in a traditional note — scoped stats came back empty. It now
matches on any surface. StatsView also gains invalidateCaches(), because
refreshStatsViews() only re-renders and would otherwise keep serving
note-scope surfaces and example sentences computed under the old trie.

(loadTriageContext needed no change after all — it already tries every entry
of rec.surfaces, so example sentences were being found in traditional notes.)

AI: buildUserPrompt and buildRepairPrompt take the script, and the repair
prompt no longer hardcodes "the exact simplified-Chinese surface form". The
clause asks for Taiwanese Mandarin usage rather than only Traditional glyphs
— characters alone yield Mainland vocabulary in Traditional clothing, and
CC-CEDICT carries both halves of every common pair (網路/網絡, 影片/視頻).

validateStory accepts a target in either script, so a model that ignores the
instruction is still scored on whether it used the word, and the generated
note's checklist ticks on either form so the boxes are truthful. The script
check tests for the PRESENCE of traditional-only characters rather than the
absence of simplified ones: 台, 只, 后 and 里 are simplified headwords yet
entirely normal in Traditional writing, so the absence test is unreliable.
Advisory only — a short story can legitimately contain none.

Discoverability, three ways: the settings group, a one-tap "Traditional
characters" toggle in the reading view's overflow menu (which runs the full
invalidation rather than the plain redecorate the colour checkboxes use), and
a conservative one-time prompt when an opened note contains at least three
traditional-only characters. The prompt never switches on its own.

398 tests pass; lint 0 errors, 0 warnings; coverage 91.96% lines /
80.11% branches; check-release 39 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A
@davadev @claude
0.6.0 — docs and version bump for the first release candidate
de8cbb8 
Adds docs/traditional-chinese.md, linked from the new settings group via
docLink() and from the README's guide list, plus a short README section.

The doc is explicit about the two things a Taiwan learner would otherwise
discover the hard way: the Taiwan readings cover CC-CEDICT's ~500
idiosyncratic re-readings and NOT the neutral-tone difference you actually
hear most (謝謝 xièxiè, 東西 dōngxī), and HSK is a Mainland standard so the
colours, "Top HSK" label and story level are calibrated to a syllabus they do
not use.

It also explains why the plugin never converts between scripts, with the data
behind it: 1,078 Simplified headwords map to more than one Traditional form,
and with no frequency data the naive conversion gives 干 -> 乹 and 历 -> 厤.

Version bumped once for the 0.6.0 target; later -rc.N tags leave it alone.

check-release --tag 0.6.0-rc.1 --with-build: 40 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A
@davadev @claude
0.6.0 — fix regressions found in self-review of the Traditional work
a859e0e 
Four defects found reviewing the 0.6.0 branch against 0.5.1, three of them
introduced by this branch.

1. WordPopup could freeze after marking a word. refresh() read the surface
   back out of the rendered headword and called vocab.bySurface() with it.
   That was safe while the headword was always rec.simplified, but this
   branch let displaySurface return the counterpart form for an unambiguous
   mapping — a form the learner may never have encountered, and therefore
   absent from rec.surfaces, which is what bySurface() falls back to
   scanning. The popup now remembers the tapped surface and both renders and
   re-reads by it, so the headword is exactly the characters on the page and
   refresh() no longer depends on the DOM. Guarded by a test asserting that
   displaySurface can legitimately return a form outside rec.surfaces.

2. index() could throw on a hand-edited dictionary file. It now reads
   e.definitions to extract the Taiwan reading, but the vault-side loader
   only validates `simplified` and `pinyin`. A truncated or hand-written
   .cci-dictionary.json entry would have thrown mid-load and taken the
   plugin's dictionary with it. extractTaiwanReading is now defensive about
   a missing or non-string definitions list.

3. pronunciationRegion had the same silent-path bug the script guard was
   built to fix. It was handled only in the settings tab, so a region change
   arriving via importSettings() or a remote SettingsMirror apply left the
   RubyWidget's snapshotted pinyin stale. Both settings now run through one
   applyScriptSideEffects() guard, which rebuilds the trie only when the
   script actually changed and drops cached tokens for either.

4. The Words table still printed the raw r.pinyin snapshot, so it showed
   pre-repair readings (nü3) and ignored the Taiwan region while the
   flashcard, popup and detail modal all used the corrected path. It now
   goes through displayPinyin like everything else. The table is capped at
   500 rows, so the extra lookups are bounded.

Also moved StatsView.invalidateCaches() out of the middle of the field
declarations.

Regression evidence against the real 125,052-entry dictionary:
- lookup()[0] identical for all 198,106 surfaces, so no makeKey can move.
- 0 entries whose canonical key changes after the pinyin repair.
- 0 segmentation differences over 20,000 synthetic sentences in Simplified
  mode, and 0 over the 109 CJK spans of the real vault note.
- New upgradeRegression.test.ts covers the 0.5.1 -> 0.6.0 path: a record
  keyed 女|nü53 re-keys to 女|nü3, an old-key and new-key pair for the same
  word merge into one (strongest status wins, exposure counts summed)
  rather than duplicating, untouched records are preserved byte for byte,
  and a traditional surface still resolves to the same record.

404 tests pass; lint 0 errors, 0 warnings; coverage 92.01% lines /
80.71% branches; check-release 40 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A
@davadev @claude
0.6.0 — make the invalidation guard testable, and verify the wiring e…
58a3ea6 
…nd to end

The script/region invalidation decision was the subtlest thing on this
branch and the only piece with no test, because it lived inside a method on
CciPlugin that a unit test cannot stand up. Extracted to a pure
planScriptChange() in settings/scriptChange.ts and covered, including the
property that matters: whenever either setting actually moved, retokenize is
never false. Getting that wrong repaints fresh colours over stale
segmentation and reports nothing.

The extraction also records the asymmetry explicitly — a script change
rebuilds the trie because segmentation changes, a region change does not,
but BOTH must drop cached tokens because RubyWidget snapshots its pinyin at
construction time.

Verified end to end against the real 125,052-entry dictionary, driving the
actual DictionaryService + TokenizerService rather than the pieces:
  - Simplified mode, traditional text: 台|灣|的|天|氣|很|熱|我|昨天|去|圖|書|館|學|習|中文
  - After the flip:                    台灣|的|天氣|很|熱|我|昨天|去|圖書館|學習|中文
  - Simplified text before vs after the flip: character-for-character identical
  - Flipping back WITHOUT invalidate() returns the stale traditional
    segmentation, which is direct evidence the guard is load-bearing rather
    than defensive; invalidate() restores the correct result
  - 垃圾 reads lā jī / lè sè by region through the real lookup path
  - 女 indexes as nǚ and 九 as jiǔ with no dictionary re-download
  - The detector answers true on the traditional sample and false on the
    simplified one
  - Whole run, including reading and parsing the 17 MB dictionary: 326 ms

Also confirmed main.js is gitignored, so the release workflow's clean build
is what ships rather than a stale local artifact.

410 tests pass; lint 0 errors, 0 warnings; check-release 40 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A
@davadev @claude
0.6.0 — don't let the override migration touch the mirror before boot…
9a2a59b 
…strap

Three sync-path defects found while deciding whether this was ready to
prerelease. The first is the serious one.

1. The onload override migration wrote through saveDictionaryUserData(),
   which flushes via vocab.flushSave() and therefore SCHEDULES A MIRROR
   WRITE. At that point bootstrapMirrorAfterLoad() has not run, so the vault
   mirror still holds a remote envelope this device has never merged —
   writing local-only vocabulary over it loses whatever another device had.
   The 5s mirror debounce means bootstrap usually wins the race, but "usually"
   is not a guarantee on a mobile device with stalled Nextcloud I/O, which is
   exactly the environment this plugin documents elsewhere as hostile.
   It now writes the plugin data blob directly and leaves the mirror alone;
   the corrected keys reach it on the next natural write, after the merge.

   The same call was also a floating promise. An unhandled rejection during
   onload surfaces as Obsidian's generic "plugin encountered an error while
   loading" — the failure mode vocab.load() already carries a comment about.
   Errors are now caught and logged; the migration is idempotent, so a failed
   write just means it runs again next launch.

2. Remote overrides bypassed the migration entirely. The onload pass only
   sees this device's own blob, but mergeMirroredDictionaryData() inserts
   whatever the mirror holds verbatim — so a peer still on 0.5.1 would
   re-introduce legacy keys after every sync, silently re-orphaning those
   overrides. Incoming overrides are now migrated on the way in.

3. Wired clearSurfaceCache() into all eight dictionary.reload() sites. This
   is the pre-existing bug noted during review: reload() changes what
   lookup() resolves a surface to, bySurface() memoises that, and nothing
   ever cleared it — so a newly added custom word could keep resolving to a
   cached miss until some unrelated vocabulary mutation happened to flush it.

412 tests pass; lint 0 errors, 0 warnings; check-release 40 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcoqjUfHtUJ7VAuSyAF99A
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@davadev