Skip to content

Commit 74ad822

Browse files
authored
fix(cli/scoring): make TextNormalizer replacement order deterministic (#921)
## Summary Fixes #911 — the CI ASR benchmark scored byte-identical hypothesis/reference pairs differently between runs (the `1688-142285-0002.flac` record moved between WER 0% and 10% with no code change). ## Root cause (proven, and stronger than suspected) The issue guessed per-**process** dictionary hashing. It's per-**instance**: Swift seeds each hash table from its storage allocation, so two identical dictionary literals in the *same process* iterate in different orders. Since `normalize()` builds fresh `contractions`/`abbreviations` dictionaries on every call, the reference and hypothesis calls inside one `calculateWERAndCER()` get independent orders. With overlapping keys, `don't` expands to `do not` when `n't` is applied first but `don not` when `'t` wins — so ref and hyp can disagree within a single scoring call. Evidence (standalone harness compiled from the unmodified `TextNormalizer.swift`): - 300 fresh processes on the exact #911 pair: **144 diverged** (`do not` vs `don not`, exactly the 1-word/1-char gap in the artifact: WER 1/10, CER 1/35). - Two identical dict literals built back-to-back in one process: different key order 5/5 runs. ## Fix Apply all four replacement tables (`britishToAmerican`, `abbreviations`, `contractions`, `numberWords`) sorted **longest key first**, lexical tie-break. Deterministic, and the most specific rule now always beats its substring — which is also the semantically correct expansion: | input | before (order-dependent) | after | |---|---|---| | `don't` | `do not` **or** `don not` | `do not` | | `can't` | `can not` **or** `ca not` | `can not` | | `I'd been` | `i had been` **or** `i would been` | `i had been` | | `it's been` | `it has been` **or** `it is been` | `it has been` | Verified with the patched file across 50 fresh processes: byte-identical output every run. ## Tests `AsrTextNormalizerTests`: the exact #911 record scores WER 0/CER 0 across 100 repeated calls; repeated-call byte-determinism; overlapping-key expansion expectations. Note for reviewers: CI benchmark baselines may shift slightly on merge — historical numbers were sampled from random orderings; the deterministic ordering picks the correct expansions, so WER should move down or stay flat.
1 parent 71242fa commit 74ad822

3 files changed

Lines changed: 251 additions & 170 deletions

File tree

Sources/FluidAudio/ASR/Parakeet/Unified/benchmark.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ Measured through the FluidAudio Swift managers (`UnifiedAsrManager` for batch,
55
scored with the repo's `TextNormalizer` (same normalization as `asr-benchmark`).
66
Encoder precision: **int8**. Run with `swift run -c release fluidaudiocli unified-benchmark`.
77

8+
> WER rows below predate the deterministic normalizer ordering (#911); reruns
9+
> may differ by a few tenths of a point with no model change.
10+
811
| Mode | Avg WER | Aggregate WER | Median WER | Median RTFx | Overall RTFx | Long files (>15s) |
912
|------|---------|---------------|------------|-------------|--------------|-------------------|
1013
| batch | 2.16% | 1.68% | 0.00% | 130.0x | 143.6x | 238 |

0 commit comments

Comments
 (0)