Commit 74ad822
authored
fix(cli/scoring): make TextNormalizer replacement order deterministic (#921)
## Summary
Fixes #911 — the CI ASR benchmark scored byte-identical
hypothesis/reference pairs differently between runs (the
`1688-142285-0002.flac` record moved between WER 0% and 10% with no code
change).
## Root cause (proven, and stronger than suspected)
The issue guessed per-**process** dictionary hashing. It's
per-**instance**: Swift seeds each hash table from its storage
allocation, so two identical dictionary literals in the *same process*
iterate in different orders. Since `normalize()` builds fresh
`contractions`/`abbreviations` dictionaries on every call, the reference
and hypothesis calls inside one `calculateWERAndCER()` get independent
orders. With overlapping keys, `don't` expands to `do not` when `n't` is
applied first but `don not` when `'t` wins — so ref and hyp can disagree
within a single scoring call.
Evidence (standalone harness compiled from the unmodified
`TextNormalizer.swift`):
- 300 fresh processes on the exact #911 pair: **144 diverged** (`do not`
vs `don not`, exactly the 1-word/1-char gap in the artifact: WER 1/10,
CER 1/35).
- Two identical dict literals built back-to-back in one process:
different key order 5/5 runs.
## Fix
Apply all four replacement tables (`britishToAmerican`, `abbreviations`,
`contractions`, `numberWords`) sorted **longest key first**, lexical
tie-break. Deterministic, and the most specific rule now always beats
its substring — which is also the semantically correct expansion:
| input | before (order-dependent) | after |
|---|---|---|
| `don't` | `do not` **or** `don not` | `do not` |
| `can't` | `can not` **or** `ca not` | `can not` |
| `I'd been` | `i had been` **or** `i would been` | `i had been` |
| `it's been` | `it has been` **or** `it is been` | `it has been` |
Verified with the patched file across 50 fresh processes: byte-identical
output every run.
## Tests
`AsrTextNormalizerTests`: the exact #911 record scores WER 0/CER 0
across 100 repeated calls; repeated-call byte-determinism;
overlapping-key expansion expectations.
Note for reviewers: CI benchmark baselines may shift slightly on merge —
historical numbers were sampled from random orderings; the deterministic
ordering picks the correct expansions, so WER should move down or stay
flat.1 parent 71242fa commit 74ad822
3 files changed
Lines changed: 251 additions & 170 deletions
File tree
- Sources
- FluidAudioCLI/Utils
- FluidAudio/ASR/Parakeet/Unified
- Tests/FluidAudioTests/CLI
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
8 | 11 | | |
9 | 12 | | |
10 | 13 | | |
| |||
0 commit comments