fix(cli/scoring): make TextNormalizer replacement order deterministic - #921
Merged
Merged
Conversation
The CI ASR benchmark scored byte-identical hypothesis/reference pairs differently between runs (#911): the same test-other record moved between WER 0% and 10% with no code change. Root cause is stronger than the per-process hashing suspected in the issue: Swift dictionaries iterate in a per-INSTANCE random order (the hash seed incorporates the storage allocation), so the two normalize() calls inside one calculateWERAndCER() each build fresh contraction / abbreviation dictionaries with independent orders. With overlapping keys, "don't" expands to "do not" when "n't" is applied first but "don not" when "'t" wins — reference and hypothesis can disagree within a single process. Reproduced in a standalone harness: 144 of 300 processes diverged on the exact #911 record, and two identical dictionary literals in one process iterate differently 5/5 times. Fix: apply all four replacement tables (britishToAmerican, abbreviations, contractions, numberWords) sorted longest-key-first with a lexical tie-break. This is deterministic and also picks the semantically correct expansion — the specific rule now always beats its substring ("'d been" -> "had been" over "'d" -> "would", "can't" -> "can not" over "n't", "it's been" -> "it has been"). Verified with the patched file across 50 fresh processes: identical output every run, correct expansions for all overlapping-key cases. Fixes #911
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 144.0s processing • Test runtime: 2m 40s • 09/14/2026, 10:58 AM EST |
PocketTTS Smoke Test ✅
Runtime: 0m12s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Supertonic3 Smoke Test ✅
Runtime: 0m36s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m30s • 09/14/2026, 10:45 AM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 41.4s diarization time • Test runtime: 2m 16s • 09/14/2026, 10:46 AM EST |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 9m9s • 09/14/2026, 10:56 AM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 47s • 2026-09-14T14:55:27.490Z |
…fixes Post-review cleanup on the #911 fix: - Hoist abbreviations/contractions/numberWords out of normalize() into static tables and pre-sort all four (incl. britishToAmerican) once via longestKeyFirst static lets, instead of rebuilding and sorting per call. Output verified byte-identical to the per-call-sort version across 30 fresh processes on the harness corpus. This also makes the ordering structural: a future table reuses the sorted statics rather than relying on remembering .sorted(by:) at each loop. - Rename AsrTextNormalizerTests -> AsrScoringNormalizerTests: the trait-gating CI job selects classes with an unanchored --filter 'TextNormalizerTests|...' (tests.yml), which substring-matched the old name and swept this suite into a job that exists only to prove NemoTextProcessing trait behavior. - Cut test loops 100 -> 10 iterations (~6s of -O runtime at 100; each iteration still samples fresh state, 10 is ample). - Note in Unified benchmark.md that recorded WER rows predate the deterministic ordering (#911) so reruns are not comparable to the tenth of a point.
Alex-Wengg
added a commit
that referenced
this pull request
Sep 14, 2026
…deterministic (#923) ## Summary Fixes #922 — same nondeterminism class as #911/#921, in the diarization scorer. `computeSpeakerMapping` built its speaker index orders with `Array(predicted.keys)` / `Array(groundTruth.keys)`. Swift dictionary iteration order is per-instance random (the hash seed incorporates the storage allocation — re-confirmed empirically on Swift 6.2.3), and both assignment paths (the subset DP and the greedy fallback) tie-break by first-encountered index via strict `>` comparisons. The confusion matrix rounds overlaps to integer milliseconds, so exact ties are realistic — and whenever two pairings tie, the winning speaker mapping (and JER) depended on random key enumeration order. ## Evidence Standalone harness on the scenario in the new test (two predicted speakers each overlapping one ground-truth speaker by exactly 4.75 s after the 0.25 s collar): the pre-fix code produced **2 distinct speaker mappings within a single 200-call process** (3/3 runs), and fails the new test's predicate in **20/20 processes**. With the fix: 1 mapping, always (`right → spk_a`, pinned in the test). ## Fixes (commit 2 adds the full sweep from adversarial review) 1. `computeSpeakerMapping`: sort both key arrays — ties resolve by the solver's fixed index-order rule over lexicographically sorted speaker ids; unambiguous assignments unchanged. 2. `offlineMetrics`: the `correctlyAssigned`, `jaccardScores`, and unmapped-pred loops accumulated Doubles in dictionary order. Addition is not associative, and both DER and JER pass through near-zero cancellation (`max(0, overlapSpeech - correctlyAssigned)`, `1.0 - averageJaccard`) where a ulp-level reorder survives the Float cast — with 3+ speakers, DER/JER were still not bit-deterministic. All three loops now iterate in sorted key order. 3. Streaming path (`DiarizationBenchmark.calculateStreamingMetrics`): `overlapsByGtSpeaker.max(by: { $0.value < $1.value })` kept the first-encountered element among tied overlaps in random dictionary order — same class. Ties now break by smaller speaker id. ## Tests `DiarizationSpeakerMappingTests`: tied-overlap stability across 50 calls (each call regroups segments into fresh dictionaries, each with an independent per-allocation key order — the unfixed code fails this 20/20 processes), the exact tied winner pinned so silent tie-break changes fail loudly, plus an unambiguous-mapping correctness check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #911 — the CI ASR benchmark scored byte-identical hypothesis/reference pairs differently between runs (the
1688-142285-0002.flacrecord moved between WER 0% and 10% with no code change).Root cause (proven, and stronger than suspected)
The issue guessed per-process dictionary hashing. It's per-instance: Swift seeds each hash table from its storage allocation, so two identical dictionary literals in the same process iterate in different orders. Since
normalize()builds freshcontractions/abbreviationsdictionaries on every call, the reference and hypothesis calls inside onecalculateWERAndCER()get independent orders. With overlapping keys,don'texpands todo notwhenn'tis applied first butdon notwhen'twins — so ref and hyp can disagree within a single scoring call.Evidence (standalone harness compiled from the unmodified
TextNormalizer.swift):do notvsdon not, exactly the 1-word/1-char gap in the artifact: WER 1/10, CER 1/35).Fix
Apply all four replacement tables (
britishToAmerican,abbreviations,contractions,numberWords) sorted longest key first, lexical tie-break. Deterministic, and the most specific rule now always beats its substring — which is also the semantically correct expansion:don'tdo notordon notdo notcan'tcan notorca notcan notI'd beeni had beenori would beeni had beenit's beenit has beenorit is beenit has beenVerified with the patched file across 50 fresh processes: byte-identical output every run.
Tests
AsrTextNormalizerTests: the exact #911 record scores WER 0/CER 0 across 100 repeated calls; repeated-call byte-determinism; overlapping-key expansion expectations.Note for reviewers: CI benchmark baselines may shift slightly on merge — historical numbers were sampled from random orderings; the deterministic ordering picks the correct expansions, so WER should move down or stay flat.
🤖 Generated with Claude Code