fix(cli/scoring): sort speaker ids so diarization speaker mapping is deterministic - #923
Merged
Merged
Conversation
…istic computeSpeakerMapping enumerated predicted/ground-truth speakers via Array(dict.keys) — a per-instance random order (see #911/#921) — and both assignment solvers keep the first-encountered winner among tied confusion-matrix entries, so the reported speaker mapping (and JER in asymmetric cases) could differ between runs whenever two pairings tied in millisecond-rounded overlap. Reproduced in a standalone harness: two predicted speakers tying at 4.75 s overlap against one ground-truth speaker produced 2 distinct mappings within a single 200-call process on the old code; sorted key arrays yield 1. Ties now resolve by the assignment solver's fixed index-order rule over lexicographically sorted speaker ids. Fixes #922
Supertonic3 Smoke Test ✅
Runtime: 0m36s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Post-review fixes on the #922 branch: - offlineMetrics still iterated speakerMapping / groundTruthBySpeaker / predictedBySpeaker dictionaries while accumulating Doubles. Addition is not associative and both der and jer pass through near-zero cancellation (max(0, overlapSpeech - correctlyAssigned) and 1.0 - averageJaccard), where a ulp-level reorder survives the Float cast — so with 3+ speakers der/jer were still not run-to-run deterministic after the mapping fix. All three loops now iterate in sorted key order; debug mapping log also sorted. - Streaming path had the same class: overlapsByGtSpeaker.max(by: { $0.value < $1.value }) keeps the first-encountered element among tied overlaps, in random dictionary order. Ties now break by smaller speaker id. - Test pins the exact tied winner (["right": "spk_a"]) instead of count/values-set asserts, so a silent tie-break change that would shift recorded benchmark numbers fails the test. Winner re-verified via the standalone harness (200-call x 3-process: 1 distinct mapping; unambiguous scenario unchanged). Review also empirically re-confirmed per-ALLOCATION dictionary seeding (Swift 6.2.3): the unfixed code fails the new test's predicate in 20/20 processes, so the in-process 50-iteration loop is a real regression guard.
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 51s • 2026-09-14T16:03:19.389Z |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m35s • 09/14/2026, 11:59 AM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
PocketTTS Smoke Test ✅
Runtime: 0m5s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 8m13s • 09/14/2026, 12:03 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
…lation Closes the review's remaining test gap: - Extract the streaming overlap-winner selection into StreamDiarizationBenchmark.bestOverlapMatch (internal, testable) and test it directly: 100 fresh tied dictionaries always yield the smaller speaker id; unambiguous max and empty-dict cases covered. - Add a 3-speaker offlineMetrics test asserting mapping plus exact der/jer/speakerErrorRate stability across 50 calls, exercising the sorted Double-accumulation loops. Expected values verified with the standalone harness: mapping p1/p2/p3 -> gt_a/gt_b/gt_c, metrics bitwise identical within and across 5 processes.
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 120.3s processing • Test runtime: 2m 3s • 09/14/2026, 11:56 AM EST |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 49.7s diarization time • Test runtime: 2m 57s • 09/14/2026, 12:04 PM EST |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #922 — same nondeterminism class as #911/#921, in the diarization scorer.
computeSpeakerMappingbuilt its speaker index orders withArray(predicted.keys)/Array(groundTruth.keys). Swift dictionary iteration order is per-instance random (the hash seed incorporates the storage allocation — re-confirmed empirically on Swift 6.2.3), and both assignment paths (the subset DP and the greedy fallback) tie-break by first-encountered index via strict>comparisons. The confusion matrix rounds overlaps to integer milliseconds, so exact ties are realistic — and whenever two pairings tie, the winning speaker mapping (and JER) depended on random key enumeration order.Evidence
Standalone harness on the scenario in the new test (two predicted speakers each overlapping one ground-truth speaker by exactly 4.75 s after the 0.25 s collar): the pre-fix code produced 2 distinct speaker mappings within a single 200-call process (3/3 runs), and fails the new test's predicate in 20/20 processes. With the fix: 1 mapping, always (
right → spk_a, pinned in the test).Fixes (commit 2 adds the full sweep from adversarial review)
computeSpeakerMapping: sort both key arrays — ties resolve by the solver's fixed index-order rule over lexicographically sorted speaker ids; unambiguous assignments unchanged.offlineMetrics: thecorrectlyAssigned,jaccardScores, and unmapped-pred loops accumulated Doubles in dictionary order. Addition is not associative, and both DER and JER pass through near-zero cancellation (max(0, overlapSpeech - correctlyAssigned),1.0 - averageJaccard) where a ulp-level reorder survives the Float cast — with 3+ speakers, DER/JER were still not bit-deterministic. All three loops now iterate in sorted key order.DiarizationBenchmark.calculateStreamingMetrics):overlapsByGtSpeaker.max(by: { $0.value < $1.value })kept the first-encountered element among tied overlaps in random dictionary order — same class. Ties now break by smaller speaker id.Tests
DiarizationSpeakerMappingTests: tied-overlap stability across 50 calls (each call regroups segments into fresh dictionaries, each with an independent per-allocation key order — the unfixed code fails this 20/20 processes), the exact tied winner pinned so silent tie-break changes fail loudly, plus an unambiguous-mapping correctness check.🤖 Generated with Claude Code