Follow-up from the #911 review (fixed for the WER scorer in #921). The same nondeterminism class exists in the CLI diarization scorer.
DiarizationMetrics.computeSpeakerMapping (Sources/FluidAudioCLI/Utils/DiarizationMetrics.swift:596-597) builds its speaker index orders directly from dictionary iteration:
let predictedIds = Array(predicted.keys)
let groundTruthIds = Array(groundTruth.keys)
Swift dictionaries iterate in a per-instance random order (see #911 — the seed is per allocation, not even per process). The confusion matrix rounds overlaps to Int milliseconds (scale = 1_000.0, line 613), and both assignment paths tie-break by first-encountered index via strict comparisons (candidateScore > bestResult.score at line 670 in the DP, score > bestScore at line 697 in the greedy fallback). So whenever two speaker pairings tie in rounded overlap — realistic for zero-overlap spurious speakers or symmetric short segments — the winning speaker mapping depends on random key enumeration order, and DER/JER/mapping rows can move between runs with no code change. Same CI-noise symptom #911 was filed for, surfacing in diarization-benchmark.yml instead of asr-benchmark.yml.
Fix shape: .sorted() on both key arrays (one line each) makes the tie-break deterministic. Optionally audit the CLI for other Array(<dict>.keys) / for (k, v) in <dict> sites that feed order-sensitive scoring.
Found by adversarial review of #921; not yet observed in a CI run — the mechanism is confirmed from the code, the drift frequency depends on how often rounded-overlap ties occur in the benchmark corpora.
Follow-up from the #911 review (fixed for the WER scorer in #921). The same nondeterminism class exists in the CLI diarization scorer.
DiarizationMetrics.computeSpeakerMapping(Sources/FluidAudioCLI/Utils/DiarizationMetrics.swift:596-597) builds its speaker index orders directly from dictionary iteration:Swift dictionaries iterate in a per-instance random order (see #911 — the seed is per allocation, not even per process). The confusion matrix rounds overlaps to Int milliseconds (
scale = 1_000.0, line 613), and both assignment paths tie-break by first-encountered index via strict comparisons (candidateScore > bestResult.scoreat line 670 in the DP,score > bestScoreat line 697 in the greedy fallback). So whenever two speaker pairings tie in rounded overlap — realistic for zero-overlap spurious speakers or symmetric short segments — the winning speaker mapping depends on random key enumeration order, and DER/JER/mapping rows can move between runs with no code change. Same CI-noise symptom #911 was filed for, surfacing indiarization-benchmark.ymlinstead ofasr-benchmark.yml.Fix shape:
.sorted()on both key arrays (one line each) makes the tie-break deterministic. Optionally audit the CLI for otherArray(<dict>.keys)/for (k, v) in <dict>sites that feed order-sensitive scoring.Found by adversarial review of #921; not yet observed in a CI run — the mechanism is confirmed from the code, the drift frequency depends on how often rounded-overlap ties occur in the benchmark corpora.