Add experimental CTC zh-CN Mandarin ASR - #476
Merged
Merged
Conversation
Fixes actor isolation violations that appeared with stricter Swift 6 concurrency checking in newer Xcode versions. The issue was caused by extracting actor references from properties into local variables using if-let/guard-let, which changes isolation context and risks data races. Solution uses optional chaining with proper scoping: - Avoids force unwrapping (repository rule) - Prevents actor isolation violations (Swift 6 requirement) - Handles actor reentrancy safely (asrManager can become nil after await) - Uses if-let for conditional blocks to avoid skipping critical state updates Changes: - reset(): Optional chaining for resetDecoderState - finish(): Guard-let on processTranscriptionResult return value - processWindow(): Guard-let for required results, if-let for optional rescoring - All early-return guards use guard-let at function level - Conditional block uses if-let to avoid premature function exit Fixes prevent partial state mutations and ensure subscriber notifications always occur even if optional vocabulary rescoring fails.
Moves state mutations to occur AFTER all required async calls complete, preventing inconsistent state if asrManager becomes nil during suspension. Previously, if the second guard-let failed (line 408), the function would return after having already mutated: - accumulatedTokens - lastProcessedFrame - segmentIndex - processedChunks This created inconsistency where tokens were accumulated but transcript state and subscriber notifications were skipped. Solution: Delay all state mutations until after both required async calls (transcribeChunk and processTranscriptionResult) complete successfully.
Integrates Parakeet CTC 0.6B zh-CN model for Mandarin Chinese speech recognition. - Add CtcZhCnManager for full pipeline transcription (preprocessor → encoder → CTC decoder) - Add CtcZhCnModels for model loading from HuggingFace - Support int8 (0.55GB) and fp32 (1.1GB) encoder variants - Add ctc-zh-cn-transcribe CLI command - Add ctc-zh-cn-benchmark CLI command (placeholder) - Greedy CTC decoding with proper blank/repeat handling - 10.22% CER on FLEURS Mandarin Chinese (100 samples) Performance: - Mean CER: 10.22% (matches Python baseline: 10.45%) - 46% of samples < 5% CER (near perfect) - Auto-download from HuggingFace on first use Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add GitHub Actions workflow for CI benchmarking - Implement THCHS-30 dataset auto-download from HuggingFace - Add Swift CLI benchmark command with local/remote dataset support - Add Python benchmark scripts for alternative testing - Expected performance: 8.37% mean CER (100 samples) Dataset: FluidInference/THCHS-30-tests Model: parakeet-ctc-0.6b-zh-cn (int8, 571 MB)
CTC zh-CN Benchmark Results ❌Status: Benchmark failed (see logs) THCHS-30 (Mandarin Chinese)
CER Distribution
Model: parakeet-ctc-0.6b-zh-cn (int8, 571 MB) • Dataset: THCHS-30 (Tsinghua University) CER = Character Error Rate • Lower is better • Calculated using Levenshtein distance with normalized text |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 235.8s processing • Test runtime: 3m 57s • 04/02/2026, 11:22 PM EST |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 0m48s • 04/02/2026, 11:15 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 5m10s • 04/02/2026, 11:18 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 10s • 2026-04-03T03:14:33.346Z |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Qwen3-ASR int8 Smoke Test ✅
Performance Metrics
Runtime: 3m52s Note: CI VM lacks physical GPU — CoreML MLState (macOS 15) KV cache produces degraded results on virtualized runners. On Apple Silicon: ~1.3% WER / 2.5x RTFx. |
Alex-Wengg
changed the title
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 41.3s diarization time • Test runtime: 1m 59s • 04/02/2026, 11:09 PM EST |
…lection - Add .ctcZhCn to .v3 case in decoder selection switch - CTC zh-CN models use TdtDecoderV3 like v3 models - Fixes build failure in CI
- Remove FLEURS 100-sample validation benchmark - Make THCHS-30 full benchmark (2,495 samples, 8.23% CER) the primary result - Clarify command runs full dataset by default
PocketTTS Smoke Test ✅
Runtime: 0m50s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality may differ from Apple Silicon. |
Kokoro TTS Smoke Test ✅
Runtime: 0m57s Note: Kokoro TTS uses CoreML flow matching + Vocos vocoder. CI VM lacks physical ANE — performance may differ from Apple Silicon. |
- Add digit-to-Chinese conversion (0→零, 1→一, etc.) to normalizeChineseText - Add English punctuation removal and ASCII quote handling - Fix CI workflow cache paths (THCHS-30 dataset, parakeet-ctc-zh-cn model) - Fix CI workflow job name (FLEURS → THCHS-30) - Add comprehensive unit tests for text normalization, CER calculation, and Levenshtein distance Fixes: - 🔴 Missing digit conversion was inflating CER by ~1.66% - 🟡 Dataset cache was never effective (wrong path) - 🟡 Model cache was never effective (wrong path) - 🔴 No unit tests for new pure functions
- Add experimental warning to benchmark documentation - Clarify this is an early preview with potential API changes
Alex-Wengg
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
- Replace literal curly quotes with Unicode escape sequences - Avoids Swift parser treating quotes as string terminators - Fixes CI build error: consecutive statements on a line must be separated by semicolon Uses Unicode escapes U+201C, U+201D, U+2018, U+2019 for Chinese quotation marks
1. Fix CI workflow CER threshold mismatch (12% → 10%) - PR comment cerStatus now uses 10.0 threshold to match validation - Previously: validation used 10%, but PR comment used 12% - Result: Inconsistent status indicators (❌ header with ✅ CER row) 2. Fix saveResults NaN crash with empty results array - Add guard to return early if results array is empty - Prevents division by zero (0.0 / 0.0 = NaN) - Prevents JSONEncoder throwing error on NaN values - Logs clear warning instead of cryptic encoding error
…quotes
- Replace literal curly quotes with Unicode escape sequences on lines 25, 102
- Matches fix in CtcZhCnBenchmark.swift
- Fixes: 'consecutive statements on a line must be separated by ;'
Both test functions now use:
- \u{201C}\u{201D}\u{2018}\u{2019} instead of literal ''
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
- CTC zh-CN is experimental and doesn't need automated CI - Reduces CI runtime on every PR - Users can run benchmark manually: swift run fluidaudiocli ctc-zh-cn-benchmark
1. Fix FP32 encoder download issue (🔴 Critical) - Include both encoder variants in requiredModels set - Previously only int8 encoder was downloaded - Now downloads both int8 and fp32 encoders - Users can select which to use at runtime via --fp32 flag 2. Fix AsrManager to reject CTC-only models (🟡 Warning) - Split .ctcZhCn case from .v3 in TDT decoder switch - Throw explicit error for CTC-only model misuse - Prevents silent routing to incompatible TDT decoder - Error: CTC-only model .ctcZhCn does not support TDT decoding. Use CtcZhCnManager instead. Changes: - ModelNames.CTCZhCn.requiredModels: Now includes both encoderFile and encoderFp32File - Removed requiredModelsFp32 (no longer needed) - AsrManager.tdtDecodeWithTimings: Separate case for .ctcZhCn with error
Alex-Wengg
changed the title
- Remove Scripts/test_ctc_zh_cn_hf.py (192 lines) - Remove Scripts/benchmark_ctc_zh_cn.py (177 lines) Reasoning: - Scripts were for development validation (Swift vs Python baseline) - Swift CLI already has built-in benchmark: swift run fluidaudiocli ctc-zh-cn-benchmark - Python scripts depend on local mobius/ directory structure - Reduces maintenance burden for experimental feature - Validation complete: Swift achieves 8.23% CER on THCHS-30
Comment on lines
+123
to
+125
| if !force && modelsExist(at: targetDir) { | ||
| logger.info("CTC zh-CN models already present at: \(targetDir.path)") | ||
| return targetDir |
Contributor
There was a problem hiding this comment.
🟡 download() skips downloading when requested encoder variant is missing
The download() function at line 123 uses modelsExist(at:) to skip downloading, but modelsExist (CtcZhCnModels.swift:207-209) checks if EITHER the int8 OR fp32 encoder exists. This means download(useInt8Encoder: true) will return early with "models already present" when only the fp32 encoder exists (and vice versa). The specifically requested encoder variant is never checked. This breaks the pre-download-for-offline-use pattern: a user calling download(useInt8Encoder: true) to ensure models are available offline would get no error, but a subsequent load(useInt8Encoder: true) without network would fail because Encoder-v2-int8.mlmodelc was never actually downloaded.
Prompt for agents
In CtcZhCnModels.download() (CtcZhCnModels.swift:111-165), the early-return check at line 123 calls modelsExist(at:) which accepts either encoder variant as sufficient. But the download function knows which specific variant was requested via the useInt8Encoder parameter.
The fix should make the existence check variant-aware. Either:
1. Add a useInt8Encoder parameter to modelsExist() and check for the specific encoder file, or
2. Add an additional check after modelsExist() that verifies the specific encoder file (int8 or fp32) exists before returning early.
For example, after the modelsExist check passes, also verify:
let encoderFileName = useInt8Encoder ? ModelNames.CTCZhCn.encoderFile : ModelNames.CTCZhCn.encoderFp32File
let encoderPath = targetDir.appendingPathComponent(encoderFileName)
guard FileManager.default.fileExists(atPath: encoderPath.path) else { /* proceed to download */ }
This ensures that download() actually downloads the requested variant even if the other variant already exists.
Was this helpful? React with 👍 or 👎 to provide feedback.
Issue: Range 1...0 is invalid when m or n is 0 (empty arrays) Error: 'Range requires lowerBound <= upperBound' Fix: Guard against empty arrays before entering main loop - When m=0 or n=0, dp[m][n] is already correctly initialized - Skip loops that would create invalid ranges 1...0 Affected tests: - testLevenshteinDistance_EmptyStrings - testLevenshteinDistance_OneEmpty - testCalculateCER_EmptyReference - testCalculateCER_EmptyHypothesis - testCalculateCER_BothEmpty Fixed in both: - Sources/FluidAudioCLI/Commands/ASR/CtcZhCnBenchmark.swift - Tests/FluidAudioTests/ASR/Parakeet/CtcZhCnTests.swift
3 tasks
Alex-Wengg
added a commit
that referenced
this pull request
May 5, 2026
reports.md: trim filler — collapse benchmark sub-tables to bullets, drop the empty greedy/beam A/B table, merge the "PR #476 doc update" one-liner into the digest, fold the mobius-folder map and the disk cleanup into shorter notes, drop the duplicated atan2 fix code block (lives in mobius PR #50), keep one verification table instead of two, add PR #570 row, condense section 13. ~407 → 255 lines. KokoroAne.md: rename section to "KokoroNoise — atan2 phase fix", condense the I/O-unchanged paragraph, drop the inline source link, and add the ANE-zh cache path so users on the Mandarin variant get the same invalidation guidance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds experimental Mandarin Chinese ASR support via the CTC zh-CN model and includes critical Swift 6 concurrency fixes for
SlidingWindowAsrManager.Swift 6 Concurrency Fixes
Fixed Issues
processWindow()that violated Swift 6 actor isolationaccumulatedTokens,lastProcessedFrame,segmentIndex,processedChunks) now occur after all async calls complete successfullyChanges
SlidingWindowAsrManager.processWindow(): Moved state mutation to after async guard statementsCTC zh-CN Mandarin ASR Integration (Experimental)
New Features
Models
FluidInference/parakeet-ctc-0.6b-zh-cn-coremlCLI Commands
Benchmark Results (THCHS-30 Full Test Set)
Full dataset (2,495 samples):
Dataset
THCHS-30 - Mandarin Chinese speech corpus from Tsinghua University
FluidInference/THCHS-30-testsText Normalization
CER calculation includes:
Devin Review Fixes ✅
Addressed all issues from Devin code review:
Review #1 (4 issues)
Review #2 (2 issues)
Review #3 (2 issues)
requiredModelssetAdditional Fixes
\u{201C}etc.) in both source and testsFiles Changed
Swift 6 Concurrency
Sources/FluidAudio/ASR/Parakeet/SlidingWindow/SlidingWindowAsrManager.swiftSources/FluidAudio/ASR/Parakeet/AsrManager.swift(added .ctcZhCn case + error handling)CTC zh-CN Integration
Sources/FluidAudio/ASR/Parakeet/CtcZhCnManager.swift(new)Sources/FluidAudio/ASR/Parakeet/CtcZhCnModels.swift(new)Sources/FluidAudioCLI/Commands/ASR/CtcZhCnTranscribeCommand.swift(new)Sources/FluidAudioCLI/Commands/ASR/CtcZhCnBenchmark.swift(new)Sources/FluidAudio/ModelNames.swift(updated - both encoder variants)Documentation/Benchmarks.md(updated - marked experimental)Tests
Tests/FluidAudioTests/ASR/Parakeet/CtcZhCnTests.swift(new - 13 test cases)Testing
Notes
🤖 Generated with Claude Code