Kokoro Text-to-Speech - #112
Conversation
The CI was failing because models weren't being downloaded. Added the --auto-download flag to ensure models are fetched before synthesis. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Added support for --auto-download flag in the TTS CLI command to download required models before synthesis. This fixes CI failures where models weren't available. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
The frontend model was being skipped during download but required during load, causing "Model not found" errors. Removed the skip condition to ensure all 7 models are downloaded. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Added verification step after downloading models to ensure they can be loaded. If loading fails, explicitly compile the model using MLModel.compileModel(). This fixes CI errors where downloaded models weren't in the correct compiled format. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixed compilation error by adding await for the async compileModel call and replaced unused testModel variable with underscore. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Changed to download the complete CoreML model structure including: - model.mil - weights/weight.bin - coremldata.bin - metadata.json - analytics/ directory This matches the actual compiled model format on HuggingFace instead of trying to create our own Manifest.json. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
The kokoro_frontend model has metadata.json instead of Manifest.json. Added special handling to: - Download metadata.json and save it as Manifest.json for frontend - Download Manifest.json directly for other models - Only download coremldata.bin and analytics for frontend This fixes the "A valid manifest does not exist" error. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Changed from using current working directory to TTSModels.cacheDirectoryURL() for storing Kokoro models. This ensures models are cached in: - macOS: ~/Library/Caches/fluidaudio/ - Linux: ~/.cache/fluidaudio/ This is consistent with all other models in FluidAudio and prevents models from being downloaded to random directories. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
| ### Kokoro-82M MLX Pipeline | ||
|
|
||
| ```bash | ||
| uv run python benchmark_kokoro_mlx.py |
There was a problem hiding this comment.
the scripts are not here right
There was a problem hiding this comment.
they are in the morbius internal
| ### Kokoro-82M PyTorch Pipeline | ||
|
|
||
| ```bash | ||
| uv run python benchmark_kokoro_pip.py |
There was a problem hiding this comment.
oh also, this is probably not a fair apples to apples comparison since the torch is using "cpu" as the backend. It might be a lot faster with "mps" as the backend. Might even be almost on par with MLX
EIther copy the scripts here or provide some documentation on what thtese scripts are doing?
There was a problem hiding this comment.
as discussed, just leave this to me after you merge, I will run the scripts on my laptop
- .github/workflows/swift-format.yml 2. Documentation/EspeakFramework.md - Reorganized documentation to clarify that both macOS and iOS use the same primary flow (packaged bundle first) - Separated fallback behavior - only macOS falls back to downloading zip if bundle missing - Removed confusion about platform differences in bundling approach 3. sources/FluidAudio/Resources/.gitkeep & espeak-ng/.gitkeep ❌ DELETED - Removed placeholder files (no longer needed if committing actual bundle) 4. Sources/FluidAudio/TextToSpeech/Kokoro/Assets/Lexicon/EspeakG2P.swift - Added configurable espeak voice parameter (defaults to en-us) - Tracks current voice and only calls espeak_SetVoiceByName() when it changes - Enables multi-language support (British English, Spanish, French, etc.) 5. Sources/FluidAudio/TextToSpeech/Kokoro/Assets/TtsResourceDownloader.swift - Removed PyTorch .pt file download logic (58 lines removed) - Simplified voice embedding download to only fetch usable JSON files - Better error messages when embeddings unavailable 6. Sources/FluidAudio/TextToSpeech/Kokoro/Pipeline/Preprocess/KokoroChunker.swift - Removed #if canImport(ESpeakNG) checks (96 lines removed) - Removed #available checks for macOS 10.14/iOS 12.0 (unnecessary with current deployment targets) - Removed NSLinguisticTagger fallback code (always uses NaturalLanguage now) - Cleaner code without redundant platform checks
ESpeakNG xcframework supports iOS but CocoaPods fails to link during pob lib on iOS.
| # iOS Configuration | ||
| # Exclude TTS module from iOS builds to avoid ESpeakNG xcframework linking issues. | ||
| # CocoaPods has known limitations with vendored xcframeworks during pod lib lint on iOS: | ||
| # the framework symbols aren't properly linked in the temporary build environment, | ||
| # causing "Undefined symbols" linker errors even though the binary is valid. | ||
| # iOS builds include: ASR (speech recognition), Diarization, and VAD (voice activity detection). | ||
| spec.ios.exclude_files = "Sources/FluidAudio/TextToSpeech/**/*" | ||
| spec.ios.frameworks = "CoreML", "AVFoundation", "Accelerate", "UIKit" | ||
|
|
||
| # macOS Configuration | ||
| # ESpeakNG framework is only vendored for macOS in the podspec (not a framework limitation). | ||
| # The xcframework supports iOS, but CocoaPods fails to link it during iOS validation. | ||
| # This enables TTS (text-to-speech) functionality with G2P (grapheme-to-phoneme) conversion. | ||
| # macOS builds include: ASR, Diarization, VAD, and TTS with ESpeakNG support. | ||
| spec.osx.vendored_frameworks = "Sources/FluidAudio/Frameworks/ESpeakNG.xcframework" | ||
| spec.osx.frameworks = "CoreML", "AVFoundation", "Accelerate", "Cocoa" |
There was a problem hiding this comment.
So not supported on iOS when they use cocoa pods?
There was a problem hiding this comment.
CocoaPods/CocoaPods#11344
https://stackoverflow.com/questions/73644559/xcframework-pod-lib-lint-fail
Yeah, i think this is part of why it got phased out
There was a problem hiding this comment.
I see.. Im okay with the limitation, it will jsut ahve to be something to call out. Can you link this issue in the comments?
BrandonWeng
left a comment
BrandonWeng
left a comment
There was a problem hiding this comment.
Great work on this and iterating with the open source folks. Lets just merge this in and iterate on it for now. Just make sure to mark it as beta and send any feedback our way
| /// Non-English voices are present in the model but are not yet quality-assured for production use. | ||
| public static let availableVoices: [String] = [ | ||
| // American English | ||
| // American English (supported, beta) |
There was a problem hiding this comment.
thought you tested some of these
|
@Alex-Wengg Is this tested to work only on the 10s and 15s versions? I downloaded versions of those models from huggingface that didn't specify any length but it looks like those have been taken down and replaced with only versions that have 10s or 15s in their names. Are these strict limitations of the model output? If so why can't I have the coreml version be more like the onnx version that generates for varying lengths of audio without issue? |
|
@danieloi as for ios development we do have some users in the discord server that were able to develop on their iOS apps i can point you to . just ping me as @bobasaur on discord and we can discuss more . https://huggingface.co/FluidInference/kokoro-82m-coreml/tree/main/kokoro_21_5s.mlmodelc |
3 participants
This PR introduces text-to-speech functionality to FluidAudio using the Kokoro TTS model from Core ML.
docs
Documentation/KokoroChunkerOverview.md # chunking algorithm priority details