Kokoro Text-to-Speech (#112)

Author: Alex-Wengg

.github/workflows/tts-test.yml

@@ -0,0 +1,43 @@
+name: TTS Test
+
+on:
+  pull_request:
+    branches: [ main ]
+  workflow_dispatch:
+
+jobs:
+  test-tts:
+    runs-on: macos-14
+    
+    steps:
+    - uses: actions/checkout@v4
+    
+    - name: Build FluidAudio
+      run: |
+        swift build -c release
+    
+    - name: Generate TTS Audio
+      run: |
+        echo "🎤 Generating TTS audio (ground truth test)..."
+        TEXT="I can't believe we finally made it to the summit after climbing for twelve exhausting hours through wind and rain, but wow, this view of the endless mountain ranges stretching to the horizon makes every single difficult step completely worth the journey."
+        
+        # This will auto-download model and generate audio
+        swift run --configuration release fluidaudio tts "$TEXT" --output kokoro_output.wav --auto-download
+        
+        # Verify output
+        if [ -f kokoro_output.wav ]; then
+          SIZE=$(ls -l kokoro_output.wav | awk '{print $5}')
+          echo "✅ TTS successful: kokoro_output.wav ($SIZE bytes)"
+        else
+          echo "❌ Output file not created"
+          exit 1
+        fi
+    
+    - name: Upload Audio Output
+      if: always()
+      uses: actions/upload-artifact@v4
+      with:
+        name: kokoro-tts-output
+        path: kokoro_output.wav
+        retention-days: 7
+    

.gitignore

@@ -97,6 +97,10 @@ FluidAudioDatasets/
 *.wav
 *.mp3
 Resources/
+!Sources/FluidAudio/Resources/
+!Sources/FluidAudio/Resources/**
 scripts/
 Documentation/parakeet-tdt/
 docs/parakeet-tdt/
+
+fluidaudio_cli/*

Documentation/Benchmarks.md

@@ -80,6 +80,91 @@ iPhone 16 Pro Max run, and only for models that were reloaded during the session
 | Decoder       |                       88.49 |                        8.11 |              146.01 | MLComputeUnits(rawValue: 1) |
 | JointDecision |                       48.46 |                        7.97 |               71.85 | MLComputeUnits(rawValue: 1) |
 
+## Text-to-Speech
+
+We generated the same strings with to gerneate audio between 1s to ~300s in order to test the speed across a range of varying inputs on Pytorch CPU, MPS, and MLX pipeline, and compared it against the native Swift version with Core ML models.
+
+Each pipeline warmed up the models by running through it once with pesudo inputs, and then comparing the raw inference time with the model already loaded. You can see that for the Core ML model, we traded lower memory and very slightly faster inference for longer initial warm-up.
+
+Note that the Pytorch kokoro model in Pytorch has a memory leak issue: https://github.com/hexgrad/kokoro/issues/152
+
+The following tests were ran on M4 Pro, 48GB RAM, Macbook Pro. If you have another device, please do try replicating it as well!
+
+### Kokoro-82M PyTorch (CPU)
+
+```bash
+KPipeline benchmark for voice af_heart (warm-up took 0.175s) using hexgrad/kokoro
+Test   Chars    Output (s)   Inf(s)       RTFx       Peak GB
+1      42       2.750        0.187        14.737x    1.44
+2      129      8.625        0.530        16.264x    1.85
+3      254      15.525       0.923        16.814x    2.65
+4      93       6.125        0.349        17.566x    2.66
+5      104      7.200        0.410        17.567x    2.70
+6      130      9.300        0.504        18.443x    2.72
+7      197      12.850       0.726        17.711x    2.83
+8      6        1.350        0.098        13.823x    2.83
+9      1228     76.200       4.342        17.551x    3.19
+10     567      35.200       2.069        17.014x    4.85
+11     4615     286.525      17.041       16.814x    4.78
+Total  -        461.650      27.177       16.987x    4.85    
+```
+
+### Kokoro-82M PyTorch (MPS)
+
+I wasn't able to run the MPS model for longer durations, even with `PYTORCH_ENABLE_MPS_FALLBACK=1` enabled, it kept crashing for the longer strings.
+
+```bash
+KPipeline benchmark for voice af_heart (warm-up took 0.568s) using pip package
+Test   Chars    Output (s)   Inf(s)       RTFx       Peak GB
+1      42       2.750        0.414        6.649x     1.41
+2      129      8.625        0.729        11.839x    1.54
+Total  -        11.375       1.142        9.960x     1.54    
+```
+
+### Kokoro-82M MLX Pipeline
+
+```bash
+TTS benchmark for voice af_heart (warm-up took an extra 2.155s) using model prince-canuma/Kokoro-82M
+Test   Chars    Output (s)   Inf(s)       RTFx       Peak GB
+1      42       2.750        0.347        7.932x     1.12
+2      129      8.650        0.597        14.497x    2.47
+3      254      15.525       0.825        18.829x    2.65
+4      93       6.125        0.306        20.039x    2.65
+5      104      7.200        0.343        21.001x    2.65
+6      130      9.300        0.560        16.611x    2.65
+7      197      12.850       0.596        21.573x    2.65
+8      6        1.350        0.364        3.706x     2.65
+9      1228     76.200       2.979        25.583x    3.29
+10     567      35.200       1.374        25.615x    3.37
+11     4615     286.500      11.112       25.783x    3.37
+Total  -        461.650      19.401       23.796x    3.37
+```
+
+#### Swift + Fluid Audio Core ML models
+
+Note that it does take `~15s` to compile the model on the first run, subsequent runs are shorter, we expect ~2s to load. 
+
+```bash
+> swift run fluidaudio tts --benchmark
+...
+FluidAudio TTS benchmark for voice af_heart (warm-up took an extra 2.348s)
+Test   Chars    Ouput (s)    Inf(s)       RTFx
+1      42       2.825        0.440        6.424x
+2      129      7.725        0.594        13.014x
+3      254      13.400       0.776        17.278x
+4      93       5.875        0.587        10.005x
+5      104      6.675        0.613        10.889x
+6      130      8.075        0.621        13.008x
+7      197      10.650       0.627        16.983x
+8      6        0.825        0.360        2.290x
+9      1228     67.625       2.362        28.625x
+10     567      33.025       1.341        24.619x
+11     4269     247.600      9.087        27.248x
+Total  -        404.300      17.408       23.225
+
+Peak memory usage (process-wide): 1.503 GB
+```
+
 ## Voice Activity Detection
 
 Model is nearly identical to the base model in terms of quality, perforamnce wise we see an up to ~3.5x improvement compared to the silero Pytorch VAD model with the 256ms batch model (8 chunks of 32ms)

Documentation/EspeakFramework.md

@@ -0,0 +1,30 @@
+# eSpeak-NG Framework Packaging
+
+FluidAudio bundles the eSpeak-NG phoneme resources so Kokoro can fall back to G2P lookups when the US lexicons don’t contain a word. The Core ML pipeline expects the resources under `Resources/espeak-ng/espeak-ng-data.bundle` with the canonical `voices/` directory inside.
+
+## All Platforms (Primary Flow)
+- `TtsResourceDownloader.ensureEspeakDataBundle` first attempts to stage the packaged `espeak-ng-data.bundle` from SwiftPM resources (`Sources/FluidAudio/Resources/espeak-ng/`).
+- The bundle is copied to `~/.cache/fluidaudio/Models/kokoro/Resources/espeak-ng/`.
+- The `voices/` directory is validated after staging; if missing, `TTSError.downloadFailed` is raised.
+
+## Fallback Behavior (macOS Only)
+- If the packaged bundle is unavailable, **macOS only** falls back to downloading `espeak-ng.zip` from HuggingFace and extracting it with `/usr/bin/unzip`.
+- **iOS/tvOS/watchOS** do not support fallback downloads and will throw `TTSError.downloadFailed` if the packaged bundle is missing.
+- For mobile platforms, ensure the packaged bundle is present in the Swift package resources before building.
+
+## Best practices
+- Keep the `espeak-ng-data.bundle` (packaged copy) and the optional `espeak-ng.zip` fallback in sync with any updates to the Kokoro phoneme mapper.
+- If you customize the cache location, be sure the `Resources/espeak-ng/espeak-ng-data.bundle/voices/` directory is present before running TTS.
+- When testing on iOS, bundle the extracted resources with the app or seed the simulator cache in advance to avoid runtime failures.
+
+## CocoaPods integration notes
+- The `ESpeakNG.xcframework` now includes support for iOS device (arm64), iOS Simulator (arm64 + x86_64), and macOS (arm64 + x86_64).
+- iOS Simulator support is provided via a stub framework that allows building and linking but returns failure values for ESpeakNG function calls.
+- Pod validation passes successfully with `pod lib lint FluidAudio.podspec --allow-warnings` for all platforms.
+- On iOS Simulator, ESpeakNG initialization will fail gracefully and phonemization requests will return `nil` due to the stub implementation.
+- Full ESpeakNG functionality is available on iOS device and macOS platforms.
+
+## Licensing notes
+- eSpeak-NG is distributed under the GNU GPL v3 (or later). Both the core library and the `espeak-ng-data` voices inherit the same license.
+- The full license text now lives at `Licenses/ESpeakNG_LICENSE.txt`; ship this file (or the upstream `COPYING`) anywhere the framework is redistributed and surface it in your third-party notices UI.
+- If you republish the prebuilt `ESpeakNG.xcframework`, keep the license alongside the binary and ensure downstream consumers can obtain the corresponding source per GPL requirements.

Documentation/TTS/README.md

@@ -0,0 +1,67 @@
+# Text-To-Speech (TTS) Code Examples
+
+> **⚠️ Beta:** The TTS system is currently in beta and only supports American English. Additional language support is planned for future releases.
+
+Quick recipes for running the Kokoro synthesis stack.
+
+## CLI quick start
+
+```bash
+swift run fluidaudio tts "Welcome to FluidAudio text to speech" \
+  --output ~/Desktop/demo.wav \
+  --voice af_heart
+```
+
+The first invocation downloads Kokoro models, phoneme dictionaries, and voice embeddings; later runs reuse the
+cached assets.
+
+## Swift async usage
+
+```swift
+import FluidAudio
+import Foundation
+
+@main
+struct DemoTTS {
+    static func main() async {
+        guard #available(macOS 13.0, *) else {
+            print("FluidAudio TTS requires macOS 13 or newer.")
+            return
+        }
+
+        let manager = TtSManager()
+
+        do {
+            try await manager.initialize()
+            let audioData = try await manager.synthesize(text: "Hello from FluidAudio!")
+
+            let outputURL = URL(fileURLWithPath: "/tmp/fluidaudio-demo.wav")
+            try audioData.write(to: outputURL)
+            print("Saved synthesized audio to: \(outputURL.path)")
+        } catch {
+            print("Synthesis failed: \(error)")
+        }
+    }
+}
+```
+
+Swap in `manager.initialize(models:)` when you want to preload only the long-form `.fifteenSecond` variant.
+
+## Inspecting chunk metadata
+
+```swift
+let manager = TtSManager()
+try await manager.initialize()
+
+let detailed = try await manager.synthesizeDetailed(
+    text: "FluidAudio can report chunk splits for you.",
+    variantPreference: .fifteenSecond
+)
+
+for chunk in detailed.chunks {
+    print("Chunk #\(chunk.index) -> variant: \(chunk.variant), tokens: \(chunk.tokenCount)")
+    print("  text: \(chunk.text)")
+}
+```
+
+`KokoroSynthesizer.SynthesisResult` also exposes `diagnostics` for per-run variant and audio footprint totals.

FluidAudio.podspec

@@ -18,7 +18,28 @@ Pod::Spec.new do |spec|
 
   spec.source       = { :git => "https://github.com/FluidInference/FluidAudio.git", :tag => "v#{spec.version}" }
   spec.source_files = "Sources/FluidAudio/**/*.swift"
-  
+
+  # iOS Configuration
+  # Exclude TTS module from iOS builds to avoid ESpeakNG xcframework linking issues.
+  # CocoaPods has known limitations with vendored xcframeworks during pod lib lint on iOS:
+  # the framework symbols aren't properly linked in the temporary build environment,
+  # causing "Undefined symbols" linker errors even though the binary is valid.
+  # iOS builds include: ASR (speech recognition), Diarization, and VAD (voice activity detection).
+  spec.ios.exclude_files = "Sources/FluidAudio/TextToSpeech/**/*"
+  spec.ios.frameworks = "CoreML", "AVFoundation", "Accelerate", "UIKit"
+
+  # macOS Configuration
+  # ESpeakNG framework is only vendored for macOS in the podspec (not a framework limitation).
+  # The xcframework supports iOS, but CocoaPods fails to link it during iOS validation.
+  # This enables TTS (text-to-speech) functionality with G2P (grapheme-to-phoneme) conversion.
+  # macOS builds include: ASR, Diarization, VAD, and TTS with ESpeakNG support.
+  spec.osx.vendored_frameworks = "Sources/FluidAudio/Frameworks/ESpeakNG.xcframework"
+  spec.osx.frameworks = "CoreML", "AVFoundation", "Accelerate", "Cocoa"
+
   spec.swift_versions = ["5.10"]
-  spec.frameworks = "CoreML", "AVFoundation", "Accelerate"
+
+  # Enable module definition for proper framework imports
+  spec.pod_target_xcconfig = {
+    'DEFINES_MODULE' => 'YES'
+  }
 end

Package.swift

@@ -19,9 +19,15 @@ let package = Package(
     ],
     dependencies: [],
     targets: [
+        .binaryTarget(
+            name: "ESpeakNG",
+            path: "Sources/FluidAudio/Frameworks/ESpeakNG.xcframework"
+        ),
         .target(
             name: "FluidAudio",
-            dependencies: [],
+            dependencies: [
+                "ESpeakNG",
+            ],
             path: "Sources/FluidAudio",
             exclude: []
         ),

README.md

@@ -281,11 +281,61 @@ swift run fluidaudio vad-analyze path/to/audio.wav --streaming
 # Benchmark accuracy/precision trade-offs
 swift run fluidaudio vad-benchmark --num-files 50 --threshold 0.3
 ```
-
 `swift run fluidaudio vad-analyze --help` lists every tuning option, including
 negative-threshold overrides, max-speech splitting, padding, and chunk size.
 Offline mode also reports RTFx using the model's per-chunk processing time.
 
+## Text‑To‑Speech (TTS)
+
+> **⚠️ Beta:** The TTS system is currently in beta and only supports American English. Additional language support is planned for future releases.
+
+- Model: Kokoro (CoreML unified model)
+- Language: American English (beta)
+- G2P: Dictionary first, then eSpeak NG (CEspeakNG) for OOV words
+- Output: 24 kHz mono WAV
+
+Requirements (macOS)
+Ensure eSpeak NG headers/libs are available via pkg-config (`espeak-ng`).
+https://github.com/espeak-ng/espeak-ng/tree/master 
+
+### Quick Start (CLI)
+
+```bash
+# First run will download the Kokoro model and vocab
+swift run fluidaudio tts "Hello from FluidAudio." --auto-download --output out.wav
+
+# Another example with punctuation and OOV handling
+swift run fluidaudio tts "Edge-cases: URLs like https://example.com and e-mail test@example.com." --output out2.wav
+```
+
+Notes
+- The TTS pipeline uses a word→phoneme dictionary first; unknown words are phonemized with eSpeak NG (C API) and mapped to the model’s token set.
+- OOV words are printed with their IPA and mapped tokens for visibility during synthesis.
+- We do not prepend any “language token” to avoid leading vowel artifacts.
+
+### Quick Start (Code)
+
+```swift
+import FluidAudio
+
+Task {
+  do {
+    let data = try await KokoroModel.synthesize(text: "Hello from FluidAudio.")
+    try data.write(to: URL(fileURLWithPath: "out.wav"))
+  } catch {
+    print("TTS error: \(error)")
+  }
+}
+```
+
+Troubleshooting
+Build requires eSpeak NG headers/libs for the C API discoverable via pkg-config (`espeak-ng`).
+- If SwiftPM cannot find headers, build with explicit paths:
+  - `swift build -Xcc -I/opt/homebrew/include -Xlinker -L/opt/homebrew/lib`
+- Dictionary and model assets are cached under `~/.cache/fluidaudio/Models/kokoro`.
+
+## Showcase 
+
 ## Showcase
 
 Make a PR if you want to add your app!
@@ -326,6 +376,8 @@ Parakeet-mlx: https://github.com/senstella/parakeet-mlx
 
 silero-vad: https://github.com/snakers4/silero-vad
 
+Kokoro-82M: https://huggingface.co/hexgrad/Kokoro-82M
+
 ### Citation
 
 If you use FluidAudio in your work, please cite: