OmniVoice Lab

Test and evaluation lab for OmniVoice, an open-weights zero-shot text-to-speech model by k2-fsa.

This repository evaluates German (de) synthesis quality, voice cloning accuracy, and offline inference performance on desktop hardware.

---

Environment and hardware setup

Component	Specification / Path
GPU	NVIDIA GeForce RTX 5070 Ti (16 GB VRAM, Blackwell)
CUDA & PyTorch	CUDA 12.8 / torch 2.8.0+cu128 (required for Blackwell)
Python & Tooling	Python 3.12.9, managed with uv
Model Weights	F:\KI_Model\Huggingface\k2-fsa--OmniVoice (~3.0 GB, stored off C:)
Virtual Environment	OmniVoice\.venv (installed from the cloned submodule)

---

Key architectural insights and pitfalls

1. Generation modes and real-time performance

OmniVoice provides three core generation modes:

• Voice cloning (ref_audio + ref_text or VoiceClonePrompt): The most reliable mode. Conditioning the diffusion model on reference audio narrows the search space and reaches 0.465 warm RTF (about 1.8 seconds of GPU compute per 4-second sentence).
• Voice design (instruct="female, low pitch"): Steers speaker timbre through descriptive text attributes. Prompts were trained primarily on English and Chinese data, so German styling works but has higher variance.
• Auto voice (text only): The model samples a random synthetic voice for each generation call.

Note

The paper's advertised RTF of 0.025 comes from large batch processing in datacenter environments. Single-sentence interactive generation on a single desktop GPU with standard 32 diffusion steps runs at roughly 0.46 to 0.57 RTF.

---

2. Whisper ASR auto-download trap

• Behavior: If you omit ref_text during voice cloning, OmniVoice downloads OpenAI Whisper Large v3 Turbo (1.6 GB) to C: to transcribe the audio on the fly.
• Fix: Always provide ref_text alongside ref_audio, or pre-encode and reuse .pt prompts with create_voice_clone_prompt(). Reference recordings and transcripts live in data/audio_ref/.

---

3. Language detection and text normalization

• Behavior: OmniVoice defaults non-CJK text to English if language is omitted (text.py:319). English text normalization routes through WeTextProcessing, which requires pynini and C++ build tools on Windows.
• Fix: Always pass language="de" explicitly (for example model.generate(text=..., language="de", normalize_text=True)). For German, OmniVoice falls back to num2words, a pure-Python library that needs no compiler.

---

4. Long text handling (10 to 30 second range)

• Behavior: OmniVoiceGenerationConfig sets audio_chunk_threshold = 30.0 seconds. Text producing under 30 seconds of audio generates as a single utterance without chunking.
• Issue: Utterances in the 10 to 30 second range often suffer from unnatural pacing and loss of clarity.
• Fix: Split paragraphs into individual sentences and pass them as a list (text=["Sentence 1", "Sentence 2", ...]). This processes sentences in a single parallel GPU batch and joins them with natural pauses.

---

5. Licensing constraints

• Code: The lab repository code is released under the MIT License. Upstream OmniVoice code is licensed under Apache 2.0 (Xiaomi Corp. / k2-fsa).
• Pre-trained model weights: Licensed under CC-BY-NC (Non-Commercial) because upstream training datasets (such as Emilia) prohibit commercial use.
• Commercial use: You cannot use pre-trained OmniVoice weights in revenue-generating services.

---

Manufacturer baselines vs. lab environment

To put local numbers in context, the table below compares the upstream paper benchmarks (Zhu et al., arXiv:2604.00688) with our desktop setup:

Dimension	Manufacturer Baseline (k2-fsa / Xiaomi)	OmniVoice Lab (Local Setup)
Hardware	Datacenter cluster (NVIDIA A100 80 GB / H100)	Single desktop GPU (NVIDIA RTX 5070 Ti 16 GB, Blackwell)
Execution Mode	Large batch inference (omnivoice-infer-batch, 64+ items)	Sequential single-sentence synthesis (interactive TTS)
Throughput (RTF)	RTF = 0.025 (40x faster than real-time)	Warm RTF = 0.465 (2.15x faster than real-time)
Attention Kernel	flash_attn and flex_attention with packed batches	PyTorch SDPA (Scaled Dot-Product Attention)
Official Evaluation Suites	• FLEURS (102 languages, Omnilingual-ASR WER) • LibriSpeech-PC (HuBERT WER + SIM-o) • Seed-TTS (Whisper/Paraformer WER) • MiniMax (Multilingual WER + UTMOS)	• Standardized 6-sentence German suite (benchmarks/) • Voice cloning with regional accents • Duration budgeting and nonverbal tags (scripts/)
Quality Assessment	Automated ASR WER/CER, ECAPA-TDNN/WavLM cosine similarity, UTMOS neural predictor	Direct human listening tests, exact token budgeting, CUDA-synchronized profiling

---

Empirical lab benchmark results

Measurements taken on the standardized 6-sentence German evaluation suite using an NVIDIA GeForce RTX 5070 Ti (16 GB VRAM) with torch 2.8.0+cu128:

Generation Mode	Benchmark Script	Model Load	Prompt Prep	Cold Start (Call 1)	Warm RTF (Calls 2–6)	Peak Sentence RTF	Report
Voice cloning	benchmark_voiceclone.py	3.18 s	0.45 s	0.721	0.465 (~2.15x real-time)	0.377	Report
Voice cloning (.pt cached)	benchmark_voiceclone_berlin.py	3.24 s	0.54 s	0.669	0.501 (~2.00x real-time)	0.395	Report
Voice design	benchmark_voicedesign.py	3.06 s	N/A	0.699	0.481 (~2.08x real-time)	0.384	Report
Auto voice	benchmark_autovoice.py	3.18 s	N/A	0.842	0.573 (~1.75x real-time)	0.480	Report

Benchmark takeaways

• Conditioned generation runs faster: Voice cloning (0.465 RTF) and voice design (0.481 RTF) beat unconditioned auto voice (0.573 RTF). Conditioning the diffusion model on speaker audio or text attributes restricts the trajectory search space and stabilizes compute time.
• Initial warm-up cost: The first call runs 1.3x to 1.6x slower due to CUDA context setup and kernel selection before reaching steady state.
• Throughput scales with sentence length: Longer sentences (70 to 72 characters) amortize pipeline overhead and reach peak speeds between 0.377 and 0.384 RTF.

---

Repository structure

omnivoice_lab/
├── data/
│   ├── audio_ref/         # Curated reference WAVs (24 kHz PCM_16 mono) + matching .txt transcripts
│   ├── voices/            # Pre-encoded .pt VoiceClonePrompt files
│   └── output/            # Generated audio outputs (gitignored)
├── docs_dw/               # Internal German project notes, journals, and developer memory
│   ├── JOURNAL_1.md       # Chronological development log (part 1)
│   ├── JOURNAL_2.md       # Chronological development log (part 2)
│   ├── memories.md        # Full project rules and environment specifications
│   └── rules.md           # Operating rules summary cheat sheet
├── utils/                 # Environment, audio inspection, and model utilities
│   ├── README.md          # Utilities documentation and usage reference
│   ├── check_env.py       # Environment, CUDA/GPU, and model path verification
│   ├── wav_info.py        # Validates audio format, sample rate, and channels
│   ├── download_model.py  # Full Hugging Face model downloader with license manifest
│   └── download_model_files.py # Selective Hugging Face file downloader
├── benchmarks/            # Latency and RTF measurement benchmarks
│   ├── README.md          # Benchmark suite documentation and RTF analysis
│   ├── benchmark_autovoice.py
│   ├── benchmark_voicedesign.py
│   ├── benchmark_voiceclone.py
│   └── benchmark_voiceclone_berlin.py
├── scripts/               # Test, evaluation, and feature experiment scripts
│   ├── auto_voice.py      # Basic auto voice test
│   ├── voice_clone.py     # Basic voice cloning test
│   ├── voice_design.py    # Basic voice design test
│   ├── long_text_batch.py # Compares single utterance vs. batched sentences
│   ├── speed_sweep.py     # Speaking rate and duration control evaluation
│   ├── nonverbal.py       # Base model 13 non-verbal sound symbols
│   ├── singing.py         # Singing and emotion finetune (unprompted)
│   ├── singing_clone.py   # Singing and emotion finetune + voice cloning
│   └── debug_singing_duration.py # Duration budget diagnostic
├── OmniVoice/             # Cloned upstream repository (gitignored, contains .venv)
│   └── omnivoice/         # Upstream Python package source
├── LICENSE                # MIT License
└── README.md

---

Quick start

Verify environment and model availability:

uv run --project OmniVoice python utils/check_env.py

Run voice cloning with reference audio:

uv run --project OmniVoice python scripts/voice_clone.py

---

License

• Lab repository & scripts: MIT License
• Upstream OmniVoice codebase: Apache 2.0 License (Xiaomi Corp. / k2-fsa)
• Pre-trained model weights: CC-BY-NC 4.0 (Non-Commercial) due to the Emilia training dataset