Skip to content

README.md

Latest commit

 

History

History
130 lines (89 loc) · 6.41 KB

README.md

File metadata and controls

130 lines (89 loc) · 6.41 KB
Raw

Benchmark suite and latency analysis

This directory contains latency, throughput, and Real-Time Factor (RTF) benchmarks for OmniVoice.

All scripts evaluate zero-shot synthesis on German (de) text using the same six test sentences and hardware setup.

Voice cloning benchmark: benchmark_voiceclone.py

Description

benchmark_voiceclone.py measures synthesis speed and Real-Time Factor for the voice cloning mode.

  • Real-Time Factor (RTF): $$\text{RTF} = \frac{\text{Compute Time (seconds)}}{\text{Generated Audio Duration (seconds)}}$$ An RTF below 1.0 means generation runs faster than real-time audio playback. For example, RTF = 0.465 means one second of speech takes 0.465 seconds of GPU time to synthesize.
  • Reference recordings: Reads a 3 to 10 second WAV file (data/audio_ref/<name>.wav) and its transcript (reference_txt_<name>.txt or <name>.txt), configurable via OMNIVOICE_REF_NAME in .env (default: my_voice_berlin_k).
  • One-time prompt encoding: Encodes reference audio once into a VoiceClonePrompt object before the benchmark loop starts. This avoids re-encoding the voice for every sentence.
  • CUDA synchronization: Calls torch.cuda.synchronize() before stopping timers so measurements reflect completed GPU execution instead of asynchronous kernel launch times.

Standard test sentences

All benchmarks in this suite use the same six German sentences:

# Chars German Test Sentence
1 59 "Guten Tag, das hier ist ein erster Test der Sprachsynthese."
2 61 "Der Zug fährt heute leider mit zwanzig Minuten Verspätung ab."
3 62 "Kannst du mir bitte kurz erklären, wie das genau funktioniert?"
4 72 "Am Wochenende soll das Wetter deutlich besser werden als bisher gedacht."
5 70 "Ich hätte nicht gedacht, dass das so schnell geht, mein lieber Freund."
6 60 "Bitte legen Sie die Unterlagen auf den Tisch im Nebenzimmer."

Measured results (NVIDIA RTX 5070 Ti 16 GB, PyTorch 2.8.0+cu128)

Benchmark Model Load Prompt Encoding Cold Start (Call 1) Warm RTF (Calls 2–6) Peak Sentence RTF
Voice cloning (my_voice_berlin_k) 3.18 s 0.45 s 0.721 RTF 0.465 RTF 0.377 RTF
Voice cloning (.pt cached) 3.24 s 0.54 s 0.669 RTF 0.501 RTF 0.395 RTF
Voice design (female, high pitch) 3.06 s N/A 0.699 RTF 0.481 RTF 0.384 RTF
Auto voice (unprompted) 3.18 s N/A 0.842 RTF 0.573 RTF 0.480 RTF

Note

The paper's advertised RTF of 0.025 relies on large multi-GPU batches in datacenters. Single-sentence interactive generation on a single desktop GPU with standard 32 diffusion steps runs at roughly 0.46 to 0.57 RTF.

How to run

uv run --project OmniVoice python benchmarks/benchmark_voiceclone.py

Auto voice benchmark: benchmark_autovoice.py

Description

benchmark_autovoice.py measures unconditioned synthesis speed in auto voice mode. The model samples a random synthetic voice for each sentence.

Key details

How to run

uv run --project OmniVoice python benchmarks/benchmark_autovoice.py

Voice cloning and prompt persistence: benchmark_voiceclone_berlin.py

Description

benchmark_voiceclone_berlin.py tests voice cloning with the Berlin speaker recording (my_voice_berlin_k) and saves the encoded prompt tensor to disk.

Key details

  • Regional accent test: Checks how the Berlin speech pattern carries across unseen sentences.
  • Prompt saving (prompt.save): Exports the encoded voice prompt to data/voices/<name>.pt.
  • Voice caching: Subsequent scripts can call VoiceClonePrompt.load("data/voices/<name>.pt") to skip audio loading and encoding, saving about 0.30 to 0.50 seconds per session.
  • Automatic reporting: Writes results to data/output/benchmark_voiceclone_berlin/benchmark_results.md.

How to run

uv run --project OmniVoice python benchmarks/benchmark_voiceclone_berlin.py

Voice design benchmark: benchmark_voicedesign.py

Description

benchmark_voicedesign.py measures generation speed when steering voice characteristics through text attributes (instruct="female, young adult, high pitch").

Key details

  • Conditioning cost: Measures the compute overhead of text cross-attention compared to unprompted auto voice.
  • Stable speaker identity: Unlike auto voice, constant instruct attributes keep the speaker consistent across sentences.
  • Automatic reporting: Writes results to data/output/benchmark_voicedesign/benchmark_results.md.

How to run

uv run --project OmniVoice python benchmarks/benchmark_voicedesign.py

Benchmark suite overview

Benchmark Script Generation Mode Voice Setup Evaluation Focus
benchmark_voiceclone.py Voice cloning Configured in .env (OMNIVOICE_REF_NAME) Reference encoding overhead and warm RTF.
benchmark_voiceclone_berlin.py Voice cloning Berlin speaker (my_voice_berlin_k) + .pt export Prompt caching to disk and regional speech cadence.
benchmark_autovoice.py Auto voice Randomly sampled per sentence Baseline diffusion speed without guidance.
benchmark_voicedesign.py Voice design Text prompt ("female, young adult, high pitch") Latency cost of text attribute conditioning.

Best practices for benchmarking

  1. Keep runs offline: Run with HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 (set automatically in all benchmark scripts).
  2. Provide reference text: Always supply transcripts next to reference audio to prevent automatic Whisper downloads.
  3. Separate cold starts: The first GPU call includes kernel compilation and context setup. Measure steady-state RTF on calls 2 and later.