Skip to content

README.md

Latest commit

 

History

History
296 lines (216 loc) · 16.8 KB

README.md

File metadata and controls

296 lines (216 loc) · 16.8 KB
Raw

Scripts overview and usage

This directory contains test, evaluation, and experiment scripts for OmniVoice.

All scripts resolve model paths dynamically via environment variables (OMNIVOICE_MODEL_DIR and OMNIVOICE_SINGING_MODEL_DIR), configured in .env. All scripts run in offline mode (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1) to prevent downloads to C:.

Auto voice: auto_voice.py

Description

auto_voice.py is the minimal smoke test for auto voice generation.

  • Input: A single German sentence ("Guten Tag, das hier ist ein erster Test der Sprachsynthese auf meinem Rechner.").
  • Generation mode: Auto voice (no ref_audio and no instruct textual style prompt). The model samples a random voice for the sentence.
  • Language setting: Set explicitly to language="de" to ensure correct tokenization and avoid defaulting to English and WeTextProcessing.
  • Output: Saves a 24 kHz WAV file to data/output/auto_voice_de.wav and prints execution time, audio duration, and RTF to stdout.

How to run

uv run --project OmniVoice python scripts/auto_voice.py

Voice cloning: voice_clone.py

Description

voice_clone.py clones a speaker voice from a short reference recording and synthesizes a new German sentence with matching timbre.

  • Input reference: A 3 to 10 second WAV file (data/audio_ref/<name>.wav) and its text transcript (data/audio_ref/reference_txt_<name>.txt or <name>.txt). Configured via OMNIVOICE_REF_NAME in .env (default: my_voice_berlin_k).
  • Generation mode: Voice cloning via ref_audio and ref_text.
  • Explicit transcript: Supplying ref_text prevents OmniVoice from downloading and running Whisper to transcribe the audio.
  • Output: Saves a 24 kHz WAV file to data/output/clone_<REFERENCE>.wav alongside runtime metrics.

How to run

uv run --project OmniVoice python scripts/voice_clone.py

Voice design: voice_design.py

Description

voice_design.py tests voice design mode, building target speaker characteristics from natural language attributes (instruct) without reference audio files.

Supported attribute categories

Attributes combine freely in any order, case is ignored, and omitted categories are filled automatically by the model:

Category Supported Attribute Values
Gender (gender) male, female
Age (age) child, teenager, young adult, middle-aged, elderly
Pitch (pitch) very low pitch, low pitch, moderate pitch, high pitch, very high pitch
Style (style) whisper

(English accents and Chinese dialects exist in the base model, but only affect synthesis when target text is in those languages).

Generation example

model.generate(
    text="Guten Tag, das hier ist ein Test der Stimmgestaltung ohne Referenzaufnahme.",
    language="de",
    instruct="female, young adult, high pitch",
)

Output

Saves a 24 kHz WAV file to data/output/voice_design_de.wav.

How to run

uv run --project OmniVoice python scripts/voice_design.py

Long text and batching: long_text_batch.py

Description

long_text_batch.py compares two synthesis strategies for multi-sentence paragraphs to fix pacing degradation in the 10 to 30 second range:

  1. Method A (Single utterance):

    • Passes the entire paragraph as a single continuous string to model.generate().
    • Stays in one piece because it falls below the default chunk threshold (audio_chunk_threshold = 30.0 s), often causing unnatural pacing.
    • Output: data/output/long_text/naive.wav.

  2. Method B (Batched sentence dispatch):

    • Splits the paragraph into individual sentences via regex.
    • Passes the sentences as a batch list (text=["Sentence 1", "Sentence 2", ...]) in a single GPU call.
    • Joins resulting audio segments with an inter-sentence pause (GAP_SECONDS = 0.2).
    • Output: Full audio at data/output/long_text/batched.wav plus sentence clips (sentence_01.wav to sentence_05.wav).

Key takeaway

Sentence batching keeps every segment within OmniVoice's optimal utterance length, improves natural rhythm, and runs in parallel on the GPU.

How to run

uv run --project OmniVoice python scripts/long_text_batch.py

Non-verbal vocalizations: nonverbal.py

Description

nonverbal.py tests the base OmniVoice model's 13 non-verbal vocalization symbols (such as laughter, sighing, conversational reactions, and particle sounds).

Unlike emotion tags (which apply overall mood in the singing finetune), non-verbal tags insert a concrete sound at that exact position before speech continues.

Test matrix (16 cases)

  • Control baseline: 00_control_de.wav (neutral reference sentence without tags).
  • Core vocalizations: [laughter], [sigh] (tested with German text).
  • Reaction tags: [confirmation-en], [question-en] (tested across English and German text).
  • Particle shapes: Question particles (-ah, -oh, -ei, -yi), surprise particles (-ah, -oh, -wa, -yo), and dissatisfaction (-hnn).
  • Speaker consistency: Uses a cloned speaker prompt (my_voice_berlin_k) so differences between files isolate tag behavior.

Output

Saves 16 WAV files to data/output/nonverbal/.

Key observations

  • Native acoustic events: Because non-verbal tags were part of the base model's pre-training (Emilia dataset), OmniVoice treats them as acoustic sound events. The base model does not read tag words (such as "surprise" or "laughter") as literal text.
  • Cloning compatibility: Non-verbal vocalizations blend into the timbre of the cloned reference voice before transitioning into German speech.
  • Base model scope (no emotion tags): The base model does not support mood tags like [whisper], [sad], or [angry]. For whisper styling, use instruct="whisper" in voice design (scripts/voice_design.py) or a whispered reference recording in voice cloning (scripts/voice_clone.py).

How to run

uv run --project OmniVoice python scripts/nonverbal.py

Singing and emotion cloning: singing_clone.py

Description

singing_clone.py tests the community finetune ModelsLab/omnivoice-singing combined with voice cloning from reference audio. It checks whether a cloned voice can be steered into singing or emotional expressions via text tags.

Test design (16 cases)

  • Constant carrier sentence: All emotion tests use the same neutral German sentence ("Ich habe heute Morgen deine Nachricht gelesen.") to isolate tag influence from text bias.
  • Consistent speaker: Uses a single cloned reference voice (my_voice_berlin_k) throughout all runs.
  • Control baseline: 00_control.wav (neutral sentence untagged).
  • Strong emotion tags: [happy], [sad], [angry], [nervous], [whisper].
  • Weak emotion tags: [calm], [excited].
  • Singing evaluation: German song lyrics vs. English control lyrics ([singing]).
  • Combined tags: Multi-tag styling ([singing] [sad], [whisper] [angry]).

Output

Saves 16 WAV files to data/output/singing_clone/.

Key observations

  • Literal tag readout: Listening tests confirm that combining emotion tags ([angry], [happy]) with a cloned reference voice breaks conditioning and causes the model to speak the tag words aloud as text (for example pronouncing "angry" before the sentence).
  • Language limitation: Emotion and singing tags were trained on English audio corpora (GTSinger, CREMA-D, RAVDESS, Expresso). German singing and emotional inflections do not generalize reliably.
  • No special tokens: tokenizer.json in the singing model is identical to the base model. Tags are ordinary text strings that rely on context association learned during fine-tuning.

How to run

uv run --project OmniVoice python scripts/singing_clone.py

Singing and emotion tags: singing.py

Description

singing.py evaluates the ModelsLab/omnivoice-singing finetune in auto voice mode (where the model generates its own synthetic voice rather than cloning from a reference file).

What is guidance_scale? (Classifier-Free Guidance)

guidance_scale is a native parameter in the OmniVoice diffusion pipeline (default = 2.0 in OmniVoiceGenerationConfig): $$\text{logits} = \text{c_log_probs} + \text{guidance_scale} \times (\text{c_log_probs} - \text{u_log_probs})$$

  • Default (2.0): Balances naturalness with text adherence.
  • Elevated value (3.0): Recommended by ModelsLab to amplify the conditioning signal of the finetuned emotion and singing tags.
  • Trade-off: Values above 4.0 cause audible distortion and clipping artifacts.

Test matrix (18 cases)

  • Baseline plain speech: 00_plain_de, 01_plain_en (confirms untagged speech synthesis).
  • Singing evaluation: 02_singing_de vs. 03_singing_en (paired German and English nursery rhymes).
  • Strong emotions (paired DE/EN):
    • [happy]: 04_happy_de, 05_happy_en
    • [sad]: 06_sad_de, 07_sad_en
    • [angry]: 08_angry_de, 09_angry_en
    • [nervous]: 10_nervous_de, 11_nervous_en
    • [whisper]: 12_whisper_de, 13_whisper_en
  • Weak emotions: 14_calm_de, 15_excited_de.
  • Combined tags: 16_singing_sad_de, 17_singing_sad_en (paired DE/EN).

Output

Saves 18 WAV files to data/output/singing/.

Key observations

  • Tag readout on German ([nervous], [excited]): The model speaks the English tag words aloud before the German sentence. On English text, tags apply without being read aloud.
  • German singing instability: Singing in German rarely succeeds, whereas English nursery rhymes sing much more reliably.
  • Premature singing cutoff: Without explicit duration budgeting, the model calculates time for standard speech rates (~15 chars/second) and runs out of budget before finishing the song.

How to run

uv run --project OmniVoice python scripts/singing.py

Speaking rate and duration control: speed_sweep.py

Description

speed_sweep.py evaluates the two native tempo and length control mechanisms in OmniVoice using a cloned speaker voice (my_voice_berlin_k).

Two control mechanisms

  1. Speed factor (speed): Sweeps speaking rate from 0.6 (slow) to 1.4 (fast) in 0.1 increments. In voice cloning mode, speed=1.0 corresponds to the natural speaking rate of the reference recording.
  2. Fixed duration (duration): Forces exact target output lengths (duration=6.0 s and duration=3.0 s).

Pre-generation token budgeting

OmniVoice does not apply post-hoc digital time-stretching or pitch-shifting filters. Instead, speed and duration adjust the audio token sequence budget before diffusion generation begins. Slower speeds produce genuinely held vowels and natural pauses without warble artifacts.

Test matrix (11 cases)

  • Speed sweep: speed_06.wav through speed_14.wav (9 factor variations on the same German carrier sentence).
  • Fixed duration: duration_6s.wav and duration_3s.wav.

Output

Saves 11 WAV files to data/output/speed_sweep/.

How to run

uv run --project OmniVoice python scripts/speed_sweep.py

Duration diagnostic and starvation analysis: debug_singing_duration.py

Description

debug_singing_duration.py tests the time starvation hypothesis: whether singing and emotion tags collapse when combined with voice cloning due to insufficient token budgets.

The time starvation hypothesis

  1. Budget calculation: In voice cloning mode, model._estimate_target_tokens() derives the output token budget from the reference audio: $$\text{Rate} = \frac{\text{Audio Tokens in Reference}}{\text{Characters in Reference Transcript}}$$ This estimates token count based on the reference speaker's spoken rate (~15 chars/second).
  2. Singing discrepancy: Singing needs several times more audio frames per syllable to sustain vowels (~3 to 5 chars/second).
  3. Collapse mechanism: Given a spoken reference budget, the model receives too few tokens to sustain singing. The diffusion process rushes or falls back into plain speech.

Test matrix (8 cases across 3 groups)

Group Case Purpose / Design Text & Tag Mode Budget Control
A: Unprompted (No Clone) a1_noclone_sing_estimated Default heuristic [singing] Twinkle... (EN) Auto Voice Heuristic (~25 tokens)
a2_noclone_sing_duration12 Generous duration [singing] Twinkle... (EN) Auto Voice duration=12.0 s
a3_noclone_sing_speed05 Halved tempo [singing] Twinkle... (EN) Auto Voice speed=0.5
B: Cloned Voice b1_clone_sing_estimated Default budget [singing] Twinkle... (EN) Cloned Spoken rate budget (~3.5 s)
b2_clone_sing_duration12 Generous duration [singing] Twinkle... (EN) Cloned duration=12.0 s
b3_clone_sing_speed05 Halved tempo [singing] Twinkle... (EN) Cloned speed=0.5
C: Emotion Readout c1_clone_angry_estimated Default budget [angry] Ich habe... (DE) Cloned Spoken rate budget (~3.1 s)
c2_clone_angry_duration6 Padded duration [angry] Ich habe... (DE) Cloned duration=6.0 s

Output

Saves 8 WAV files to data/output/debug_duration/.

Key observations and empirical verdict

  • English singing verified: Listening tests confirm that in English, [singing] works reliably both in auto voice mode (a2, a3) and with a cloned speaker voice (b2, b3) once given sufficient duration (duration=12.0 s) or reduced tempo (speed=0.5).
  • Duration control essential for singing: Without explicit duration budgeting (a1, b1), the model defaults to a fast speaking-rate budget and cuts singing short.
  • German language barrier confirmed: In German (c1, c2), emotion tags like [angry] continue to be read aloud as text, and German singing remains unstable regardless of duration budgeting.
  • Conclusion: Time starvation causes rushed singing (solvable with duration or speed), whereas the failure on German is an upstream training limitation of ModelsLab/omnivoice-singing.

How to run

uv run --project OmniVoice python scripts/debug_singing_duration.py

Additional utility and test scripts

Script Mode / Purpose Command
check_env.py Verifies Python, PyTorch, CUDA 12.8, GPU VRAM, and model files uv run --project OmniVoice python utils/check_env.py
wav_info.py Validates reference WAV audio headers in data/audio_ref/ uv run --project OmniVoice python utils/wav_info.py
download_model.py Full Hugging Face model downloader with license manifest uv run --project OmniVoice python utils/download_model.py <repo>
download_model_files.py Selective Hugging Face file downloader with patterns uv run python utils/download_model_files.py <repo> <patterns>
voice_clone.py Clones a speaker from reference WAV + transcript uv run --project OmniVoice python scripts/voice_clone.py
voice_design.py Generates voice tailored by textual style prompts (instruct) uv run --project OmniVoice python scripts/voice_design.py
benchmark_*.py Measures latency and RTF across generation modes uv run --project OmniVoice python benchmarks/benchmark_voiceclone.py
long_text_batch.py Evaluates sentence batching vs. single long utterances uv run --project OmniVoice python scripts/long_text_batch.py
speed_sweep.py Sweeps speaking rate factors (0.6–1.4) and fixed target durations uv run --project OmniVoice python scripts/speed_sweep.py
nonverbal.py Tests the 13 non-verbal tags (such as [laughter]) uv run --project OmniVoice python scripts/nonverbal.py
singing.py Tests singing and emotion tags with unprompted synthetic voices uv run --project OmniVoice python scripts/singing.py
singing_clone.py Evaluates singing and emotion tags combined with voice cloning uv run --project OmniVoice python scripts/singing_clone.py
debug_singing_duration.py Duration diagnostic on voice cloning vs. singing token budgets uv run --project OmniVoice python scripts/debug_singing_duration.py
utils/README.md Full documentation for all environment and download tools N/A