VibeVoice-awq-engine

Embeddable Python engine for Microsoft VibeVoice TTS, with AWQ-INT4 quantization of the LLM portion for ~50% VRAM cut and ~2× faster inference on Ampere GPUs.

This is a soft fork of microsoft/VibeVoice that adds:

• A minimal VibeVoiceEngine Python class — single generate(text, voices) call, no FastAPI / no service layer
• A drop-in unified AWQ-INT4 model at ncoder-ai/VibeVoice-Large-AWQ — from_pretrained auto-loads AWQ kernels (no manual graft step)
• Legacy 2-step graft path still supported via awq_llm_path for users with custom AWQ checkpoints

The unmodified upstream model code lives in vibevoice/. Files added by this fork:

• vibevoice_engine/ — the engine wrapper (engine.py, __init__.py)
• examples/ — runnable scripts (basic.py, multi_speaker.py)
• test_e2e.py — smoke test verifying download + AWQ swap + generation
• UPSTREAM_README.md — original microsoft/VibeVoice README, preserved verbatim

When to use this vs alternatives

You want to...	Use
Run a local HTTP server (OpenAI-compatible API)	ncoder-ai/VibeVoice-FastAPI
Embed VibeVoice in a Python script / notebook / your own service	This repo (vibevoice-awq-engine)
Train, finetune, or use the gradio demos	microsoft/VibeVoice (upstream)

Install

git clone https://github.com/ncoder-ai/VibeVoice-awq-engine.git
cd VibeVoice-awq-engine
pip install -e ".[awq]"

For FP16-only use (no AWQ extras):

pip install -e .

transformers is pinned to 4.51.3 because VibeVoice's modular modeling code targets that version. Newer versions have generation API shifts that break the custom _prepare_cache_for_generation hook.

Quickstart

AWQ-INT4 (recommended — ~8.4 GB VRAM, RTF ~0.70)

from vibevoice_engine import VibeVoiceEngine
import soundfile as sf

engine = VibeVoiceEngine(
    model_path="ncoder-ai/VibeVoice-Large-AWQ",  # drop-in unified model
    inference_steps=7,
)

audio = engine.generate(
    "Speaker 0: The morning sun cast long shadows across the cobblestone street.",
    voices=["voices/David_Attenborough.wav"],
)
sf.write("out.wav", audio, 24000)

The unified model embeds quantization_config in its config.json, so transformers wires AWQ kernels automatically — no separate graft step. First run downloads ~9 GB; subsequent runs reuse the HF cache.

FP16 baseline (~17 GB VRAM, RTF ~0.54)

engine = VibeVoiceEngine(
    model_path="rsxdalv/VibeVoice-Large",
    inference_steps=7,
)

Legacy 2-step graft (for custom AWQ checkpoints)

engine = VibeVoiceEngine(
    model_path="rsxdalv/VibeVoice-Large",
    awq_llm_path="ncoder-ai/VibeVoice-Large-AWQ-INT4",  # LLM-only repo
    inference_steps=7,
)

Same end result as the unified model — kept for users with their own AWQ-quantized Qwen2 checkpoints.

Multi-speaker dialogue

script = """\
Speaker 0: Hello, my name is Alice.
Speaker 1: And I'm Bob. Pleased to meet you.
Speaker 0: Likewise!
"""
audio = engine.generate(
    script,
    voices=["voices/female.wav", "voices/male.wav"],
    cfg_scale=1.3,
)
sf.write("dialogue.wav", audio, 24000)

VibeVoice supports up to 4 distinct speakers per generation call.

Configuration

from vibevoice_engine import EngineConfig, VibeVoiceEngine

config = EngineConfig(
    model_path="rsxdalv/VibeVoice-Large",
    awq_llm_path="ncoder-ai/VibeVoice-Large-AWQ-INT4",
    device="cuda:0",
    dtype="float16",            # match AWQ's internal compute path
    attn_implementation="sdpa", # or "flash_attention_2" if installed
    inference_steps=7,          # 5-50; 7 is the sweet spot
    cfg_scale=1.3,              # 1.0-3.0
    torch_compile=False,        # True → ~40% speedup, slow first call
    sample_rate=24000,
)
engine = VibeVoiceEngine(config)
audio = engine.generate("Speaker 0: Hello.", voices=["voice.wav"])

Performance

Real-world benchmarks on a single RTX 3090 (24 GB), 277-character prompt at 7 inference steps:

Configuration	VRAM	Generation time	RTF
rsxdalv/VibeVoice-Large (FP16)	17.4 GB	~9 s for 16 s audio	0.54
FabioSarracino/VibeVoice-Large-Q8 (bnb-Q8)	10.8 GB	~20 s for 16 s audio	1.22
ncoder-ai/VibeVoice-Large-AWQ	8.4 GB	~11 s for 16 s audio	0.70

All three sound identical at 7 inference steps — the AWQ quantization is not audible.

How AWQ works in this model

The unified ncoder-ai/VibeVoice-Large-AWQ checkpoint has its Qwen2-7B LLM pre-quantized to INT4 with AWQ + Marlin GEMM kernels. The audio tokenizer, diffusion head, and connectors remain FP16 inside the same checkpoint — controlled by modules_to_not_convert in quantization_config. Transformers reads that config at load time and wires AWQ layers into the LLM only, leaving everything else untouched. Result: only the LLM matmuls run in INT4, so audio quality is preserved while the LLM (which dominates compute and memory) is ~50% smaller and ~2× faster.

Examples

See examples/:

• basic.py — single-speaker AWQ generation
• multi_speaker.py — 4-speaker dialogue scene

Smoke test

After install:

python test_e2e.py

Downloads the FP16 base + AWQ-INT4 LLM (if not cached), loads them, generates a single-speaker and multi-speaker sample, validates non-trivial audio output. ~1-2 min on a 3090 once cached.

Quantization recipe

The AWQ-INT4 weights at ncoder-ai/VibeVoice-Large-AWQ-INT4 were produced with AutoAWQ using:

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",  # Marlin kernel
}

Calibration: 250 samples (200 prose snippets + 50 wikitext), 512-token max. Quantization runtime: ~12 minutes on a single RTX 3090.

Tracking upstream

This is a soft fork of microsoft/VibeVoice. To pull upstream changes:

git remote add upstream https://github.com/microsoft/VibeVoice.git
git fetch upstream
git merge upstream/main

The fork only adds files; the original vibevoice/ modular code is unchanged so upstream merges should be conflict-free except for README.md / pyproject.toml (which we override).

License

MIT — inherited from upstream microsoft/VibeVoice. See LICENSE.

Credits

• microsoft/VibeVoice — the underlying TTS model and architecture
• rsxdalv/VibeVoice-Large — FP16 base used for AWQ quantization
• casper-hansen/AutoAWQ — quantization toolkit
• ncoder-ai/VibeVoice-FastAPI — sister project; HTTP server wrapper around similar logic