Qwen3-TTS Custom Voice Server

High-concurrency async TTS server that serves Qwen3-TTS-12Hz-1.7B-Base with pre-computed speaker embeddings on a custom vLLM-Omni fork (vllm-omni-myfi/). Voice cloning runs with zero ECAPA-TDNN overhead per request — the 1024-dim x-vector is extracted once offline, cached on GPU, and injected directly into the AR Talker.

                ┌─────────────────────────────────────────────┐
                │   FastAPI  (inference_server.py)            │
                │   /synthesize · /synthesize/stream          │
                └─────────────┬───────────────────────────────┘
                              │ speaker_id + text
                              ▼
            ┌──────────────────────────────────┐
            │  EmbeddingCacheManager (GPU)     │  ← voice_profiles/*.safetensors
            │  speaker_id → 1024-dim x-vector  │
            └──────────────────┬───────────────┘
                               ▼
        ┌──────────────────────────────────────────────┐
        │   vLLM-Omni AsyncOmni  (two-stage pipeline)  │
        │                                              │
        │   Stage 0: AR Talker    (Qwen3 backbone)     │
        │     text + x-vector  →  codec tokens         │
        │               │                              │
        │               ▼  SharedMemoryConnector       │
        │   Stage 1: Code2Wav     (Speech Tokenizer)   │
        │     codec tokens  →  24 kHz waveform         │
        └──────────────────────────────────────────────┘

---

What's in this repo

Path	Purpose
inference_server.py	FastAPI async server (main entry)
batch_inference.py	Offline batch mode (no server)
embedding_cache.py	GPU-resident speaker embedding cache
inference_config.py	Env-var configuration
extract_speaker_embedding.py	One-time voice profile extraction
stage_config_optimized.yaml	Tuned vLLM-Omni pipeline config
qwen_tts/	Qwen3-TTS model library (used only by the extractor)
vllm-omni-myfi/	Custom vLLM-Omni fork with the patches below
qwen_custom.md	Model architecture reference
vllm_custom.md	vLLM-Omni integration reference

---

Prerequisites

	Minimum	Recommended
OS	Linux (Ubuntu 20.04+)	Ubuntu 22.04
Python	3.10	3.12
GPU	NVIDIA CC 7.0+ (V100/T4)	A100 / H100 / L40
VRAM	16 GB	24 GB+
CUDA	12.x	12.4+
RAM	32 GB	64 GB
Disk	20 GB	50 GB

Both pipeline stages run on a single GPU. The optimized config allocates 40 % + 30 % = 70 % VRAM, leaving headroom for CUDA overhead and the embedding cache.

---

Install

# 1. Fresh env (do NOT install vLLM via conda — its static NCCL conflicts)
conda create -n qwen3-tts python=3.12 -y && conda activate qwen3-tts

# 2. Base vLLM engine (install BEFORE vllm-omni)
pip install vllm==0.15.1

# 3. Custom vLLM-Omni fork (this repo's patched vllm-omni-myfi/)
cd vllm-omni-myfi && pip install -e . && cd ..

# 4. qwen-tts library (used by the embedding extractor)
pip install -e .

# 5. Server extras
pip install fastapi uvicorn safetensors

# 6. (Optional) FlashAttention 2 — cuts VRAM usage
pip install -U flash-attn --no-build-isolation

Verify

python -c "
import vllm, vllm_omni, qwen_tts, fastapi, torch
print(f'vLLM       {vllm.__version__}')
print(f'vLLM-Omni  {vllm_omni.__version__}')
print(f'PyTorch    {torch.__version__}')
print(f'CUDA       {torch.cuda.is_available()} ({torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"N/A\"})')
"

Expected: vLLM 0.15.1, vLLM-Omni 0.14.0, CUDA True.

---

Download the model

pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base

Or ModelScope (faster in CN):

pip install -U modelscope
modelscope download --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --local_dir ./models/Qwen3-TTS-12Hz-1.7B-Base
export TTS_MODEL_PATH=./models/Qwen3-TTS-12Hz-1.7B-Base

Why the Base model, not the CustomVoice checkpoint? The CustomVoice variant ships with a fixed set of predefined speakers (vivian, ryan, …). For arbitrary user-provided voices, the Base checkpoint is the correct choice — it exposes the ECAPA-TDNN speaker encoder that produces the 1024-dim x-vector we cache.

---

Extract voice profiles

Each profile is a ~4 KB .safetensors file (the 1024-dim x-vector). One-time cost per speaker; the server itself never touches ECAPA-TDNN.

python extract_speaker_embedding.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --device cuda \
    --ref-audio /path/to/speaker.wav \
    --ref-text "Exact transcript of the reference audio." \
    --output-name alice \
    --output-dir voice_profiles

Audio requirements: clean speech, 5–30 s, any sample rate (auto-resampled to 24 kHz). The --ref-text must be the exact transcript for good cloning quality.

---

Run the server

python inference_server.py

Defaults: http://0.0.0.0:8090, loads every .safetensors in voice_profiles/ at startup.

Environment variables

Variable	Default	Description
TTS_MODEL_PATH	Qwen/Qwen3-TTS-12Hz-1.7B-Base	HF model ID or local path
TTS_STAGE_CONFIG	stage_config_optimized.yaml	Pipeline config YAML
TTS_PROFILES_DIR	voice_profiles	Directory of .safetensors profiles
TTS_DEVICE	cuda:0	GPU for embedding cache
TTS_HOST / TTS_PORT	0.0.0.0 / 8090	Bind address
TTS_MAX_CONCURRENT	16	Backpressure semaphore
TTS_STAGE_INIT_TIMEOUT	300	Stage init wait (seconds)
TTS_TEMPERATURE / TTS_TOP_K / TTS_MAX_NEW_TOKENS	0.9 / 50 / 2048	Sampling defaults

Startup sequence

1. Loading speaker embeddings from voice_profiles   ~instant
2. Initializing AsyncOmni                           ~30–120 s (first run: model download)
3. Stage 0 (AR Talker) loading                      ~30–60 s
4. Stage 1 (Code2Wav) loading                       ~15–30 s
5. Server ready: N speakers loaded, max_concurrent=16

---

API

POST /synthesize

curl -X POST http://localhost:8090/synthesize \
    -H "Content-Type: application/json" \
    -d '{
        "speaker_id": "alice",
        "text": "Hello! This is a test of the text to speech system.",
        "language": "English",
        "response_format": "wav"
    }' \
    --output out.wav

Request body:

Field	Type	Default	Description
speaker_id	string	—	Profile filename stem (e.g. alice)
text	string	—	Text to synthesize
language	string	"Auto"	Auto, English, Chinese, …
response_format	string	"wav"	wav, pcm, mp3, flac
max_new_tokens	int	2048	Max codec tokens
stream	bool	false	(use /synthesize/stream instead)

Response headers: X-Request-Id, X-Audio-Duration, X-Latency, X-Sample-Rate.

POST /synthesize/stream

Chunked raw PCM float32 LE @ 24 kHz:

curl -X POST http://localhost:8090/synthesize/stream \
    -H "Content-Type: application/json" \
    -d '{"speaker_id":"alice","text":"Streaming test.","language":"English"}' \
    --output out.pcm
ffmpeg -f f32le -ar 24000 -ac 1 -i out.pcm out.wav

GET /speakers · GET /health

curl http://localhost:8090/speakers | python -m json.tool
curl http://localhost:8090/health   | python -m json.tool

Python client

import httpx
resp = httpx.post("http://localhost:8090/synthesize", json={
    "speaker_id": "alice",
    "text": "Hello from Python!",
    "language": "English",
})
open("out.wav", "wb").write(resp.content)

---

Offline batch mode

# manifest.json
# [
#   {"speaker_id": "alice", "text": "First.",  "language": "English", "output": "a.wav"},
#   {"speaker_id": "bob",   "text": "Second.", "language": "English", "output": "b.wav"}
# ]

python batch_inference.py \
    --manifest manifest.json \
    --profiles-dir voice_profiles \
    --output-dir output_audio \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --stage-configs-path stage_config_optimized.yaml

---

Built-in optimizations

Optimization	Effect	Location
Pre-computed embeddings	Skips ECAPA-TDNN per request (~200 ms → 0)	embedding_cache.py
Top-level ref_spk_embedding fast path	Bypasses msgspec tensor-drop in vLLM-Omni IPC	vllm-omni-myfi/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py
Cached bos/eos/pad embeds	Special-token projection computed once	qwen3_tts_talker.py
torch.compile on code predictor	Fused kernels for residual code head	qwen3_tts_code_predictor_vllm.py
Non-streaming text prefill	All text in prefill, fewer decode steps	inference_server.py
codec_chunk_frames: 12	Halves time-to-first-audio in streaming	stage_config_optimized.yaml
connector_get_sleep_s: 0.002	Tighter inter-stage polling	stage_config_optimized.yaml
max_batch_size: 4, max_inflight: 4	Concurrent AR + decoding	stage_config_optimized.yaml

---

Tuning

CUDA OOM — edit stage_config_optimized.yaml:

# Stage 0 (AR Talker)
gpu_memory_utilization: 0.30   # default 0.40
max_batch_size: 2              # default 4

# Stage 1 (Code2Wav)
gpu_memory_utilization: 0.20   # default 0.30
max_batch_size: 2              # default 4

runtime:
  defaults:
    max_inflight: 2            # default 4

Higher throughput on H100/A100 80 GB — raise gpu_memory_utilization (0.5 / 0.4), max_batch_size (8), max_inflight (8), max_num_batched_tokens (4096).

Stage init timeout on slow disk / first download:

TTS_STAGE_INIT_TIMEOUT=600 python inference_server.py

---

Troubleshooting

ModuleNotFoundError: No module named 'vllm' — install vllm==0.15.1 before the vllm-omni fork (Step 2). vllm-omni intentionally does not depend on vllm to avoid entrypoint overwrites.

No speaker embeddings found — voice_profiles/ is empty; run extract_speaker_embedding.py first.

RuntimeError: Cannot re-initialize CUDA in forked subprocess — VLLM_WORKER_MULTIPROC_METHOD=spawn must be set before any CUDA import. The server does this automatically; if writing a custom script, export it manually.

torch.compile warnings on first request — expected. The first 1–2 requests trace the code-predictor graph; subsequent requests use the compiled kernels.

Speaker X not found — the server loads profiles at startup only. Add new .safetensors to voice_profiles/ and restart (lazy load is a fallback, not a hot-reload).

---

Architecture notes

For model-internals details, see qwen_custom.md. For the vLLM-Omni serialization constraints and stage-config reference, see vllm_custom.md.

The two knowledge-base docs exist because the tensor passed via additional_information["ref_spk_embedding"] is the critical integration point: msgspec drops tensors nested inside list-serialized dicts, so the embedding must be a top-level key — which is exactly what the patched talker picks up.

---

License

See LICENSE.