Skip to content

Commit 89341d7

Browse files
authored
Improve README.md and model.md for Voxtral REaltime (pytorch#17598)
1 parent 255e9e0 commit 89341d7

2 files changed

Lines changed: 256 additions & 243 deletions

File tree

‎examples/models/voxtral_realtime/README.md‎

Lines changed: 50 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -4,21 +4,33 @@ Self-contained ExecuTorch implementation of Mistral's
44
[Voxtral-Mini-4B-Realtime-2602](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602),
55
a ~4B parameter streaming speech-to-text model. No HuggingFace Transformers
66
dependency — weights are loaded directly from the Mistral checkpoint.
7+
See [model.md](model.md) for architecture and implementation details.
8+
9+
## Overview
10+
11+
The pipeline has two stages: **export** (Python, once) and **inference**
12+
(C++ runner, repeated). Export converts the Mistral checkpoint into a
13+
`model.pte` file. A separate `preprocessor.pte` handles audio-to-mel
14+
conversion. At inference time, the C++ runner loads both `.pte` files
15+
and the Tekken tokenizer, then transcribes audio to text.
16+
17+
Two modes are supported: **offline** (encode full audio, then decode)
18+
and **streaming** (process 80ms chunks in real time, including live
19+
microphone input).
720

821
## Prerequisites
922

1023
- ExecuTorch installed from source (see [building from source](../../../docs/source/using-executorch-building-from-source.md))
1124
- [safetensors](https://pypi.org/project/safetensors/) (`pip install safetensors`)
12-
- [torchao](https://github.com/pytorch/ao) (`pip install torchao`)
13-
- Model weights downloaded from [HuggingFace](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602)
14-
(requires access approval). The directory should contain `params.json`,
15-
`consolidated.safetensors`, and `tekken.json`.
25+
- Model weights downloaded from [HuggingFace](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602).
26+
The directory should contain `params.json`, `consolidated.safetensors`,
27+
and `tekken.json`.
1628

1729
## Preprocessor
1830

19-
The model takes mel spectrogram input. Export a preprocessor `.pte` to
20-
convert raw 16kHz mono audio into the mel tensor expected by the audio
21-
encoder (128 bins, hop=160, n_fft=400):
31+
Export a preprocessor `.pte` to convert raw audio into the format the
32+
model expects. `--max_audio_len 300` supports audio up to 5 minutes
33+
(300 seconds):
2234

2335
```bash
2436
python -m executorch.extension.audio.mel_spectrogram \
@@ -27,29 +39,20 @@ python -m executorch.extension.audio.mel_spectrogram \
2739
--output_file ./voxtral_rt_exports/preprocessor.pte
2840
```
2941

30-
This produces `preprocessor.pte` which takes a 1-D waveform tensor
31-
`(num_samples,)` and outputs a mel spectrogram `(1, 128, T_mel)`. The
32-
`--max_audio_len 300` flag supports audio up to 5 minutes.
33-
34-
For streaming, add `--streaming` to skip the 30-second chunk padding so
35-
that 1280 samples (80ms) produces exactly 8 mel frames:
42+
For streaming, use a separate preprocessor with `--streaming` (no audio
43+
length limit):
3644

3745
```bash
3846
python -m executorch.extension.audio.mel_spectrogram \
3947
--feature_size 128 \
4048
--streaming \
41-
--output_file ./voxtral_rt_exports/preprocessor.pte
49+
--output_file ./voxtral_streaming_exports/preprocessor.pte
4250
```
4351

4452
## Export
4553

46-
Export produces a single `.pte` file with three methods:
47-
48-
| Method | Input | Output |
49-
|--------|-------|--------|
50-
| `audio_encoder` | mel spectrogram `(1, 128, T_mel)` | audio embeddings `(1, T_mel//8, 3072)` |
51-
| `text_decoder` | embeddings `(1, seq_len, 3072)` + positions `(seq_len,)` | logits `(1, seq_len, 131072)` |
52-
| `token_embedding` | token IDs `(1, seq_len)` | embeddings `(1, seq_len, 3072)` |
54+
Export produces a single `.pte` containing the audio encoder, text decoder,
55+
and token embedding.
5356

5457
```bash
5558
python export_voxtral_rt.py \
@@ -61,9 +64,8 @@ python export_voxtral_rt.py \
6164
--qembedding 8w
6265
```
6366

64-
For streaming, add `--streaming` to export the encoder with KV caches for
65-
incremental processing. This replaces `audio_encoder` with
66-
`encode_audio_chunk` which processes 8 mel frames at a time:
67+
For streaming, add `--streaming` to export the encoder for incremental
68+
processing (80ms audio chunks):
6769

6870
```bash
6971
python export_voxtral_rt.py \
@@ -81,16 +83,10 @@ python export_voxtral_rt.py \
8183
| Backend | Offline | Streaming | Quantization |
8284
|---------|---------|-----------|--------------|
8385
| `xnnpack` | ✓ | ✓ | `4w`, `8w`, `8da4w`, `8da8w` |
84-
| `metal` | ✓ | ✗ | `fpa4w` (Metal-specific 4-bit) |
85-
| `portable` | ✓ | ✓ | `4w`, `8w`, `8da4w`, `8da8w` |
86+
| `metal` | ✓ | ✗ | none (fp32) or `fpa4w` (Metal-specific 4-bit) |
8687

87-
Metal backend uses AOTInductor for Apple GPU acceleration. For the text
88-
decoder, it uses `StandardSDPA` (standard PyTorch `F.scaled_dot_product_attention`)
89-
and `StaticKVCache` (using `index_copy_`) instead of XNNPACK's custom ops,
90-
as AOTI has compatibility issues with custom ops. The offline audio encoder
91-
uses standard `F.scaled_dot_product_attention` which works on all backends.
92-
93-
**Note:** Metal does not yet support streaming mode (no `--streaming` flag).
88+
Metal backend provides Apple GPU acceleration. It does not yet support
89+
streaming mode.
9490

9591
#### Metal export example
9692

@@ -120,10 +116,16 @@ python export_voxtral_rt.py \
120116
| `--streaming` | off | Export streaming encoder with KV cache |
121117
| `--max-enc-len` | `750` | Encoder sliding window size (streaming only) |
122118

123-
**Note:** `fpa4w` (floating-point activation, 4-bit weight) quantization requires `--backend metal`.
119+
**Notes:**
120+
- `fpa4w` quantization requires `--backend metal`.
121+
- The model was trained with `--delay-tokens 6`. Other values may degrade accuracy.
124122

125123
## Build
126124

125+
ExecuTorch must be installed from source first (see
126+
[Prerequisites](#prerequisites)). The `make` targets below handle
127+
building core libraries and the runner binary.
128+
127129
### XNNPACK (CPU)
128130

129131
```bash
@@ -158,9 +160,9 @@ cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
158160
--audio_path input.wav
159161
```
160162

161-
For streaming, add `--streaming`. The runner processes audio in 80ms steps
162-
(one audio token per step), computing mel and running the encoder+decoder
163-
incrementally.
163+
For streaming, add `--streaming`. This requires a model exported with
164+
`--streaming`. The runner processes audio in 80ms steps, computing mel
165+
and running the encoder+decoder incrementally.
164166

165167
```bash
166168
cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
@@ -172,7 +174,8 @@ cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
172174
```
173175

174176
For live microphone input, use `--mic` to read raw 16kHz float32 PCM from
175-
stdin. Pipe from any audio capture tool:
177+
stdin. This requires a model exported with `--streaming` and a streaming
178+
preprocessor. Pipe from any audio capture tool:
176179

177180
```bash
178181
# macOS
@@ -197,23 +200,13 @@ Ctrl+C stops recording and flushes remaining text.
197200
| `--streaming` | off | Use streaming transcription (from WAV file) |
198201
| `--mic` | off | Live microphone mode (reads raw f32le PCM from stdin) |
199202

200-
### Example output
201-
202-
```
203-
$ cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
204-
--model_path voxtral_rt_exports/model.pte \
205-
--tokenizer_path tekken.json \
206-
--preprocessor_path voxtral_rt_exports/preprocessor.pte \
207-
--audio_path output.wav
208-
209-
Mr. Quilter is the apostle of the middle classes, and we are glad to
210-
welcome his gospel. Nor is Mr. Quilter's manner less interesting than
211-
his matter. He tells us that at this festive season of the year, with
212-
Christmas and roast beef looming before us, similes drawn from eating
213-
and its results occur most readily to the mind.
214-
```
215-
216-
## Architecture
203+
## Troubleshooting
217204

218-
See [model.md](model.md) for architecture details, design choices, and
219-
checkpoint format.
205+
- **Audio format**: Input must be 16kHz mono WAV. Convert with
206+
`ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav`.
207+
- **OOM during export**: Reduce `--max-seq-len` or skip encoder
208+
quantization (`--qlinear-encoder`).
209+
- **"Model was not exported with --streaming"**: Re-export with the
210+
`--streaming` flag. Both `--streaming` and `--mic` runner modes
211+
require a streaming-exported model.
212+
- **`fpa4w` error**: This quantization requires `--backend metal`.

0 commit comments

Comments
 (0)