@@ -4,21 +4,33 @@ Self-contained ExecuTorch implementation of Mistral's
44[ Voxtral-Mini-4B-Realtime-2602] ( https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 ) ,
55a ~ 4B parameter streaming speech-to-text model. No HuggingFace Transformers
66dependency — weights are loaded directly from the Mistral checkpoint.
7+ See [ model.md] ( model.md ) for architecture and implementation details.
8+
9+ ## Overview
10+
11+ The pipeline has two stages: ** export** (Python, once) and ** inference**
12+ (C++ runner, repeated). Export converts the Mistral checkpoint into a
13+ ` model.pte ` file. A separate ` preprocessor.pte ` handles audio-to-mel
14+ conversion. At inference time, the C++ runner loads both ` .pte ` files
15+ and the Tekken tokenizer, then transcribes audio to text.
16+
17+ Two modes are supported: ** offline** (encode full audio, then decode)
18+ and ** streaming** (process 80ms chunks in real time, including live
19+ microphone input).
720
821## Prerequisites
922
1023- ExecuTorch installed from source (see [ building from source] ( ../../../docs/source/using-executorch-building-from-source.md ) )
1124- [ safetensors] ( https://pypi.org/project/safetensors/ ) (` pip install safetensors ` )
12- - [ torchao] ( https://github.com/pytorch/ao ) (` pip install torchao ` )
13- - Model weights downloaded from [ HuggingFace] ( https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 )
14- (requires access approval). The directory should contain ` params.json ` ,
15- ` consolidated.safetensors ` , and ` tekken.json ` .
25+ - Model weights downloaded from [ HuggingFace] ( https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 ) .
26+ The directory should contain ` params.json ` , ` consolidated.safetensors ` ,
27+ and ` tekken.json ` .
1628
1729## Preprocessor
1830
19- The model takes mel spectrogram input. Export a preprocessor ` .pte ` to
20- convert raw 16kHz mono audio into the mel tensor expected by the audio
21- encoder (128 bins, hop=160, n_fft=400 ):
31+ Export a preprocessor ` .pte ` to convert raw audio into the format the
32+ model expects. ` --max_audio_len 300 ` supports audio up to 5 minutes
33+ (300 seconds ):
2234
2335``` bash
2436python -m executorch.extension.audio.mel_spectrogram \
@@ -27,29 +39,20 @@ python -m executorch.extension.audio.mel_spectrogram \
2739 --output_file ./voxtral_rt_exports/preprocessor.pte
2840```
2941
30- This produces ` preprocessor.pte ` which takes a 1-D waveform tensor
31- ` (num_samples,) ` and outputs a mel spectrogram ` (1, 128, T_mel) ` . The
32- ` --max_audio_len 300 ` flag supports audio up to 5 minutes.
33-
34- For streaming, add ` --streaming ` to skip the 30-second chunk padding so
35- that 1280 samples (80ms) produces exactly 8 mel frames:
42+ For streaming, use a separate preprocessor with ` --streaming ` (no audio
43+ length limit):
3644
3745``` bash
3846python -m executorch.extension.audio.mel_spectrogram \
3947 --feature_size 128 \
4048 --streaming \
41- --output_file ./voxtral_rt_exports /preprocessor.pte
49+ --output_file ./voxtral_streaming_exports /preprocessor.pte
4250```
4351
4452## Export
4553
46- Export produces a single ` .pte ` file with three methods:
47-
48- | Method | Input | Output |
49- | --------| -------| --------|
50- | ` audio_encoder ` | mel spectrogram ` (1, 128, T_mel) ` | audio embeddings ` (1, T_mel//8, 3072) ` |
51- | ` text_decoder ` | embeddings ` (1, seq_len, 3072) ` + positions ` (seq_len,) ` | logits ` (1, seq_len, 131072) ` |
52- | ` token_embedding ` | token IDs ` (1, seq_len) ` | embeddings ` (1, seq_len, 3072) ` |
54+ Export produces a single ` .pte ` containing the audio encoder, text decoder,
55+ and token embedding.
5356
5457``` bash
5558python export_voxtral_rt.py \
@@ -61,9 +64,8 @@ python export_voxtral_rt.py \
6164 --qembedding 8w
6265```
6366
64- For streaming, add ` --streaming ` to export the encoder with KV caches for
65- incremental processing. This replaces ` audio_encoder ` with
66- ` encode_audio_chunk ` which processes 8 mel frames at a time:
67+ For streaming, add ` --streaming ` to export the encoder for incremental
68+ processing (80ms audio chunks):
6769
6870``` bash
6971python export_voxtral_rt.py \
@@ -81,16 +83,10 @@ python export_voxtral_rt.py \
8183| Backend | Offline | Streaming | Quantization |
8284| ---------| ---------| -----------| --------------|
8385| ` xnnpack ` | ✓ | ✓ | ` 4w ` , ` 8w ` , ` 8da4w ` , ` 8da8w ` |
84- | ` metal ` | ✓ | ✗ | ` fpa4w ` (Metal-specific 4-bit) |
85- | ` portable ` | ✓ | ✓ | ` 4w ` , ` 8w ` , ` 8da4w ` , ` 8da8w ` |
86+ | ` metal ` | ✓ | ✗ | none (fp32) or ` fpa4w ` (Metal-specific 4-bit) |
8687
87- Metal backend uses AOTInductor for Apple GPU acceleration. For the text
88- decoder, it uses ` StandardSDPA ` (standard PyTorch ` F.scaled_dot_product_attention ` )
89- and ` StaticKVCache ` (using ` index_copy_ ` ) instead of XNNPACK's custom ops,
90- as AOTI has compatibility issues with custom ops. The offline audio encoder
91- uses standard ` F.scaled_dot_product_attention ` which works on all backends.
92-
93- ** Note:** Metal does not yet support streaming mode (no ` --streaming ` flag).
88+ Metal backend provides Apple GPU acceleration. It does not yet support
89+ streaming mode.
9490
9591#### Metal export example
9692
@@ -120,10 +116,16 @@ python export_voxtral_rt.py \
120116| ` --streaming ` | off | Export streaming encoder with KV cache |
121117| ` --max-enc-len ` | ` 750 ` | Encoder sliding window size (streaming only) |
122118
123- ** Note:** ` fpa4w ` (floating-point activation, 4-bit weight) quantization requires ` --backend metal ` .
119+ ** Notes:**
120+ - ` fpa4w ` quantization requires ` --backend metal ` .
121+ - The model was trained with ` --delay-tokens 6 ` . Other values may degrade accuracy.
124122
125123## Build
126124
125+ ExecuTorch must be installed from source first (see
126+ [ Prerequisites] ( #prerequisites ) ). The ` make ` targets below handle
127+ building core libraries and the runner binary.
128+
127129### XNNPACK (CPU)
128130
129131``` bash
@@ -158,9 +160,9 @@ cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
158160 --audio_path input.wav
159161```
160162
161- For streaming, add ` --streaming ` . The runner processes audio in 80ms steps
162- (one audio token per step) , computing mel and running the encoder+decoder
163- incrementally.
163+ For streaming, add ` --streaming ` . This requires a model exported with
164+ ` --streaming ` . The runner processes audio in 80ms steps , computing mel
165+ and running the encoder+decoder incrementally.
164166
165167``` bash
166168cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
@@ -172,7 +174,8 @@ cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
172174```
173175
174176For live microphone input, use ` --mic ` to read raw 16kHz float32 PCM from
175- stdin. Pipe from any audio capture tool:
177+ stdin. This requires a model exported with ` --streaming ` and a streaming
178+ preprocessor. Pipe from any audio capture tool:
176179
177180``` bash
178181# macOS
@@ -197,23 +200,13 @@ Ctrl+C stops recording and flushes remaining text.
197200| ` --streaming ` | off | Use streaming transcription (from WAV file) |
198201| ` --mic ` | off | Live microphone mode (reads raw f32le PCM from stdin) |
199202
200- ### Example output
201-
202- ```
203- $ cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
204- --model_path voxtral_rt_exports/model.pte \
205- --tokenizer_path tekken.json \
206- --preprocessor_path voxtral_rt_exports/preprocessor.pte \
207- --audio_path output.wav
208-
209- Mr. Quilter is the apostle of the middle classes, and we are glad to
210- welcome his gospel. Nor is Mr. Quilter's manner less interesting than
211- his matter. He tells us that at this festive season of the year, with
212- Christmas and roast beef looming before us, similes drawn from eating
213- and its results occur most readily to the mind.
214- ```
215-
216- ## Architecture
203+ ## Troubleshooting
217204
218- See [ model.md] ( model.md ) for architecture details, design choices, and
219- checkpoint format.
205+ - ** Audio format** : Input must be 16kHz mono WAV. Convert with
206+ ` ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ` .
207+ - ** OOM during export** : Reduce ` --max-seq-len ` or skip encoder
208+ quantization (` --qlinear-encoder ` ).
209+ - ** "Model was not exported with --streaming"** : Re-export with the
210+ ` --streaming ` flag. Both ` --streaming ` and ` --mic ` runner modes
211+ require a streaming-exported model.
212+ - ** ` fpa4w ` error** : This quantization requires ` --backend metal ` .
0 commit comments