Skip to content

Commit 3880914

Browse files
Update README with Metal 4-bit quantization notes (pytorch#17600)
Added installation instructions for torchao with Metal support. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 8972b69 commit 3880914

1 file changed

Lines changed: 28 additions & 0 deletions

File tree

‎examples/models/voxtral_realtime/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,18 @@ python export_voxtral_rt.py \
9999
--qlinear fpa4w
100100
```
101101

102+
**Note:** Metal 4-bit quantization requires torchao built with experimental MPS (Metal) ops.
103+
104+
You can install torchao with Metal support from the `ao` repo (in third-party/ao/)
105+
```bash
106+
USE_CPP=1 TORCHAO_BUILD_EXPERIMENTAL_MPS=1 pip install . --no-build-isolation
107+
```
108+
109+
Alternatively, you can build torchao with Metal support while installing ExecuTorch from source
110+
```bash
111+
EXECUTORCH_BUILD_KERNELS_TORCHAO=1 TORCHAO_BUILD_EXPERIMENTAL_MPS=1 ./install_executorch.sh
112+
```
113+
102114
### Options
103115

104116
| Flag | Default | Description |
@@ -210,3 +222,19 @@ Ctrl+C stops recording and flushes remaining text.
210222
`--streaming` flag. Both `--streaming` and `--mic` runner modes
211223
require a streaming-exported model.
212224
- **`fpa4w` error**: This quantization requires `--backend metal`.
225+
- **Metal runner fails with `Library not loaded: @rpath/libc++.1.dylib`**:
226+
The AOTInductor-compiled `.so` inside the `.pte` references `libc++` via
227+
`@rpath`, which can't be resolved when extracted to a temp directory.
228+
Add `/usr/lib` to `DYLD_LIBRARY_PATH` so dyld finds it in the shared cache:
229+
```bash
230+
DYLD_LIBRARY_PATH=/usr/lib \
231+
cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner ...
232+
```
233+
- **Metal runner fails with `Library not loaded: libomp.dylib`**:
234+
The AOTInductor-compiled `.so` links against OpenMP. Install it via
235+
Homebrew and add it to `DYLD_LIBRARY_PATH`:
236+
```bash
237+
brew install libomp
238+
DYLD_LIBRARY_PATH=/usr/lib:$(brew --prefix libomp)/lib \
239+
cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner ...
240+
```

0 commit comments

Comments
 (0)