Skip to content

Commit e7ca2b2

Browse files
[MLX] C++ runner for HF LLMs with the off-graph KV cache (pytorch#21683)
**Summary** This runner builds an MLXSequenceCache, installs it, and passes the cache key, so it's the run path for .pte files exported with --use-offgraph-cache. The cache's shape is read from the .pte metadata. The flags left are policy the model can't imply: --kv-max-capacity, --kv-storage-dtype, --kv-initial-capacity, --kv-max-write, --kv-windows. Depends on pytorch#21680; the new CI job fails until that lands. **Files** - run_llm_hf.cpp — the runner: chat templates, greedy decode, benchmarking, and an interactive mode with /reset and /undo [N]. - CMakeLists.txt — standalone find_package(executorch) project. - .github/workflows/mlx.yml — test-mlx-llm-offgraph for llama-1b, gemma3-1b and gemma4-e2b. **Test** CI builds the runner, then for llama-1b, gemma3-1b and gemma4-e2b exports off-graph and asserts the same "Paris" answer test-mlx-llm checks.
1 parent 61d0880 commit e7ca2b2

6 files changed

Lines changed: 1048 additions & 18 deletions

File tree

‎.github/workflows/mlx.yml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -678,3 +678,113 @@ jobs:
678678
exit 1
679679
fi
680680
echo "::endgroup::"
681+
682+
# Off-graph KV cache: the cache is a runtime object, so this path can only be
683+
# exercised by the C++ runner (pybindings cannot bind a cache_key). Also the
684+
# only coverage that the layout the export publishes is actually consumable.
685+
test-mlx-llm-offgraph:
686+
# Requires HuggingFace secrets — skip on fork PRs.
687+
needs: run-decision
688+
if: |
689+
(github.event.pull_request.head.repo.full_name == github.repository || github.event_name != 'pull_request') &&
690+
(github.event_name == 'pull_request' || needs.run-decision.outputs.is-full-run == 'true')
691+
strategy:
692+
fail-fast: false
693+
matrix:
694+
model:
695+
- id: "unsloth/Llama-3.2-1B-Instruct"
696+
name: "llama-1b"
697+
chat: "llama3"
698+
runner: "macos-14-xlarge"
699+
- id: "unsloth/gemma-3-1b-it"
700+
name: "gemma3-1b"
701+
chat: "gemma"
702+
runner: "macos-14-xlarge"
703+
# Only model with KV sharing: 15 caches for 35 layers, and sliding and
704+
# full-attention caches with different head shapes.
705+
- id: "google/gemma-4-E2B-it"
706+
name: "gemma4-e2b"
707+
chat: "gemma4"
708+
runner: "macos-15-xlarge"
709+
uses: pytorch/test-infra/.github/workflows/macos_job.yml@main
710+
secrets: inherit
711+
with:
712+
default-packages: ""
713+
job-name: test-mlx-llm-offgraph-${{ matrix.model.name }}
714+
runner: ${{ matrix.model.runner }}
715+
python-version: "3.12"
716+
submodules: recursive
717+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
718+
secrets-env: EXECUTORCH_HF_TOKEN
719+
# Higher than the sibling LLM job: this one also builds and installs
720+
# ExecuTorch so the standalone runner project can link against it.
721+
timeout: 120
722+
script: |
723+
set -eux
724+
export HF_HUB_DISABLE_XET=1
725+
726+
MODEL_ID="${{ matrix.model.id }}"
727+
MODEL_NAME="${{ matrix.model.name }}"
728+
CHAT="${{ matrix.model.chat }}"
729+
730+
echo "::group::Install ExecuTorch and build the MLX runtime"
731+
${CONDA_RUN} python install_executorch.py > /dev/null
732+
${CONDA_RUN} cmake --preset mlx-release
733+
${CONDA_RUN} cmake --build cmake-out --target install -j$(( $(sysctl -n hw.ncpu) - 1 ))
734+
echo "::endgroup::"
735+
736+
echo "::group::Build the C++ runner"
737+
# Standalone find_package(executorch) project, so it needs the install above.
738+
${CONDA_RUN} cmake -S backends/mlx/examples/llm \
739+
-B cmake-out/backends/mlx/examples/llm -DCMAKE_BUILD_TYPE=Release
740+
${CONDA_RUN} cmake --build cmake-out/backends/mlx/examples/llm \
741+
-j$(( $(sysctl -n hw.ncpu) - 1 ))
742+
RUNNER=cmake-out/backends/mlx/examples/llm/mlx_run_llm_hf
743+
if [ ! -x "${RUNNER}" ]; then
744+
echo "Failed: runner not found at ${RUNNER}"
745+
exit 1
746+
fi
747+
echo "::endgroup::"
748+
749+
echo "::group::Install LLM requirements"
750+
${CONDA_RUN} pip install -U "huggingface_hub[cli]<1.0"
751+
${CONDA_RUN} huggingface-cli login --token $SECRET_EXECUTORCH_HF_TOKEN
752+
OPTIMUM_ET_VERSION=$(cat .ci/docker/ci_commit_pins/optimum-executorch.txt)
753+
${CONDA_RUN} pip install transformers "optimum-executorch @ git+https://github.com/huggingface/optimum-executorch.git@${OPTIMUM_ET_VERSION}"
754+
if [ "${MODEL_ID}" = "google/gemma-4-E2B-it" ]; then
755+
# Gemma 4 needs a newer Transformers than the CI-wide pin. Keep this
756+
# on the same commit test-mlx-llm validated against.
757+
GEMMA4_TRANSFORMERS_COMMIT=61461a7bcb458db7cf6eeea49678b9ab776a7821
758+
${CONDA_RUN} pip install -U "transformers @ git+https://github.com/huggingface/transformers.git@${GEMMA4_TRANSFORMERS_COMMIT}"
759+
fi
760+
echo "::endgroup::"
761+
762+
echo "::group::Export ${MODEL_NAME} off-graph"
763+
${CONDA_RUN} python -m executorch.backends.mlx.examples.llm.export_llm_hf \
764+
--model-id "${MODEL_ID}" \
765+
--output /tmp/${MODEL_NAME}_offgraph.pte \
766+
--use-offgraph-cache \
767+
--max-seq-len 1024 \
768+
--dtype bf16 \
769+
--qlinear 4w
770+
echo "::endgroup::"
771+
772+
echo "::group::Run ${MODEL_NAME} off-graph inference"
773+
# The cache geometry comes from the .pte; only capacity is given here.
774+
TOKENIZER=$(${CONDA_RUN} python -c \
775+
"from huggingface_hub import hf_hub_download; print(hf_hub_download('${MODEL_ID}', 'tokenizer.json'))")
776+
OUTPUT=$(${RUNNER} \
777+
--pte /tmp/${MODEL_NAME}_offgraph.pte \
778+
--tokenizer "${TOKENIZER}" \
779+
--chat "${CHAT}" \
780+
--kv-max-capacity 1024 \
781+
--prompt "What is the capital of France?" \
782+
--max-new-tokens 50 2>&1)
783+
echo "$OUTPUT"
784+
if echo "$OUTPUT" | grep -iq "Paris"; then
785+
echo "Success: 'Paris' found in output"
786+
else
787+
echo "Failed: Expected 'Paris' not found in output"
788+
exit 1
789+
fi
790+
echo "::endgroup::"
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
# C++ runner for off-graph-cache HF models (export_llm_hf --use-offgraph-cache).
8+
# Unlike run_llm_hf.py (pybindings), this binds the cache via cache_key, so it
9+
# is the run path for off-graph .pte files.
10+
11+
cmake_minimum_required(VERSION 3.24)
12+
project(mlx_run_llm_hf)
13+
14+
set(CMAKE_CXX_STANDARD 17)
15+
set(CMAKE_CXX_STANDARD_REQUIRED ON)
16+
17+
set(EXECUTORCH_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/../../../..)
18+
include(${EXECUTORCH_ROOT}/tools/cmake/Utils.cmake)
19+
20+
set(_common_include_directories ${EXECUTORCH_ROOT}/..)
21+
set(_json_include
22+
${EXECUTORCH_ROOT}/extension/llm/tokenizers/third-party/json/single_include
23+
)
24+
# MLXExecutor.h reaches schema_generated.h, which needs flatbuffers. The MLX
25+
# backend exposes that include under BUILD_INTERFACE, which does not reach a
26+
# separate find_package() project like this one.
27+
set(_flatbuffers_include ${EXECUTORCH_ROOT}/third-party/flatbuffers/include)
28+
29+
list(APPEND CMAKE_FIND_ROOT_PATH ${CMAKE_CURRENT_BINARY_DIR}/../../../..)
30+
find_package(executorch CONFIG REQUIRED FIND_ROOT_PATH_BOTH)
31+
executorch_target_link_options_shared_lib(executorch)
32+
33+
# As in examples/models/llama: point gflags_DIR at the copy the executorch build
34+
# configured, since a separate find_package() project cannot see it.
35+
set(gflags_DIR ${CMAKE_CURRENT_BINARY_DIR}/../../../../third-party/gflags)
36+
find_package(gflags REQUIRED)
37+
38+
set(link_libraries
39+
executorch
40+
extension_module
41+
extension_tensor
42+
extension_llm_cache
43+
extension_llm_runner
44+
extension_llm_sampler
45+
gflags
46+
)
47+
48+
if(NOT TARGET mlxdelegate)
49+
message(FATAL_ERROR "mlx_run_llm_hf requires the MLX backend (mlxdelegate)")
50+
endif()
51+
list(APPEND link_libraries mlxdelegate mlx)
52+
executorch_target_link_options_shared_lib(mlxdelegate)
53+
54+
# CPU kernels for the ops that stay outside the delegate (e.g. the cache
55+
# bookkeeping copy_ the HF export wrappers emit).
56+
if(TARGET optimized_native_cpu_ops_lib)
57+
list(APPEND link_libraries optimized_native_cpu_ops_lib)
58+
executorch_target_link_options_shared_lib(optimized_native_cpu_ops_lib)
59+
endif()
60+
61+
list(APPEND link_libraries tokenizers::tokenizers)
62+
63+
add_executable(mlx_run_llm_hf run_llm_hf.cpp)
64+
target_include_directories(
65+
mlx_run_llm_hf PUBLIC ${_common_include_directories} ${_json_include}
66+
${_flatbuffers_include}
67+
)
68+
target_link_libraries(mlx_run_llm_hf PUBLIC ${link_libraries})
69+
70+
# The copy helper is gated on EXECUTORCH_BUILD_MLX, which the installed config
71+
# does not set; reaching here means mlxdelegate exists.
72+
set(EXECUTORCH_BUILD_MLX ON)
73+
executorch_target_copy_mlx_metallib(mlx_run_llm_hf)

‎backends/mlx/examples/llm/README.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ This example demonstrates how to export and run LLMs using the MLX delegate for
77
- **Export**: Convert HuggingFace LLMs to ExecuTorch format with MLX delegate
88
- **Quantization**: Optional INT4/INT8 weight quantization via TorchAO
99
- **KV Cache**: Efficient KV cache implementation for autoregressive generation
10+
- **Off-graph KV Cache**: Optional runtime-owned cache, sized and configured per run instead of at export
1011
- **Custom Ops**: Uses `mlx::custom_sdpa` and `mlx::kv_cache_update` for optimal execution on MLX
1112
- **Pybindings**: Run inference using ExecuTorch Python bindings
1213
- **Gemma 4**: Text-only export and run flow supports processor-backed checkpoints such as `google/gemma-4-E2B-it`
@@ -23,6 +24,7 @@ pip install transformers optimum-executorch
2324
|--------|-------------|
2425
| `export_llm_hf` | Export LLMs using optimum-executorch pipeline, with optional custom MLX SDPA/KV cache |
2526
| `run_llm_hf` | Run exported models with token-by-token generation |
27+
| `run_llm_hf.cpp` | C++ runner; the run path for off-graph-cache exports, and runs in-graph ones too |
2628

2729
For exporting via the ExecuTorch LLM pipeline (e.g. `examples/models/llama`), use `--mlx` to enable the MLX delegate.
2830

@@ -54,6 +56,13 @@ python -m executorch.backends.mlx.examples.llm.export_llm_hf \
5456
--qlinear 4w \
5557
--qembedding 4w
5658

59+
# Off-graph KV cache: the cache is owned by the runtime, not baked into the .pte
60+
python -m executorch.backends.mlx.examples.llm.export_llm_hf \
61+
--model-id "unsloth/gemma-3-1b-it" \
62+
--output gemma3_offgraph.pte \
63+
--use-offgraph-cache \
64+
--max-seq-len 1024
65+
5766
# Gemma 4 text-only export
5867
python -m executorch.backends.mlx.examples.llm.export_llm_hf \
5968
--model-id "google/gemma-4-E2B-it" \
@@ -86,6 +95,12 @@ pip install -U "transformers @ git+https://github.com/huggingface/transformers.g
8695
| `--no-tie-word-embeddings` | `False` | Disable re-tying lm_head to embedding after quantization |
8796
| `--use-custom-sdpa` | `False` | Use MLX custom SDPA (`mlx::custom_sdpa`) |
8897
| `--use-custom-kv-cache` | `False` | Use MLX custom KV cache (`mlx::kv_cache_update`) |
98+
| `--use-offgraph-cache` | `False` | Use the off-graph KV cache (`kvcache::update_and_attend`); replaces the two flags above |
99+
| `--prefill-chunk-size` | `512` | Off-graph: tokens per prefill step, published as `get_prefill_chunk_size` for the runner. It is the largest single write, so a ring layer is sized `window + chunk - 1`; it may not exceed the sliding window or the exported sequence length |
100+
101+
Off-graph exports keep no cache in the `.pte`, so the pybindings `run_llm_hf`
102+
cannot run them — use [`mlx_run_llm_hf`](#mlx_run_llm_hf-c) below, which builds
103+
the cache and binds it at load time.
89104

90105
---
91106

@@ -123,6 +138,107 @@ python -m executorch.backends.mlx.examples.llm.run_llm_hf \
123138

124139
---
125140

141+
## `mlx_run_llm_hf` (C++)
142+
143+
Native runner for models exported with `--use-offgraph-cache`. That cache lives
144+
outside the graph and is handed to the backend by key at load time, which the
145+
pybindings `run_llm_hf` above cannot do. It also runs in-graph `.pte` files
146+
unchanged — omit `--kv-max-capacity` — so the same binary compares both cache
147+
paths. Greedy decode.
148+
149+
### Build
150+
151+
A standalone `find_package(executorch)` project, so ExecuTorch must be installed
152+
first:
153+
154+
```bash
155+
cmake --preset mlx-release
156+
cmake --build cmake-out --target install -j$(( $(sysctl -n hw.ncpu) - 1 ))
157+
158+
cmake -S backends/mlx/examples/llm -B cmake-out/backends/mlx/examples/llm \
159+
-DCMAKE_BUILD_TYPE=Release
160+
cmake --build cmake-out/backends/mlx/examples/llm -j$(( $(sysctl -n hw.ncpu) - 1 ))
161+
```
162+
163+
The binary lands at `cmake-out/backends/mlx/examples/llm/mlx_run_llm_hf`.
164+
165+
### Run
166+
167+
```bash
168+
python -m executorch.backends.mlx.examples.llm.export_llm_hf \
169+
--model-id unsloth/gemma-3-1b-it \
170+
--output gemma3_offgraph.pte \
171+
--use-offgraph-cache \
172+
--max-seq-len 1024
173+
174+
cmake-out/backends/mlx/examples/llm/mlx_run_llm_hf \
175+
--pte gemma3_offgraph.pte \
176+
--tokenizer ~/.cache/huggingface/hub/models--unsloth--gemma-3-1b-it/snapshots/*/tokenizer.json \
177+
--chat gemma \
178+
--kv-max-capacity 1024 \
179+
--prompt "What is the capital of France?" \
180+
--max-new-tokens 50
181+
```
182+
183+
`--chat` selects the instruct template — `llama3`, `gemma`, `gemma4`, or `0` for
184+
raw text. It matters: raw text confuses an instruct model into emitting turn
185+
markers, and using the wrong template invalidates a comparison between two
186+
`.pte` files.
187+
188+
### Configuring the cache at run time
189+
190+
Only the cache *geometry* is fixed at export — how many caches, their KV heads,
191+
head dims and windows, which the export publishes as constant methods the runner
192+
reads before building the cache. Everything else is chosen per run, with no
193+
re-export. The runner reports what it built:
194+
195+
```
196+
[cache] off-graph seq | capacity=1024 initial=512 kv_dtype=BFloat16
197+
26 layers: 4 flat + 22 ring(window 512)
198+
```
199+
200+
`--kv-windows` overrides the attention pattern, repeating a comma-separated list
201+
over the caches (`0` = flat). The geometry each cache declared is untouched, so
202+
this cannot desync from the graph. For gemma-3-1b, whose 26 layers are 22
203+
sliding at 512 and 4 full:
204+
205+
```bash
206+
# (omitted) 26 layers: 4 flat + 22 ring(window 512)
207+
--kv-windows 512,512,512,512,512,0 # the same, spelling out gemma-3's 5:1 period
208+
--kv-windows 0 # 26 layers: 26 flat
209+
--kv-windows 512 # 26 layers: 0 flat + 26 ring(window 512)
210+
--kv-windows 512,256 # 26 layers: 0 flat + 13 ring(256) + 13 ring(512)
211+
```
212+
213+
A ring layer allocates its whole `window + chunk - 1` slots up front, while a
214+
flat layer starts at `--kv-initial-capacity` and doubles as the sequence grows,
215+
so an all-flat cache can look smaller than a sliding one early in a run and
216+
larger later. Prefill runs in steps of the chunk size the export published, so
217+
the ring is bounded by the chunk rather than by the prompt: prefilling 24k
218+
tokens in one step would need 541 MiB of ring here, against 24 MiB at the 512
219+
default.
220+
221+
### Options
222+
223+
`--help` lists every flag with its default.
224+
225+
| Option | Default | Description |
226+
|--------|---------|-------------|
227+
| `--pte` | *(required)* | Path to .pte file |
228+
| `--tokenizer` | *(required)* | Path to `tokenizer.json` |
229+
| `--prompt` | `The quick brown fox` | Input prompt |
230+
| `--max-new-tokens` | `50` | Tokens to generate, excluding the prompt |
231+
| `--temperature` | `0` | Sampling temperature; 0 is greedy argmax, which is what makes two `.pte` files comparable |
232+
| `--chat` | `llama3` | Chat template: `llama3`, `gemma`, `gemma4`, or `0` to disable |
233+
| `--kv-max-capacity` | `0` | Off-graph: history the cache may hold. Setting it selects the off-graph path |
234+
| `--kv-storage-dtype` | `bf16` | Off-graph: KV storage dtype (`bf16`, `fp16`, `fp32`) |
235+
| `--kv-initial-capacity` | `-1` | Off-graph: starting pool size; grows by doubling up to capacity |
236+
| `--kv-windows` | *(model's own)* | Off-graph: attention pattern override, e.g. `512` |
237+
| `--interactive` | `false` | Multi-turn chat on stdin; off-graph only |
238+
| `--warmup` | `false` | Run once before measuring, to absorb JIT and pool growth |
239+
240+
---
241+
126242
## Architecture
127243

128244
The `export_llm_hf` script uses optimum-executorch's `CausalLMExportableModule` by default. When custom flags are enabled, it uses `TorchExportableModuleWithStaticCache` from HuggingFace transformers, with optional MLX-specific replacements:

0 commit comments

Comments
 (0)