You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e7ca2b2
Browse filesBrowse the repository at this point in the historyBrowse files
[MLX] C++ runner for HF LLMs with the off-graph KV cache (pytorch#21683)
**Summary**
This runner builds an MLXSequenceCache, installs it, and passes the
cache key, so it's the run path for .pte files exported with
--use-offgraph-cache. The cache's shape is read from the .pte metadata.
The flags left are policy the model can't imply: --kv-max-capacity,
--kv-storage-dtype, --kv-initial-capacity, --kv-max-write, --kv-windows.
Depends on pytorch#21680; the new CI job fails until that lands.
**Files**
- run_llm_hf.cpp — the runner: chat templates, greedy decode,
benchmarking, and an interactive mode with /reset and /undo [N].
- CMakeLists.txt — standalone find_package(executorch) project.
- .github/workflows/mlx.yml — test-mlx-llm-offgraph for llama-1b,
gemma3-1b and gemma4-e2b.
**Test**
CI builds the runner, then for llama-1b, gemma3-1b and gemma4-e2b
exports off-graph and asserts the same "Paris" answer test-mlx-llm
checks.
|`--no-tie-word-embeddings`|`False`| Disable re-tying lm_head to embedding after quantization |
87
96
|`--use-custom-sdpa`|`False`| Use MLX custom SDPA (`mlx::custom_sdpa`) |
88
97
|`--use-custom-kv-cache`|`False`| Use MLX custom KV cache (`mlx::kv_cache_update`) |
98
+
|`--use-offgraph-cache`|`False`| Use the off-graph KV cache (`kvcache::update_and_attend`); replaces the two flags above |
99
+
|`--prefill-chunk-size`|`512`| Off-graph: tokens per prefill step, published as `get_prefill_chunk_size` for the runner. It is the largest single write, so a ring layer is sized `window + chunk - 1`; it may not exceed the sliding window or the exported sequence length |
100
+
101
+
Off-graph exports keep no cache in the `.pte`, so the pybindings `run_llm_hf`
102
+
cannot run them — use [`mlx_run_llm_hf`](#mlx_run_llm_hf-c) below, which builds
|`--kv-initial-capacity`|`-1`| Off-graph: starting pool size; grows by doubling up to capacity |
236
+
|`--kv-windows`|*(model's own)*| Off-graph: attention pattern override, e.g. `512`|
237
+
|`--interactive`|`false`| Multi-turn chat on stdin; off-graph only |
238
+
|`--warmup`|`false`| Run once before measuring, to absorb JIT and pool growth |
239
+
240
+
---
241
+
126
242
## Architecture
127
243
128
244
The `export_llm_hf` script uses optimum-executorch's `CausalLMExportableModule` by default. When custom flags are enabled, it uses `TorchExportableModuleWithStaticCache` from HuggingFace transformers, with optional MLX-specific replacements:
0 commit comments