Skip to content

Commit c89046c

Browse files
authored
Expose committed session cloning and build prefix reuse above it (pytorch#22812)
Expose `Session::clone_async(upto)` for independently writable committed prefixes. A caller-owned `PrefixCache` handles token matching and LRU eviction; `capture_prompt(...).wrap(...)` requests the prompt clone before the first user callback, and caller-thread `collect()` inserts it after generation. Capture binds to session identity and survives moves without retaining source ownership. Lookup tries each snapshot only at its longest matching boundary. The MLX example uses this helper and forwards the final prompt token for fresh logits. The cache factory API stays unchanged; session limits are checked through the constructed cache's `BatchControl::max_seqs()`. Review the Session/Runner lifecycle first, then ModuleExecutor/backend clone contracts, prefix policy, and MLX integration. **214 shared native tests pass on Apple M1 Pro**, including 14 capture tests for timing, reuse, failures, handle lifetime, and moved/reassigned/destroyed sources. All five native targets and the standalone MLX batched runner build; lint passes. GPU execution was not repeated for this lifetime change; the following GPU/model results are from `ddf92b40d1`. All 49 MLX tests pass. Gemma 3 1B validation covers BF16/FP16, full/selected logits, both layouts, physical ring wrap, branches, closure, and concurrency: 16 smoke groups, 88 cache/work checks, and 44 cold export comparisons pass. Warm repeats forward 1 of 23/1280 prompt tokens. **Normal cold/warm initial logits exceed tolerance in 72/88 cases.** The public header recommends opt-in greedy experiments until non-greedy sampling is validated; even greedy token parity is not guaranteed. Clone controls with matching schedule/geometry are byte-exact. A 23-token control also gives byte-exact cold/split/replay/clone logits in both dtypes with one-token prefill. Isolated probes identify shape-dependent MLX projection and attention arithmetic; precision promotions did not resolve the normal cold/warm gap. The four existing exports were reused and hash-verified. Authored with OpenAI Codex.
1 parent c3bce5d commit c89046c

19 files changed

Lines changed: 3092 additions & 78 deletions

‎.github/workflows/mlx.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ on:
1111
paths:
1212
- .github/workflows/mlx.yml
1313
- backends/mlx/**
14+
- extension/llm/batching/**
15+
- extension/llm/cache/**
1416
- extension/llm/export/**
1517
- extension/audio/**
1618
- examples/models/gemma4_31b/**
@@ -68,7 +70,7 @@ jobs:
6870
echo "::endgroup::"
6971
7072
echo "::group::Build test runners"
71-
${CONDA_RUN} cmake --build cmake-out --target op_test_runner multi_thread_test_runner mlx_metallib_path_test mlx_mutable_state_test mlx_sequence_cache_test mlx_cell_cache_test -j$(( $(sysctl -n hw.ncpu) - 1 ))
73+
${CONDA_RUN} cmake --build cmake-out --target op_test_runner multi_thread_test_runner mlx_metallib_path_test mlx_mutable_state_test mlx_sequence_cache_test mlx_batched_sequence_cache_test mlx_cell_cache_test -j$(( $(sysctl -n hw.ncpu) - 1 ))
7274
echo "::endgroup::"
7375
7476
echo "::group::Check MLX artifact sizes"
@@ -112,6 +114,7 @@ jobs:
112114
113115
echo "::group::Run off-graph KV-cache op test"
114116
./cmake-out/backends/mlx/test/mlx_sequence_cache_test
117+
./cmake-out/backends/mlx/test/mlx_batched_sequence_cache_test
115118
./cmake-out/backends/mlx/test/mlx_cell_cache_test
116119
echo "::endgroup::"
117120

‎backends/mlx/examples/llm/README.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ pip install transformers optimum-executorch
2525
| `export_llm_hf` | Export LLMs using optimum-executorch pipeline, with optional custom MLX SDPA/KV cache |
2626
| `run_llm_hf` | Run exported models with token-by-token generation |
2727
| `run_llm_hf.cpp` | C++ runner; the run path for off-graph-cache exports, and runs in-graph ones too |
28+
| `run_llm_batched.cpp` | C++ runner for independent off-graph sessions, with optional shared prefix caching |
2829

2930
For exporting via the ExecuTorch LLM pipeline (e.g. `examples/models/llama`), use `--mlx` to enable the MLX delegate.
3031

@@ -161,6 +162,62 @@ cmake --build cmake-out/backends/mlx/examples/llm -j$(( $(sysctl -n hw.ncpu) - 1
161162
```
162163

163164
The binary lands at `cmake-out/backends/mlx/examples/llm/mlx_run_llm_hf`.
165+
The same build also produces `mlx_run_llm_batched` in that directory.
166+
167+
### Prefix-cache smoke test
168+
169+
`mlx_run_llm_batched` uses the backend-independent prefix cache in
170+
[`extension/llm`](../../../../extension/llm/batching/README.md).
171+
Enable it with `--prefix_cache_entries`; zero, the default, disables it.
172+
**Cold/warm logit and token parity are not guaranteed on MLX.** Prefix reuse
173+
changes prefill shapes and can change numerical results. Keep reuse opt-in and
174+
restrict experiments to greedy generation until non-greedy sampling is
175+
validated; greedy sampling itself does not guarantee matching tokens.
176+
177+
The application owns the shared `batching::PrefixCache` and captures snapshots
178+
with `capture_prompt(session, full_prompt_tokens)`. Each capture wraps one
179+
generation callback and is collected on the caller thread after
180+
`generation.wait()`. Capture binds to the original session identity, so the
181+
source Session and capture handle can move. Keep the source's current owner
182+
alive until generation completes; capture does not delay source closure.
183+
Capture is best effort and requests its clone before delivering the
184+
first tokens; user callback exceptions retain the Runner's normal failure
185+
behavior. ModuleExecutor has no prefix-policy hooks.
186+
`--prompt_rounds 2` repeats the prompt on a fresh session while retaining the
187+
cache. Use a model exported with `--use-offgraph-cache`, full or selected
188+
logits, and a matching tokenizer and chat template.
189+
190+
After the Gemma 3 off-graph export below, run from the ExecuTorch root, replacing
191+
the tokenizer path with the tokenizer used for that export:
192+
193+
```bash
194+
ET_PREFIX_TOKENIZER=/path/to/gemma3/tokenizer.json
195+
ET_PREFIX_RESULTS=$(mktemp -d)
196+
for entries in 0 2; do
197+
cmake-out/backends/mlx/examples/llm/mlx_run_llm_batched \
198+
--pte gemma3_offgraph.pte --tokenizer "$ET_PREFIX_TOKENIZER" \
199+
--chat gemma --cache_kind batched-sequence \
200+
--max_session_tokens 1024 --max_new_tokens 64 \
201+
--max_decode_sequences 1 --temperature 0 --seed 42 \
202+
--prefix_cache_entries "$entries" --prompt_rounds 2 \
203+
--out_prefix "$ET_PREFIX_RESULTS/cache-$entries" \
204+
"What is the capital of France?" \
205+
> "$ET_PREFIX_RESULTS/cache-$entries.log" 2>&1 || exit 1
206+
done
207+
cmp "$ET_PREFIX_RESULTS/cache-0_0.txt" "$ET_PREFIX_RESULTS/cache-0_1.txt"
208+
cmp "$ET_PREFIX_RESULTS/cache-0_0.txt" "$ET_PREFIX_RESULTS/cache-2_0.txt"
209+
cmp "$ET_PREFIX_RESULTS/cache-0_0.txt" "$ET_PREFIX_RESULTS/cache-2_1.txt"
210+
cat "$ET_PREFIX_RESULTS/cache-2.log"
211+
```
212+
213+
The generated text should agree across all four runs. In the cached run's
214+
request report, generation 0 should have zero cached prompt tokens and generation 1
215+
should reuse the prompt except for its final token, which is replayed for
216+
fresh logits. Repeat with `--cache_kind batched-cell` to check the alternative
217+
storage implementation. These short prompts check integration; the cache
218+
tests additionally cover sliding-window wrap, divergent continuations, and
219+
unavailable history. Model validation should also include prompts longer than
220+
the model's sliding window.
164221

165222
### Run
166223

‎backends/mlx/examples/llm/run_llm_batched.cpp‎

Lines changed: 115 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@
1313
// is a prompt, and generated text is streamed to
1414
// <out_prefix>_<prompt-index>.txt.
1515

16+
#include <algorithm>
1617
#include <cstdint>
1718
#include <fstream>
1819
#include <functional>
19-
#include <future>
2020
#include <iostream>
2121
#include <limits>
2222
#include <memory>
@@ -31,6 +31,7 @@
3131
#include <executorch/backends/mlx/examples/llm/runner_utils.h>
3232
#include <executorch/extension/llm/batching/decode_first_scheduler.h>
3333
#include <executorch/extension/llm/batching/module_executor.h>
34+
#include <executorch/extension/llm/batching/prefix_cache.h>
3435
#include <executorch/extension/llm/batching/runner.h>
3536
#include <executorch/extension/llm/cache/cache_registry.h>
3637
#include <executorch/extension/llm/runner/llm_runner_helper.h>
@@ -42,6 +43,16 @@ DEFINE_string(pte, "", "Path to the .pte exported with --use-offgraph-cache");
4243
DEFINE_string(tokenizer, "", "Path to a supported tokenizer file");
4344
DEFINE_string(out_prefix, "gen", "Output files are <prefix>_<n>.txt");
4445
DEFINE_int32(max_session_tokens, 2048, "Maximum tokens retained per session");
46+
DEFINE_int32(
47+
prefix_cache_entries,
48+
0,
49+
"Maximum immutable prompt snapshots; 0 disables prefix caching. "
50+
"Reserves additional logical cache capacity per entry.");
51+
DEFINE_int32(
52+
prompt_rounds,
53+
1,
54+
"Repeat the prompt batch on fresh sessions, retaining the prefix cache "
55+
"between rounds");
4556
DEFINE_string(
4657
kv_storage_dtype,
4758
"",
@@ -129,6 +140,9 @@ struct Emitter {
129140

130141
struct JobResult {
131142
std::optional<batching::Session> session;
143+
batching::PrefixCache::PromptCapture capture;
144+
std::vector<batching::Token> prompt_tokens;
145+
std::size_t cached_prompt_tokens = 0;
132146
batching::GenerationHandle handle;
133147
std::optional<batching::FinishReason> reason;
134148
std::optional<batching::GenerationMetrics> metrics;
@@ -158,11 +172,11 @@ const char* reason_name(const std::optional<batching::FinishReason>& reason) {
158172
return "unknown";
159173
}
160174

161-
void submit_prompt(
162-
batching::Session& session,
175+
void prepare_prompt(
176+
batching::Runner& runner,
177+
batching::PrefixCache& prefixes,
163178
const tokenizers::Tokenizer& tokenizer,
164179
const std::string& prompt,
165-
const std::vector<batching::Token>& stop_tokens,
166180
JobResult& result) {
167181
try {
168182
std::string wrapped;
@@ -184,9 +198,32 @@ void submit_prompt(
184198
return;
185199
}
186200

201+
result.prompt_tokens = std::move(*encoded);
202+
auto match = prefixes.lookup(result.prompt_tokens);
203+
if (match) {
204+
result.cached_prompt_tokens = match->matched_tokens;
205+
result.session = std::move(match->session);
206+
} else {
207+
result.session = runner.open_session_async().get();
208+
}
209+
if (!result.session) {
210+
result.message = "could not open session";
211+
}
212+
} catch (const std::exception& error) {
213+
result.message = error.what();
214+
}
215+
}
216+
217+
void submit_prompt(
218+
batching::PrefixCache& prefixes,
219+
const tokenizers::Tokenizer& tokenizer,
220+
const std::vector<batching::Token>& stop_tokens,
221+
bool retain_prefix,
222+
JobResult& result) {
223+
try {
187224
auto emitter = std::make_shared<Emitter>(
188225
tokenizer,
189-
encoded->back(),
226+
result.prompt_tokens.back(),
190227
result.output_path,
191228
static_cast<std::size_t>(FLAGS_flush_every));
192229
if (!emitter->file) {
@@ -202,22 +239,30 @@ void submit_prompt(
202239
config.stop_tokens = stop_tokens;
203240
config.seed = FLAGS_seed;
204241

205-
result.handle = session.generate_async(
206-
std::move(*encoded),
242+
std::vector<batching::Token> suffix(
243+
result.prompt_tokens.begin() + result.cached_prompt_tokens,
244+
result.prompt_tokens.end());
245+
if (retain_prefix) {
246+
result.capture =
247+
prefixes.capture_prompt(*result.session, result.prompt_tokens);
248+
}
249+
result.handle = result.session->generate_async(
250+
std::move(suffix),
207251
std::move(config),
208-
[emitter](const batching::GenerationUpdate& update) {
209-
std::size_t count = update.tokens.size();
210-
if (update.finish_reason == batching::FinishReason::StopToken &&
211-
count > 0) {
212-
--count;
213-
}
214-
for (std::size_t i = 0; i < count; ++i) {
215-
emitter->append(update.tokens[i]);
216-
}
217-
if (update.finish_reason) {
218-
emitter->finish();
219-
}
220-
});
252+
result.capture.wrap(
253+
[emitter](const batching::GenerationUpdate& update) {
254+
std::size_t count = update.tokens.size();
255+
if (update.finish_reason == batching::FinishReason::StopToken &&
256+
count > 0) {
257+
--count;
258+
}
259+
for (std::size_t i = 0; i < count; ++i) {
260+
emitter->append(update.tokens[i]);
261+
}
262+
if (update.finish_reason) {
263+
emitter->finish();
264+
}
265+
}));
221266
} catch (const std::exception& error) {
222267
result.message = error.what();
223268
}
@@ -236,10 +281,11 @@ int main(int argc, char** argv) {
236281
return 1;
237282
}
238283
if (FLAGS_max_session_tokens <= 0 || FLAGS_max_new_tokens <= 0 ||
239-
FLAGS_max_decode_sequences <= 0 || FLAGS_flush_every <= 0) {
240-
std::cerr
241-
<< "session, generation, decode, and flush limits must be positive"
242-
<< std::endl;
284+
FLAGS_max_decode_sequences <= 0 || FLAGS_flush_every <= 0 ||
285+
FLAGS_prompt_rounds <= 0 || FLAGS_prefix_cache_entries < 0) {
286+
std::cerr << "session, generation, decode, flush, and round limits must be "
287+
"positive; prefix cache entries must be non-negative"
288+
<< std::endl;
243289
return 1;
244290
}
245291
if (FLAGS_temperature < 0.0 || FLAGS_top_p <= 0.0 || FLAGS_top_p > 1.0 ||
@@ -257,6 +303,11 @@ int main(int argc, char** argv) {
257303
std::cerr << "too many prompts" << std::endl;
258304
return 1;
259305
}
306+
if (prompts.size() > std::numeric_limits<std::size_t>::max() /
307+
static_cast<std::size_t>(FLAGS_prompt_rounds)) {
308+
std::cerr << "too many prompt rounds" << std::endl;
309+
return 1;
310+
}
260311
if (FLAGS_max_new_tokens > FLAGS_max_session_tokens) {
261312
std::cerr << "--max_new_tokens exceeds --max_session_tokens" << std::endl;
262313
return 1;
@@ -310,9 +361,18 @@ int main(int argc, char** argv) {
310361
const std::vector<batching::Token> stop_tokens(
311362
resolved_stop_tokens.ids.begin(), resolved_stop_tokens.ids.end());
312363

364+
const auto snapshot_slots = std::min(
365+
prompts.size(), static_cast<std::size_t>(FLAGS_prefix_cache_entries));
366+
const auto resident_sessions = static_cast<std::uint64_t>(prompts.size()) +
367+
FLAGS_prefix_cache_entries + snapshot_slots;
368+
if (resident_sessions >
369+
static_cast<std::uint64_t>(std::numeric_limits<int>::max())) {
370+
std::cerr << "too many active and retained sessions" << std::endl;
371+
return 1;
372+
}
313373
auto executor = batching::ModuleExecutor::create(
314374
std::move(module),
315-
static_cast<int>(prompts.size()),
375+
static_cast<int>(resident_sessions),
316376
FLAGS_max_session_tokens,
317377
kv_dtype,
318378
FLAGS_kv_initial_capacity,
@@ -355,7 +415,8 @@ int main(int argc, char** argv) {
355415
}
356416

357417
batching::Runner runner(**executor, std::move(scheduler));
358-
std::vector<JobResult> results(prompts.size());
418+
batching::PrefixCache prefixes(FLAGS_prefix_cache_entries);
419+
std::vector<JobResult> results(prompts.size() * FLAGS_prompt_rounds);
359420
// Fail before opening sessions if any output path cannot be created.
360421
for (std::size_t i = 0; i < results.size(); ++i) {
361422
results[i].output_path =
@@ -369,40 +430,44 @@ int main(int argc, char** argv) {
369430
}
370431
}
371432

372-
std::vector<std::future<std::optional<batching::Session>>> session_futures;
373-
session_futures.reserve(prompts.size());
374-
for (std::size_t i = 0; i < prompts.size(); ++i) {
375-
session_futures.push_back(runner.open_session_async());
376-
}
377-
for (std::size_t i = 0; i < prompts.size(); ++i) {
378-
results[i].session = session_futures[i].get();
379-
}
380-
for (std::size_t i = 0; i < prompts.size(); ++i) {
381-
if (!results[i].session) {
382-
results[i].message = "could not open session";
383-
continue;
433+
for (int round = 0; round < FLAGS_prompt_rounds; ++round) {
434+
const std::size_t base = static_cast<std::size_t>(round) * prompts.size();
435+
for (std::size_t i = 0; i < prompts.size(); ++i) {
436+
prepare_prompt(
437+
runner, prefixes, *tokenizer, prompts[i], results[base + i]);
384438
}
385-
submit_prompt(
386-
*results[i].session, *tokenizer, prompts[i], stop_tokens, results[i]);
387-
}
388-
for (JobResult& result : results) {
389-
if (!result.handle.valid()) {
390-
continue;
439+
for (std::size_t i = 0; i < prompts.size(); ++i) {
440+
JobResult& result = results[base + i];
441+
if (!result.session) {
442+
continue;
443+
}
444+
submit_prompt(
445+
prefixes, *tokenizer, stop_tokens, i < snapshot_slots, result);
446+
}
447+
for (std::size_t i = 0; i < prompts.size(); ++i) {
448+
JobResult& result = results[base + i];
449+
if (result.handle.valid()) {
450+
result.handle.wait();
451+
result.metrics = result.handle.metrics();
452+
result.reason = result.handle.finish_reason();
453+
result.message = result.handle.error_message();
454+
}
455+
result.capture.collect();
456+
result.session.reset();
391457
}
392-
result.handle.wait();
393-
result.metrics = result.handle.metrics();
394-
result.reason = result.handle.finish_reason();
395-
result.message = result.handle.error_message();
396458
}
397459

460+
prefixes.clear();
398461
runner.shutdown();
399462
const batching::EngineMetrics engine = runner.metrics();
400463

401464
if (FLAGS_metrics) {
402465
std::cout << "\n";
403466
for (std::size_t i = 0; i < results.size(); ++i) {
404467
if (results[i].metrics) {
405-
std::cout << "[" << i << "] "
468+
std::cout << "[" << i << "] " << results[i].prompt_tokens.size()
469+
<< " prompt tokens (" << results[i].cached_prompt_tokens
470+
<< " cached)\n"
406471
<< batching::format_report(*results[i].metrics);
407472
}
408473
}
@@ -428,7 +493,7 @@ int main(int argc, char** argv) {
428493
}
429494
}
430495

431-
std::cout << prompts.size() - failures << "/" << prompts.size()
496+
std::cout << results.size() - failures << "/" << results.size()
432497
<< " generations completed" << std::endl;
433498
return failures == 0 ? 0 : 1;
434499
}

‎backends/mlx/test/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ et_cxx_test(
143143
mlx_batched_sequence_cache_test
144144
SOURCES
145145
${CMAKE_CURRENT_LIST_DIR}/mlx_batched_sequence_cache_test.cpp
146+
${CMAKE_CURRENT_LIST_DIR}/mlx_sequence_clone_test.cpp
146147
EXTRA_LIBS
147148
mlxdelegate
148149
mlx_schema

‎backends/mlx/test/mlx_cell_cache_test.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -273,7 +273,9 @@ TEST_F(MLXCellCacheTest, RegistryBuildsCellLayout) {
273273
kMLXBackendId, cache::kind::kBatchedCell, args.geometry, args.config);
274274
ASSERT_TRUE(built.ok());
275275
const std::shared_ptr<cache::Cache>& c = *built;
276-
EXPECT_NE(c->as<cache::BatchControl>(), nullptr);
276+
ASSERT_NE(c->as<cache::BatchControl>(), nullptr);
277+
EXPECT_EQ(
278+
c->as<cache::BatchControl>()->max_seqs(), cache::CellCache::kMaxSeqs);
277279
EXPECT_NE(c->as<MLXCache>(), nullptr) << "the backend face comes back too";
278280
// A cell layout is multi-sequence, so it offers no single-sequence face.
279281
EXPECT_EQ(c->as<cache::SequenceControl>(), nullptr);

0 commit comments

Comments
 (0)