From ee2db54fd67033e2f67d0a687bb100d4b2d49428 Mon Sep 17 00:00:00 2001 From: Scott Roy Date: Thu, 15 Oct 2026 17:06:08 -0700 Subject: [PATCH 1/4] up --- examples/models/muse-glimmer/README.md | 110 +++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/examples/models/muse-glimmer/README.md b/examples/models/muse-glimmer/README.md index 3c9beac41d6..912445ac859 100644 --- a/examples/models/muse-glimmer/README.md +++ b/examples/models/muse-glimmer/README.md @@ -161,8 +161,118 @@ The binaries are written to `cmake-out/examples/models/muse-glimmer/`: Run a binary with `--help` for its model-specific arguments. +## Off-graph KV cache and batching + +The off-graph path keeps KV cache state in the runtime rather than inside the +exported model graph. It supports packed multi-session batching, runtime-managed +cache growth, and optional prefix snapshots for reuse across requests. The +batching server uses a shared serving interface with backend-specific cache and +model execution implementations, with optional DFlash speculative decoding. +It uses separate batching runners and requires an off-graph export; legacy +artifacts and workers are not interchangeable with it. + +MLX is currently supported; CUDA support is planned. The commands below show +MLX as the available backend example, not a restriction of the off-graph design. + +### Export + +Select a supported backend +example, using MLX and the checkpoint variables from [Model assets](#model-assets), +export a text-only target: + +```bash +python -m executorch.examples.models.muse_glimmer.export.export_solo \ + --backend mlx \ + --gguf "$TARGET" \ + --use-offgraph-kv-cache \ + --activation-dtype float16 \ + --max-prefill-chunk 512 \ + --output-dir /path/to/exports/mg-offgraph +``` + +Or export the target and drafter for DFlash: + +```bash +python -m executorch.examples.models.muse_glimmer.export.export_dflash \ + --backend mlx \ + --target-gguf "$TARGET" \ + --draft-gguf "$DRAFT" \ + --use-offgraph-kv-cache \ + --activation-dtype float16 \ + --output-dir /path/to/exports/mg-offgraph-dflash +``` + +The server's `--max-context` sets the per-session runtime limit and must not +exceed the exported limit. Runtime cache allocation is separate from that limit. +For MLX, off-graph exports use the checkpoint's native context limit rather than +`--max-seq-len`, and KV storage grows as needed instead of allocating the full +context up front. + +### Set up the batching runtime + +After the general [environment setup](#supported-backends), use the Make target +for your backend to build and install ExecuTorch and build the batching runner +and worker. The current MLX setup command is: + +```bash +cd /path/to/executorch +make muse-glimmer-batching-mlx +pip install -r /path/to/executorch/examples/llm_server/python/requirements.txt +``` + +The Make target builds the native dependencies and both batching executables; +it does not export a model or start the server. CUDA will require its matching +batching build target when support is available; the legacy CUDA runner target +is not a substitute. + +### Start the batching server + +Replace the absolute example paths below with your checkout, export, and +tokenizer locations. Select the backend matching the export and worker build. +This MLX example uses four concurrent sessions, a 128K (131072-token) context +limit, and up to 32 prefix snapshots: + +```bash +python -m executorch.examples.models.muse_glimmer.serving.serve_batching \ + --worker-bin /path/to/executorch/cmake-out/examples/models/muse-glimmer-batching/muse_glimmer_batching_worker \ + --model-path /path/to/exports/mg-offgraph/model.pte \ + --tokenizer-path /path/to/assets/hf/tokenizer.json \ + --hf-tokenizer /path/to/assets/hf \ + --backend mlx \ + --model-id muse-glimmer-30B \ + --max-context 131072 \ + --max-sessions 4 \ + --max-decode-sequences 4 \ + --max-inflight-requests 4 \ + --prefix-cache-entries 32 \ + --tool-parser atem \ + --host 127.0.0.1 \ + --port 8000 +``` + +For DFlash, use `/path/to/exports/mg-offgraph-dflash/model.pte` and add +`--dflash --dflash-n-draft 3` to the launch command. The artifact must support +at least three draft tokens (a block size of four). Omitting `--dflash-n-draft`, +or setting it to zero, uses the exported maximum minus one. Draft proposals are +always argmax; request sampling parameters apply to the target distribution. + +Choose concurrency and prefix-cache limits to fit available memory. Prefix +snapshots are disabled by default; omit `--prefix-cache-entries` to disable them. +If the export stores weights separately, also pass `--data-path` with its data +file. For vision, export with `--mmproj "$MMPROJ"` and pass +`--pos-embed-path /path/to/exports/mg-offgraph/pos_embed.bin`, using the matching +export directory. + +The endpoint is `http://127.0.0.1:8000/v1/chat/completions`; the smoke request and +pi configuration below work with this server too. See the +[batching serving guide](BATCHING_SERVING.md) for image support, session behavior, +and detailed limits. + ## Serve +The following instructions use the legacy worker. For off-graph exports, use +[the batching server](#start-the-batching-server) above instead. + Install the shared server dependencies: ```bash From eefa878f549c55dd5d17509943c8c74f82583fff Mon Sep 17 00:00:00 2001 From: Scott Roy Date: Thu, 15 Oct 2026 18:49:27 -0700 Subject: [PATCH 2/4] up --- examples/models/muse-glimmer/README.md | 144 ++++++++++++++----------- 1 file changed, 79 insertions(+), 65 deletions(-) diff --git a/examples/models/muse-glimmer/README.md b/examples/models/muse-glimmer/README.md index 912445ac859..6a0b5bd9093 100644 --- a/examples/models/muse-glimmer/README.md +++ b/examples/models/muse-glimmer/README.md @@ -161,24 +161,67 @@ The binaries are written to `cmake-out/examples/models/muse-glimmer/`: Run a binary with `--help` for its model-specific arguments. +## Serve + +The following instructions use the regular worker built above. For off-graph +exports, use [the batching server](#start-the-batching-server) below instead. + +Install the shared server dependencies: + +```bash +pip install -r /path/to/executorch/examples/llm_server/python/requirements.txt +``` + +Serve a CUDA target export, replacing the example paths with your local paths: + +```bash +python -m executorch.examples.models.muse_glimmer.serving.serve \ + --model-path /path/to/exports/solo/model.pte \ + --data-path /path/to/exports/solo/aoti_cuda_blob.ptd \ + --tokenizer-path /path/to/assets/hf/tokenizer.json \ + --hf-tokenizer /path/to/assets/hf \ + --worker-bin /path/to/executorch/cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \ + --model-id muse-glimmer-30B \ + --tool-parser atem \ + --host 127.0.0.1 \ + --port 8000 +``` + +For MLX, use the MLX export and omit `--data-path`. For DFlash, use the matching +DFlash export paths; the server detects the exported method contract. For +vision, also pass `--pos-embed-path /path/to/exports/solo/pos_embed.bin`, using +the matching export directory. + +Smoke test: + +```bash +curl http://127.0.0.1:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"muse-glimmer-30B","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":32,"temperature":0}' +``` + +Tool calling is supported with `--tool-parser atem`. The canonical Hugging Face +template renders tool definitions, and the server converts Muse Glimmer 30B ATEM +output into OpenAI-compatible `tool_calls`. + ## Off-graph KV cache and batching The off-graph path keeps KV cache state in the runtime rather than inside the exported model graph. It supports packed multi-session batching, runtime-managed -cache growth, and optional prefix snapshots for reuse across requests. The -batching server uses a shared serving interface with backend-specific cache and -model execution implementations, with optional DFlash speculative decoding. -It uses separate batching runners and requires an off-graph export; legacy +cache growth, and optional prefix snapshots for reuse across requests. +Batching supports both text-only and vision-enabled exports, with or without +DFlash. + +It uses separate batching runners artifacts and workers are not interchangeable with it. -MLX is currently supported; CUDA support is planned. The commands below show -MLX as the available backend example, not a restriction of the off-graph design. +The commands below use MLX as an example. Select the backend matching your +hardware, export, and worker build. ### Export -Select a supported backend -example, using MLX and the checkpoint variables from [Model assets](#model-assets), -export a text-only target: +Export with `--use-offgraph-kv-cache`. For example, using MLX and the checkpoint +variables from [Model assets](#model-assets), export a text-only target: ```bash python -m executorch.examples.models.muse_glimmer.export.export_solo \ @@ -202,6 +245,11 @@ python -m executorch.examples.models.muse_glimmer.export.export_dflash \ --output-dir /path/to/exports/mg-offgraph-dflash ``` +To enable image input, add `--mmproj "$MMPROJ"` to either export command above. +This includes the vision encoder and writes its position-embedding data alongside +the model. The resulting export can serve both text-only and image requests; +a separate text-only export is not required. + The server's `--max-context` sets the per-session runtime limit and must not exceed the exported limit. Runtime cache allocation is separate from that limit. For MLX, off-graph exports use the checkpoint's native context limit rather than @@ -212,7 +260,7 @@ context up front. After the general [environment setup](#supported-backends), use the Make target for your backend to build and install ExecuTorch and build the batching runner -and worker. The current MLX setup command is: +and worker. For MLX: ```bash cd /path/to/executorch @@ -221,9 +269,8 @@ pip install -r /path/to/executorch/examples/llm_server/python/requirements.txt ``` The Make target builds the native dependencies and both batching executables; -it does not export a model or start the server. CUDA will require its matching -batching build target when support is available; the legacy CUDA runner target -is not a substitute. +it does not export a model or start the server. Use a batching build target, +not a regular runner target. ### Start the batching server @@ -259,63 +306,30 @@ always argmax; request sampling parameters apply to the target distribution. Choose concurrency and prefix-cache limits to fit available memory. Prefix snapshots are disabled by default; omit `--prefix-cache-entries` to disable them. If the export stores weights separately, also pass `--data-path` with its data -file. For vision, export with `--mmproj "$MMPROJ"` and pass -`--pos-embed-path /path/to/exports/mg-offgraph/pos_embed.bin`, using the matching -export directory. - -The endpoint is `http://127.0.0.1:8000/v1/chat/completions`; the smoke request and -pi configuration below work with this server too. See the +file. + +For a vision-enabled export, add +`--pos-embed-path /path/to/exports/mg-offgraph/pos_embed.bin` to the server command, +using the matching export directory. Send images as OpenAI `image_url` content +parts containing inline base64 JPEG or PNG data URLs. The batching server accepts +one image across each submitted conversation; remote image URLs and local image +paths are not supported. + +The endpoint is `http://127.0.0.1:8000/v1/chat/completions`; the smoke request in +[Serve](#serve) and the [Pi configuration](#use-from-pi) below work with this +server too. See the [batching serving guide](BATCHING_SERVING.md) for image support, session behavior, and detailed limits. -## Serve - -The following instructions use the legacy worker. For off-graph exports, use -[the batching server](#start-the-batching-server) above instead. - -Install the shared server dependencies: - -```bash -pip install -r examples/llm_server/python/requirements.txt -``` - -Serve a CUDA target export: - -```bash -python -m executorch.examples.models.muse_glimmer.serving.serve \ - --model-path exports/solo/model.pte \ - --data-path exports/solo/aoti_cuda_blob.ptd \ - --tokenizer-path assets/hf/tokenizer.json \ - --hf-tokenizer assets/hf \ - --worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \ - --model-id muse-glimmer-30B \ - --tool-parser atem \ - --host 127.0.0.1 \ - --port 8000 -``` - -For MLX, use the MLX `model.pte` and omit `--data-path`. For DFlash, replace -`exports/solo` with `exports/dflash` in both artifact paths; the server detects -the exported method contract. For vision, also pass -`--pos-embed-path /pos_embed.bin`. - -Smoke test: - -```bash -curl http://127.0.0.1:8000/v1/chat/completions \ - -H 'Content-Type: application/json' \ - -d '{"model":"muse-glimmer-30B","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":32,"temperature":0}' -``` - -Tool calling is supported with `--tool-parser atem`. The canonical Hugging Face -template renders tool definitions, and the server converts Muse Glimmer 30B ATEM -output into OpenAI-compatible `tool_calls`. See -[`serving/tool_parsers/atem.py`](serving/tool_parsers/atem.py) and -[`tests/test_atem_tool_parser.py`](tests/test_atem_tool_parser.py). +## Use from Pi -### Use from pi +This configuration works with both the regular server and the off-graph +batching server. Batching and DFlash are server-side choices; Pi uses the same +OpenAI-compatible endpoint for either path. Enable `--tool-parser atem` on the +server for tool calling. -Point pi at the server via `~/.pi/agent/models.json`: +Point Pi at the server via `/path/to/home/.pi/agent/models.json`, replacing +`/path/to/home` with your home directory: ```json { From 916bea0f088cf6b76790e9b79346e030ec307eef Mon Sep 17 00:00:00 2001 From: Songhao Jia Date: Fri, 16 Oct 2026 14:30:08 -0700 Subject: [PATCH 3/4] Clarify CUDA batching and DFlash sampling documentation Documentation update assisted by OpenAI Codex. --- examples/models/muse-glimmer/README.md | 28 ++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/examples/models/muse-glimmer/README.md b/examples/models/muse-glimmer/README.md index 6a0b5bd9093..b0f6630bbdd 100644 --- a/examples/models/muse-glimmer/README.md +++ b/examples/models/muse-glimmer/README.md @@ -212,8 +212,8 @@ cache growth, and optional prefix snapshots for reuse across requests. Batching supports both text-only and vision-enabled exports, with or without DFlash. -It uses separate batching runners -artifacts and workers are not interchangeable with it. +It uses separate batching runners and workers. Regular runner artifacts and +workers are not interchangeable with batching artifacts and workers. The commands below use MLX as an example. Select the backend matching your hardware, export, and worker build. @@ -245,6 +245,8 @@ python -m executorch.examples.models.muse_glimmer.export.export_dflash \ --output-dir /path/to/exports/mg-offgraph-dflash ``` +For CUDA exports, use `--backend cuda` and `--activation-dtype bfloat16` instead. + To enable image input, add `--mmproj "$MMPROJ"` to either export command above. This includes the vision encoder and writes its position-embedding data alongside the model. The resulting export can serve both text-only and image requests; @@ -298,10 +300,24 @@ python -m executorch.examples.models.muse_glimmer.serving.serve_batching \ ``` For DFlash, use `/path/to/exports/mg-offgraph-dflash/model.pte` and add -`--dflash --dflash-n-draft 3` to the launch command. The artifact must support -at least three draft tokens (a block size of four). Omitting `--dflash-n-draft`, -or setting it to zero, uses the exported maximum minus one. Draft proposals are -always argmax; request sampling parameters apply to the target distribution. +`--dflash --dflash-n-draft 3` to the launch command. `--dflash-n-draft` is a +runtime server option controlling the number of proposals per verification step +(N). Set it when starting the server; changing N does not require re-exporting. +N must be smaller than the exported drafter block size (D). Omitting the option, +or setting it to zero, uses N = D - 1. + +On CUDA, D is fixed at export time from the drafter checkpoint configuration and +cannot be changed at runtime. The exporter currently has no separate block-size +override option. To use a different D, export a new artifact with the corresponding +drafter configuration. Changing N does not change D: a D = 16 export still +evaluates 16 drafter rows per session when N = 3, and its default is N = 15. + +Sampling behavior depends on the backend. On MLX, draft proposals are always +argmax; request sampling parameters apply to the target distribution. On CUDA, +the request's temperature, top-k, and top-p settings apply to both the draft and +target distributions. Temperature zero selects greedy sampling. For stochastic +sampling, verification uses the draft probabilities for acceptance and samples +from the residual distribution when a proposal is rejected. Choose concurrency and prefix-cache limits to fit available memory. Prefix snapshots are disabled by default; omit `--prefix-cache-entries` to disable them. From ddc40b7d95fa1d695e35d1db0685772cff7a958a Mon Sep 17 00:00:00 2001 From: Songhao Jia Date: Fri, 16 Oct 2026 14:58:39 -0700 Subject: [PATCH 4/4] Remove DFlash sampling implementation details from README Documentation update assisted by OpenAI Codex. --- examples/models/muse-glimmer/README.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/examples/models/muse-glimmer/README.md b/examples/models/muse-glimmer/README.md index b0f6630bbdd..9d07039d673 100644 --- a/examples/models/muse-glimmer/README.md +++ b/examples/models/muse-glimmer/README.md @@ -312,13 +312,6 @@ override option. To use a different D, export a new artifact with the correspond drafter configuration. Changing N does not change D: a D = 16 export still evaluates 16 drafter rows per session when N = 3, and its default is N = 15. -Sampling behavior depends on the backend. On MLX, draft proposals are always -argmax; request sampling parameters apply to the target distribution. On CUDA, -the request's temperature, top-k, and top-p settings apply to both the draft and -target distributions. Temperature zero selects greedy sampling. For stochastic -sampling, verification uses the draft probabilities for acceptance and samples -from the residual distribution when a proposal is rejected. - Choose concurrency and prefix-cache limits to fit available memory. Prefix snapshots are disabled by default; omit `--prefix-cache-entries` to disable them. If the export stores weights separately, also pass `--data-path` with its data