Skip to content

Commit 98cf058

Browse files
cthottimetascroy
andauthored
[RFC] DFlash (pytorch#20840)
### Summary This PR adds support for DFlash speculative decoding to the ExecuTorch MLX delegate using Qwen3-4B as the reference model. It has hidden-state export for the target model, a configurable DFlash draft model, export support for both models, and an end-to-end speculative decoding driver. The implementation has been verified to produce an identical output to standard greedy decoding. Although the original issue was targeted at Gemma, my implementation was developed and validated on Qwen3 because publicly available DFlash draft weights are available and the model fits within local hardware constraints. The overall implementation is model-agnostic and designed to support additional architectures in follow-up work. Fixes pytorch#20701 ### DFlash Overview Unlike conventional speculative decoding, where a small autoregressive draft model predicts one token at a time, DFlash predicts an entire block of tokens in a single forward pass over a masked block. The draft model is conditioned on intermediate hidden states from the target model, which are added into the Key/Value projections of every draft rather than only at the input. This provides richer conditioning throughout the draft network while keeping the draft model relatively small. The draft model is implemented through a configurable DFlashConfig, making it adaptable to different transformer architectures. It has been numerically verified on Qwen3 and structurally validated against Llama-3.1 and Gemma-style configurations. There are additional Gemma-specific features documented, like partial RoPE and post-layer scaling, but they remain outside the scope of this PR. ### Key Implementation Details **_Phase 1: Target hidden-state export_** - Adds `TorchExportableModuleWithStaticCacheAndHidden`, which extends the existing Hugging Face export wrapper to return both logits and concatenated hidden states from the selected target layers. **_Phase 2: Draft model and export_** - Adds a configurable DFlash draft model together with its export pipeline. - The exporter supports dynamic context lengths and 4-bit quantization compatible with the existing MLX export flow. **_Phase 3: Speculative decoding driver_** - Adds an end-to-end Python implementation of the DFlash decoding algorithm (run_dflash.py), including: - Draft block generation, Target verification, First-mismatch acceptance, and KV-cache position rollback - Applies the Qwen3 chat template with enable_thinking=False, matching the format used during draft model training. ### Python Driver My PR includes the Python implementation of the speculative decoding loop rather than the earlier C++ engine recommended in the issue. Profiling showed that nearly all execution time is spent inside the exported target and draft model execution, while the Python overhead contributes only a very small fraction of the total runtime. Since both implementations execute the same exported MLX programs, the Python driver provides similar functionality and is simpler to maintain. A C++ implementation can be added later if tighter runtime integration is needed. ### Performance My primary goal of this PR is to add DFlash support and verify that the implementation is correct. In addition to correctness, I benchmarked the implementation on Apple M2 hardware to understand its performance characteristics. The most significant observation is that the target model's verification pass scales almost linearly with the number of tokens being verified. On this system, verifying a single token (**T=1**) takes approximately **32 ms**, while verifying a block of 16 tokens (**T=16**) takes approximately **239 ms**, roughly 7.5× the latency for a 16× larger verification window. Since speculative decoding relies on verifying multiple draft tokens in a single target forward pass, this scaling behavior directly impacts the achievable speedup. The measured acceptance rates (**chat: 4.17, math: 5.59, code: 6.81**) are not sufficient to offset the additional verification cost on this hardware, so the implementation is currently slower than baseline greedy decoding for a single-request inference on an Apple M2. This appears to be because of the underlying hardware rather than the DFlash implementation itself. The speculative decoding algorithm is functioning correctly and produces token-for-token identical output to baseline decoding. Platforms where verification cost grows more slowly with batch size are expected to benefit more from DFlash, since speculative decoding becomes increasingly effective as the cost of verifying larger blocks approaches the cost of verifying a single token. ### Test Plan All tests were run using exported target and draft models (`qwen3_4b_dflash_target.pte` and `qwen3_4b_dflash_draft.pte`) with a block size of 16 and 4-bit quantization. **_Lossless speculative decoding:_** `python3 examples/models/qwen3/tests/test_dflash_lossless.py` Verifies that DFlash produces token-for-token identical output to standard greedy decoding. **_Target model export_** ``` python3 examples/models/qwen3/tests/test_dflash_target.py \ qwen3_4b_dflash_target.pte ``` **_Draft model export and dynamic context lengths_** ``` python3 examples/models/qwen3/tests/test_dflash_draft.py \ qwen3_4b_dflash_draft.pte ``` ### Final Results **_Math prompt_** - baseline_runs (tok/s) = [25.73, 24.63, 25.65] - baseline_median = 25.65 - dflash_runs (tok/s) = [19.44, 19.47, 19.44] - dflash_median = 19.44 - τ = 5.59 - speedup = 0.76× **_Code prompt_** - baseline_runs (tok/s) = [28.28, 28.19, 26.10] - baseline_median = 28.19 - dflash_runs (tok/s) = [23.84, 23.81, 22.70] - dflash_median = 23.81 - τ = 6.81 - speedup = 0.84× **_Chat prompt_** - baseline_runs (tok/s) = [26.05, 26.11, 27.98] - baseline_median = 26.11 - dflash_runs (tok/s) = [14.78, 14.77, 14.75] - dflash_median = 14.77 - τ = 4.17 - speedup = 0.57× cc @mergennachin @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel @jackzhxng --------- Co-authored-by: Chet Hotti <cthotti@users.noreply.github.com> Co-authored-by: Scott Roy <scroy@meta.com>
1 parent adeb496 commit 98cf058

18 files changed

Lines changed: 2044 additions & 285 deletions

‎.github/workflows/mlx.yml‎

Lines changed: 126 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -727,6 +727,131 @@ jobs:
727727
fi
728728
echo "::endgroup::"
729729
730+
# DFlash speculative decoding: target and draft exported as two methods of one
731+
# .pte. Also the only coverage that the constants the export publishes
732+
# (get_max_ctx_len / get_prefill_chunk_size / get_max_block_len /
733+
# get_mask_token_id) match what
734+
# the runner derives from them -- the runner takes no capacity or block flags.
735+
test-mlx-dflash:
736+
needs: run-decision
737+
if: |
738+
github.event_name == 'pull_request' ||
739+
needs.run-decision.outputs.is-full-run == 'true'
740+
uses: pytorch/test-infra/.github/workflows/macos_job.yml@main
741+
with:
742+
default-packages: ""
743+
job-name: test-mlx-dflash
744+
runner: macos-14-xlarge
745+
python-version: "3.12"
746+
submodules: recursive
747+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
748+
# The 4B target is the smallest checkpoint DFlash drafts are published
749+
# for, so this job downloads and quantizes far more than the 1B LLM jobs.
750+
timeout: 120
751+
script: |
752+
set -eux
753+
export HF_HUB_DISABLE_XET=1
754+
755+
MAX_CTX_LEN=1024
756+
PREFILL_CHUNK_SIZE=128
757+
N_DRAFT=3
758+
DRAFT_MODEL=z-lab/Qwen3-4B-DFlash-b16
759+
TOKENIZER_MODEL=Qwen/Qwen3-4B
760+
PTE=/tmp/qwen3_4b_dflash.pte
761+
762+
echo "::group::Install ExecuTorch"
763+
${CONDA_RUN} python install_executorch.py > /dev/null
764+
echo "::endgroup::"
765+
766+
echo "::group::Install DFlash requirements"
767+
OPTIMUM_ET_VERSION=$(cat .ci/docker/ci_commit_pins/optimum-executorch.txt)
768+
${CONDA_RUN} pip install transformers "optimum-executorch @ git+https://github.com/huggingface/optimum-executorch.git@${OPTIMUM_ET_VERSION}"
769+
echo "::endgroup::"
770+
771+
${CONDA_RUN} pip list
772+
773+
echo "::group::Export Qwen3-4B DFlash (target + draft)"
774+
# No --block-size: it must come from the draft checkpoint's config.
775+
${CONDA_RUN} python -m executorch.backends.mlx.examples.llm.dflash.export \
776+
--output ${PTE} \
777+
--draft-model ${DRAFT_MODEL} \
778+
--target-model ${TOKENIZER_MODEL} \
779+
--max-ctx-len ${MAX_CTX_LEN} \
780+
--prefill-chunk-size ${PREFILL_CHUNK_SIZE}
781+
echo "::endgroup::"
782+
783+
echo "::group::Cache the tokenizer"
784+
# The runner loads the tokenizer with local_files_only, so fetch it here
785+
# rather than letting an unauthenticated request happen mid-run. Also
786+
# confirms the chat template is present -- a template that fails to load
787+
# renders to nothing and the prompt tokenizes to zero tokens.
788+
TOKENIZER_MODEL=${TOKENIZER_MODEL} ${CONDA_RUN} python -c "import os; from transformers import AutoTokenizer; m = os.environ['TOKENIZER_MODEL']; t = AutoTokenizer.from_pretrained(m); assert t.chat_template, f'{m} has no chat_template'; ids = t.apply_chat_template([{'role': 'user', 'content': 'hi'}], add_generation_prompt=True, tokenize=True, return_tensors='pt'); ids = getattr(ids, 'input_ids', ids); assert ids.numel() > 0, 'chat template rendered no tokens'; print(f'tokenizer cached, chat template renders {ids.numel()} tokens')"
789+
echo "::endgroup::"
790+
791+
echo "::group::Check published constant methods"
792+
# The runner reads these instead of taking flags, so a mismatch here is
793+
# a silently broken runner rather than a failed export. Kept on one line:
794+
# a YAML block scalar indents every line, which python -c would reject.
795+
BLOCK_SIZE=$(${CONDA_RUN} python -c "from huggingface_hub import snapshot_download; from executorch.backends.mlx.examples.llm.dflash.model import load_dflash_config; print(load_dflash_config(snapshot_download('${DRAFT_MODEL}', allow_patterns=['*.json'])).block_size)")
796+
MASK_TOKEN_ID=$(${CONDA_RUN} python -c "from huggingface_hub import snapshot_download; from executorch.backends.mlx.examples.llm.dflash.model import load_dflash_config; print(load_dflash_config(snapshot_download('${DRAFT_MODEL}', allow_patterns=['*.json'])).mask_token_id)")
797+
echo "draft checkpoint block_size: ${BLOCK_SIZE} mask_token_id: ${MASK_TOKEN_ID}"
798+
${CONDA_RUN} python -c "from executorch.runtime import Runtime, Verification; from executorch.backends.mlx.examples.llm.runtime_meta import read_const_int as r; p = Runtime.get().load_program('${PTE}', verification=Verification.Minimal); got = {n: r(p, n) for n in ['get_max_ctx_len', 'get_prefill_chunk_size', 'get_max_block_len', 'get_mask_token_id']}; want = {'get_max_ctx_len': ${MAX_CTX_LEN}, 'get_prefill_chunk_size': ${PREFILL_CHUNK_SIZE}, 'get_max_block_len': ${BLOCK_SIZE}, 'get_mask_token_id': ${MASK_TOKEN_ID}}; print('published:', got); assert got == want, f'mismatch: {got} != {want}'; assert {'draft', 'target'} <= set(p.method_names), p.method_names; print('Success: constants match and both methods present')"
799+
echo "::endgroup::"
800+
801+
echo "::group::Run DFlash speculative decoding"
802+
# --n-draft below the exported maximum, so a regression that silently
803+
# pins the block to BLOCK_SIZE gets caught.
804+
OUTPUT=$(${CONDA_RUN} python -m executorch.backends.mlx.examples.llm.dflash.run \
805+
--pte ${PTE} \
806+
--tokenizer ${TOKENIZER_MODEL} \
807+
--n-draft ${N_DRAFT} \
808+
--prompt "What is the capital of France?" \
809+
--max-new-tokens 32 \
810+
--verbose 2>&1)
811+
echo "$OUTPUT"
812+
if echo "$OUTPUT" | grep -iq "Paris"; then
813+
echo "Success: 'Paris' found in output"
814+
else
815+
echo "Failed: Expected 'Paris' not found in output"
816+
exit 1
817+
fi
818+
# A special token in the decoded text means skip_special_tokens regressed.
819+
if echo "$OUTPUT" | grep -q "<|im_end|>"; then
820+
echo "Failed: special tokens leaked into the decoded output"
821+
exit 1
822+
fi
823+
echo "::endgroup::"
824+
825+
echo "::group::Check speculation actually happened"
826+
# tau <= 1.0 means every round fell back to a target-only step, i.e. the
827+
# draft was never accepted and DFlash degenerated to plain decoding.
828+
# tau also cannot exceed n_draft + 1, the emitted tokens per round.
829+
TAU=$(echo "$OUTPUT" | grep "avg emitted/round" | grep -oE "[0-9]+\.[0-9]+")
830+
echo "avg emitted/round (tau): ${TAU}"
831+
awk "BEGIN { exit !(${TAU} > 1.0 && ${TAU} <= ${N_DRAFT} + 1) }" || {
832+
echo "Failed: tau ${TAU} outside (1.0, $((N_DRAFT + 1))]"
833+
exit 1
834+
}
835+
echo "Success: tau ${TAU} in (1.0, $((N_DRAFT + 1))]"
836+
echo "::endgroup::"
837+
838+
echo "::group::Reject an out-of-range --n-draft"
839+
# block_len is n_draft + 1 and the graph traced block_len <= BLOCK_SIZE,
840+
# so BLOCK_SIZE itself must be refused before reaching the runtime.
841+
if ${CONDA_RUN} python -m executorch.backends.mlx.examples.llm.dflash.run \
842+
--pte ${PTE} \
843+
--tokenizer ${TOKENIZER_MODEL} \
844+
--n-draft ${BLOCK_SIZE} \
845+
--max-new-tokens 1 > /tmp/n_draft_err.txt 2>&1; then
846+
echo "Failed: --n-draft ${BLOCK_SIZE} should have been rejected"
847+
cat /tmp/n_draft_err.txt
848+
exit 1
849+
fi
850+
cat /tmp/n_draft_err.txt
851+
grep -q "must be in \[1, $((BLOCK_SIZE - 1))\]" /tmp/n_draft_err.txt
852+
echo "Success: out-of-range --n-draft rejected"
853+
echo "::endgroup::"
854+
730855
# Off-graph KV cache: the cache is a runtime object, so this path can only be
731856
# exercised by the C++ runner (pybindings cannot bind a cache_key). Also the
732857
# only coverage that the layout the export publishes is actually consumable.
@@ -812,7 +937,7 @@ jobs:
812937
--model-id "${MODEL_ID}" \
813938
--output /tmp/${MODEL_NAME}_offgraph.pte \
814939
--use-offgraph-cache \
815-
--max-seq-len 1024 \
940+
--max-ctx-len 1024 \
816941
--dtype bf16 \
817942
--qlinear 4w
818943
echo "::endgroup::"

‎Makefile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -508,3 +508,8 @@ clean:
508508
rm -rf cmake-out \
509509
extension/llm/tokenizers/build \
510510
extension/llm/tokenizers/pytorch_tokenizers.egg-info
511+
512+
# qwen3_dflash-mlx target removed: it depended on the C++ engine sources
513+
# (CMakeLists.txt, CMakePresets.json, qwen3_dflash_engine.*), which are
514+
# gitignored/not yet landed. Restore this target in the follow-up PR that
515+
# actually lands the C++ engine.

‎backends/mlx/.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Auto-generated by backends/mlx/serialization/generate.py — do not commit.
2+
# See backends/mlx/serialization/README.md for regeneration instructions.
3+
runtime/MLXLoader.cpp
4+
runtime/MLXLoader.h
5+
runtime/schema_generated.h
6+
serialization/_generated/
7+
serialization/_generated_serializers.py
8+
serialization/mlx_graph_schema.py
9+
_generated_inspector.py

‎backends/mlx/examples/llm/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ python -m executorch.backends.mlx.examples.llm.export_llm_hf \
6161
--model-id "unsloth/gemma-3-1b-it" \
6262
--output gemma3_offgraph.pte \
6363
--use-offgraph-cache \
64-
--max-seq-len 1024
64+
--max-ctx-len 1024
6565

6666
# Gemma 4 text-only export
6767
python -m executorch.backends.mlx.examples.llm.export_llm_hf \
@@ -88,15 +88,15 @@ pip install -U "transformers @ git+https://github.com/huggingface/transformers.g
8888
|--------|---------|-------------|
8989
| `--model-id` | `unsloth/Llama-3.2-1B-Instruct` | HuggingFace model ID |
9090
| `--output` | *(required)* | Output .pte file path |
91-
| `--max-seq-len` | `1024` | Maximum sequence length for KV cache |
91+
| `--max-ctx-len` | `1024` | Maximum context length / KV cache capacity |
9292
| `--dtype` | `bf16` | Model dtype (`fp32`, `fp16`, `bf16`) |
9393
| `--qlinear` | None | Quantization for linear layers (`4w`, `8w`, `nvfp4`) |
9494
| `--qembedding` | None | Quantization for embedding layers (`4w`, `8w`, `nvfp4`) |
9595
| `--no-tie-word-embeddings` | `False` | Disable re-tying lm_head to embedding after quantization |
9696
| `--use-custom-sdpa` | `False` | Use MLX custom SDPA (`mlx::custom_sdpa`) |
9797
| `--use-custom-kv-cache` | `False` | Use MLX custom KV cache (`mlx::kv_cache_update`) |
9898
| `--use-offgraph-cache` | `False` | Use the off-graph KV cache (`kvcache::update_and_attend`); replaces the two flags above |
99-
| `--prefill-chunk-size` | `512` | Off-graph: tokens per prefill step, published as `get_prefill_chunk_size` for the runner. It is the largest single write, so a ring layer is sized `window + chunk - 1`; it may not exceed the sliding window or the exported sequence length |
99+
| `--prefill-chunk-size` | `512` | Max tokens per forward step. Bounds the traced `seq_len` dimension and is published as `get_prefill_chunk_size` for the runner. It is also the largest single cache write, so a ring layer is sized `window + chunk - 1`; it may not exceed the sliding window or the context length. Ignored on the optimum-executorch path, which owns its own `seq_len` bound |
100100

101101
Off-graph exports keep no cache in the `.pte`, so the pybindings `run_llm_hf`
102102
cannot run them — use [`mlx_run_llm_hf`](#mlx_run_llm_hf-c) below, which builds
@@ -169,7 +169,7 @@ python -m executorch.backends.mlx.examples.llm.export_llm_hf \
169169
--model-id unsloth/gemma-3-1b-it \
170170
--output gemma3_offgraph.pte \
171171
--use-offgraph-cache \
172-
--max-seq-len 1024
172+
--max-ctx-len 1024
173173

174174
cmake-out/backends/mlx/examples/llm/mlx_run_llm_hf \
175175
--pte gemma3_offgraph.pte \
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
4+
"""DFlash draft model package.
5+
6+
Public API:
7+
- arch: resolve_arch, model_type <-> HF config mapping
8+
- adapters: DFlashAdapter registry, Qwen3 draft weight loading
9+
- model: DFlashDraftModel, DFlashConfig, load_dflash_config
10+
- cache: DFlashDraftKVCache (persistent draft KV cache)
11+
- export: export entry-point (target + draft -> one .pte)
12+
- run: speculative decoding loop runner
13+
"""
14+
15+
from executorch.backends.mlx.examples.llm.dflash.adapters import ( # noqa: F401
16+
get_adapter,
17+
)
18+
from executorch.backends.mlx.examples.llm.dflash.arch import resolve_arch # noqa: F401
19+
from executorch.backends.mlx.examples.llm.dflash.cache import ( # noqa: F401
20+
DFlashDraftKVCache,
21+
)
22+
from executorch.backends.mlx.examples.llm.dflash.model import ( # noqa: F401
23+
DFlashConfig,
24+
DFlashDraftModel,
25+
load_dflash_config,
26+
)
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
"""Registry mapping a checkpoint's ``model_type`` to its DFlash adapter.
8+
9+
Adapters hold the one part of the pipeline that genuinely differs per
10+
checkpoint: draft weight loading, which depends on the keys the draft omits and
11+
where they are copied from in the target. Everything else in the DFlash export
12+
and runner path is architecture-agnostic -- chat templating in particular is
13+
carried by the checkpoint's own tokenizer config, so it lives in runtime_meta.
14+
"""
15+
16+
from typing import Callable, Dict, NamedTuple
17+
18+
19+
class DFlashAdapter(NamedTuple):
20+
load_draft_weights: Callable
21+
22+
23+
_ADAPTERS: Dict[str, DFlashAdapter] = {}
24+
25+
26+
def register_adapter(model_type: str, *, load_draft_weights: Callable) -> None:
27+
_ADAPTERS[model_type] = DFlashAdapter(load_draft_weights=load_draft_weights)
28+
29+
30+
def get_adapter(model_type: str) -> DFlashAdapter:
31+
try:
32+
return _ADAPTERS[model_type]
33+
except KeyError:
34+
raise ValueError(
35+
f"No DFlash adapter registered for model_type {model_type!r}. "
36+
f"Registered: {sorted(_ADAPTERS)}."
37+
) from None
38+
39+
40+
def load_qwen3_dflash_draft_weights(draft_model, checkpoint_path, target_state_dict):
41+
"""Load Qwen3 DFlash checkpoint weights and copy shared target weights.
42+
The draft checkpoint omits its embedding and LM head, which are copied from the target.
43+
"""
44+
from safetensors.torch import load_file
45+
46+
draft_weights = {}
47+
for f in checkpoint_path.glob("*.safetensors"):
48+
draft_weights.update(load_file(str(f)))
49+
50+
missing, unexpected = draft_model.load_state_dict(draft_weights, strict=False)
51+
assert not unexpected, f"Unexpected draft checkpoint keys: {unexpected}"
52+
still_missing = [
53+
k for k in missing if not k.startswith(("embed_tokens.", "lm_head."))
54+
]
55+
assert not still_missing, f"Missing draft checkpoint keys: {still_missing}"
56+
57+
draft_model.embed_tokens.weight.data.copy_(
58+
target_state_dict["model.embed_tokens.weight"]
59+
)
60+
lm_head_key = (
61+
"lm_head.weight"
62+
if "lm_head.weight" in target_state_dict
63+
else "model.embed_tokens.weight"
64+
)
65+
draft_model.lm_head.weight.data.copy_(target_state_dict[lm_head_key])
66+
67+
68+
register_adapter(
69+
"qwen3",
70+
load_draft_weights=load_qwen3_dflash_draft_weights,
71+
)

0 commit comments

Comments
 (0)