Skip to content

Commit 07b7c7e

Browse files
mergennachinmnachin
andauthored
Fix buffer dtype mismatch and causal mask recompute in export (pytorch#18380)
- Materialize state buffers as bf16 (not fp32) to match compute dtype, fixing "Expected bfloat16 inputs" error in Triton SDPA during AOTI lowering - Use hasattr(layer.attn, "mask") instead of isinstance(layer.attn, FullAttention) for causal mask recompute — the isinstance check fails when the module is imported via different Python paths (executorch.examples... vs examples...) - Remove unused FullAttention import from export.py - Fix model directory name in README examples (Qwen3.5-MoE-A3B → Qwen3.5-35B-A3B) Co-authored-by: mnachin <mnachin@fb.com>
1 parent 1cbc24c commit 07b7c7e

2 files changed

Lines changed: 8 additions & 5 deletions

File tree

‎examples/models/qwen3_5_moe/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ recommended — the model is too large to fit in VRAM at bf16.
3232

3333
```bash
3434
python export.py \
35-
--model-dir ~/models/Qwen3.5-MoE-A3B \
35+
--model-dir ~/models/Qwen3.5-35B-A3B \
3636
--output-dir ./qwen35_moe_exports \
3737
--qlinear 4w \
3838
--qembedding 8w
@@ -73,7 +73,7 @@ The runner requires:
7373
cmake-out/examples/models/qwen3_5_moe/qwen3_5_moe_runner \
7474
--model_path qwen35_moe_exports/model.pte \
7575
--data_path qwen35_moe_exports/aoti_cuda_blob.ptd \
76-
--tokenizer_path ~/models/Qwen3.5-MoE-A3B/tokenizer.json \
76+
--tokenizer_path ~/models/Qwen3.5-35B-A3B/tokenizer.json \
7777
--prompt "The meaning of life is" \
7878
--max_new_tokens 128
7979
```

‎examples/models/qwen3_5_moe/export.py‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
import torch
1313
import torch.nn as nn
1414

15-
from executorch.examples.models.qwen3_5_moe.model import FullAttention, Qwen35MoE
15+
from executorch.examples.models.qwen3_5_moe.model import Qwen35MoE
1616

1717

1818
# ---------------------------------------------------------------------------
@@ -129,13 +129,16 @@ def _materialize_buffers(model, config):
129129
Replaces meta buffers with real tensors on CPU, recomputes RoPE
130130
inv_freq and causal masks.
131131
"""
132+
# State buffers (KV cache, conv/recurrent state) are bf16 to match
133+
# compute dtype. Masks stay bool, inv_freq stays float32.
132134
for fqn, buf in list(model.named_buffers()):
133135
if buf.device.type == "meta":
136+
dtype = torch.bfloat16 if buf.dtype != torch.bool else torch.bool
134137
parts = fqn.rsplit(".", 1)
135138
parent = model.get_submodule(parts[0]) if len(parts) > 1 else model
136139
parent.register_buffer(
137140
parts[-1],
138-
torch.zeros(buf.shape, dtype=buf.dtype, device="cpu"),
141+
torch.zeros(buf.shape, dtype=dtype, device="cpu"),
139142
)
140143

141144
# Recompute RoPE inv_freq (zero-fill above is wrong for these)
@@ -153,7 +156,7 @@ def _materialize_buffers(model, config):
153156

154157
# Recompute causal masks for full attention layers
155158
for layer in model.layers:
156-
if isinstance(layer.attn, FullAttention):
159+
if hasattr(layer.attn, "mask"):
157160
mask = torch.tril(
158161
torch.ones(config.max_seq_len, config.max_seq_len, dtype=torch.bool)
159162
)

0 commit comments

Comments
 (0)