Repository navigation
Commit cb94e69
authored
Use compile-time autotune for the Qwen3.5 CUDA export on PyTorch 2.14 (#23140)
### Summary
`test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packed`
(and the Qwen3.5 `benchmark-cuda` job) have failed on every main run
since the PyTorch 2.14 pin bump (#22501).
The Qwen3.5 export set `autotune_at_compile_time=OFF`. That was added
with the 2.13 pin (#20967) to avoid a 2.13-only compile-time autotune
issue. On PyTorch 2.14, OFF selects AOTInductor's new lazy-compile path,
which breaks this model in two ways:
1. **Export crash.** Autotuning now runs a C++ JIT variant of the graph
(`GraphLowering._run_jit_variant_for_autotune`) directly on
`self.constants` and the real inputs. The low-memory CUDA export frees
KV-cache storage (`resize_(0)`) before compile. The #21803 rehydration
only hooks `CachingAutotuner.run`, so the null KV storage reaches the
kernels:
```
_run_jit_variant_for_autotune -> compiled_fn(input_handles)
RuntimeError: CUDA driver error: an illegal memory access was
encountered
```
The logged error was only the secondary one, from `preserve_rng_state`
-> `set_rng_state`.
2. **Unrunnable artifact.** Even with the KV storage rehydrated, the
lazy AOT code loads every Triton kernel from a cubin file at runtime and
ignores `embed_kernel_binary`. ExecuTorch only packs the wrapper `.so`,
so the runner fails on the first execute with `CUDA driver error: file
not found`.
### Fix
Drop the `autotune_at_compile_time=OFF` override. On 2.14, AOTI defaults
it to on for accelerators, so kernels are autotuned at compile time on
generated example inputs and embedded in the `.so`, the same as the
other CUDA models. The obsolete 2.13 comment is removed too.
### Test plan
CI on this PR (earlier commits carried temporary diagnostics, which have
since been removed):
-
`test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packed`
passes:
- export GPU peak 18682 MB (limit 21504 MB)
- runner produces correct output, decode 113.8 tok/s
- `Qwen3.5-MoE serving smoke passed`1 parent f626041 commit cb94e69
1 file changed
Lines changed: 2 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1138 | 1138 | | |
1139 | 1139 | | |
1140 | 1140 | | |
1141 | | - | |
1142 | | - | |
1143 | | - | |
| 1141 | + | |
| 1142 | + | |
1144 | 1143 | | |
1145 | 1144 | | |
1146 | 1145 | | |
| |||
1151 | 1150 | | |
1152 | 1151 | | |
1153 | 1152 | | |
1154 | | - | |
1155 | 1153 | | |
1156 | 1154 | | |
1157 | 1155 | | |
| |||
1162 | 1160 | | |
1163 | 1161 | | |
1164 | 1162 | | |
1165 | | - | |
1166 | 1163 | | |
1167 | 1164 | | |
1168 | 1165 | | |
| |||
0 commit comments