Repository navigation
Use compile-time autotune for the Qwen3.5 CUDA export on PyTorch 2.14 - #23140
Merged
Merged
Conversation
Run the Qwen3.5 MoE export with CUDA_LAUNCH_BLOCKING=1 so the traceback points at the kernel that faults instead of a later CUDA API call. Follow-up commits on this branch will carry the actual fix.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23140
Note: Links to docs will display an error until the docs builds have been completed. ❌ 7 Pending, 6 Unclassified FailuresAs of commit 5b24ba1 with merge base 42b7709 ( UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
Inductor re-raises codegen errors from None, and preserve_rng_state's set_rng_state failure then replaces the original exception, so the log only shows the secondary error. Wrap the export to print the full exception chain and log every CachingAutotuner.run (flagging undersized tensor storage) so the faulting kernel is identifiable.
With autotune_at_compile_time off, PyTorch 2.14 no longer autotunes through a Python first pass. GraphLowering._run_jit_variant_for_autotune compiles a C++ JIT wrapper and runs it on self.constants and the real inputs directly. The low-memory CUDA export frees KV-cache storage with resize_(0), and #21803 only rehydrated it inside CachingAutotuner.run, so the null KV storage now reaches the kernels and the Qwen3.5 MoE export fails with an illegal memory access. Rehydrate emptied constants and inputs for the duration of the JIT run, the same way CachingAutotuner.run already does.
autotune_at_compile_time=OFF was added with the PyTorch 2.13 pin (#20967) to avoid a 2.13-only compile-time autotune issue. On PyTorch 2.14 it selects the new AOTI lazy-compile path, whose AOT code loads every Triton kernel from a cubin file at runtime and ignores embed_kernel_binary. ExecuTorch only packs the wrapper .so, so the runner fails on the first execute with "CUDA driver error: file not found". Drop the override so AOTI uses its default (compile-time autotune with embedded kernel binaries).
Gasoonjia
deployed
to
upload-benchmark-results
September 25, 2026 03:49 — with 
GitHub Actions
Active
GitHub Actions
Active
Compile-time autotune is back on for the Qwen3.5 CUDA export, so AOTI no longer runs the JIT-variant autotune pass and the KV rehydration wrapper is unused. Drop it along with the temporary CUDA_LAUNCH_BLOCKING and exception-chain diagnostics. The only remaining change is the export config.
Gasoonjia
changed the title
JacobSzwejbka
approved these changes
Sep 25, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packed(and the Qwen3.5benchmark-cudajob) have failed on every main run since the PyTorch 2.14 pin bump (#22501).The Qwen3.5 export set
autotune_at_compile_time=OFF. That was added with the 2.13 pin (#20967) to avoid a 2.13-only compile-time autotune issue. On PyTorch 2.14, OFF selects AOTInductor's new lazy-compile path, which breaks this model in two ways:GraphLowering._run_jit_variant_for_autotune) directly onself.constantsand the real inputs. The low-memory CUDA export frees KV-cache storage (resize_(0)) before compile. The Fix Qwen3.5 CUDA autotuning with released KV storage #21803 rehydration only hooksCachingAutotuner.run, so the null KV storage reaches the kernels:preserve_rng_state->set_rng_state.embed_kernel_binary. ExecuTorch only packs the wrapper.so, so the runner fails on the first execute withCUDA driver error: file not found.Fix
Drop the
autotune_at_compile_time=OFFoverride. On 2.14, AOTI defaults it to on for accelerators, so kernels are autotuned at compile time on generated example inputs and embedded in the.so, the same as the other CUDA models. The obsolete 2.13 comment is removed too.Test plan
CI on this PR (earlier commits carried temporary diagnostics, which have since been removed):
test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packedpasses:Qwen3.5-MoE serving smoke passed