Skip to content

Commit 6b53322

Browse files
authored
Give the Qwen3.5 export GPU memory budget room for run to run variance (pytorch#22134)
### Summary The Qwen3.5-35B CUDA export jobs have been red on main since 2026-08-24: - `cuda-perf / export-models (SocialLocalMobile/Qwen3.5-35B-A3B-HQQ-INT4, quantized-int4-tile-packed, ...)` - `Test CUDA Builds / test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packed` The export itself succeeds and writes a valid model. What fails is the peak GPU memory gate that runs right after it: ``` EXPORT_GPU_PEAK_MEMORY_MB: 20489.70 ERROR: export exceeded GPU memory budget (20489.70 MB > 20480 MB) ``` That is an overshoot of 9.7 MB, or 0.047 percent. ### Why it happens Nothing regressed. pytorch#21803 turned max autotune on for this export. Autotuning allocates and frees large benchmark buffers, so `torch.cuda.max_memory_allocated` is not reproducible from one run to the next. Across 22 runs on main since 2026-08-20 the measured peak took four distinct values: | peak | runs | result | | --- | --- | --- | | 20328.96 MB | 3 | pass | | 20350.96 MB | 12 | pass | | 20375.95 MB | 1 | pass | | 20489.70 MB | 6 | **fail** | That is a band 160 MB wide, and the budget of 20480 MB (exactly 20 GiB) sat 9.7 MB below the top of it. Two runs on opposite sides of the line finished two minutes apart, on adjacent commits, with an identical pinned environment (same torch, triton, torchao and transformers builds, only the executorch version string differed). None of the commits in the window that first turned the job red touch the export path. So the outcome came down to which value the run happened to produce, not to any change in the code. ### Fix Raise the default budget to 21504 MB. That is about 1 GB above the highest peak observed so far, so a real memory regression of any meaningful size still trips the gate, and it keeps the export 3 GB clear of the 24 GB consumer card the budget exists to protect. `EXPORT_GPU_PEAK_MB_LIMIT` can still be overridden in the environment. ### Test plan Checked the script parses, and replayed the gate arithmetic against every peak value seen in CI: ``` 20328.96 pass 20350.96 pass 20375.95 pass 20489.70 pass 21503.99 pass 21504.01 REJECT 22000.00 REJECT ``` CI on this PR covers the real thing: both jobs above run the same script.
1 parent e831b3e commit 6b53322

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

‎.ci/scripts/export_model_artifact.sh‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -492,7 +492,10 @@ if [ "$MODEL_NAME" = "qwen3_5_moe" ]; then
492492
# Gate peak GPU memory so we keep the export viable on consumer GPUs
493493
# (e.g. RTX 4090 with 24 GB). The export script prints a machine-
494494
# parseable marker line "EXPORT_GPU_PEAK_MEMORY_MB: <float>".
495-
EXPORT_GPU_PEAK_MB_LIMIT="${EXPORT_GPU_PEAK_MB_LIMIT:-20480}"
495+
# Max autotune makes the peak vary by about 160 MB between runs of the same
496+
# commit, so the budget needs headroom above the highest value we see or the
497+
# gate rejects an export that did not actually grow.
498+
EXPORT_GPU_PEAK_MB_LIMIT="${EXPORT_GPU_PEAK_MB_LIMIT:-21504}"
496499
PEAK_LINE=$(grep -E '^EXPORT_GPU_PEAK_MEMORY_MB:' "$EXPORT_LOG" | tail -1)
497500
rm -f "$EXPORT_LOG"
498501
if [ -z "$PEAK_LINE" ]; then

0 commit comments

Comments
 (0)