Repository navigation
Commit 6b53322
authored
Give the Qwen3.5 export GPU memory budget room for run to run variance (pytorch#22134)
### Summary
The Qwen3.5-35B CUDA export jobs have been red on main since 2026-08-24:
- `cuda-perf / export-models
(SocialLocalMobile/Qwen3.5-35B-A3B-HQQ-INT4, quantized-int4-tile-packed,
...)`
- `Test CUDA Builds /
test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packed`
The export itself succeeds and writes a valid model. What fails is the
peak GPU
memory gate that runs right after it:
```
EXPORT_GPU_PEAK_MEMORY_MB: 20489.70
ERROR: export exceeded GPU memory budget (20489.70 MB > 20480 MB)
```
That is an overshoot of 9.7 MB, or 0.047 percent.
### Why it happens
Nothing regressed. pytorch#21803 turned max autotune on for this export.
Autotuning
allocates and frees large benchmark buffers, so
`torch.cuda.max_memory_allocated`
is not reproducible from one run to the next.
Across 22 runs on main since 2026-08-20 the measured peak took four
distinct
values:
| peak | runs | result |
| --- | --- | --- |
| 20328.96 MB | 3 | pass |
| 20350.96 MB | 12 | pass |
| 20375.95 MB | 1 | pass |
| 20489.70 MB | 6 | **fail** |
That is a band 160 MB wide, and the budget of 20480 MB (exactly 20 GiB)
sat
9.7 MB below the top of it.
Two runs on opposite sides of the line finished two minutes apart, on
adjacent
commits, with an identical pinned environment (same torch, triton,
torchao and
transformers builds, only the executorch version string differed). None
of the
commits in the window that first turned the job red touch the export
path. So
the outcome came down to which value the run happened to produce, not to
any
change in the code.
### Fix
Raise the default budget to 21504 MB. That is about 1 GB above the
highest peak
observed so far, so a real memory regression of any meaningful size
still trips
the gate, and it keeps the export 3 GB clear of the 24 GB consumer card
the
budget exists to protect. `EXPORT_GPU_PEAK_MB_LIMIT` can still be
overridden in
the environment.
### Test plan
Checked the script parses, and replayed the gate arithmetic against
every peak
value seen in CI:
```
20328.96 pass
20350.96 pass
20375.95 pass
20489.70 pass
21503.99 pass
21504.01 REJECT
22000.00 REJECT
```
CI on this PR covers the real thing: both jobs above run the same
script.1 parent e831b3e commit 6b53322
1 file changed
Lines changed: 4 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
492 | 492 | | |
493 | 493 | | |
494 | 494 | | |
495 | | - | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
496 | 499 | | |
497 | 500 | | |
498 | 501 | | |
| |||
0 commit comments