Repository navigation
fix(mi355x): wait for prior-job VRAM to drain below 5% before setup - #4813
Conversation
| # After a job ends, a node can hold 15-26 GiB per GPU for minutes with no process | ||
| # attached; 10% passes that, and a TP8 vLLM server at 0.90 then fails its startup | ||
| # free-memory check. 5% leaves room for the engine's own context. | ||
| wait_for_amd_gpu_clean 5 |
There was a problem hiding this comment.
🟡 (optional) The new gate still passes the 15 GiB residual this PR cites, so TP8 vLLM jobs at 0.90 can still hit the free-memory startup failure. wait_for_amd_gpu_clean compares rocm-smi's whole-number VRAM% with -le 5. On a 287.98 GiB GPU, 15 GiB is 5.2% and prints as 5, so it passes. Anything up to about 15.8 GiB (rounded) or 17.2 GiB (truncated) passes too. Fix: gate on used or free bytes instead (for example rocm-smi --showmeminfo vram), or use 4%, so every GPU holds at most about 14.4 GiB. That leaves the 14.4 GiB engine-context budget this comment relies on.
Why this was flagged
The trigger is a prior job leaving about 15-17 GiB per GPU with no process attached, which is the low end of the 15-26 GiB range in setup.sh:11. setup.sh:14 calls wait_for_amd_gpu_clean 5. common.sh:11-14 takes the integer from GPU Memory Allocated \(VRAM%\): [0-9]+ and checks -le "$threshold". 15 GiB / 287.98 GiB = 5.2%, which rocm-smi prints as 5 whether it rounds or truncates, so the wait ends at once. vLLM at 0.90 needs used memory <= 28.8 GiB, including its own context, when it starts. The PR's failing runs reported free memory of 251-259 GiB, so used was 29-37 GiB. That means a 15 GiB residual plus roughly 14 GiB of engine context still lands above 28.8 GiB. Those jobs fail the same way as on the base branch, even though the comment at setup.sh:13 says 5% leaves room for the engine's context. Nothing else in the hook re-checks memory, and the rocm-smi percent is too coarse for a gate this tight (1% is about 2.9 GiB).
Verification: nit: this fix is incomplete, not a regression. The base 10% gate already let these residuals through by the same route. The trigger is a prior job leaving about 15-15.8 GiB per GPU with no process attached. setup.sh:14 calls wait_for_amd_gpu_clean 5. 15 GiB / 287.98 GiB = 5.21%, which prints as 5 whether rocm-smi rounds or truncates, so 5 -le 5 passes on the first poll.
After a Slurm job ends on mi355x-amds, its GPUs can keep 15–26 GiB of VRAM allocated for several minutes with no process attached. The setup hook's default 10% gate (about 29 GiB) passes that state, so the next job starts on a partly occupied node. A TP8 vLLM server at
gpu-memory-utilization: 0.90then fails its startup check (Free memory on device ... (251–259/287.98 GiB) on startup is less than desired GPU memory utilization (0.9, 259.19 GiB)). In the Kimi K3.1 MI355X vLLM sweep (#4806), this happened to four points that started 3–5 minutes after the previous job on the same node.The MI355X hook now waits until per-GPU VRAM use is at most 5%. The 15-minute limit is unchanged.