Skip to content

fix(mi355x): wait for prior-job VRAM to drain below 5% before setup - #4813

Merged
cquil11 merged 1 commit into
mainfrom
cam/mi355x-gpu-clean-threshold
Nov 5, 2026
Merged

cquil11 merged 1 commit into
mainfrom
cam/mi355x-gpu-clean-threshold

Conversation

@cquil11

@cquil11 cquil11 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

After a Slurm job ends on mi355x-amds, its GPUs can keep 15–26 GiB of VRAM allocated for several minutes with no process attached. The setup hook's default 10% gate (about 29 GiB) passes that state, so the next job starts on a partly occupied node. A TP8 vLLM server at gpu-memory-utilization: 0.90 then fails its startup check (Free memory on device ... (251–259/287.98 GiB) on startup is less than desired GPU memory utilization (0.9, 259.19 GiB)). In the Kimi K3.1 MI355X vLLM sweep (#4806), this happened to four points that started 3–5 minutes after the previous job on the same node.

The MI355X hook now waits until per-GPU VRAM use is at most 5%. The 15-minute limit is unchanged.

@cquil11
cquil11 requested a review from a team November 5, 2026 01:20

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

# After a job ends, a node can hold 15-26 GiB per GPU for minutes with no process
# attached; 10% passes that, and a TP8 vLLM server at 0.90 then fails its startup
# free-memory check. 5% leaves room for the engine's own context.
wait_for_amd_gpu_clean 5

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The new gate still passes the 15 GiB residual this PR cites, so TP8 vLLM jobs at 0.90 can still hit the free-memory startup failure. wait_for_amd_gpu_clean compares rocm-smi's whole-number VRAM% with -le 5. On a 287.98 GiB GPU, 15 GiB is 5.2% and prints as 5, so it passes. Anything up to about 15.8 GiB (rounded) or 17.2 GiB (truncated) passes too. Fix: gate on used or free bytes instead (for example rocm-smi --showmeminfo vram), or use 4%, so every GPU holds at most about 14.4 GiB. That leaves the 14.4 GiB engine-context budget this comment relies on.

Why this was flagged

The trigger is a prior job leaving about 15-17 GiB per GPU with no process attached, which is the low end of the 15-26 GiB range in setup.sh:11. setup.sh:14 calls wait_for_amd_gpu_clean 5. common.sh:11-14 takes the integer from GPU Memory Allocated \(VRAM%\): [0-9]+ and checks -le "$threshold". 15 GiB / 287.98 GiB = 5.2%, which rocm-smi prints as 5 whether it rounds or truncates, so the wait ends at once. vLLM at 0.90 needs used memory <= 28.8 GiB, including its own context, when it starts. The PR's failing runs reported free memory of 251-259 GiB, so used was 29-37 GiB. That means a 15 GiB residual plus roughly 14 GiB of engine context still lands above 28.8 GiB. Those jobs fail the same way as on the base branch, even though the comment at setup.sh:13 says 5% leaves room for the engine's context. Nothing else in the hook re-checks memory, and the rocm-smi percent is too coarse for a gate this tight (1% is about 2.9 GiB).

Verification: nit: this fix is incomplete, not a regression. The base 10% gate already let these residuals through by the same route. The trigger is a prior job leaving about 15-15.8 GiB per GPU with no process attached. setup.sh:14 calls wait_for_amd_gpu_clean 5. 15 GiB / 287.98 GiB = 5.21%, which prints as 5 whether rocm-smi rounds or truncates, so 5 -le 5 passes on the first poll.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant