Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ bash "$(dirname "${BASH_SOURCE[0]}")/check-rdma.sh"
# the next model load fail much later with a misleading OOM.
# shellcheck source=runners/srt-slurm/hooks/common.sh
source "$(dirname "${BASH_SOURCE[0]}")/../common.sh"
wait_for_amd_gpu_clean
# After a job ends, a node can hold 15-26 GiB per GPU for minutes with no process
# attached; 10% passes that, and a TP8 vLLM server at 0.90 then fails its startup
# free-memory check. 5% leaves room for the engine's own context.
wait_for_amd_gpu_clean 5

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The new gate still passes the 15 GiB residual this PR cites, so TP8 vLLM jobs at 0.90 can still hit the free-memory startup failure. wait_for_amd_gpu_clean compares rocm-smi's whole-number VRAM% with -le 5. On a 287.98 GiB GPU, 15 GiB is 5.2% and prints as 5, so it passes. Anything up to about 15.8 GiB (rounded) or 17.2 GiB (truncated) passes too. Fix: gate on used or free bytes instead (for example rocm-smi --showmeminfo vram), or use 4%, so every GPU holds at most about 14.4 GiB. That leaves the 14.4 GiB engine-context budget this comment relies on.

Why this was flagged

The trigger is a prior job leaving about 15-17 GiB per GPU with no process attached, which is the low end of the 15-26 GiB range in setup.sh:11. setup.sh:14 calls wait_for_amd_gpu_clean 5. common.sh:11-14 takes the integer from GPU Memory Allocated \(VRAM%\): [0-9]+ and checks -le "$threshold". 15 GiB / 287.98 GiB = 5.2%, which rocm-smi prints as 5 whether it rounds or truncates, so the wait ends at once. vLLM at 0.90 needs used memory <= 28.8 GiB, including its own context, when it starts. The PR's failing runs reported free memory of 251-259 GiB, so used was 29-37 GiB. That means a 15 GiB residual plus roughly 14 GiB of engine context still lands above 28.8 GiB. Those jobs fail the same way as on the base branch, even though the comment at setup.sh:13 says 5% leaves room for the engine's context. Nothing else in the hook re-checks memory, and the rocm-smi percent is too coarse for a gate this tight (1% is about 2.9 GiB).

Verification: nit: this fix is incomplete, not a regression. The base 10% gate already let these residuals through by the same route. The trigger is a prior job leaving about 15-15.8 GiB per GPU with no process attached. setup.sh:14 calls wait_for_amd_gpu_clean 5. 15 GiB / 287.98 GiB = 5.21%, which prints as 5 whether rocm-smi rounds or truncates, so 5 -le 5 passes on the first poll.


# Some MI355X experiments reserve large 2 MiB HugeTLB pools and leave the
# reservation behind after their Slurm allocation exits. Those free hugepages
Expand Down
Loading