Repository navigation
fix(mi355x): start jobs only once prior-job VRAM has drained - #4814
Conversation
| # process attached. vLLM then fails its startup free-memory check, and SGLang | ||
| # silently sizes a smaller mamba cache and caps max_running_requests below the | ||
| # benchmark's concurrency. Clean nodes read 0%, so wait for that. | ||
| wait_for_amd_gpu_clean 2 30 |
There was a problem hiding this comment.
🔴 On MI355X jobs, the new 30-minute VRAM wait never runs to the end. The host-setup timeout kills the hook at 20 minutes, so nodes that drain after minute 20 still fail. wait_for_amd_gpu_clean 2 30 can poll for up to 1800s. The mi355x-amds host-setup in inferencex-e2e/configs/runners.yaml:817 sets timeout-s: 1200, and check-rdma.sh also runs inside that budget first. Fix: make the hook timeout cover the full wait plus the RDMA check and hugepage steps (e.g. raise timeout-s to about 2100), or keep minutes below the timeout.
Why this was flagged
Trigger: a mi355x-amds node whose leftover VRAM takes more than about 20 minutes to drop to 2% or below. The PR description says this can take up to ~30 minutes. setup.sh:15 now calls wait_for_amd_gpu_clean 2 30, which runs $((minutes * 6)) = 180 polls 10s apart (common.sh:10), about 1800s. The host-setup entry at inferencex-e2e/configs/runners.yaml:813-818 still has timeout-s: 1200. config.py:173 passes it to srtctl as timeout_seconds, so the hook is killed at 1200s, before the loop finishes. The operator then gets a host-setup timeout instead of the hook's own "Error: GPUs still draining prior job's memory after 30min" message. The stated 30-minute drain window is really just under 20 minutes, because check-rdma.sh (setup.sh:4) uses part of the same budget. On the base branch the 15-minute (900s) wait fit inside 1200s, so a slow drain reached the function's own clear error.
Verification: setup.sh:15 now calls wait_for_amd_gpu_clean 2 30, so it can poll for 180 × 10s = 1800s. inferencex-e2e/configs/runners.yaml:813-818 leaves the mi355x-amds host-setup at timeout-s: 1200. A node that drains between about minute 20 and minute 30 still fails host setup. Before this PR the wait was 900s, which fit inside 1200s, so nothing regresses.
Follow-up to #4813. The leftover VRAM on mi355x-amds after a job is 15–26 GiB per GPU and can take up to about 30 minutes to drain. Even below 5% it changes what SGLang measures, and SGLang gives no error: on the Kimi K3.1 MI355X SGLang sweep (#4808), the HiCache points started 31 s to 6 min after the previous job, at 7–8% VRAM. With only 53–66 GB free per GPU after weights (about 90 GB on a clean GPU), SGLang sized a smaller mamba state cache and capped
max_running_requestsat 31 for c52 and 22 for c60. vLLM fails such a start loudly; SGLang benchmarks it at the wrong concurrency.The MI355X hook now waits until VRAM is at most 2% (clean nodes read 0%), for up to 30 minutes.
wait_for_amd_gpu_cleantakes the wait in minutes as an optional second argument, defaulting to 15.