Skip to content

fix(mi355x): start jobs only once prior-job VRAM has drained - #4814

Merged
cquil11 merged 1 commit into
mainfrom
cam/mi355x-gpu-clean-strict
Nov 5, 2026
Merged

cquil11 merged 1 commit into
mainfrom
cam/mi355x-gpu-clean-strict

Conversation

@cquil11

@cquil11 cquil11 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #4813. The leftover VRAM on mi355x-amds after a job is 15–26 GiB per GPU and can take up to about 30 minutes to drain. Even below 5% it changes what SGLang measures, and SGLang gives no error: on the Kimi K3.1 MI355X SGLang sweep (#4808), the HiCache points started 31 s to 6 min after the previous job, at 7–8% VRAM. With only 53–66 GB free per GPU after weights (about 90 GB on a clean GPU), SGLang sized a smaller mamba state cache and capped max_running_requests at 31 for c52 and 22 for c60. vLLM fails such a start loudly; SGLang benchmarks it at the wrong concurrency.

The MI355X hook now waits until VRAM is at most 2% (clean nodes read 0%), for up to 30 minutes. wait_for_amd_gpu_clean takes the wait in minutes as an optional second argument, defaulting to 15.

@cquil11
cquil11 requested a review from a team November 5, 2026 01:27
@cquil11
cquil11 merged commit 522f0e9 into main Nov 5, 2026
2 checks passed
@cquil11
cquil11 deleted the cam/mi355x-gpu-clean-strict branch November 5, 2026 01:28
# process attached. vLLM then fails its startup free-memory check, and SGLang
# silently sizes a smaller mamba cache and caps max_running_requests below the
# benchmark's concurrency. Clean nodes read 0%, so wait for that.
wait_for_amd_gpu_clean 2 30

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 On MI355X jobs, the new 30-minute VRAM wait never runs to the end. The host-setup timeout kills the hook at 20 minutes, so nodes that drain after minute 20 still fail. wait_for_amd_gpu_clean 2 30 can poll for up to 1800s. The mi355x-amds host-setup in inferencex-e2e/configs/runners.yaml:817 sets timeout-s: 1200, and check-rdma.sh also runs inside that budget first. Fix: make the hook timeout cover the full wait plus the RDMA check and hugepage steps (e.g. raise timeout-s to about 2100), or keep minutes below the timeout.

Why this was flagged

Trigger: a mi355x-amds node whose leftover VRAM takes more than about 20 minutes to drop to 2% or below. The PR description says this can take up to ~30 minutes. setup.sh:15 now calls wait_for_amd_gpu_clean 2 30, which runs $((minutes * 6)) = 180 polls 10s apart (common.sh:10), about 1800s. The host-setup entry at inferencex-e2e/configs/runners.yaml:813-818 still has timeout-s: 1200. config.py:173 passes it to srtctl as timeout_seconds, so the hook is killed at 1200s, before the loop finishes. The operator then gets a host-setup timeout instead of the hook's own "Error: GPUs still draining prior job's memory after 30min" message. The stated 30-minute drain window is really just under 20 minutes, because check-rdma.sh (setup.sh:4) uses part of the same budget. On the base branch the 15-minute (900s) wait fit inside 1200s, so a slow drain reached the function's own clear error.

Verification: setup.sh:15 now calls wait_for_amd_gpu_clean 2 30, so it can poll for 180 × 10s = 1800s. inferencex-e2e/configs/runners.yaml:813-818 leaves the mi355x-amds host-setup at timeout-s: 1200. A node that drains between about minute 20 and minute 30 still fails host setup. Before this PR the wait was 900s, which fit inside 1200s, so nothing regresses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant