[cquil11]

Follow-up to #4814/#4815. After a vLLM job ends on mi355x-amds, its node can keep 3–7% of VRAM per GPU allocated with no process attached for more than 30 minutes. It drains eventually (g09 and g16 later read 0%), but the 30-minute gate fails the next job first. A Kimi K3.1 SGLang job (#4808, Slurm 52795) on g50 read 7% → 4% → 3% over 30 minutes and then failed setup. It has now happened on five nodes, each right after a vLLM job.

The MI355X hook now waits up to 60 minutes, and the host-setup timeout rises from 2400 s to 4200 s to cover the wait. The 2% threshold is unchanged: at 3–7% leftover, SGLang caps max_running_requests below the high-concurrency points and vLLM fails its startup free-memory check.

[claude]

🔴 Multi-node MI355X jobs can now spend their whole 1-hour Slurm allocation waiting for VRAM to drain, then get killed with no benchmark run. The mi355x-amds SRT_MULTI lane sets time_limit="01:00:00" (lanes.py:86), but the hook now waits up to 60 minutes and timeout-s is 4200. Slurm kills the job before the hook timeout can fire. Fix: keep the drain wait plus the benchmark inside every lane's limit. Either raise the multi-node lane time_limit by the extra 30 minutes, or keep the 60-minute wait for single-node only.

Why this was flagged

The multi-node lane for mi355x-amds sets time_limit="01:00:00" (infx/launch/drivers/srt/lanes.py:86). srt_time_limit returns it (lanes.py:137-138), and render() writes it to default_time_limit (config.py:120). The same cluster-wide host-setup hook with timeout_seconds runs on every node (config.py:168-176, nodes: all). After this change, a node holding 3-7% VRAM after a vLLM job makes wait_for_amd_gpu_clean 2 60 (setup.sh:15) poll for up to 60 minutes, and check-rdma.sh runs before that. A node that drains after 30-55 minutes passes setup with only minutes left, so the benchmark is killed by Slurm TIMEOUT partway through. A node that does not drain uses the full hour of a multi-node allocation and hits Slurm TIMEOUT instead of the clear "GPUs still draining" error. On the base branch the 30-minute gate failed after 30 minutes. timeout-s: 4200 cannot help here because it is longer than the 3600 s Slurm limit. The single-node path (single-node-time-limit: 500) has room for the wait, so only multi-node is affected.

Verification: The multi-node lane sets time_limit="01:00:00" (infx/launch/drivers/srt/lanes.py:86), but the diff raises the wait to wait_for_amd_gpu_clean 2 60 (setup.sh:15) and timeout-s: 4200 (runners.yaml:824). On the base, an undrained node failed at 30 minutes with its own error. Now Slurm hits the 60-minute limit first. A 4200 s timeout inside a 3600 s allocation can never fire.

[Duyi-Wang]

It looks like has been fixed in torvalds/linux@c178e53 and need to upgrade DKMS @cquil11