Repository navigation
fix(mi355x): let host setup outlast the 30-minute VRAM drain wait - #4815
Conversation
| IBDEVICES: '@fabric.rdma-devices' | ||
| timeout-s: 1200 | ||
| # Covers the hook's 30-minute wait for prior-job VRAM to drain. | ||
| timeout-s: 2400 |
There was a problem hiding this comment.
🟡 (optional) Multi-node MI355X jobs on a slowly draining node can now spend 30+ minutes of their one-hour Slurm limit in setup, then die as a Slurm TIMEOUT. The base failed them at the 20-minute setup timeout instead. The new timeout-s: 2400 lets wait_for_amd_gpu_clean 2 30 run to completion. That hook runs inside the allocation, and the mi355x-amds SRT_MULTI lane pins time_limit="01:00:00" (lanes.py:86). Fix: for multi-node runs, keep setup plus model load plus benchmark inside the Slurm limit, e.g. raise that lane's time_limit by the 40-minute setup budget.
Why this was flagged
The trigger is a multi-node mi355x-amds job (sglang-disagg, atom-disagg, tilert) that lands on a node still holding VRAM for 20-30 minutes. config.py:167-176 renders the cluster's host-setup into default_host_setup for single-node and multi-node jobs alike, with timeout_seconds=2400 and nodes: all. srt_time_limit (lanes.py:124-143) gives multi-node jobs the lane's time_limit="01:00:00" (lanes.py:86), because mi355x-amds sets no default-time-limit. The wait in common.sh:7-26 polls 180 times, with a 10 s sleep plus a rocm-smi call each time, so it can take about 31 minutes or more. That leaves under 29 minutes for multi-node model load, the benchmark and any eval before Slurm kills the job. On the base, the 1200 s timeout ended such jobs at 20 minutes and released the nodes. After this change they hold 2+ nodes for up to the full hour and can still fail mid-benchmark as a Slurm TIMEOUT.
Verification: runners.yaml:818 now sets timeout-s: 2400 with nodes: all, and wait_for_amd_gpu_clean 2 30 in common.sh:7-21 can last about 30 minutes. lanes.py:77-88 pins the mi355x-amds SRT_MULTI lane to time_limit="01:00:00". On the base, the same jobs already failed when the 1200 s setup timeout killed them at 20 minutes.
#4814 made the MI355X setup hook wait up to 30 minutes for prior-job VRAM to drain, but the cluster's host-setup
timeout-swas still 1200 s. A node that drains slowly was therefore killed at 20 minutes as a setup timeout instead of getting the full wait. The timeout is now 2400 s, which covers the 30-minute wait plus the rest of setup.