Skip to content

fix(mi355x): let host setup outlast the 30-minute VRAM drain wait - #4815

Merged
cquil11 merged 1 commit into
mainfrom
cam/mi355x-host-setup-timeout
Nov 5, 2026
Merged

cquil11 merged 1 commit into
mainfrom
cam/mi355x-host-setup-timeout

Conversation

@cquil11

@cquil11 cquil11 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

#4814 made the MI355X setup hook wait up to 30 minutes for prior-job VRAM to drain, but the cluster's host-setup timeout-s was still 1200 s. A node that drains slowly was therefore killed at 20 minutes as a setup timeout instead of getting the full wait. The timeout is now 2400 s, which covers the 30-minute wait plus the rest of setup.

@cquil11
cquil11 requested a review from a team November 5, 2026 02:54
@cquil11
cquil11 merged commit 0ad48c5 into main Nov 5, 2026
4 checks passed
@cquil11
cquil11 deleted the cam/mi355x-host-setup-timeout branch November 5, 2026 02:57

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

IBDEVICES: '@fabric.rdma-devices'
timeout-s: 1200
# Covers the hook's 30-minute wait for prior-job VRAM to drain.
timeout-s: 2400

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Multi-node MI355X jobs on a slowly draining node can now spend 30+ minutes of their one-hour Slurm limit in setup, then die as a Slurm TIMEOUT. The base failed them at the 20-minute setup timeout instead. The new timeout-s: 2400 lets wait_for_amd_gpu_clean 2 30 run to completion. That hook runs inside the allocation, and the mi355x-amds SRT_MULTI lane pins time_limit="01:00:00" (lanes.py:86). Fix: for multi-node runs, keep setup plus model load plus benchmark inside the Slurm limit, e.g. raise that lane's time_limit by the 40-minute setup budget.

Why this was flagged

The trigger is a multi-node mi355x-amds job (sglang-disagg, atom-disagg, tilert) that lands on a node still holding VRAM for 20-30 minutes. config.py:167-176 renders the cluster's host-setup into default_host_setup for single-node and multi-node jobs alike, with timeout_seconds=2400 and nodes: all. srt_time_limit (lanes.py:124-143) gives multi-node jobs the lane's time_limit="01:00:00" (lanes.py:86), because mi355x-amds sets no default-time-limit. The wait in common.sh:7-26 polls 180 times, with a 10 s sleep plus a rocm-smi call each time, so it can take about 31 minutes or more. That leaves under 29 minutes for multi-node model load, the benchmark and any eval before Slurm kills the job. On the base, the 1200 s timeout ended such jobs at 20 minutes and released the nodes. After this change they hold 2+ nodes for up to the full hour and can still fail mid-benchmark as a Slurm TIMEOUT.

Verification: runners.yaml:818 now sets timeout-s: 2400 with nodes: all, and wait_for_amd_gpu_clean 2 30 in common.sh:7-21 can last about 30 minutes. lanes.py:77-88 pins the mi355x-amds SRT_MULTI lane to time_limit="01:00:00". On the base, the same jobs already failed when the 1200 s setup timeout killed them at 20 minutes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant