Skip to content

fix(mi355x): also exclude g14, which keeps 7% VRAM after a job - #4824

Merged
cquil11 merged 1 commit into
mainfrom
cam/mi355x-exclude-g14
Nov 5, 2026
Merged

cquil11 merged 1 commit into
mainfrom
cam/mi355x-exclude-g14

Conversation

@cquil11

@cquil11 cquil11 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #4819. mia1-p01-g14 now shows the same plateau as g09 and g16: after a job ends, VRAM stays at 7% per GPU with no process attached. A Kimi K3.1 eval (#4808, Slurm 52773) is stuck in the setup hook's drain gate there. mi355x-amds submissions now exclude g14 as well, until the three nodes get a GPU reset.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@cquil11
cquil11 merged commit b7773f0 into main Nov 5, 2026
8 checks passed
@cquil11
cquil11 deleted the cam/mi355x-exclude-g14 branch November 5, 2026 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant