Skip to content

fix(gb300): exclude c003 and c008 until their local NVMe is checked - #4821

Merged
cquil11 merged 2 commits into
mainfrom
cam/gb300-exclude-c003
Nov 5, 2026
Merged

cquil11 merged 2 commits into
mainfrom
cam/gb300-exclude-c003

Conversation

@cquil11

@cquil11 cquil11 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

On im-gb300-r01-c003, pyxis/enroot extracted the SGLang K3.1 container onto local /scratch with zero-filled blocks. In job 43599 (#4802), libcuda.so, libnvidia-ml.so and libcublas.so under cuda/lib64/stubs had the right sizes but contained only zeros. The same files on c004 and in the shared squash are valid ELF. As a result, SGLang's custom all-reduce JIT failed on c003's ranks only, and two #4802 jobs failed at model load there. The staged model under /scratch/models on that disk is suspect for the same reason.

im-gb300-r01-c008 failed two Kimi K3.1 vLLM evals (#4800) the same way: a decode rank read back an empty Triton CompiledKernel metadata file from the container-local cache on /scratch. Ranks with identical arguments on other nodes never hit this.

gb300-nv submissions now exclude both nodes until ops checks their NVMe.

@cquil11
cquil11 requested a review from a team November 5, 2026 04:05
@cquil11 cquil11 changed the title fix(gb300): exclude c003 until its local NVMe is checked fix(gb300): exclude c003 and c008 until their local NVMe is checked Nov 5, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread inferencex-e2e/configs/runners.yaml Outdated
exclusive: false
# 2026-11-05: c003's local NVMe extracted containers with zero-filled blocks (libcuda stub);
# its staged model copy is suspect too. Remove after ops checks the disk.
exclude: [im-gb300-r01-c003]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟣 pre-existing, not blocking: pre-existing: OperatorX and CollectiveX GB300 jobs can still be placed on c003 and run its corrupted container extraction. This exclusion only reaches infx launches. Both tools run pyxis containers on cluster:gb300-nv but build their salloc from collectivex/configs/platform_config.json. That file's gb300.operator sets no exclude_nodes. Fix: quarantine c003 everywhere that submits pyxis jobs to this cluster. In this change, add "exclude_nodes": "im-gb300-r01-c003" to the gb300 operator block. Both operatorx/ci.py:560 and collectivex/launchers/launch_gb-nv.sh:48 already pass that key to Slurm as --exclude.
A small fix can ride a push you are already making; otherwise a short reply is enough.

Why this was flagged

Only the infx launcher reads slurm.exclude from runners.yaml, through infx/launch/backends/slurm/init.py:371-385 and :429-430 and infx/launch/drivers/srt/config.py:158-159. OperatorX (operatorx/ci.py:546-563) and CollectiveX (collectivex/launchers/launch_gb-nv.sh:43-49) allocate GB300 nodes with their own salloc. Their partition is batch_1, the same partition that holds the node with the bad NVMe. They take --exclude only from the operator.exclude_nodes field of collectivex/configs/platform_config.json, and the gb300 entry (lines 130-155) does not set it. Both tools then start pyxis containers on the allocated node (operatorx/ci.py:660-665 with --container-writable, collectivex/runtime/common.sh:1189). Per the PR description, c003's local /scratch enroot extraction zero-fills libcuda.so, libnvidia-ml.so and libcublas.so. So these jobs can still fail or measure the wrong thing on c003, exactly as on the base branch. The PR description says gb300-nv submissions exclude c003, but that holds only for infx-launched jobs.

Verification: The diff adds exclude: [im-gb300-r01-c003] only to the gb300-nv slurm: block in inferencex-e2e/configs/runners.yaml (lines 650-652). Only the infx launcher reads that block. In collectivex/configs/platform_config.json, the gb300 operator block has no exclude_nodes, so operatorx/ci.py:546-563 and collectivex/launchers/launch_gb-nv.sh:43-48 add no --exclude. The base branch had no exclusion anywhere, so merging makes nothing worse.

@cquil11
cquil11 merged commit e1c78e0 into main Nov 5, 2026
12 checks passed
@cquil11
cquil11 deleted the cam/gb300-exclude-c003 branch November 5, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant