Repository navigation
fix(gb300): exclude c003 and c008 until their local NVMe is checked - #4821
Conversation
| exclusive: false | ||
| # 2026-11-05: c003's local NVMe extracted containers with zero-filled blocks (libcuda stub); | ||
| # its staged model copy is suspect too. Remove after ops checks the disk. | ||
| exclude: [im-gb300-r01-c003] |
There was a problem hiding this comment.
🟣 pre-existing, not blocking: pre-existing: OperatorX and CollectiveX GB300 jobs can still be placed on c003 and run its corrupted container extraction. This exclusion only reaches infx launches. Both tools run pyxis containers on cluster:gb300-nv but build their salloc from collectivex/configs/platform_config.json. That file's gb300.operator sets no exclude_nodes. Fix: quarantine c003 everywhere that submits pyxis jobs to this cluster. In this change, add "exclude_nodes": "im-gb300-r01-c003" to the gb300 operator block. Both operatorx/ci.py:560 and collectivex/launchers/launch_gb-nv.sh:48 already pass that key to Slurm as --exclude.
A small fix can ride a push you are already making; otherwise a short reply is enough.
Why this was flagged
Only the infx launcher reads slurm.exclude from runners.yaml, through infx/launch/backends/slurm/init.py:371-385 and :429-430 and infx/launch/drivers/srt/config.py:158-159. OperatorX (operatorx/ci.py:546-563) and CollectiveX (collectivex/launchers/launch_gb-nv.sh:43-49) allocate GB300 nodes with their own salloc. Their partition is batch_1, the same partition that holds the node with the bad NVMe. They take --exclude only from the operator.exclude_nodes field of collectivex/configs/platform_config.json, and the gb300 entry (lines 130-155) does not set it. Both tools then start pyxis containers on the allocated node (operatorx/ci.py:660-665 with --container-writable, collectivex/runtime/common.sh:1189). Per the PR description, c003's local /scratch enroot extraction zero-fills libcuda.so, libnvidia-ml.so and libcublas.so. So these jobs can still fail or measure the wrong thing on c003, exactly as on the base branch. The PR description says gb300-nv submissions exclude c003, but that holds only for infx-launched jobs.
Verification: The diff adds exclude: [im-gb300-r01-c003] only to the gb300-nv slurm: block in inferencex-e2e/configs/runners.yaml (lines 650-652). Only the infx launcher reads that block. In collectivex/configs/platform_config.json, the gb300 operator block has no exclude_nodes, so operatorx/ci.py:546-563 and collectivex/launchers/launch_gb-nv.sh:43-48 add no --exclude. The base branch had no exclusion anywhere, so merging makes nothing worse.
On im-gb300-r01-c003, pyxis/enroot extracted the SGLang K3.1 container onto local
/scratchwith zero-filled blocks. In job 43599 (#4802),libcuda.so,libnvidia-ml.soandlibcublas.soundercuda/lib64/stubshad the right sizes but contained only zeros. The same files on c004 and in the shared squash are valid ELF. As a result, SGLang's custom all-reduce JIT failed on c003's ranks only, and two #4802 jobs failed at model load there. The staged model under/scratch/modelson that disk is suspect for the same reason.im-gb300-r01-c008 failed two Kimi K3.1 vLLM evals (#4800) the same way: a decode rank read back an empty Triton
CompiledKernelmetadata file from the container-local cache on/scratch. Ranks with identical arguments on other nodes never hit this.gb300-nv submissions now exclude both nodes until ops checks their NVMe.