ci: add --bind-to none to Phoenix syscheck smoke-test (fixes GPU-node MPI bind failure) - #1834
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, CI-scoped, and matches existing MPI launch behavior to address a clearly identified failure mode.
Pull request overview
This PR fixes Phoenix CI syscheck smoke-test failures on shared GPU nodes by aligning the mpirun invocation in the build workflow with the repository’s existing MPI launch pattern (--bind-to none), preventing Open MPI from failing early during CPU binding on fragmented SLURM cpusets.
Changes:
- Add
--bind-to noneto the Phoenix post-buildsyschecksmoke-testmpiruncommand. - Expand the surrounding comment to document why binding must be disabled on busy GPU nodes.
File summaries
| File | Description |
|---|---|
| .github/workflows/common/build.sh | Updates the Phoenix syscheck smoke-test to use mpirun --bind-to none to avoid hwloc/CPU-bind failures under SLURM cpusets. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1834 +/- ##
=======================================
Coverage 62.35% 62.35%
=======================================
Files 84 84
Lines 21621 21621
Branches 3199 3199
=======================================
Hits 13482 13482
Misses 5939 5939
Partials 2200 2200 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Symptom
Post-merge, Phoenix
Test Suitejobs began failing (runs926a3a5,5829daf) with:Root cause
build.sh's post-build syscheck smoke-test launchedmpirun -np 1 "$syscheck_bin"without--bind-to none. On a busy GPU node SLURM hands the job an offset/partial cpuset (e.g. cores32-47), and Open MPI's default--bind-to corethen fails athwloc_set_cpubindbefore the binary starts.This was a latent bug: every other MPI launch already passes
--bind-to none— the.makorun templates,coverage_build.py, andpreflight.sh. This smoke-test was the lone omission. It stayed hidden while test jobs landed on idle nodes; #1833's partition-list change let them backfill onto shared GPU nodes with awkward cpusets, exposing it.Evidence
The failing CI job already contains the perfect A/B — same node, same binary,
-np 1:mpirun --bind-to none -np 1 syscheckmpirun -np 1 syscheckhwloc_set_cpubind)Reproduced directly on multiple, distinct GPU nodes (h100-HGX
atl1-1-03-008-22-0, h200atl1-1-02-012-23-0— neither is a CI node), confirming it is not node-specific:A CPU node with an equally fragmented cpuset did not reproduce — it is specific to the GPU-node topology.
Fix
Add
--bind-to noneto the smoke-testmpirun, matching every other launch in the repo. One line.Verification
bash -nandpython3 toolchain/mfc/lint_source.pypass locally.mpirunlaunch left in.github/.