Skip to content

ci: add --bind-to none to Phoenix syscheck smoke-test (fixes GPU-node MPI bind failure) - #1834

Merged
sbryngelson merged 1 commit into
masterfrom
ci-fix-syscheck-bind-to-none
Sep 8, 2026
Merged

ci: add --bind-to none to Phoenix syscheck smoke-test (fixes GPU-node MPI bind failure)#1834
sbryngelson merged 1 commit into
masterfrom
ci-fix-syscheck-bind-to-none

Conversation

@sbryngelson

Copy link
Copy Markdown
Member

Symptom

Post-merge, Phoenix Test Suite jobs began failing (runs 926a3a5, 5829daf) with:

hwloc_set_cpubind returned "Error" for bitmap "0"   (orte/mca/rtc/hwloc/rtc_hwloc.c:382)
Open MPI tried to bind a new process ... process was killed without launching
Application name: build/install/gpu-acc-*/bin/syscheck

Root cause

build.sh's post-build syscheck smoke-test launched mpirun -np 1 "$syscheck_bin" without --bind-to none. On a busy GPU node SLURM hands the job an offset/partial cpuset (e.g. cores 32-47), and Open MPI's default --bind-to core then fails at hwloc_set_cpubind before the binary starts.

This was a latent bug: every other MPI launch already passes --bind-to none — the .mako run templates, coverage_build.py, and preflight.sh. This smoke-test was the lone omission. It stayed hidden while test jobs landed on idle nodes; #1833's partition-list change let them backfill onto shared GPU nodes with awkward cpusets, exposing it.

Evidence

The failing CI job already contains the perfect A/B — same node, same binary, -np 1:

Launch Command Result
preflight.sh mpirun --bind-to none -np 1 syscheck passed
build.sh smoke-test mpirun -np 1 syscheck failed (hwloc_set_cpubind)

Reproduced directly on multiple, distinct GPU nodes (h100-HGX atl1-1-03-008-22-0, h200 atl1-1-02-012-23-0 — neither is a CI node), confirming it is not node-specific:

mpirun -np 2/4 hostname            (default)   -> rc=131 "binding generic error"
mpirun --bind-to none -np 2/4 …    (suppressed) -> rc=0

A CPU node with an equally fragmented cpuset did not reproduce — it is specific to the GPU-node topology.

Fix

Add --bind-to none to the smoke-test mpirun, matching every other launch in the repo. One line.

Verification

  • bash -n and python3 toolchain/mfc/lint_source.py pass locally.
  • Grep confirms this was the only bare mpirun launch left in .github/.

Copilot AI lite review requested due to automatic review settings September 8, 2026 03:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, CI-scoped, and matches existing MPI launch behavior to address a clearly identified failure mode.

Pull request overview

This PR fixes Phoenix CI syscheck smoke-test failures on shared GPU nodes by aligning the mpirun invocation in the build workflow with the repository’s existing MPI launch pattern (--bind-to none), preventing Open MPI from failing early during CPU binding on fragmented SLURM cpusets.

Changes:

  • Add --bind-to none to the Phoenix post-build syscheck smoke-test mpirun command.
  • Expand the surrounding comment to document why binding must be disabled on busy GPU nodes.
File summaries
File Description
.github/workflows/common/build.sh Updates the Phoenix syscheck smoke-test to use mpirun --bind-to none to avoid hwloc/CPU-bind failures under SLURM cpusets.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.35%. Comparing base (b44c811) to head (1bcf036).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1834   +/-   ##
=======================================
  Coverage   62.35%   62.35%           
=======================================
  Files          84       84           
  Lines       21621    21621           
  Branches     3199     3199           
=======================================
  Hits        13482    13482           
  Misses       5939     5939           
  Partials     2200     2200           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sbryngelson
sbryngelson merged commit 0c849aa into master Sep 8, 2026
147 of 148 checks passed
@sbryngelson
sbryngelson deleted the ci-fix-syscheck-bind-to-none branch September 8, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants