ci(phoenix): run the post-build syscheck under mpirun

[sbryngelson]

Description

Every Phoenix job has failed on every PR since the cluster came back from the August 11-13 PACE maintenance, blocking the queue. The failure is in CI, not in MFC.

The post-build smoke test in .github/workflows/common/build.sh launches the syscheck binary directly:

validate_cmd='syscheck_bin=$(find build/install -name syscheck -type f 2>/dev/null | head -1); [ -z "$syscheck_bin" ] || "$syscheck_bin" > /dev/null 2>&1'

The maintenance upgraded Phoenix to Slurm 26.05.2 with PMIx 4.2.6. Phoenix's openmpi/4.1.5 predates that, so an MPI binary started bare inside an allocation now reads the PMIx environment as an srun launch and aborts in MPI_Init:

OPAL ERROR: Unreachable in file ext3x_client.c at line 112
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot execute.
*** An error occurred in MPI_Init

retry_build then nukes the build directory, rebuilds, fails validation a second time, and exits 1 before the test step ever runs.

This changes the smoke test to launch through mpirun, which is unaffected and is how MFC launches every other binary. The guard still does what it was added for in #1304: a binary compiled for the wrong microarchitecture still dies on SIGILL, and mpirun propagates the non-zero exit.

Output is no longer redirected to /dev/null. That redirection is why this presented as a bare Post-build validation failed on attempt 1. with no cause in the log.

Type of change

• Bug fix

Testing

Reproduced and verified on a Phoenix compute node with the CPU CI modules (. ./mfc.sh load -c p -m c, gcc/12.3.0 + openmpi/4.1.5), on a clean build of syscheck:

$ ./build/install/cpu-*/bin/syscheck; echo "rc=$?"
 [TEST] MPI: call mpi_init(ierr)
[...] OPAL ERROR: Unreachable in file ext3x_client.c at line 112
*** An error occurred in MPI_Init
rc=1

$ mpirun -np 1 ./build/install/cpu-*/bin/syscheck; echo "rc=$?"
 [TEST] MPI: call mpi_init(ierr)
 [...]
 Syscheck: PASSED.
rc=0

Corroborating evidence that only the bare launch is broken: the Case Opt Phoenix jobs pass on the same runs that fail here, and they run mpirun -np 2 .../bin/syscheck to Syscheck: PASSED. followed by five full pre_process/simulation/post_process cases.

./mfc.sh precheck passes (all seven lint-gate checks).

Checklist

• I added or updated tests for new behavior
• I updated documentation if user-facing behavior changed

Notes for follow-up (not in this PR)

Two unrelated things surfaced while tracking this down:

1. cmake/MFCTargets.cmake:294 links CUDA::nvToolsExt on the CUDA <= 12.8 branch. CUDA 12.9 removed that shared library (NVTX v3 is header-only), and an NVHPC-built tree here carries a direct NEEDED libnvToolsExt.so.1. Worth checking against the upgraded CUDA on Phoenix.
2. Target.get_slug() in toolchain/mfc/build.py:301 hashes the target name, config slug, and fpp, but not the compiler. A gcc build and an nvfortran build of the same config both land in build/install/cpu-<hash>. CI hides this by wiping build/ each Phoenix job; it does bite locally.

[Copilot]

Pull request overview

This PR updates the Phoenix CI post-build smoke validation to run the syscheck MPI binary via mpirun instead of launching it directly, addressing failures caused by the recent Phoenix Slurm/PMIx upgrade and improving diagnosability by keeping output in the CI logs.

Changes:

• Run the post-build syscheck smoke test under mpirun -np 1 on Phoenix to avoid bare-launch MPI_Init failures.
• Stop redirecting syscheck output to /dev/null so failures are visible in CI logs.
• Document the Slurm/PMIx/OpenMPI interaction that motivated the change.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.