Skip to content

Commit a4a5d3c

Browse files
authored
ci(phoenix): run the post-build syscheck under mpirun (#1728)
The post-build smoke test launched the syscheck binary directly. Since the August 11-13 PACE maintenance, which upgraded Phoenix to Slurm 26.05.2 with PMIx 4.2.6, an OpenMPI 4.1.5 binary started bare inside an allocation reads the PMIx environment as an srun launch and aborts in MPI_Init: OPAL ERROR: Unreachable in file ext3x_client.c at line 112 The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. Every Phoenix job then failed validation twice and exited 1 before the test step ran, blocking all pull requests. mpirun is unaffected, and it is how MFC launches every other binary, so the smoke test now goes through it. The guard still catches what it was added for: a binary compiled for the wrong microarchitecture still dies on SIGILL and mpirun propagates the failure. Output is no longer sent to /dev/null, so the next failure is readable in the CI log instead of appearing as a bare "Post-build validation failed".
1 parent bb29773 commit a4a5d3c

1 file changed

Lines changed: 8 additions & 1 deletion

File tree

.github/workflows/common/build.sh

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,16 @@ source .github/scripts/retry-build.sh
3434

3535
# Phoenix: smoke-test the syscheck binary to catch architecture mismatches
3636
# (SIGILL from binaries compiled on a different compute node).
37+
#
38+
# Launch under mpirun rather than directly. Phoenix's openmpi/4.1.5 predates the
39+
# PMIx 4.2.6 that came with the Slurm 26.05.2 upgrade, so an MPI binary started
40+
# bare inside an allocation misreads the PMIx environment as an srun launch and
41+
# aborts in MPI_Init ("OPAL ERROR: Unreachable in file ext3x_client.c"). mpirun
42+
# is unaffected, and it is how MFC launches every binary anyway. Output is left
43+
# on stdout so a future failure is diagnosable from the CI log.
3744
validate_cmd=""
3845
if [ "$job_cluster" = "phoenix" ]; then
39-
validate_cmd='syscheck_bin=$(find build/install -name syscheck -type f 2>/dev/null | head -1); [ -z "$syscheck_bin" ] || "$syscheck_bin" > /dev/null 2>&1'
46+
validate_cmd='syscheck_bin=$(find build/install -name syscheck -type f 2>/dev/null | head -1); [ -z "$syscheck_bin" ] || mpirun -np 1 "$syscheck_bin"'
4047
fi
4148

4249
# --- Variant selection ---

0 commit comments

Comments
 (0)