From 2ae80855f20b0fb918d781638cad374ee0b5181a Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 15 Aug 2026 18:16:12 -0400 Subject: [PATCH] ci(phoenix): run the post-build syscheck under mpirun The post-build smoke test launched the syscheck binary directly. Since the August 11-13 PACE maintenance, which upgraded Phoenix to Slurm 26.05.2 with PMIx 4.2.6, an OpenMPI 4.1.5 binary started bare inside an allocation reads the PMIx environment as an srun launch and aborts in MPI_Init: OPAL ERROR: Unreachable in file ext3x_client.c at line 112 The application appears to have been direct launched using "srun", but OMPI was not built with SLURM's PMI support and therefore cannot execute. Every Phoenix job then failed validation twice and exited 1 before the test step ran, blocking all pull requests. mpirun is unaffected, and it is how MFC launches every other binary, so the smoke test now goes through it. The guard still catches what it was added for: a binary compiled for the wrong microarchitecture still dies on SIGILL and mpirun propagates the failure. Output is no longer sent to /dev/null, so the next failure is readable in the CI log instead of appearing as a bare "Post-build validation failed". --- .github/workflows/common/build.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/common/build.sh b/.github/workflows/common/build.sh index b05a2a49b3..45c58ae0e7 100755 --- a/.github/workflows/common/build.sh +++ b/.github/workflows/common/build.sh @@ -34,9 +34,16 @@ source .github/scripts/retry-build.sh # Phoenix: smoke-test the syscheck binary to catch architecture mismatches # (SIGILL from binaries compiled on a different compute node). +# +# Launch under mpirun rather than directly. Phoenix's openmpi/4.1.5 predates the +# PMIx 4.2.6 that came with the Slurm 26.05.2 upgrade, so an MPI binary started +# bare inside an allocation misreads the PMIx environment as an srun launch and +# aborts in MPI_Init ("OPAL ERROR: Unreachable in file ext3x_client.c"). mpirun +# is unaffected, and it is how MFC launches every binary anyway. Output is left +# on stdout so a future failure is diagnosable from the CI log. validate_cmd="" if [ "$job_cluster" = "phoenix" ]; then - validate_cmd='syscheck_bin=$(find build/install -name syscheck -type f 2>/dev/null | head -1); [ -z "$syscheck_bin" ] || "$syscheck_bin" > /dev/null 2>&1' + validate_cmd='syscheck_bin=$(find build/install -name syscheck -type f 2>/dev/null | head -1); [ -z "$syscheck_bin" ] || mpirun -np 1 "$syscheck_bin"' fi # --- Variant selection ---