ci: identify faulted node via sacct when the .out marker is missing

[sbryngelson]

Problem

In run 34183404644 the Case Opt gpu-acc job hit a dead-GPU V100 (atl1-1-02-006-34-0, cuInit returned error 999 CUDA_ERROR_UNKNOWN). Preflight correctly flagged it (exit 77) and resubmitted — but both attempts ran on the exact same node, so it exhausted the resubmit budget and red-crossed.

Root cause

submit-slurm-job.sh identifies the faulted node only from the MFC_FAULT_NODE= marker in the job's .out:

faulted_node=$(bash "$SCRIPT_DIR/node-exclude.sh" node-from "$output_file")

When the job dies before its .out is flushed / visible over NFS, the monitor reports "reached terminal state (FAILED) without creating output file", node-from returns empty, the merge adds nothing, and the --exclude list stays at the seed only. SLURM then re-schedules onto the same bad node. The logs show it plainly: failed preflight on an unidentified node … Excluding: <seed only>.

Fix

Fall back to SLURM's own accounting when the marker is unreadable — sacct knows the node whether or not the .out exists:

if [ -z "$faulted_node" ]; then
    faulted_node=$(sacct -j "$job_id" -X -n -o NodeList 2>/dev/null | head -n1 | tr -d ' ')
    case "$faulted_node" in ""|None*|*[,\[]*) faulted_node="" ;; esac   # reject "None assigned" / ranges
fi

Now the offending node is excluded on resubmit, so the retry lands elsewhere. With reliable identification, the existing MFC_MAX_NODE_RESUBMITS=1 budget gives one genuine retry on a different node — which would have turned this run green (both attempts were the same node).

Notes / scope

• Independent of the --bind-to none fix (ci: add --bind-to none to Phoenix syscheck smoke-test (fixes GPU-node MPI bind failure) #1834): that addressed a binding false-failure; this addresses a genuine bad node that wasn't being excluded.
• Deliberately keeps MFC_MAX_NODE_RESUBMITS=1. Riding out multiple distinct dead-GPU nodes would need a higher cap, but "two distinct bad nodes in a row after excluding the first" is a legitimate cluster-health signal — better surfaced than silently absorbed. Bad nodes should still be reported to PACE (e.g. atl1-1-02-006-34-0, still undrained).

Verification

• bash -n and python3 toolchain/mfc/lint_source.py pass.
• Guard unit-checked: keeps a clean single hostname; rejects "", None assigned, atl1-[1-2] ranges, and comma-lists.

[github-actions]

Claude Code Review

Head SHA: f1f9cce

Files changed:

• 1
• .github/scripts/submit-slurm-job.sh

Findings:

• .github/scripts/submit-slurm-job.sh (new fallback, ~line 312): the added sacct command substitution lacks the || true guard that the pre-existing sacct call at line 185 uses. The script runs under set -euo pipefail (line 8), so if sacct -j "$job_id" -X -n -o NodeList fails or is transiently unavailable right after a node fault (a plausible condition, since this code path exists specifically to handle degraded/unreliable state), the pipeline's non-zero exit propagates through the faulted_node=$(...) assignment and aborts the whole script immediately — defeating the surrounding fault-handling logic (the node_attempt retry and the ::error:: cluster-wide-problem message) and turning a recoverable node fault into an unhandled crash. Add || true (matching the existing sacct call) so a failed lookup falls through to the empty-faulted_node case instead of terminating the script.

[Copilot]

🟡 Changes recommended

The new sacct | head pipeline is unguarded under set -euo pipefail and can abort the resubmit flow on non-zero/SIGPIPE unless it’s made failure-tolerant (as in other sacct usages).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR improves CI resilience for SLURM-based GitHub Actions jobs by ensuring a faulted compute node can still be identified and excluded from resubmission even when the job’s .out marker file is missing/unreadable (e.g., due to early job death or NFS latency).

Changes:

• Adds a fallback path to query SLURM accounting (sacct) for the job’s NodeList when MFC_FAULT_NODE= cannot be read from the output file.
• Filters out non-actionable sacct node outputs (empty, None…, ranges, comma-lists) before merging into the --exclude list.

File summaries

File	Description
.github/scripts/submit-slurm-job.sh	Adds sacct fallback node identification to ensure resubmitted jobs avoid the same faulted node when .out is missing.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.81%. Comparing base (b44c811) to head (f046780).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1835      +/-   ##
==========================================
+ Coverage   62.35%   62.81%   +0.46%     
==========================================
  Files          84       84              
  Lines       21621    21975     +354     
  Branches     3199     3224      +25     
==========================================
+ Hits        13482    13804     +322     
- Misses       5939     5949      +10     
- Partials     2200     2222      +22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.