ci: run Phoenix GPU benchmarks in one single-node job (backfill + same-hardware)

[sbryngelson]

What

Phoenix GPU benchmarks now build and bench both the master and PR trees in one single-node SLURM job (new bench-pair.sh), instead of two separate jobs pinned to one partition. The single job needs only -G2 on one node, so it submits to a partition list and backfills.

Frontier and Phoenix-CPU benchmarks are untouched — they keep the existing two-job concurrent path.

Stacked on #1832 (base branch = ci-restore-gpu-partition-list-for-tests). Retarget to master once #1832 merges.

Why

The old bench model selected one partition requiring two idle nodes in that same partition at once (GPU_PARTITION_MIN_NODES=2) and pinned both PR and master jobs to it. That is the single hardest scheduling condition on Phoenix, so bench routinely starved in the queue (recent bench GitHub jobs were 3.5–4.2h wall-clock, almost entirely queue wait).

Running both trees on one node inverts the tradeoff:

• Stronger fairness. PR and master run on the same physical node and GPUs, back-to-back — not merely the same GPU model on two different nodes (which is what the old path gave, and it already drifted across h200/h100/a100/v100 run-to-run).
• Backfill. One node + -G2 means SLURM can start on whichever partition frees first (gpu-h200,gpu-h100,gpu-a100,gpu-v100), instead of waiting for two idle nodes in one partition.
• No contention. Builds (compile-only, no GPU) run first; the two benchmarks run strictly sequentially on the same GPUs.

How it's structured

• common/bench-pair.sh (new): builds master, builds pr, runs preflight once, then benches master and pr back-to-back. Writes bench-<device>-<interface>.yaml in each tree (same paths downstream already expects). Build/bench commands mirror common/bench.sh.
• run_parallel_benchmarks.sh: for Phoenix GPU, submits the one pair job and lets submit-slurm-job.sh monitor it (so bench now also gets preemption-resubmit and node-exclude-on-fault, exit 77 — which the old two-job bench path never had). All other clusters keep the two-job path byte-for-byte.
• submit-slurm-job.sh: the GPU partition block collapses to the shared partition list for both tests and benchmarks (the bench-specific BENCH_GPU_PARTITION/selector branch added in ci: submit Phoenix GPU test jobs to a partition list for backfill #1832 is no longer needed). select-gpu-partition.sh is now unused and can be removed in a follow-up.

Timing (from real Phoenix runs)

Per-tree build+bench decomposes into build ~14 min + bench ~6 min (SLURM accounting + a real bench log). Sequential single-node total ≈ ~40 min (builds sequential, benches back-to-back), vs the bench_time=04:00:00 budget — ~6× headroom.

Verification done

• bash -n on all three scripts.
• Live on Phoenix: --test-only confirms the -G2 + partition-list shape schedules; a real -G2 probe backfilled onto idle v100 in ~1s and saw exactly 2 GPUs as indices 0,1 (so -g 0 1 -n 2 is unchanged).
• Full dry-run of bench-pair.sh with stubbed trees: correct order (build master → build pr → preflight once → bench master → bench pr), ../master resolution, and both YAMLs written to the right paths.

Not yet verified

An actual end-to-end 2×(build+bench) inside one allocation — needs a real ~40-min allocation. Each run is identical to today's working bench, so the residual risk is orchestration (dry-run-checked) and time budget (measured, ample). Best confirmed by a live bench CI run on this branch.

[github-actions]

Claude Code Review

Head SHA: bbe6d6e

Files changed:

• 3
• .github/scripts/run_parallel_benchmarks.sh
• .github/scripts/submit-slurm-job.sh
• .github/workflows/common/bench-pair.sh

Findings:

• run_parallel_benchmarks.sh derives job_slug="bench-${device}-${interface}" once at the top and reuses it in Phase 3's failure-diagnostic tails (pr/${job_slug}.out, master/${job_slug}.out). That's correct for the two-job path (script basename bench), but for the new Phoenix-GPU path the submitted script is bench-pair.sh, and submit-slurm-job.sh's own job_slug (used for its .out/.slurm_job_id filenames) is derived from that basename, producing bench-pair-${device}-${interface} instead. If the single-node job exits 0 but the YAML is still missing when Phase 3 checks (the NFS-propagation race the surrounding code explicitly guards against), the fallback tail -n 100 "pr/${job_slug}.out" looks for the wrong filename and silently prints "Could not read PR log" instead of the real job output, losing diagnostics for the new path.
• bench-pair.sh now performs two full builds plus two full benchmark runs (master and PR, sequentially) inside a single Phoenix SLURM job, but submit-slurm-job.sh still assigns it bench_time="04:00:00" (unchanged), the same budget previously given to a job that built+benched only one tree via bench.sh. Doubling the work under the same wall-clock limit risks new timeouts for the Phoenix GPU benchmark job that didn't exist under the old two-job model.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.35%. Comparing base (70589fd) to head (95a7ca8).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1833      +/-   ##
==========================================
+ Coverage   62.30%   62.35%   +0.04%     
==========================================
  Files          84       84              
  Lines       21583    21621      +38     
  Branches     3195     3199       +4     
==========================================
+ Hits        13448    13482      +34     
- Misses       5937     5939       +2     
- Partials     2198     2200       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

This PR updates Phoenix GPU benchmarking to run both master and PR builds/benches inside a single SLURM allocation to improve scheduling (backfill) and ensure same-hardware comparisons.

Changes:

• Add bench-pair.sh to build+bench master and PR sequentially on one node/GPU allocation.
• Update Phoenix GPU benchmark submission to use the single-job path and rely on shared partition lists.
• Remove the now-obsolete GPU partition selector and broaden log-collection patterns.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
.github/workflows/common/bench-pair.sh	New single-allocation script to build+bench master and PR back-to-back on the same Phoenix GPU node.
.github/workflows/bench.yml	Adjust log collection globs to include the new bench-pair job log naming.
.github/scripts/submit-slurm-job.sh	Simplify Phoenix GPU partition selection to always use the partition list for both tests and benchmarks.
.github/scripts/select-gpu-partition.sh	Remove obsolete partition selection helper no longer needed with single-node bench-pair.
.github/scripts/run_parallel_benchmarks.sh	Route Phoenix GPU to the one-job bench-pair path; keep other clusters on the existing two-job flow.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.