ci: submit Phoenix GPU test jobs to a partition list for backfill

[sbryngelson]

What

Phoenix GPU test jobs now submit to a partition list

#SBATCH -p gpu-h200,gpu-h100,gpu-a100,gpu-v100

instead of a single partition chosen at submit time by select-gpu-partition.sh. Benchmark jobs are unchanged — they still pick and pin one partition via BENCH_GPU_PARTITION.

Why

Before #1299, Phoenix test jobs already used a partition list (-p gpu-v100,gpu-a100,gpu-h100,gpu-l40s,gpu-h200), so SLURM could start the job on whichever partition freed first. #1299 unified test + bench submission onto the single-partition select-gpu-partition.sh selector. That selector exists for benchmarks, which must run PR and master on the same GPU type to be comparable — a constraint tests don't have.

The side effect was that test jobs lost multi-partition backfill: each job is now pinned to one partition and queues behind everything else there. In practice CI has been landing on gpu-a100 and sitting behind ~80 higher-priority jobs while idle nodes existed on other partitions (e.g. gpu-v100). This PR restores the old backfill behavior for tests while keeping the bench selector intact.

Notes

• gpu-l40s is intentionally omitted (out of rotation for bad hardware; 4/10 nodes currently drained by PACE) and gpu-rtx6000 is omitted (too slow for the test time limit, deep queue), matching the current selector's rotation.
• Bench PR/master consistency is preserved: the job_type = "bench" branch is unchanged.
• No change to the per-node --exclude seed logic or the preflight resubmit path.

[Copilot]

🟢 Approval recommended

The change is localized to Phoenix GPU submission logic and preserves benchmark partition pinning while restoring multi-partition backfill for tests.

Pull request overview

Restores Phoenix GPU test job backfill behavior by submitting test (and build+test) jobs to a multi-partition -p list, while keeping benchmark jobs pinned to a single, consistently selected GPU partition for PR/master comparability.

Changes:

• Split Phoenix GPU partition selection by job_type: benchmarks remain single-partition (via BENCH_GPU_PARTITION / select-gpu-partition.sh), while tests use a fixed partition list for SLURM backfill.
• Add clearer in-script rationale/comments for why bench vs test scheduling differs.

File summaries

File	Description
.github/scripts/submit-slurm-job.sh	Adjusts Phoenix GPU partition selection to use a partition list for test jobs while preserving single-partition benchmark consistency.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.30%. Comparing base (70589fd) to head (667a96f).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1832   +/-   ##
=======================================
  Coverage   62.30%   62.30%           
=======================================
  Files          84       84           
  Lines       21583    21583           
  Branches     3195     3195           
=======================================
  Hits        13448    13448           
  Misses       5937     5937           
  Partials     2198     2198           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.