ci: request 1 GPU for case-optimization jobs (they only use one)

[sbryngelson]

Problem

Phoenix Case Opt GPU jobs have been red-crossing by never getting scheduled. Example (run 34246706326):

shb-run-case-optimization  gpu-h200,gpu-h100,gpu-a100,gpu-v100
Submit 13:21   Start: None   Elapsed 00:00:00   State CANCELLED (8h GitHub timeout)

It sat PENDING ~7h and was killed at the job's 8h timeout-minutes. The GPU partitions are heavily oversubscribed (hundreds pending, 0 idle), so the scheduler couldn't place it.

Why it's worse than it needs to be

submit-slurm-job.sh hardcodes -G2 for every Phoenix GPU job. But run_case_optimization.sh runs tiny validation cases — mfc.sh run <case> --case-optimization -n $ngpus -- --gbpp 1 --steps 10 — and $ngpus falls back to 1. It uses however many GPUs it's granted; it does not need two.

Requesting -G2 forces SLURM to find a node with two free GPUs, which is dramatically harder under contention than one. So case-opt starves while single-GPU work could have backfilled.

Fix

Give case-optimization jobs -G1; keep -G2 for everything else (the test suite exercises multi-GPU MPI; bench-pair uses 2). Implemented via a new caseopt job type keyed off the submitted script name:

run_case_optimization*) job_type="caseopt" ;;   # -> -G1

Render check (from the exact case logic):

Submitted script	job_type	-G
run_case_optimization.sh	caseopt	1
common/test.sh	test	2
common/build-and-test.sh	buildtest	2
common/bench-pair.sh	bench	2

Case-opt validates that case-optimized binaries build and run; one GPU is sufficient for that (the run becomes -n 1 instead of -n 2). This does not reduce coverage of the optimization itself.

Verification

• bash -n and python3 toolchain/mfc/lint_source.py pass.
• Only the case-opt path changes; test/buildtest/bench are byte-identical (-G2).

Note

This eases scheduling but the underlying cause is GPU-cluster oversubscription; it does not create capacity. Bad/oversubscribed-cluster starvation of the 2-GPU test jobs is a separate, capacity-level issue.

[Copilot]

🟢 Approval recommended

The change is small, scoped to Phoenix GPU SBATCH rendering, and aligns the requested GPU count with how the case-optimization script actually runs (-n defaults to 1).

Pull request overview

This PR fixes Phoenix “Case Opt” GPU CI jobs getting stuck pending by requesting 1 GPU for case-optimization runs (which execute single-GPU smoke cases), while keeping 2 GPUs for the main test/build/bench GPU jobs.

Changes:

• Extend job-type detection to classify run_case_optimization*.sh submissions as caseopt.
• For Phoenix GPU jobs, set #SBATCH -G dynamically: 1 for caseopt, 2 otherwise.
• Preserve existing behavior for other job types and clusters.

File summaries

File	Description
.github/scripts/submit-slurm-job.sh	Detect caseopt job type from the submitted script name and request -G1 for Phoenix case-optimization GPU jobs while keeping -G2 for other GPU jobs.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.76%. Comparing base (d2d8cac) to head (6dcf5c5).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1836      +/-   ##
==========================================
- Coverage   62.81%   62.76%   -0.05%     
==========================================
  Files          84       84              
  Lines       21975    21993      +18     
  Branches     3224     3224              
==========================================
+ Hits        13804    13805       +1     
- Misses       5949     5967      +18     
+ Partials     2222     2221       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.