[lucasjia-aws]

Summary

The Nova model training team reported that our deep integ tests in account 784379639078 (us-east-1) occupy most of the shared Nova p5 pool. Each nova-textgeneration-lite-v2 serverless SFT/RLVR job needs 4 p5 instances, and because the deep suite runs under pytest-xdist, the SFT, SFT data-mixing and RLVR tests submit their jobs at the same moment. With ~11 p5 available, our tests held 8, which left no room for a customer job. The training team's ask was that these jobs not run at the same time.

CI history shows the same contention: across 127 batches of three simultaneously submitted Nova jobs since August, all three started within 15 minutes in only 30 batches; in the rest one or more waited for another to finish, or for the pool to free up.

Changes

• Add tests/integ/train/nova_capacity.py:
  • nova_capacity_slot() reuses tests/integ/lock.py to serialize Nova serverless jobs across all xdist workers of a run. The slot is held from submission until the job is terminal, and is released automatically if the process dies.
  • wait_for_nova_training_job() replaces the hand-rolled poll loops. If the job is still Starting/Pending after 60 minutes, it stops the job and skips the test, so we give capacity back instead of queueing behind customers. It also stops the job whenever the wait ends with the job non-terminal (timeout, error, interrupt), so killed or failed tests do not leave orphaned jobs that start training hours later.
• test_sft_trainer_nova_workflow, test_sft_trainer_nova_lite2_with_data_mixing and test_rlvr_trainer_nova_workflow now submit and wait inside the slot. Assertions are unchanged.

Testing

• black --check (26.3.1) and flake8 (7.1.2) pass on the changed files.
• pytest --collect-only -m "gpu_intensive and us_east_1" collects the three tests.
• Exercised the helpers locally with fake job objects: normal completion, pending timeout (stop + skip), training past the pending threshold (no skip), max-wait timeout (stop), refresh error (stop), failed job (no stop), stop errors swallowed, and three processes contending for the slot (max one holder).
• Not yet run against real AWS; the next scheduled CI Health run will exercise it.

Notes

• Serializing the three jobs makes the Nova chain ~2h05m–2h15m with no queueing. The existing gpu-integ job timeouts (HyperPod tests, capacity waits) are tracked separately and are not addressed here.

[github-actions]

Claude finished @lucasjia-aws's task in 2m 1s —— View job

---

Review — PR #6387: Serialize Nova serverless integ jobs

• Read PR diff
• Review nova_capacity.py helpers
• Review the three modified test files
• Post review

Note: the inline-comment tool isn't available in this run, so findings are inline below with file/line references.

Overall this is a clean, test-only change that solves a real resource-contention problem (3 Nova SFT/RLVR jobs × 4 p5 instances firing simultaneously under xdist). The helper is well documented, the fcntl cross-worker lock reuses the existing tests/integ/lock.py the same way test_local_model_trainer.py does, and secondary_status/training_job_status are real attributes on sagemaker.core.resources.TrainingJob (sagemaker-core/.../resources.py:31262-31263). The try/finally correctly stops a non-terminal job on skip, timeout, error, or interrupt. No blocking issues found. A few minor observations:

1. Lock is released before the stopped job is actually terminal (nova_capacity.py:84-133)
In the stop path (capacity pytest.skip or max_wait_time exceeded), _stop_quietly() issues training_job.stop() and returns immediately — stop() is async, so the job is still in Stopping and holding its 4 p5 instances when the with nova_capacity_slot() block exits and the next worker acquires the slot. This slightly contradicts the docstring ("Hold it from submission until the job is terminal" / "stopped it to release the shared pool"). In practice the overlap is brief and the whole point is to reduce contention, so this is acceptable — but the docstring/skip wording slightly overstates it. Consider either a short poll-until-terminal after stop(), or softening the wording to "requested a stop."

2. A transient refresh() error aborts the job (nova_capacity.py:109)
If training_job.refresh() raises (e.g. a throttle/transient describe failure), it propagates straight to the finally, which stops the job and fails the test. A single transient API blip would then kill an otherwise-healthy run. Given these jobs are expensive and long-running, a small retry/backoff around refresh() (or catching and continuing the poll loop) would make the wait more robust. Minor, since integ flakiness is tolerable, but worth considering.

3. secondary_status heuristic (nova_capacity.py:61,115)
Keying "waiting for capacity" off secondary_status in ("Starting", "Pending") is a reasonable proxy, but Starting also covers legitimate instance launch / image download / input download. The 1-hour pending_timeout makes a false positive unlikely, and you've documented the rationale — just flagging that the signal isn't exclusively "no capacity." Fine as-is.

4. Nits (non-blocking)

• All three call sites pass poll_interval=30, which equals the default — the explicit arg is redundant.
• No acquisition timeout on nova_capacity_slot(): a worker can block for roughly the sum of the other serialized jobs (up to ~3h each). That's the intended serialization, but confirm the CI/xdist suite timeout accommodates fully-serialized Nova execution so a blocked worker isn't reported as a hang.
• import time is still used elsewhere in all three modified files (unique_id generation), so removing the manual loops left no unused imports. 👍