[lucasjia-aws]

Summary

test_sft_trainer_serverful_smtj submits an SFT job on one ml.g6.12xlarge in us-east-1 and polled for up to 3 hours. When there is no capacity the job stays in Pending ("Preparing the instances for training") and does not fail, so the test held the CI Health V3 GPU integ build until CodeBuild hit its 180-minute timeout. The queued job was then left behind and trained later with nobody waiting for it (for example, on 2026-10-08 the job was submitted at 07:22 UTC and only started training at 14:08 UTC, long after the build was killed).

Across the 169 SMTJ jobs this test created since 2026-07-20, the wait for an instance was under 60 minutes for 38% of jobs, under 90 minutes for 47%, and under 180 minutes for 58%; once an instance is acquired, download, training and upload take at most ~21 minutes.

Changes

• Replace the hand-rolled poll loop with wait_for_training_job_with_capacity_timeout from tests/integ/capacity.py (the shared helper introduced in test(train): Bound capacity wait in gated JumpStart train test #6397 and used by the Nova serverless tests).
• If the job is still waiting for instances after 90 minutes, it is stopped and the test is skipped with ml.g6.12xlarge capacity unavailable. The job is also stopped whenever the wait ends with it still running (timeout, error, interrupt), so no orphaned job is left queueing.
• Total wait is 2 hours (capacity wait plus the job's run time), down from 3 hours, which keeps the test inside the build timeout.
• The final assertion now includes the job's failure_reason.

Testing

• black --check (26.3.1) and flake8 (7.1.2) pass on the changed file.
• pytest --collect-only -m "gpu_intensive and us_east_1" collects the test.
• The helper's behavior (completion, pending timeout with stop and skip, max-wait stop, stop on error) was covered when it was introduced. Not yet run against real AWS; the next scheduled CI Health run will exercise it.

Notes

• At current capacity this test will skip roughly half the time instead of timing out the build. Raising the pass rate would need a different instance type (for example ml.g5.12xlarge), which is out of scope here.

[github-actions]

🤖 AI code review

This PR replaces a hand-rolled 3-hour poll loop in the SMTJ SFT integ test with the shared wait_for_training_job_with_capacity_timeout helper, bounding the capacity wait to 90 min and total wait to 2 h, stopping orphaned jobs and skipping on capacity shortage. I verified the helper's signature matches the call, the ..capacity import path resolves correctly, time is still used (for unique_id), and secondary_status/failure_reason exist on the TrainingJob shape. The change is correct and well-scoped; no issues found.

_{Reviewed commit 1d68380. Automated review; verify before acting.}