You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c9886ea
Browse filesBrowse the repository at this point in the historyBrowse files
test(train): Bound capacity wait in SMTJ SFT integ test (#6409)
test_sft_trainer_serverful_smtj trains on one ml.g6.12xlarge in
us-east-1. When there is no capacity the job stays in Pending, and the
test polled for up to 3 hours, which held the GPU integ build until
CodeBuild timed out. The job was then left queued behind the killed
build and trained hours later with nobody waiting for it.
Poll with the shared bounded capacity wait from tests/integ/capacity.py:
after 90 minutes in Pending the job is stopped and the test is skipped,
and the job is stopped whenever the wait ends with it still running.
The total wait is 2 hours, covering the capacity wait plus the job's
~21 minute run time, so the test stays well inside the build timeout.
0 commit comments