Repository navigation
Commit 4f4f488
authored
test(train): Bound capacity wait in gated JumpStart train test (#6397)
test_jumpstart_train_from_gated_reference trains on the gated model's
default GPU instance (ml.g5.2xlarge) with train(wait=True). When there
is no capacity the job stays in Pending ("waiting for capacity")
indefinitely: MaxRuntimeInSeconds does not count queue time and
TrainingJob.wait() has no timeout. The test then blocked until the
CI Health V3 release canary build timed out, and the queued job was
left behind holding the account's single ml.g5.2xlarge training quota,
which in turn failed the V2 slow test suite with ResourceLimitExceeded.
Submit with wait=False and poll with a bounded capacity wait: after an
hour in Pending the job is stopped and the test is skipped. The total
wait covers queue time plus the job's MaxRuntimeInSeconds, so the test
never ends before the job does.
Move the bounded wait out of the Nova-only helper into a shared
tests/integ/capacity.py. wait_for_nova_training_job keeps its signature
and defaults and now delegates to the shared helper.1 parent 4ff8e11 commit 4f4f488
3 files changed
Lines changed: 149 additions & 63 deletions
File tree
- sagemaker-train/tests/integ
- jumpstart
- train
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
| |||
50 | 52 | | |
51 | 53 | | |
52 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
53 | 62 | | |
54 | 63 | | |
55 | 64 | | |
| |||
361 | 370 | | |
362 | 371 | | |
363 | 372 | | |
364 | | - | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
365 | 391 | | |
366 | 392 | | |
367 | 393 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
28 | 27 | | |
29 | 28 | | |
30 | 29 | | |
| |||
35 | 34 | | |
36 | 35 | | |
37 | 36 | | |
38 | | - | |
39 | | - | |
40 | 37 | | |
| 38 | + | |
41 | 39 | | |
42 | 40 | | |
43 | 41 | | |
| |||
51 | 49 | | |
52 | 50 | | |
53 | 51 | | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | 52 | | |
58 | 53 | | |
59 | 54 | | |
| |||
65 | 60 | | |
66 | 61 | | |
67 | 62 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | 63 | | |
79 | 64 | | |
80 | 65 | | |
81 | 66 | | |
82 | 67 | | |
83 | 68 | | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
| 69 | + | |
94 | 70 | | |
95 | | - | |
96 | | - | |
97 | | - | |
| 71 | + | |
| 72 | + | |
98 | 73 | | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
0 commit comments