Repository navigation
Commit 02343e4
committed
fix: stop deploy waits hanging; follow empty list pages
ModelBuilder.deploy() could wait forever. The live-logging done-check
only returned once it could read the endpoint's CloudWatch log group,
but that group only exists after a container starts. An endpoint that
fails before provisioning (InsufficientInstanceCapacity), an
inference-component based endpoint (which hosts no model itself) or an
endpoint deleted mid-wait therefore kept _wait_until polling with no
timeout, which hung the sagemaker-serve CI canaries until the CodeBuild
timeout killed them.
- Return the DescribeEndpoint response once the endpoint reaches a
terminal status whether or not its log group exists, in both
sagemaker-serve's _live_logging_deploy_done_with_progress and
sagemaker-core's _live_logging_deploy_done. Treat Updating as in
progress in the serve checker, matching _deploy_done_with_progress.
- Tolerate a missing endpoint for at most 10 consecutive polls instead
of waiting on it forever.
- Make ModelBuilder._wait_for_endpoint raise UnexpectedStatusException
(CapacityError for capacity failures) when the endpoint ends in any
state other than InService, as Session.wait_for_endpoint does,
instead of logging an error and returning a failed endpoint.
- Wait on status only for inference-component based endpoints, which
never get an endpoint log group.
ResourceIterator stopped at the first empty page even when the response
carried a NextToken. ListAIRecommendationJobs returns such pages, so
AIRecommendationJob.get_all() and list_recommendations() always came
back empty. Keep following NextToken past empty pages, and stop if a
token repeats so a misbehaving list call cannot loop.
Add an xfail_on_insufficient_capacity marker to the sagemaker-serve
integ tests and apply it to the GPU tests that hit ICE: a test whose
endpoint or job SageMaker fails for lack of capacity reports XFAIL
instead of FAIL, while any other failure still fails the test.1 parent 9a4a9a0 commit 02343e4
20 files changed
Lines changed: 631 additions & 93 deletions
File tree
- sagemaker-core
- src/sagemaker/core
- helper
- utils
- tests/unit
- generated
- helper
- sagemaker-serve
- src/sagemaker/serve
- tests
- integ
- unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1274 | 1274 | | |
1275 | 1275 | | |
1276 | 1276 | | |
| 1277 | + | |
1277 | 1278 | | |
1278 | 1279 | | |
1279 | | - | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
1280 | 1286 | | |
1281 | 1287 | | |
1282 | 1288 | | |
| |||
2979 | 2985 | | |
2980 | 2986 | | |
2981 | 2987 | | |
2982 | | - | |
2983 | | - | |
| 2988 | + | |
| 2989 | + | |
| 2990 | + | |
| 2991 | + | |
| 2992 | + | |
| 2993 | + | |
| 2994 | + | |
| 2995 | + | |
| 2996 | + | |
| 2997 | + | |
| 2998 | + | |
| 2999 | + | |
| 3000 | + | |
| 3001 | + | |
| 3002 | + | |
| 3003 | + | |
| 3004 | + | |
| 3005 | + | |
| 3006 | + | |
| 3007 | + | |
| 3008 | + | |
| 3009 | + | |
| 3010 | + | |
| 3011 | + | |
| 3012 | + | |
| 3013 | + | |
| 3014 | + | |
| 3015 | + | |
| 3016 | + | |
| 3017 | + | |
| 3018 | + | |
| 3019 | + | |
| 3020 | + | |
| 3021 | + | |
| 3022 | + | |
| 3023 | + | |
| 3024 | + | |
| 3025 | + | |
| 3026 | + | |
| 3027 | + | |
| 3028 | + | |
2984 | 3029 | | |
2985 | 3030 | | |
2986 | 3031 | | |
2987 | 3032 | | |
2988 | 3033 | | |
2989 | 3034 | | |
2990 | 3035 | | |
| 3036 | + | |
| 3037 | + | |
2991 | 3038 | | |
2992 | 3039 | | |
2993 | 3040 | | |
| 3041 | + | |
| 3042 | + | |
2994 | 3043 | | |
2995 | | - | |
2996 | | - | |
2997 | | - | |
2998 | | - | |
2999 | | - | |
3000 | | - | |
3001 | | - | |
3002 | | - | |
3003 | | - | |
3004 | | - | |
| 3044 | + | |
| 3045 | + | |
| 3046 | + | |
| 3047 | + | |
| 3048 | + | |
| 3049 | + | |
| 3050 | + | |
| 3051 | + | |
| 3052 | + | |
3005 | 3053 | | |
| 3054 | + | |
3006 | 3055 | | |
3007 | 3056 | | |
3008 | 3057 | | |
| |||
3016 | 3065 | | |
3017 | 3066 | | |
3018 | 3067 | | |
3019 | | - | |
3020 | | - | |
3021 | | - | |
3022 | | - | |
3023 | 3068 | | |
3024 | | - | |
3025 | | - | |
3026 | | - | |
3027 | | - | |
| 3069 | + | |
| 3070 | + | |
| 3071 | + | |
3028 | 3072 | | |
3029 | | - | |
| 3073 | + | |
| 3074 | + | |
3030 | 3075 | | |
3031 | 3076 | | |
3032 | 3077 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
505 | 505 | | |
506 | 506 | | |
507 | 507 | | |
508 | | - | |
509 | | - | |
510 | | - | |
511 | | - | |
512 | | - | |
513 | | - | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
518 | | - | |
519 | | - | |
520 | | - | |
521 | | - | |
522 | | - | |
523 | | - | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
524 | 530 | | |
525 | 531 | | |
526 | 532 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
177 | 212 | | |
178 | 213 | | |
179 | 214 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
23 | 28 | | |
24 | 29 | | |
25 | 30 | | |
| |||
375 | 380 | | |
376 | 381 | | |
377 | 382 | | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
378 | 496 | | |
379 | 497 | | |
380 | 498 | | |
| |||
0 commit comments