Repository navigation
Commit 60a051c
test(serve): stop unit tests resolving IAM roles over the network (#6496)
ModelBuilder.__post_init__ resolves a default execution role when none is
supplied, and resolve_and_validate_role does that with sts:GetCallerIdentity
followed by a paginated iam:SimulatePrincipalPolicy. Every unit test that
builds a ModelBuilder without role_arn therefore makes live IAM calls, which
succeed only because CI credentials are present.
IAM throttles on an account-wide quota, so those builds fail with
Throttling ... SimulatePrincipalPolicy (reached max retries: 4): Rate exceeded
on whichever test happened to be running. Over the 184 decided builds of
sagemaker-python-sdk-ci-health-unit-test-v3 with SUBMODULE=sagemaker-serve
since 2026-09-23, throttling accounts for 37 of 42 failures, and 100% of it
(37/37 builds, 90/90 per-env instances) comes from
tests/unit/test_model_builder_methods.py -- the one remaining file whose tests
construct ModelBuilder without a role and without stubbing the resolver. All
23 of its unprotected tests have thrown it in production.
The failures are not reproducible: each build runs the suite three times
(py310/py311/py312) and these hit only one of the three 89% of the time, while
genuine failures hit all three every time.
This generalises the fixture 5432663^..5a65c55 ("fix PR check tests", #6272)
added to test_model_builder_coverage_boost.py, which drove that file from 83%
of throttling occurrences to zero. test_bedrock_model_builder.py and
test_ai_inference_recommender/test_model_builder_recommendations.py carry the
same fixture and have never thrown a throttle in 531 builds. Moving it into
tests/unit/conftest.py covers the package rather than one file at a time.
Two details worth noting:
- The stub passes the provided role through rather than returning a fixed
value, matching the real contract. Four of the five call sites pass a real
role, and tests assert on what comes back.
- resolve_and_validate_role is patched once per importing module. model_builder
and bedrock_model_builder each bind it with a module-level `from ... import`,
so each holds its own reference and patching the definition site in
sagemaker.core.helper.iam_role_resolver would rebind neither.
Tests that patch the resolver themselves still win, since their patch is applied
inside the fixture's and so rebinds the attribute last.
Verified with deliberately invalid credentials plus a transport-level request
recorder (with a positive control, so a silent failure to record cannot be read
as absence of traffic):
- serve unit suite: 67 failed -> 44 failed, 2087 -> 2110 passed
- newly broken tests: 0
- tests issuing STS calls: 24 -> 1 (the remaining one is the telemetry
daemon thread, attributed outside any test, and is unrelated)
- STS/IAM errors in the run: 0
The 44 remaining failures are all s3:GetObject against the JumpStart cache.
Those succeed in CI, so they produce no CI failures and are left for a separate
change.
Coverage is held whole. serve/model_builder.py gains 5 covered lines because 23
tests now run to completion. 18 lines of sagemaker-core's iam_role_resolver stop
being covered by serve, having previously been reached only by making real IAM
calls; 17 were already covered by that module's own tests, and the new
TestConfigDefaultRole covers the last one (the early return for role types with
no config path, which serve was exercising incidentally via role_type="serving").
No production code changes.
Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>1 parent 4f486d7 commit 60a051c
2 files changed
Lines changed: 97 additions & 0 deletions
File tree
- sagemaker-core/tests/unit/helper
- sagemaker-serve/tests/unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| |||
1391 | 1393 | | |
1392 | 1394 | | |
1393 | 1395 | | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
| 1405 | + | |
| 1406 | + | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
0 commit comments