Repository navigation
Commit 943ce0a
authored
Fix/v3 tests (#5996)
* test: make sudo install resilient in integ test Docker image
The dummy_container_with_user_and_workdir fixture failed to build because
the standalone `RUN apt install -y sudo` step ran several layers after the
initial apt-get update, using a stale package index and failing to connect
to deb.debian.org when fetching the sudo .deb.
Install sudo in the same layer as the initial apt-get update against a fresh
index, switch from `apt` to `apt-get`, and add Acquire::Retries so transient
network failures are retried automatically.
* test(train): resolve LLMAJ base model against SageMakerPublicHub in integ test
* test: xfail when InsufficientInstanceCapacity
* test: add test level rerun logic to reduce flackyness
* fix(train): Replace EOL Claude judge model in LLM-as-Judge eval
The judge model anthropic.claude-3-5-haiku-20241022-v1:0 has reached
end-of-life in Bedrock, causing integ test evaluation pipelines to fail
with a ValidationException. Switch to the active successor
anthropic.claude-haiku-4-5-20251001-v1:0 and add it to the allowed
evaluator models list.
* test(serve): tolerate IAM SimulatePrincipalPolicy throttling in integ tests
Add tests/integ/conftest.py with an autouse session fixture that gives
boto3's default session adaptive retries (absorbed by the SDK's internal
IAM role-validation client), plus a makereport hook that xfails residual
SimulatePrincipalPolicy throttling under concurrent test load.
* test(train): validate LLMAJ pipeline via step status, make show_results best-effort
* test(serve): use canonical HF repo id openai-community/gpt2 in integ test
* test(train): fix integ test region/hub/marker issues for v3
- Route Nova evaluator/recipe tests to us-east-1 via us_east_1 marker
and us-east-1 session (InspectAI, sft max_steps below minimum)
- Pin LLMAsJudge full-flow evaluator to us-west-2 region
- Provide explicit session to no-recipe SFTTrainer test to avoid
region resolution failure under xdist
- Correct extract_evaluator_arn integ test: Lambda ARN is rejected by
_extract_evaluator_arn (dispatched upstream), expect ValueError
* test(train): Resolve Nova model from public hub in InspectAI integ tests
test_inspect_ai_evaluator's two tests constructed InspectAIEvaluator with
the public-hub model 'nova-textgeneration-lite' while the session-scoped
use_private_hub fixture pinned SAGEMAKER_HUB_NAME to the private 'sdktest'
hub, causing DescribeHubContent ResourceNotFound and a pydantic
ValidationError at construction time. Override SAGEMAKER_HUB_NAME to
SageMakerPublicHub in both tests, matching test_llmaj_custom_model.
* test(mlops): use unique S3 prefixes to fix concurrent build races
The HPO, Clarify, and pipeline train-registry integ tests used hardcoded
S3 prefixes and deleted the entire prefix in their finally blocks. When
two mlops integ builds ran concurrently against the same default bucket,
one build's cleanup wiped the input data the other build was actively
reading, causing spurious "No S3 objects found" / "matched no files on
s3" failures.
Append a uuid suffix to each test's S3 prefix so concurrent builds no
longer share paths and cleanup only removes each run's own data.
* test(train): fix hub resolution, region isolation, and stale error matches in recipe integ tests
Fix three independent failure classes in the sagemaker-train recipe
override / evaluator integ tests, all surfaced under xdist (-n auto).
Evaluator hub resolution: TestBenchMarkEvaluatorRecipeOverrideInteg
constructs BenchMarkEvaluator with the public-hub model
'meta-textgeneration-llama-3-2-1b-instruct' while the session-scoped
use_private_hub fixture pins SAGEMAKER_HUB_NAME to the private 'sdktest'
hub. Unlike the SFTTrainer path, the evaluator's JumpStart model
resolution (_resolve_jumpstart_model) does not fall back to
SageMakerPublicHub, so DescribeHubContent returned ResourceNotFound and
construction failed with a pydantic ValidationError. Override
SAGEMAKER_HUB_NAME to SageMakerPublicHub in both tests, matching the
existing InspectAI tests.
Region isolation: tests that construct SFTTrainer without an explicit
sagemaker_session build a default Session() that reads the region from
the environment. AWS_DEFAULT_REGION was only set as a side effect of the
module-scoped sagemaker_session fixture, so a worker running only such
tests had no region set and Session() raised "Must setup local AWS
configuration with a region supported by SageMaker.". Add a
session-scoped autouse ensure_default_region fixture that pins
AWS_DEFAULT_REGION up front (without clobbering an externally provided
value) so region resolution no longer depends on test order.
Stale error matches: two get_resolved_recipe no-recipe tests matched
"requires a 'recipe' or 'overrides'", but the raised message is now
"requires a 'recipe', 'overrides', or direct hyperparameter
assignments...". Update both regexes to the current wording.
* test(train): align SFT recipe override assertions with real recipe structure
The SFTTrainer recipe-override integ tests were written against a
synthetic recipe shape (flat training_config.<hyperparameter>) that
matches the unit-test mocks but not the real Hub recipe. The private
"sdktest" hub does not contain meta-textgeneration-llama-3-2-1b-instruct,
so resolution falls back to SageMakerPublicHub, whose SFT LoRA recipe
nests hyperparameters under training_config.training_args and uses
different field names. As a result overrides were correctly placed (or
intentionally dropped when the key does not exist) while the assertions
read the wrong paths, producing KeyErrors.
Fix the tests to match the real recipe:
- Read hyperparameters under training_config.training_args (learning_rate,
seed, max_len, etc.) instead of directly under training_config.
- Use real recipe field names: max_epochs (not num_epochs),
train_batch_size (not batch_size), max_len (not max_length), and the
non-spec fields micro_train_batch_size / max_norm (not the invented
max_length / save_top_k).
- Write recipe-file values at their real nested path, since user recipe
files are not field-name remapped the way the overrides dict is. This
also makes test_sft_recipe_file_with_invalid_value_raises actually
reach validation and raise "above maximum".
- Skip test_sft_save_steps_equal_to_max_steps_passes: save_steps/max_steps
are not part of the llama SFT recipe (it trains by epochs, not steps),
so the boundary cannot be exercised against this model.
No production code changes: the resolver's field-name remapping and
unknown-key dropping are working as designed and covered by unit tests.
* fix(train): fall back to public hub for base model resolution
The evaluator's JumpStart model resolution (_resolve_jumpstart_model)
only queried the hub named by SAGEMAKER_HUB_NAME. When that points at a
private recipe hub, base models (which live in SageMakerPublicHub) could
not be resolved and construction failed. Fall back to SageMakerPublicHub
when the base model isn't found in the configured hub, mirroring the
behavior already in recipe_utils.
This lets the integ tests drop the monkeypatch(SAGEMAKER_HUB_NAME=
SageMakerPublicHub) workarounds and run the real customer flow, keeping
the private hub pinned by the use_private_hub fixture.
Also deduplicate _ALLOWED_EVALUATOR_MODELS: a master merge reintroduced
stale entries (with conflicting region lists) that #5987 had removed.
Keep the refreshed allowlist and drop the EOL claude-3-5-haiku-20241022
entry from the matching unit test.
* test: trim verbose comments in v3 integ tests
Condense the lengthy explanatory comments and docstrings added across
the test files in this PR down to a line or two each, keeping the
essential "why" and any TODOs. No logic changes.
* test: de-flake recipe override and lake formation integ tests
Pass explicit sagemaker_session to two SFTTrainer recipe-override tests
so they no longer depend on a global AWS_DEFAULT_REGION that can be torn
down by another fixture under xdist.
Rewrite test_enable_lake_formation_fails_for_non_created_status to pin a
non-'Created' status deterministically (mock refresh) instead of racing
the creation window, avoiding a fall-through to the real RegisterResource
call.
* test: fix pydantic-unsafe mock.patch on FeatureGroupManager instance
test_enable_lake_formation_fails_for_non_created_status patched refresh()
on the FeatureGroupManager instance, but the manager is a pydantic model
whose validate_assignment rejects assigning a MagicMock to a method,
raising "Object has no attribute 'refresh'" before the assertion ran.
Patch refresh() on the class (type(fg)) instead to stay pydantic-safe.
* test: stop sagemaker_session fixture from deleting global AWS region
The module-scoped sagemaker_session fixture deleted AWS_DEFAULT_REGION on
teardown, clobbering the value set by the session-scoped autouse
ensure_default_region fixture. Under xdist a worker runs multiple modules,
so a later test that builds a trainer without an explicit session (SDK
falls back to Session()) intermittently failed with "Must setup local AWS
configuration with a region supported by SageMaker".
Reuse the region guaranteed by ensure_default_region and no longer mutate
the global env var on teardown.1 parent 7e7b6f1 commit 943ce0a
20 files changed
Lines changed: 326 additions & 105 deletions
File tree
- requirements/extras
- sagemaker-mlops/tests/integ
- workflow
- sagemaker-serve/tests/integ
- sagemaker-train
- src/sagemaker/train/common_utils
- tests
- integ/train
- unit/train
- common_utils
- evaluate
- tests/integ_helpers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
59 | 60 | | |
60 | 61 | | |
61 | 62 | | |
62 | | - | |
| 63 | + | |
63 | 64 | | |
64 | 65 | | |
65 | 66 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
179 | 180 | | |
180 | 181 | | |
181 | 182 | | |
| 183 | + | |
| 184 | + | |
182 | 185 | | |
183 | 186 | | |
184 | 187 | | |
| |||
350 | 353 | | |
351 | 354 | | |
352 | 355 | | |
353 | | - | |
354 | | - | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
355 | 362 | | |
356 | 363 | | |
357 | 364 | | |
358 | 365 | | |
359 | 366 | | |
360 | | - | |
| 367 | + | |
361 | 368 | | |
362 | 369 | | |
| 370 | + | |
363 | 371 | | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
368 | 384 | | |
369 | 385 | | |
370 | 386 | | |
| |||
373 | 389 | | |
374 | 390 | | |
375 | 391 | | |
376 | | - | |
377 | 392 | | |
378 | 393 | | |
379 | 394 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
30 | | - | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
130 | 131 | | |
131 | 132 | | |
132 | 133 | | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
138 | 138 | | |
139 | 139 | | |
140 | | - | |
| 140 | + | |
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
262 | 262 | | |
263 | 263 | | |
264 | 264 | | |
265 | | - | |
266 | | - | |
267 | | - | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
272 | 287 | | |
273 | 288 | | |
274 | 289 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
150 | 160 | | |
151 | 161 | | |
152 | 162 | | |
| |||
156 | 166 | | |
157 | 167 | | |
158 | 168 | | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
173 | 178 | | |
174 | 179 | | |
175 | 180 | | |
| |||
0 commit comments