Skip to content

Commit 1b14335

Browse files
authored
fix(ci,train): stop integ-tests rerunning the shallow suite (#6216)
* fix(ci,train): stop integ-tests rerunning the shallow suite The shallow suite lives under tests/integ/train/shallow, and the deep integ-tests CodeBuild project invokes pytest through tox over the whole tests/integ tree. So every PR ran the same ~100 shallow tests twice on the same commit: once in fast-integ-tests, then again inside integ-tests' parallel pass. That is pure waste, and not free. Each shallow test submits a real CreateTrainingJob, so the duplicate pass doubles this suite's draw on the training-job quota the two projects share -- the exact contention the harness's concurrency cap exists to avoid -- and adds nothing, since both passes select the same tests by the same marker expression. Fixed by passing --ignore=tests/integ/train/shallow to the pytest invocation in tox.ini. Three properties make that the right lever: - The deep project goes through tox, so it picks the ignore up from the PR's own checked-out source. The project's buildspec is not defined in this repo, and the CDK package only defines the staging copy of it, so the buildspec is not something a PR here can change. - fast-integ-tests runs `python3.10 -m pytest tests/integ/train/shallow` directly rather than through tox, so it is unaffected and still runs the whole suite. - --ignore only prunes directory recursion; it does not override an explicitly named path. `tox -- tests/integ/train/shallow` still collects all 100 tests, so no local or scheduled workflow that names the directory loses coverage. Verified by collection against the deep project's own selection, `tests/integ -m "not serial and not gpu_intensive and not us_east_1"`: 348 collected / 200 selected before, 248 / 116 after. Exactly 100 collected and 84 selected drop out -- the shallow suite and nothing else -- matching the 82 shallow tests observed in the integ-tests log for run 32994923468. Naming the directory explicitly still collects 100. (One pre-existing collection error in test_list_hyperparameters_integration.py is a stale local install, present identically before and after.) Only the parallel pass was affected: the shallow tests carry no `serial` mark, so the serial pass never collected them -- 0 references in that half of the log against 211 in the other. X-AI-Prompt: also shallow tests are duplicated : in fast-integ-tests as well as integ-tests : Can we skip or remove shallow tests from integ-tests run for sagemaker-train ? X-AI-Tool: claude-code * docs(train): drop --dist loadfile from the shallow suite's README The flag was removed from createCIShallowIntegBuildSpec in SageMakerMLFPySDKInfraCDK: it pins one file's tests to one xdist worker, and each test here holds a concurrency slot until its training job reaches a terminal state (~75s), so the 17-test RLVR file alone took 19m45s against the project's 30-minute timeout. Job names are unique per invocation rather than per test function (see unique_name in harness.py), so tests are free to spread across workers, which is what the wall-clock estimate in harness.py assumes. Documentation only -- the invocation itself lives in the CDK package. --- X-AI-Prompt: The fast-integ-tests CodeBuild job is still failing on PRs now that the CDK project is deployed -- diagnose and fix it X-AI-Tool: claude-code
1 parent b2f88c5 commit 1b14335

2 files changed

Lines changed: 16 additions & 2 deletions

File tree

‎sagemaker-train/tests/integ/train/shallow/README.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,12 @@ What this suite changes about the gate is not which job runs, but what the exist
88
one selects: the `gpu_intensive` marks added here deselect the deep tests that
99
submit a job and wait for it, and this suite covers those code paths instead.
1010

11+
`integ-tests` does not rerun this suite. It invokes pytest through tox over the
12+
whole `tests/integ` tree, which would otherwise sweep this directory in and submit
13+
every job here a second time on the same commit. `tox.ini` passes
14+
`--ignore=tests/integ/train/shallow` to keep that from happening; `fast-integ-tests`
15+
calls pytest directly, so the ignore does not apply to it.
16+
1117
> **Where this runs.** The `fast-integ-tests` job in `pr-checks-master.yml` does not
1218
> execute this suite on the GitHub runner — it starts the CodeBuild project
1319
> `sagemaker-python-sdk-ci-sagemaker-train-fast-integ-tests` via
@@ -19,7 +25,7 @@ submit a job and wait for it, and this suite covers those code paths instead.
1925
> — which is nearly all of them — without exposing those credentials to PR code.
2026
>
2127
> **Consequence for editing this suite:** the marker selection above (`-n 8`,
22-
> `--dist loadfile`, `-m "not gpu_intensive and not us_east_1"`) lives in
28+
> `-m "not gpu_intensive and not us_east_1"`) lives in
2329
> `createCIShallowIntegBuildSpec` in the `SageMakerMLFPySDKInfraCDK` package, not in
2430
> this repo. Adding a file under `shallow/` is picked up automatically, but changing
2531
> *how* the suite is invoked means a change there, which deploys through a pipeline

‎sagemaker-train/tox.ini‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,15 @@ commands =
9494
pip install 'torchvision==0.18.1+cpu' -f 'https://download.pytorch.org/whl/torch_stable.html'
9595
pip install 'dill>=0.3.9'
9696

97-
pytest {posargs}
97+
# --ignore keeps the shallow suite out of directory sweeps like
98+
# `tox -- tests/integ`, which is how the deep integ-tests CodeBuild project
99+
# invokes pytest. Without it, that project reruns all ~100 shallow tests
100+
# that fast-integ-tests has already run on the same commit -- duplicate
101+
# CreateTrainingJob calls against the shared job quota, for no extra
102+
# coverage. fast-integ-tests calls pytest directly rather than through tox,
103+
# so it is unaffected. --ignore only prunes recursion, so naming the
104+
# directory explicitly (`tox -- tests/integ/train/shallow`) still runs it.
105+
pytest --ignore=tests/integ/train/shallow {posargs}
98106
deps =
99107
-r ../requirements/extras/test_requirements.txt
100108
../sagemaker-core

0 commit comments

Comments
 (0)