Repository navigation
Commit 1b14335
authored
fix(ci,train): stop integ-tests rerunning the shallow suite (#6216)
* fix(ci,train): stop integ-tests rerunning the shallow suite
The shallow suite lives under tests/integ/train/shallow, and the deep
integ-tests CodeBuild project invokes pytest through tox over the whole
tests/integ tree. So every PR ran the same ~100 shallow tests twice on the same
commit: once in fast-integ-tests, then again inside integ-tests' parallel pass.
That is pure waste, and not free. Each shallow test submits a real
CreateTrainingJob, so the duplicate pass doubles this suite's draw on the
training-job quota the two projects share -- the exact contention the harness's
concurrency cap exists to avoid -- and adds nothing, since both passes select
the same tests by the same marker expression.
Fixed by passing --ignore=tests/integ/train/shallow to the pytest invocation in
tox.ini. Three properties make that the right lever:
- The deep project goes through tox, so it picks the ignore up from the PR's own
checked-out source. The project's buildspec is not defined in this repo, and
the CDK package only defines the staging copy of it, so the buildspec is not
something a PR here can change.
- fast-integ-tests runs `python3.10 -m pytest tests/integ/train/shallow` directly
rather than through tox, so it is unaffected and still runs the whole suite.
- --ignore only prunes directory recursion; it does not override an explicitly
named path. `tox -- tests/integ/train/shallow` still collects all 100 tests,
so no local or scheduled workflow that names the directory loses coverage.
Verified by collection against the deep project's own selection, `tests/integ -m
"not serial and not gpu_intensive and not us_east_1"`: 348 collected / 200
selected before, 248 / 116 after. Exactly 100 collected and 84 selected drop
out -- the shallow suite and nothing else -- matching the 82 shallow tests
observed in the integ-tests log for run 32994923468. Naming the directory
explicitly still collects 100. (One pre-existing collection error in
test_list_hyperparameters_integration.py is a stale local install, present
identically before and after.)
Only the parallel pass was affected: the shallow tests carry no `serial` mark,
so the serial pass never collected them -- 0 references in that half of the log
against 211 in the other.
X-AI-Prompt: also shallow tests are duplicated : in fast-integ-tests as well as integ-tests : Can we skip or remove shallow tests from integ-tests run for sagemaker-train ?
X-AI-Tool: claude-code
* docs(train): drop --dist loadfile from the shallow suite's README
The flag was removed from createCIShallowIntegBuildSpec in
SageMakerMLFPySDKInfraCDK: it pins one file's tests to one xdist worker, and
each test here holds a concurrency slot until its training job reaches a
terminal state (~75s), so the 17-test RLVR file alone took 19m45s against the
project's 30-minute timeout. Job names are unique per invocation rather than
per test function (see unique_name in harness.py), so tests are free to spread
across workers, which is what the wall-clock estimate in harness.py assumes.
Documentation only -- the invocation itself lives in the CDK package.
---
X-AI-Prompt: The fast-integ-tests CodeBuild job is still failing on PRs now that the CDK project is deployed -- diagnose and fix it
X-AI-Tool: claude-code1 parent b2f88c5 commit 1b14335
2 files changed
Lines changed: 16 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
11 | 17 | | |
12 | 18 | | |
13 | 19 | | |
| |||
19 | 25 | | |
20 | 26 | | |
21 | 27 | | |
22 | | - | |
| 28 | + | |
23 | 29 | | |
24 | 30 | | |
25 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
98 | 106 | | |
99 | 107 | | |
100 | 108 | | |
| |||
0 commit comments