Skip to content

Commit b8b02f8

Browse files
authored
periodic: give dl3 and edsr the time they need on the portable backend (pytorch#21693)
Scheduled `periodic` runs never report success. Two jobs are responsible, and both are timeouts rather than test failures: - `test-models-linux (cmake, dl3, portable, linux.4xlarge.memory, 90)` - `test-models-linux (cmake, edsr, portable, linux.2xlarge, 90)` Each one is killed at the 90 minute cap and reports `cancelled`, on every tick, going back to at least 2026-07-31. A cancelled job makes the whole run cancelled, so the run level result carries no information about anything else in the matrix. That is roughly 9 runner hours a day producing no signal. Neither model is broken. Both export correctly on the portable backend and both pass their tests. They just need more than 90 minutes. This raises the cap for each one to a number backed by a measurement. ## The measurements Each job was re-run with a larger cap and nothing else changed. | job | runner | cap | result | | --- | --- | --- | --- | | dl3, portable | `linux.4xlarge.memory` | 150 min | success at 100.8 min | | dl3, portable (second sample) | `linux.4xlarge.memory` | 240 min | success at 101.2 min | | dl3, portable (third sample) | `linux.4xlarge.memory` | 240 min | success at 101.4 min | | edsr, portable | `linux.2xlarge` | 330 min | success at 204.1 min | dl3 is tightly reproducible: three samples within 0.6 minutes of each other. A cap of 150 leaves about 49 minutes of headroom, which covers the tens of minutes of variance the setup steps ahead of the script have shown on that runner label. edsr had never been allowed to run to completion before, so 204.1 minutes is the first real number for it. A cap of 300 leaves about 96 minutes of headroom, the same proportion as dl3. There is only one sample here, so the margin is deliberately generous rather than tight. ## The change In `.ci/scripts/gather_test_models.py`: ```python CUSTOM_TIMEOUT = { "linux": { "mobilebert": 90, "dl3": 150, "edsr": 300, "emformer_predict": 360, "llama3_2_text_decoder": 360, }, ``` `emformer_predict` and `llama3_2_text_decoder` already sit at 360 in this same table, so a per model cap is the normal mechanism here, not a workaround. `CUSTOM_TIMEOUT` is keyed by model, not by model and backend, so this also raises the cap for the `xnnpack-quantization-delegation` variant of each model. Those two jobs finish in about 12 minutes (dl3) and 11 minutes (edsr) and are unaffected: a timeout is a ceiling, not a reservation. ## What changed since the first version of this pull request The first version skipped dl3 on portable entirely and moved edsr to `linux.4xlarge.memory`. The measurements show both of those were wrong: - Skipping dl3 would have deleted coverage for a job that passes fine given the time. - A bigger runner does nothing for edsr. At a 150 minute cap it was cancelled at 150.6 minutes on `linux.2xlarge` and at 151.1 minutes on `linux.4xlarge.memory`. Those are the same number. edsr is simply a 205 minute job, so it stays on `linux.2xlarge` and gets the time instead. Both of those changes have been removed. Nothing is skipped and no runner moves. ## Test plan The generated matrix was diffed against the one `main` produces. Both have 58 entries, and exactly four differ, all of them timeouts: ``` dl3 portable 90 -> 150 dl3 xnnpack-quantization-delegation 90 -> 150 edsr portable 90 -> 300 edsr xnnpack-quantization-delegation 90 -> 300 ``` No entry is added, removed, or moved to a different runner. `periodic` also runs on this pull request through the `ciflow/periodic` label, which makes it gather the full scheduled matrix at this branch's head. That run is the end to end check that dl3 and edsr both report `success` instead of `cancelled`. It is in flight here, and `gather-models` has already produced the two jobs with their new caps: ``` test-models-linux (cmake, dl3, portable, linux.4xlarge.memory, 150) test-models-linux (cmake, edsr, portable, linux.2xlarge, 300) ``` https://github.com/pytorch/executorch/actions/runs/31452370979
1 parent c88f5fe commit b8b02f8

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

‎.ci/scripts/gather_test_models.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,12 @@
4242
# Just some examples on how custom timeout can be set
4343
"linux": {
4444
"mobilebert": 90,
45+
# dl3 and edsr on the portable backend take about 100 and 205 minutes, so at
46+
# the default 90 they were killed on every periodic run and never reported a
47+
# result. Both pass given the time. This also raises the cap for their xnnpack
48+
# variants, which still finish in about 10 minutes and are unaffected.
49+
"dl3": 150,
50+
"edsr": 300,
4551
"emformer_predict": 360,
4652
"llama3_2_text_decoder": 360,
4753
},

0 commit comments

Comments
 (0)