Repository navigation
Commit b8b02f8
authored
periodic: give dl3 and edsr the time they need on the portable backend (pytorch#21693)
Scheduled `periodic` runs never report success. Two jobs are
responsible, and
both are timeouts rather than test failures:
- `test-models-linux (cmake, dl3, portable, linux.4xlarge.memory, 90)`
- `test-models-linux (cmake, edsr, portable, linux.2xlarge, 90)`
Each one is killed at the 90 minute cap and reports `cancelled`, on
every tick,
going back to at least 2026-07-31. A cancelled job makes the whole run
cancelled,
so the run level result carries no information about anything else in
the matrix.
That is roughly 9 runner hours a day producing no signal.
Neither model is broken. Both export correctly on the portable backend
and both
pass their tests. They just need more than 90 minutes. This raises the
cap for each
one to a number backed by a measurement.
## The measurements
Each job was re-run with a larger cap and nothing else changed.
| job | runner | cap | result |
| --- | --- | --- | --- |
| dl3, portable | `linux.4xlarge.memory` | 150 min | success at 100.8
min |
| dl3, portable (second sample) | `linux.4xlarge.memory` | 240 min |
success at 101.2 min |
| dl3, portable (third sample) | `linux.4xlarge.memory` | 240 min |
success at 101.4 min |
| edsr, portable | `linux.2xlarge` | 330 min | success at 204.1 min |
dl3 is tightly reproducible: three samples within 0.6 minutes of each
other. A cap
of 150 leaves about 49 minutes of headroom, which covers the tens of
minutes of
variance the setup steps ahead of the script have shown on that runner
label.
edsr had never been allowed to run to completion before, so 204.1
minutes is the
first real number for it. A cap of 300 leaves about 96 minutes of
headroom, the
same proportion as dl3. There is only one sample here, so the margin is
deliberately generous rather than tight.
## The change
In `.ci/scripts/gather_test_models.py`:
```python
CUSTOM_TIMEOUT = {
"linux": {
"mobilebert": 90,
"dl3": 150,
"edsr": 300,
"emformer_predict": 360,
"llama3_2_text_decoder": 360,
},
```
`emformer_predict` and `llama3_2_text_decoder` already sit at 360 in
this same
table, so a per model cap is the normal mechanism here, not a
workaround.
`CUSTOM_TIMEOUT` is keyed by model, not by model and backend, so this
also raises
the cap for the `xnnpack-quantization-delegation` variant of each model.
Those two
jobs finish in about 12 minutes (dl3) and 11 minutes (edsr) and are
unaffected: a
timeout is a ceiling, not a reservation.
## What changed since the first version of this pull request
The first version skipped dl3 on portable entirely and moved edsr to
`linux.4xlarge.memory`. The measurements show both of those were wrong:
- Skipping dl3 would have deleted coverage for a job that passes fine
given the
time.
- A bigger runner does nothing for edsr. At a 150 minute cap it was
cancelled at
150.6 minutes on `linux.2xlarge` and at 151.1 minutes on
`linux.4xlarge.memory`.
Those are the same number. edsr is simply a 205 minute job, so it stays
on
`linux.2xlarge` and gets the time instead.
Both of those changes have been removed. Nothing is skipped and no
runner moves.
## Test plan
The generated matrix was diffed against the one `main` produces. Both
have 58
entries, and exactly four differ, all of them timeouts:
```
dl3 portable 90 -> 150
dl3 xnnpack-quantization-delegation 90 -> 150
edsr portable 90 -> 300
edsr xnnpack-quantization-delegation 90 -> 300
```
No entry is added, removed, or moved to a different runner.
`periodic` also runs on this pull request through the `ciflow/periodic`
label,
which makes it gather the full scheduled matrix at this branch's head.
That run is
the end to end check that dl3 and edsr both report `success` instead of
`cancelled`. It is in flight here, and `gather-models` has already
produced the
two jobs with their new caps:
```
test-models-linux (cmake, dl3, portable, linux.4xlarge.memory, 150)
test-models-linux (cmake, edsr, portable, linux.2xlarge, 300)
```
https://github.com/pytorch/executorch/actions/runs/314523709791 parent c88f5fe commit b8b02f8
1 file changed
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
45 | 51 | | |
46 | 52 | | |
47 | 53 | | |
| |||
0 commit comments