Repository navigation
Commit 6766736
authored
Drop the coreml_static_int8 operators job, which has never once finished (pytorch#21695)
Addresses pytorch#21623.
## What is broken
Every night, the CoreML backend workflow runs this job on a macOS
runner:
```
test-backend-macos (coreml_static_int8, operators)
```
It has never finished. It runs for the full 180 minute limit, gets
killed, and reports `cancelled`. That is about three macOS runner hours
per night, every night, for no result at all, on a runner pool that is
already the scarcest resource we have.
For comparison, in a recent nightly run:
| job | outcome |
| --- | --- |
| `coreml / operators` | passed in 11m41s |
| `coreml_static_int8 / models` | passed in 26m41s |
| `coreml_static_int8 / operators` | killed at 3h00m50s |
So it is specifically the combination of the heavier quantized flow with
the much larger operators suite that does not fit.
## Why we cannot just make it faster
The suite already runs with `pytest -n auto`, so it is using every core
on the runner. There is no sharding support in `_test_backend.yml` to
split it across several runners. Adding that is real work, and it is the
right long term answer, but it is not something to land while the job is
burning runner hours nightly.
## The fix
Stop scheduling that one combination.
`_test_backend.yml` builds its matrix as every `flow` crossed with every
`suite`, so today the only way to shape it is to change the flow list,
which would also drop `coreml_static_int8 / models`, and that one passes
and is worth keeping.
This adds an optional `exclude` input to `_test_backend.yml` that is
wired straight into `strategy.matrix.exclude`. It defaults to an empty
list, so nothing changes for any other backend.
`test-backend-coreml.yml` then uses it to drop the single
`coreml_static_int8` plus `operators` pair.
After this change the nightly CoreML run is `coreml / models`, `coreml /
operators` and `coreml_static_int8 / models`, all three of which
actually complete.
## When to revert this
When the operators suite can be sharded across runners, or when the
quantized flow gets fast enough to fit in the limit. pytorch#21623 tracks that.
## How this was verified
Two separate things needed checking: that the new input is inert for
every other
caller, and that the exclude entry removes the job we mean and nothing
else.
- Inert by default. `_test_backend.yml` has 8 callers in total. With
this
branch's version of it, the 7 other callers still expand to every flow
crossed
with both suites. An `exclude` list that matches no combination is
simply ignored by
GitHub Actions, so the empty default is a no-op.
- Correct entry. `test-backend-coreml.yml` only runs
`coreml_static_int8` on the
nightly schedule, so a pull request run would not show the effect at
all. It
was therefore dispatched on a scratch branch with both flows forced on.
The
resulting job list was `(coreml, models)`, `(coreml, operators)` and
`(coreml_static_int8, models)`. The `(coreml_static_int8, operators)`
pair was
gone and nothing else moved.
## Overlap with other pull requests
`.github/workflows/test-backend-coreml.yml` is also edited by pytorch#21691,
which adds a
schedule discriminator to the `concurrency:` block at the top of the
file. This PR
adds a matrix `exclude:` entry further down. The hunks do not touch and
the two
merge cleanly in either order.1 parent 8b48cc6 commit 6766736
2 files changed
Lines changed: 11 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
14 | 19 | | |
15 | 20 | | |
16 | 21 | | |
| |||
50 | 55 | | |
51 | 56 | | |
52 | 57 | | |
| 58 | + | |
53 | 59 | | |
54 | 60 | | |
55 | 61 | | |
| |||
126 | 132 | | |
127 | 133 | | |
128 | 134 | | |
| 135 | + | |
129 | 136 | | |
130 | 137 | | |
131 | 138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
48 | 52 | | |
49 | 53 | | |
50 | 54 | | |
0 commit comments