Skip to content

Commit 6766736

Browse files
authored
Drop the coreml_static_int8 operators job, which has never once finished (pytorch#21695)
Addresses pytorch#21623. ## What is broken Every night, the CoreML backend workflow runs this job on a macOS runner: ``` test-backend-macos (coreml_static_int8, operators) ``` It has never finished. It runs for the full 180 minute limit, gets killed, and reports `cancelled`. That is about three macOS runner hours per night, every night, for no result at all, on a runner pool that is already the scarcest resource we have. For comparison, in a recent nightly run: | job | outcome | | --- | --- | | `coreml / operators` | passed in 11m41s | | `coreml_static_int8 / models` | passed in 26m41s | | `coreml_static_int8 / operators` | killed at 3h00m50s | So it is specifically the combination of the heavier quantized flow with the much larger operators suite that does not fit. ## Why we cannot just make it faster The suite already runs with `pytest -n auto`, so it is using every core on the runner. There is no sharding support in `_test_backend.yml` to split it across several runners. Adding that is real work, and it is the right long term answer, but it is not something to land while the job is burning runner hours nightly. ## The fix Stop scheduling that one combination. `_test_backend.yml` builds its matrix as every `flow` crossed with every `suite`, so today the only way to shape it is to change the flow list, which would also drop `coreml_static_int8 / models`, and that one passes and is worth keeping. This adds an optional `exclude` input to `_test_backend.yml` that is wired straight into `strategy.matrix.exclude`. It defaults to an empty list, so nothing changes for any other backend. `test-backend-coreml.yml` then uses it to drop the single `coreml_static_int8` plus `operators` pair. After this change the nightly CoreML run is `coreml / models`, `coreml / operators` and `coreml_static_int8 / models`, all three of which actually complete. ## When to revert this When the operators suite can be sharded across runners, or when the quantized flow gets fast enough to fit in the limit. pytorch#21623 tracks that. ## How this was verified Two separate things needed checking: that the new input is inert for every other caller, and that the exclude entry removes the job we mean and nothing else. - Inert by default. `_test_backend.yml` has 8 callers in total. With this branch's version of it, the 7 other callers still expand to every flow crossed with both suites. An `exclude` list that matches no combination is simply ignored by GitHub Actions, so the empty default is a no-op. - Correct entry. `test-backend-coreml.yml` only runs `coreml_static_int8` on the nightly schedule, so a pull request run would not show the effect at all. It was therefore dispatched on a scratch branch with both flows forced on. The resulting job list was `(coreml, models)`, `(coreml, operators)` and `(coreml_static_int8, models)`. The `(coreml_static_int8, operators)` pair was gone and nothing else moved. ## Overlap with other pull requests `.github/workflows/test-backend-coreml.yml` is also edited by pytorch#21691, which adds a schedule discriminator to the `concurrency:` block at the top of the file. This PR adds a matrix `exclude:` entry further down. The hunks do not touch and the two merge cleanly in either order.
1 parent 8b48cc6 commit 6766736

2 files changed

Lines changed: 11 additions & 0 deletions

File tree

‎.github/workflows/_test_backend.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,11 @@ on:
1111
description: 'JSON array of flows to test'
1212
required: true
1313
type: string
14+
exclude:
15+
description: 'JSON array of {flow, suite} pairs to drop from the matrix'
16+
required: false
17+
type: string
18+
default: '[]'
1419
ref:
1520
description: 'Git ref to checkout'
1621
required: false
@@ -50,6 +55,7 @@ jobs:
5055
matrix:
5156
flow: ${{ fromJSON(inputs.flows) }}
5257
suite: [models, operators]
58+
exclude: ${{ fromJSON(inputs.exclude) }}
5359

5460
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
5561
with:
@@ -126,6 +132,7 @@ jobs:
126132
matrix:
127133
flow: ${{ fromJSON(inputs.flows) }}
128134
suite: [models, operators]
135+
exclude: ${{ fromJSON(inputs.exclude) }}
129136

130137
uses: pytorch/test-infra/.github/workflows/macos_job.yml@main
131138
with:

‎.github/workflows/test-backend-coreml.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,10 @@ jobs:
4545
${{ github.event_name == 'schedule'
4646
&& '["coreml", "coreml_static_int8"]'
4747
|| '["coreml"]' }}
48+
# The operators suite under coreml_static_int8 has never finished inside the
49+
# 180 minute limit, so it is cancelled every night after burning about three
50+
# macOS runner hours for no result. See issue #21623.
51+
exclude: '[{"flow": "coreml_static_int8", "suite": "operators"}]'
4852
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
4953
timeout: 180
5054
run-macos: true

0 commit comments

Comments
 (0)