You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3e107c5
Browse filesBrowse the repository at this point in the historyBrowse files
Stop nightly scheduled runs from cancelling in-flight push CI on main (pytorch#21691)
## What is broken
Every so often a commit on main shows a cancelled ARM or CoreML job on
HUD even though nobody cancelled anything and the commit is fine. It
looks like a real failure, so the oncall investigates a phantom.
## Why it is broken
GitHub Actions cancels an older run when a newer run lands in the same
concurrency group. These workflows use a group key built like this:
```
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}
cancel-in-progress: true
```
The key already tells apart pull requests, branches, commits and manual
runs. It does not tell apart a scheduled run.
These same workflows also have a nightly `schedule` trigger. When the
nightly cron fires, `github.ref_name` is `main` and `github.sha` is the
current head of main, which are exactly the same values the push run for
that commit used. Same key, `cancel-in-progress: true`, so the cron run
kills the push run that was still going.
The result is a commit whose CI reads as cancelled with no explanation.
## The fix
Add `-${{ github.event_name == 'schedule' }}` to the concurrency group,
so a scheduled run and a push run on the same commit sit in different
groups and no longer cancel each other.
This is the same pattern already used by other workflows in this repo.
It is applied here to every workflow that has both a `schedule` and a
`push` trigger:
- `test-backend-arm.yml`
- `test-backend-coreml.yml`
- `test-backend-openvino.yml`
- `test-backend-qnn.yml`
- `test-backend-vulkan.yml`
- `test-backend-webgpu.yml`
- `test-backend-xnnpack.yml`
- `test-webgpu-native.yml`
- `build-cmsis-pack.yml`
- `docker-builds.yml`
`doc-build.yml` also has both triggers but is deliberately left alone:
it sets `cancel-in-progress: ${{ github.event_name == 'pull_request'
}}`, so outside a pull request it never cancels anything and the
collision cannot happen there.
This only separates schedule from push. It does not change how two
scheduled runs
of the same workflow behave: `github.sha` is still in the key, so two
scheduled
runs share a group only while `main` has not moved between them, and
that is
unchanged by this PR.
## How this was verified
Every workflow under `.github/workflows` with a `schedule:` trigger was
listed
together with its concurrency group, to make sure the change is both
correct and
complete:
- `apple.yml`, `periodic.yml` and `riscv64.yml` already carry exactly
the
`-${{ github.event_name == 'schedule' }}` suffix this PR adds, so this
is not a
new idea, it is existing practice in this repo. `periodic.yml` goes one
step
further and also appends `${{ github.event.schedule }}` to separate its
individual crons.
- The 10 workflows changed here are precisely the ones still on the
older key
that also have a `push` trigger, so none were missed.
- `build-cadence-runner.yml` and `test-pico2-build.yml` already put
`${{ github.event_name }}` in the key, which covers the same collision.
- `nightly.yml` has both triggers but only pushes on `ciflow/nightly/*`
tags, so
its `github.ref_name` is a tag and can never match the scheduled run on
`main`.
- `doc-build.yml` is the case described above.
- The other scheduled workflows have no `concurrency` block at all.
Note that the fix only takes effect once it is on `main`, because a
scheduled run
always uses the workflow file from the default branch.
CI on this pull request: 213 checks reported, 195 success, 15 skipped, 2
cancelled, 0 failures. Both cancellations were manual, and in one of
them every
step had already finished successfully before the cancel landed.
## Overlap with other pull requests
Two of the files here are also touched by other open pull requests, in
both cases
far away from the `concurrency:` block this PR edits:
- `.github/workflows/test-webgpu-native.yml` also gets a `timeout:`
change in
pytorch#21690, in the job block further down.
- `.github/workflows/test-backend-coreml.yml` also gets a matrix
`exclude:` entry
in pytorch#21695, likewise further down.
The hunks do not touch and the changes merge cleanly in any order.
0 commit comments