Skip to content

Commit 478ecbc

Browse files
leggetterclaude
andcommitted
docs(ci): the acceptance concurrency group cancels pending runs; say so
The comment on the acceptance-suite concurrency group said "Queueing costs wall-clock and nothing else." That is wrong. A concurrency group holds one running run and one pending run; when another joins, the older pending run is cancelled. cancel-in-progress: false protects only the running one. So several pull requests pushed in a burst leave all but the newest with every acceptance job CANCELLED -- not a failure, not coverage, and not blocking, since acceptance is not a required check. It happened to three of four v3.0.2 PRs in one afternoon. Worse, a release gate that is pending rather than running is cancelled by a PR joining behind it, and the builds and npm publish that need it never run. The same comment also cited "a single named test failing while a release ran concurrently" as a second instance of the collision. That was the outpost custom-domain test, and it was not a collision: the API returns a 500 after the domain is already created, and the harness retry then gets a 409 for the test's own hostname. The misdiagnosis had become evidence; it is now recorded as a counterexample instead. The review skill gains "a cancelled acceptance run is not a run either", with how to re-run one at a time. The release checklist gains a gate: no acceptance run in progress or queued before `gh release create`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012XtSQ2kfpcRXXqweskgXkH
1 parent 492bb68 commit 478ecbc

3 files changed

Lines changed: 57 additions & 7 deletions

File tree

‎.agents/skills/hookdeck-cli-release/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ Follow **in order**. Treat items with **gate** as blocking unless the maintainer
3030
- [ ] **Release notes:** Draft complete (see **Drafting release notes** and [references/release-notes-template.md](references/release-notes-template.md)); includes **Full Changelog** compare link; **contributor shout-outs only when warranted** (see that section).
3131
- [ ] **CI gate:** Latest commit on the **target branch** has **green** GitHub checks (mandatory for GA on `main`; required for betas on the branch being tagged).
3232
- [ ] **Approval:** Maintainer signed off on tag name, notes, and branch — no unilateral surprise tags.
33+
- [ ] **Acceptance group is idle (gate):** no acceptance run in progress or queued — `gh run list --workflow=test-acceptance.yml --status in_progress` and `--status queued` both empty. The release gate shares the `acceptance-suite` concurrency group, which keeps only one *pending* run: if the release's acceptance run is left pending, a pull request pushed behind it cancels it, and the builds and npm publish that depend on it never run. Don't open or push PRs while a release is building.
3334
- [ ] **Publish:** Write notes to a **temporary file**, run **`gh release create`** (see **Publish with GitHub CLI (`gh`)**), then **`rm`** the temp file. Use `--prerelease` for betas. (Humans may still use the GitHub UI per README.)
3435
- [ ] **Post-publish (optional):** Confirm the **`release`** workflow in Actions completed successfully for the new tag.
3536

‎.agents/skills/hookdeck-cli-review/SKILL.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,41 @@ than the colour:
6666
gh run view --job=<id> --log | grep -E -- '--- (SKIP|PASS): <TestName>'
6767
```
6868

69+
### A cancelled acceptance run is not a run either
70+
71+
Every acceptance run shares one concurrency group, `acceptance-suite`, so runs
72+
against the same test projects cannot collide. The group holds **one running
73+
run and one pending run**. When a third joins, the older *pending* one is
74+
cancelled; `cancel-in-progress: false` protects only the one that is running.
75+
76+
So when several pull requests are opened or pushed close together, every one
77+
but the newest can end with all five acceptance jobs `CANCELLED`. That happened
78+
to four v3.0.2 PRs in one afternoon. It is not a failure, and because
79+
acceptance is not a required check, the merge button stays green.
80+
81+
**Read `CANCELLED` as "never ran", and re-run it — one at a time.** Re-running
82+
two together puts both in the single pending slot and cancels one again. Queue
83+
the next only once the previous has started running:
84+
85+
```
86+
gh run rerun <run-id>
87+
gh run view <run-id> --json status -q .status # wait for in_progress, then the next
88+
```
89+
90+
The release gate lives in the same group. A release whose acceptance run is
91+
*pending* is cancelled by a pull request that joins behind it, and the builds
92+
and npm publish that depend on it never run. Before a release, check that no
93+
acceptance run is in progress or queued:
94+
95+
```
96+
gh run list --workflow=test-acceptance.yml --status in_progress
97+
gh run list --workflow=test-acceptance.yml --status queued
98+
```
99+
100+
And if a workflow change is in the diff: opening that pull request triggers
101+
acceptance too (`.github/workflows/**` is in the path allowlist), so it can
102+
cancel someone else's pending run.
103+
69104
### The one case where it genuinely does not run
70105

71106
`pull_request` does not fire when a pull request's head branch is updated by

‎.github/workflows/acceptance.yml‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,28 @@ on:
1616
# The failure is nasty because it does not look like rate limiting. It surfaces
1717
# as ordinary assertion failures in whichever slice lost the race -- a config
1818
# that came back empty, a filter that returned nothing -- sending whoever reads
19-
# it hunting for a regression that does not exist. It has happened twice: once
20-
# as four jobs timing out with zero assertion failures, and once as a single
21-
# named test failing while a release ran concurrently.
19+
# it hunting for a regression that does not exist. The clear case on record is
20+
# four jobs timing out with zero assertion failures.
2221
#
23-
# cancel-in-progress is deliberately false. Cancelling would be faster, but the
24-
# release workflow calls this as its publish gate, and killing that run to make
25-
# room for a pull request is the wrong trade. Queueing costs wall-clock and
26-
# nothing else.
22+
# (A red TestOutpostTenantPortalAndCustomDomain during a concurrent release was
23+
# once blamed on this too. It was not: the custom-domain API can return a 500
24+
# after the domain is already created, and the retry then gets a 409 for the
25+
# test's own hostname. Not every red run during an overlap is the overlap.)
26+
#
27+
# cancel-in-progress is deliberately false, so a RUNNING run is never killed to
28+
# make room for another -- the release gate included.
29+
#
30+
# But this is not a queue. A concurrency group holds at most one running run and
31+
# ONE pending run. When another run joins, the older pending run is CANCELLED.
32+
# So several pull requests opened or pushed in a burst leave all but the newest
33+
# with acceptance "CANCELLED": not a failure, but not coverage either, and since
34+
# acceptance is not a required check nothing blocks the merge. Re-run them one
35+
# at a time -- re-running two at once cancels one of them again.
36+
#
37+
# The same applies to the release gate when it is pending rather than running:
38+
# a pull-request run joining the group cancels it, and the builds and npm
39+
# publish that `need` it never run. Before `gh release create`, make sure no
40+
# acceptance run is in progress or queued.
2741
concurrency:
2842
group: acceptance-suite
2943
cancel-in-progress: false

0 commit comments

Comments
 (0)