Repository navigation
Commit 85ec16e
Promote commits to viable/strict on a schedule, not by commit count (pytorch#22064)
# PR pytorch#22064: Promote commits to viable/strict on a schedule, not by
commit count
## Problem
viable/strict only advanced from commits whose position in main's
history happened to be
divisible by 4. Every other commit was permanently ineligible, and the
manual escape
hatch for promoting one did not work, so a commit could sit outside
viable/strict
indefinitely and miss the nightly build that is cut from it.
Two independent reasons the escape hatch did not work:
1. `viable-strict-gate` ran on every push and failed on commits outside
the sample. The
viable/strict updater reads one row per workflow **run** rather than per
workflow, so
pushing a `ciflow/trunk` tag afterwards added a second, successful row
beside the
failed one instead of replacing it. Any single failure is fatal, so the
commit stayed
ineligible forever. A comment in `promote-to-viable-strict.yml` claimed
the updater
"resolves multiple runs by taking the latest conclusion", which is not
what it does.
2. `promote-to-viable-strict` pushed its tag using the default
`GITHUB_TOKEN`. A ref
pushed with that token does not start any workflow run, so the tag
triggered neither
the full CI it was meant to force nor the gate.
Counting commits is also the wrong unit. When the repository is busy,
four commits can be
twenty minutes apart and full CI runs more often than it needs to. When
it is quiet, four
commits can span two days. Measured over the last two weeks of main, the
longest a commit
waited to become eligible was **56.7 hours**; over 30 days it was **58.0
hours**. Either
is long enough to miss a whole nightly build.
## Changes
- `viable-strict-gate` runs only on `ciflow/trunk` tags. A plain push
produces no run of
it, so the updater reports the required check as missing and declines to
advance.
The gate never writes a failure row, so it can no longer be the thing
that poisons a
commit. To be clear about the scope of that: the other five required
checks still run
on push and can still leave a permanent failure row, so this removes one
source of
poisoning rather than all of them.
- `promote-to-viable-strict` runs every 4 hours and promotes the tip of
main. On a
scheduled tick it exits without doing anything when viable/strict
already contains the
commit, and on any run it exits when the tag already exists, so repeated
ticks over a
quiet period do not re-run CI. One tick sits ahead of the nightly branch
cut, so the
last commits of a quiet night still reach that day's nightly build.
- `promote-to-viable-strict` checks out with a bot token so its tag push
actually starts
CI, and its `sha` input is now optional and defaults to the tip of main.
- Dropped the step that deleted failed gate runs. It could not help: the
updater's data
comes from webhook events, and deleting a run does not retract the row
it already
produced. With the gate no longer running on push there is nothing left
to delete
either.
The depth based sample in `_ci-run-decision.yml` is unchanged, and must
stay. Seven
workflows re-run from a `ciflow/trunk` tag, but only four of them call
`_ci-run-decision.yml`. The other seven callers (cuda, cuda-windows,
cuda-perf, metal,
mlx, rocm, vulkan) have their own ciflow tags but not `ciflow/trunk`, so
a promotion tag
never starts them, and they hold 32 of the 48 jobs whose `if:` gates on
`is-full-run`.
Most of those 32 also have a changed-files branch and still run on main
when their own
paths are touched. The 12 in `mlx.yml` do not, so for those the depth
sample is the only
thing that ever runs them on main.
## Cost and the alternative that was considered
`pull`, `trunk`, `Apple`, `Windows MSVC Build`, `Test RISC-V Backend`
and
`Test QNN Windows MSVC build` trigger on both a push to main and a
`ciflow/trunk` tag,
and their concurrency keys differ, so neither run cancels the other. A
promoted commit
therefore runs those workflows twice.
The obvious cheaper alternative is to make `is-full-run` true for the
first commit in
each 4 hour wall clock window, computed from committer timestamps,
instead of every
fourth commit by depth. No tag, no new workflow, no token. Measured over
main's history:
| | full-run events per day | vs today | longest observed wait to become
eligible |
|---|---|---|---|
| today, every 4th commit | 4.14 | 1.00x | 56.7 h |
| this PR | 8.21 | 1.98x | 4 h plus CI time |
| first commit per 4 h window | 4.07 | 0.98x | 20.2 h |
Figures are over 14 days. Over 30 days: 3.30 / 7.30 / 4.00 events per
day, 1.00x / 2.21x
/ 1.21x, and longest observed waits of 58.0 h / 4 h plus CI time / 28.3
h.
Three things about that table need saying, because a quick reading of it
gets all three
wrong.
**The alternative does not actually fix the problem.** Its worst
observed wait is 20 to
28 hours, not 4, and its true worst case is unbounded. A wall clock
window only opens
when a commit arrives, so during a quiet night it promotes nothing,
which is exactly the
case the nightly build cares about. This change promotes the tip of main
on a timer
whether or not anything new landed, so a commit is eligible one tick
after it lands.
That is the difference between missing a nightly and not missing one,
and it is the
reason to prefer this approach despite the cost.
**The 4 hours is the cron spacing, not a guarantee.** Add CI wall clock
and up to 30
minutes for the next updater tick, and GitHub can delay or drop
scheduled runs under
load. On the one real promotion tag I could measure, all four required
tag-started
workflows were green 2.2 hours after the tag, and that figure includes a
hand re-run of
a cancelled `trunk` attempt, so it is not a clean unattended
measurement. The 08:00 UTC
tick is about 3.5 hours ahead of the nightly cut, so the margin is real
but it is not
large.
**The cost is worse than the event count suggests, not better.** A
promotion tag does not
just re-run the jobs gated on `is-full-run`; it starts seven whole
workflows over again.
Measured on commit `cff6f4dd`, which received a real `ciflow/trunk` tag:
the tag run cost
6,046 runner-minutes (latest attempt of each job, skipped jobs
excluded), and about 85
percent of that by runner-minutes, 82 percent by job count, repeated
work the push to
main had already finished. Per event, a promotion is more expensive than
a depth sample,
not less.
Where that lands in absolute terms: roughly 4.07 promotions a day at
about 6,000
runner-minutes each is about 24,600 runner-minutes a day added, against
roughly 128,000
a day for main CI today. So about **1.19x** on the total main CI bill.
That is the honest
number, and it is smaller than 1.98x only because full runs are a
minority of the bill,
not because the added events are cheap.
One further cost not in the table: a promoted commit produces roughly
twice the job rows
the updater must find green, so it roughly doubles the exposure to a
flake blocking
advancement.
If the cost is judged too high, the direction I would look first is not
the wall clock
alternative but adding `ciflow/trunk/*` to the seven callers that do not
listen for it,
so that a tag unlocks all 48 gated jobs instead of 16. A rough estimate
puts that at
about 1.16x against this change's 1.19x, so the saving is real but
small, and I have not
measured it properly. Note that it would not allow deleting the depth
sample: the sample
guarantees at most 3 consecutive commits without a full run, whereas
promoting only the
tip at each tick guarantees nothing within a window. The busiest 4 hour
window in the
last two weeks held 16 commits.
## Correction to an earlier version of this description
An earlier version said `GH_PYTORCHBOT_TOKEN` was not set on this
repository and asked
for it to be added as a repository secret. That was wrong on both
counts.
The secret is set. It is an **environment** secret on the
`update-commit-hash`
environment, alongside `UPDATEBOT_TOKEN`. The real defect was in this
change: the promote
job did not declare `environment: update-commit-hash`, so it could not
read the secret
and would have failed on every run. That is now fixed, following the
same pattern as
`weekly-pytorch-pin-bump.yml`.
Adding it as a repository secret would also have been the wrong fix,
because a repository
secret is readable by any workflow on any ref.
The earlier version also said `nightly.yml`, `cherry-pick.yml`,
`ghstack_land.yml`,
`apple.yml` and `weekly-pytorch-pin-bump.yml` reference unset bot
tokens. That was a
consequence of the same mistake and is not true. All five are correctly
wired:
cherry-pick and ghstack_land use `GH_PYTORCHBOT_CHERRY_PICK_TOKEN` on
the
`cherry-pick-bot` environment, weekly-pytorch-pin-bump uses
`UPDATEBOT_TOKEN`, apple.yml
falls back to `GITHUB_TOKEN`, and nightly.yml already declares the
environment. No action
is needed on any of them.
## Test plan
Ran the promote script against a real clone with the push and the remote
tag lookup
stubbed out, covering every branch:
- No `sha` input, which is the scheduled path: resolves the tip of main,
passes
validation and reachability, and stops cleanly on the already existing
tag. Exit 0.
- A hand dispatched SHA that is already contained in viable/strict: now
proceeds to the
tag push instead of exiting early. This was a real defect. 13272 of the
13274
first-parent commits on main are already ancestors of viable/strict, so
the old
behaviour turned nearly every deliberate dispatch into a silent no-op
that still
reported success.
- A malformed SHA, and a well-formed SHA that is not a commit: both
rejected with an
explicit error before anything is pushed.
- All three workflow files parse as YAML and the extracted shell passes
`bash -n`.
## Known limitation
The `update-commit-hash` environment's deployment branch policy allows
`main` only, so
the promote job cannot be dispatched from this branch and is first
exercised after merge.
Everything except the tag push and the secret read is covered by the
runs above.
---------
Co-authored-by: pytorchbot <pytorchbot@users.noreply.github.com>1 parent b8b5e0f commit 85ec16e
3 files changed
Lines changed: 143 additions & 86 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | | - | |
19 | | - | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
20 | 34 | | |
21 | 35 | | |
22 | 36 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
4 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
5 | 10 | | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
12 | 14 | | |
13 | | - | |
14 | | - | |
15 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
16 | 18 | | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
22 | 27 | | |
23 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
24 | 33 | | |
25 | 34 | | |
26 | 35 | | |
27 | | - | |
28 | | - | |
| 36 | + | |
| 37 | + | |
29 | 38 | | |
30 | 39 | | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | 40 | | |
38 | 41 | | |
39 | 42 | | |
| |||
43 | 46 | | |
44 | 47 | | |
45 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
46 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
47 | 78 | | |
48 | 79 | | |
49 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
50 | 86 | | |
51 | 87 | | |
52 | 88 | | |
53 | | - | |
| 89 | + | |
54 | 90 | | |
55 | 91 | | |
56 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
57 | 100 | | |
58 | 101 | | |
59 | 102 | | |
| |||
90 | 133 | | |
91 | 134 | | |
92 | 135 | | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
93 | 150 | | |
94 | 151 | | |
95 | | - | |
96 | | - | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
97 | 155 | | |
98 | 156 | | |
99 | 157 | | |
| |||
104 | 162 | | |
105 | 163 | | |
106 | 164 | | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
8 | | - | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
9 | 9 | | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
15 | 17 | | |
16 | | - | |
17 | | - | |
18 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
19 | 25 | | |
20 | 26 | | |
21 | 27 | | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | 28 | | |
26 | 29 | | |
27 | 30 | | |
| |||
37 | 40 | | |
38 | 41 | | |
39 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
40 | 57 | | |
41 | 58 | | |
42 | 59 | | |
| |||
47 | 64 | | |
48 | 65 | | |
49 | 66 | | |
50 | | - | |
51 | | - | |
| 67 | + | |
52 | 68 | | |
0 commit comments