ci: retry exit-status 128 more times before giving up - #38588
Draft
bosconi wants to merge 1 commit into
Draft
Conversation
`set_retry_on_agent_lost` already treats git exit 128 as a temporary connection issue, but with `limit: 2` alongside the other rules. That limit is calibrated for a one-off failure, and the failure mode it actually meets is a burst. Measured on 2026-08-31, when GitHub began refusing the Hetzner agents' anonymous git requests: 9901 job attempts across `test`, `nightly` and `release-qualification` failed with exit 128, 99.0% of every failed attempt that day. The rule fired as designed - affected steps show `[original, automatic, automatic]` - and was exhausted anyway, because Buildkite requeues an automatic retry within seconds while the bursts last 1-6 minutes. In build 133236, `Lint dependencies` burned all three attempts between 17:54:28 and 17:55:05, 37 seconds, inside a burst that ran for several minutes. Raise the limit to 8 for this exit status only. Buildkite's automatic retry has no delay knob, so more attempts is the only lever available and this is mitigation rather than a fix: it buys a couple of minutes of coverage and lets a job that straddles the end of a burst land, instead of failing the whole build. The durable fixes are authenticating the checkout or giving the fleet a git cache, neither of which lives here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft - mitigation for an ongoing problem, opened for discussion rather than as the fix.
Motivation
On 2026-08-31, GitHub began refusing the Hetzner CI agents' anonymous git requests in recurring bursts. Measured across the day:
testnightlyrelease-qualificationConfined to Hetzner -
buildkite-*,buildkite-aarch64-*andmac-*agents took zero exit-128 failures while cloning the same public repos at the same time. 18 bursts between 10:39 and 16:48 UTC, each 1-6 minutes.Description
set_retry_on_agent_lostalready classifies exit 128 as a temporary connection issue, and the rule fires correctly - affected steps show[original, automatic, automatic]. It is exhausted anyway, because Buildkite requeues an automatic retry within seconds while the bursts last minutes. In build 133236,Lint dependenciesspent all three attempts between 17:54:28 and 17:55:05 - 37 seconds - inside a burst that ran for several minutes. 171 of that build's job attempts failed, every one with exit 128.This raises the limit to 8 for exit 128 only, leaving the neighbouring rules at 2.
What this is not
Buildkite's
retry.automatichas no delay option, so attempt count is the only lever available here. That makes this mitigation, not a fix: it buys a couple of minutes of coverage and lets a job straddling the end of a burst land rather than failing the whole build. It will not save a job that starts in the middle of a six-minute burst.The durable fixes are elsewhere: authenticate the agent checkout, or give the fleet a git cache/mirror. Both have in-repo precedent for the authenticated-git pattern (
ci/license/bump-change-date.shpushes asmaterializebot:$GITHUB_TOKEN,ci/test/console/e2e.shclones over SSH) but neither is a change to this file.Tradeoff for the reviewer
Eight attempts on a genuinely broken git operation wastes more agent time than two. Mitigating factors: exit 128 is already declared temporary by the existing comment, and a persistent 128 is rare compared to what we just measured. If you would rather cap the blast radius, 4 or 5 still roughly doubles the coverage.
Known related gap, deliberately not addressed here
set_retry_on_agent_lostbails out withif "automatic" in retry: continue, so any step that defines its own automatic retry gets none of these infra-failure defaults, including the exit-128 rule. There are 9 such occurrences inci/nightly/pipeline.template.ymland 2 inci/test/pipeline.template.yml. Merging the defaults instead of skipping them is a behaviour change for those steps and belongs in its own PR.🤖 Generated with Claude Code