[ci]: make merge-gate startup reliable - #1762
Conversation
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
The trigger check failed while cancelling stale Buildkite builds, which ends the job before the merge-gate build is started, so this PR has no gate result rather than a failing one. Nothing here needs changing; this empty commit only produces the synchronize event that runs the workflow again. The underlying fault is fixed in hao-ai-lab#1762. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
This reproduced again today, so it is deterministic rather than a transient API blip.
The same workflow succeeded seven minutes earlier on two other branches, so the Buildkite API was not down:
#1725 has no failing test. Its only red check is this one, and re-triggering does not clear it, because the cancel step fails the same way every time and the job ends before step 7 starts the gate build. That PR therefore has no gate result rather than a failing one. Why this branch and not the others is still unknown, and the reason it is unknown is the second change in this PR: the current step does not print the response body, so the log shows only that jq could not parse something. With Because the workflow runs |
…fails The Trigger Merge Gate workflow cancels stale Buildkite builds in step 2 and starts the merge-gate build in step 7. A step failure ends the job, so when the cancel step dies the gate build is never started at all. The check then goes red for a reason unrelated to the pull request, and nothing in the log distinguishes "the tests failed" from "the tests never ran". That is what happened on hao-ai-lab#1710: jq: error (at <stdin>:1): Cannot index string with string "env" Process completed with exit code 5 `curl` is called without `--fail-with-body`, so an HTTP error is treated as success and its body is piped onward. Buildkite answers a rate limit or an unauthorized read with an object such as {"message": "Not Found"}, and `.[]` over an object yields its values, so `.env` then runs against a string and jq exits non-zero. Three changes, all to the same step: `continue-on-error: true`, because cancelling stale builds only saves agent time. Failing to cancel wastes an agent; failing to trigger means untested code, and step 7 keeps its hard failure. `--fail-with-body` on the lookup, with the response echoed as a warning, so a lookup that fails says why instead of feeding an error body to the parser. `if type == "array" then .[] else empty end` in the filter, so a response that is not a build list yields no matches rather than aborting. Verified against six response shapes: a matching build, a build for another PR, the error object from hao-ai-lab#1710, an empty array, a build with no env, and a bare string. The first returns the build number and the rest return nothing, where the error object previously exited 5.
cd978e6 to
2dc322b
Compare
|
/merge |
|
/test full |
|
/merge |
|
/test full |
Problem
The merge gate had two independent startup failures:
/mergerefreshed thereadylabel withGITHUB_TOKEN, but GitHub suppresses workflow runs caused by that token forlabeledevents. On a stable PR head, the command therefore did not startci-trigger-full-suiteat all.failingbuild can also retain active GPU jobs, as build #5043 demonstrated.Changes
ci-trigger-full-suite.ymla local reusable workflow and call it directly after the/mergeauthorization and label job succeeds. No PAT, GitHub App token, oractions: writepermission is introduced; only the existing Buildkite secret is passed explicitly.ready, and use its number/head/base/title outputs throughout the privileged workflow. PR head code is never checked out there.continue-on-error: true; the replacement Buildkite POST remains hard-fail.state[]=running,state[]=scheduled, andstate[]=failingparameters, and omit jobs/pipeline payloads.curl --fail-with-bodyfor list and cancel requests.This incorporates and supersedes #1760.
Verification
pytest -q fastvideo/tests/contract: 88 passed on exact head4c65c8e6.git diff --check upstream/main...HEAD: clean.curlverified that a failed first PUT still attempts the second, rejects an unrelated PR, and exits non-zero only forcontinue-on-errorhandling.The new privileged
/mergepath cannot run from a PR head before this workflow exists on the default branch; one post-merge canary should validate it end to end.The contributor author/date/message are preserved; the Claude co-author trailer was removed during the reviewed rewrite.