feat(hybridcloud): Chain a strict drain into its mailbox's next claim - #123268
Merged
vaind merged 0 commit intoAug 31, 2026
Merged
Conversation
vaind
force-pushed
the
ivan/webhook-drain-chain-dispatch
branch
2 times, most recently
from
August 31, 2026 22:51
e9b953e to
4fd8fa1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #123265 (which stacks on #122893).
Problem
A strict provider's mailbox burns one claim per scheduler round trip. The gap between claims isn't the ~10s cycle cadence: under a wide backlog the mailbox competes with every other due head for each cycle's dispatch budget, so a deep strict mailbox can wait many cycles between drains. Re-discovery, not delivery, dominates its burn rate — and strict providers are precisely the ones lanes-style concurrent claims can't serve (see #122897's closing rationale).
Change
A drain that ends healthy with due work behind it dispatches the mailbox's next claim itself —
_maybe_chain, gated byhybridcloud.webhookpayload.drain_chaining(default off). "Healthy with work behind" means exactly two endings:Never after a failure-stop or a lapse; backoff and the scheduler own those.
Why strict-only: the absolute-head gate admits one claim at a time, so a chain stays a single lineage per mailbox. A skip-on-failure provider in due-head mode would have the scheduler start a fresh claim behind every in-flight chain — one new self-sustaining pipeline per cycle, unbounded lanes.
Safety: the chain claims under the same drain lock and due-gate as every other dispatcher, so losing the race to a scheduler or push dispatch just continues the lineage under their attribution.
chain_depthrides the task args as a runaway backstop (MAX_CHAIN_DEPTH), far above what a healthy chain reaches — each link is bounded by its own claim, and the chain ends at the first short claim. A chain failure is swallowed: the drain's work is already delivered, and the scheduler covers the mailbox.Observability and rollout
Chain dispatches land in the existing
dispatch/dispatch.claimedmetrics asdispatcher:chain. Enable the option only after this deploy settles —chain_depthis only sent on chained dispatches, so workers from the previous deploy never see the new argument. Turning the option off stops new links immediately; in-flight ones drain out.