Skip to content

feat(hybridcloud): Chain a strict drain into its mailbox's next claim - #123268

Merged
vaind merged 0 commit into
ivan/webhook-drain-required-claimfrom
ivan/webhook-drain-chain-dispatch
Aug 31, 2026
Merged

feat(hybridcloud): Chain a strict drain into its mailbox's next claim#123268
vaind merged 0 commit into
ivan/webhook-drain-required-claimfrom
ivan/webhook-drain-chain-dispatch

Conversation

@vaind

@vaind vaind commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Stacked on #123265 (which stacks on #122893).

Problem

A strict provider's mailbox burns one claim per scheduler round trip. The gap between claims isn't the ~10s cycle cadence: under a wide backlog the mailbox competes with every other due head for each cycle's dispatch budget, so a deep strict mailbox can wait many cycles between drains. Re-discovery, not delivery, dominates its burn rate — and strict providers are precisely the ones lanes-style concurrent claims can't serve (see #122897's closing rationale).

Change

A drain that ends healthy with due work behind it dispatches the mailbox's next claim itself — _maybe_chain, gated by hybridcloud.webhookpayload.drain_chaining (default off). "Healthy with work behind" means exactly two endings:

  • the drain consumed a claim that had filled the cap (the due prefix continues past it), or
  • it released a tail at its soft-stop after delivering something — a drain that delivered nothing before the soft-stop spent its window in the queue, which is saturation, exactly when a chain would add queue load.

Never after a failure-stop or a lapse; backoff and the scheduler own those.

Why strict-only: the absolute-head gate admits one claim at a time, so a chain stays a single lineage per mailbox. A skip-on-failure provider in due-head mode would have the scheduler start a fresh claim behind every in-flight chain — one new self-sustaining pipeline per cycle, unbounded lanes.

Safety: the chain claims under the same drain lock and due-gate as every other dispatcher, so losing the race to a scheduler or push dispatch just continues the lineage under their attribution. chain_depth rides the task args as a runaway backstop (MAX_CHAIN_DEPTH), far above what a healthy chain reaches — each link is bounded by its own claim, and the chain ends at the first short claim. A chain failure is swallowed: the drain's work is already delivered, and the scheduler covers the mailbox.

Observability and rollout

Chain dispatches land in the existing dispatch / dispatch.claimed metrics as dispatcher:chain. Enable the option only after this deploy settles — chain_depth is only sent on chained dispatches, so workers from the previous deploy never see the new argument. Turning the option off stops new links immediately; in-flight ones drain out.

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 31, 2026
@vaind
vaind force-pushed the ivan/webhook-drain-chain-dispatch branch 2 times, most recently from e9b953e to 4fd8fa1 Compare August 31, 2026 22:51
@vaind
vaind merged commit 4fd8fa1 into master Aug 31, 2026
5 of 6 checks passed
@vaind
vaind deleted the ivan/webhook-drain-chain-dispatch branch August 31, 2026 22:53
@vaind
vaind restored the ivan/webhook-drain-chain-dispatch branch August 31, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant