Skip to content

Exponential workqueue backoff resets after 64 failures due to Java shift masking #4985

Closed

Description

@brendandburns

Problem

ItemExponentialFailureRateLimiter.when uses an unbounded per-item failure count as the shift distance:

long exp = failures.computeIfAbsent(item, k -> new AtomicLong()).getAndIncrement();
long d = maxDelay.toMillis() >> exp;
return d > baseDelay.toMillis() ? baseDelay.multipliedBy(1L << exp) : maxDelay;

Java masks long shift distances to their low six bits. At exp == 64, both shifts behave as though the distance were zero, causing the delay to reset to the base instead of staying capped.

Location: extended/src/main/java/io/kubernetes/client/extended/workqueue/ratelimiter/ItemExponentialFailureRateLimiter.java:37-40.

Upstream precedent: kubernetes/kubernetes#142641 describes a different arithmetic defect with the same important invariant: repeated failures must not defeat the configured maximum backoff. The Java defect is shift-distance wraparound, not the exact Go floating-point conversion defect.

Reproduction verified against Java source

Baseline: 9872badf5f69ee7d3bdb5fb7dc6183d2da935821.
Instantiate the limiter with Duration.ofMillis(5) and Duration.ofSeconds(1000) (the default controller's per-item settings). Call when("failing-object") 65 times without forget.

Observed:

  • Call 64: 1,000,000 milliseconds.
  • Call 65: 5 milliseconds.

Expected: after reaching the cap, subsequent calls remain at 1,000,000 milliseconds until the item is forgotten. This reproduction requires no sleeping or cluster.

Impact

Suggested priority: medium. Long-lived failures periodically restart fast retries, potentially producing retry bursts and excess API load during outages. Custom backoff configurations may expose additional overflow boundaries.

Acceptance criteria

  • Saturate at the maximum delay safely before unsafe shifting/multiplication; do not allow shift masking to restart backoff.
  • Preserve ordinary exponential behavior and forget/numRequeues semantics.
  • Add regression coverage across calls 63-66, 127-129, and a longer repeated sequence, checking every result rather than only the final call.
  • Cover default settings and large positive custom durations; preserve type safety and avoid arithmetic overflow.
  • Existing tests cover negative durations; assess compatibility explicitly rather than silently changing unrelated behavior.

Copilot instructions

Implement a focused fix with regression tests, run relevant tests, and open a PR referencing and closing this issue. Follow repository contribution conventions and the PR template. Do not bundle unrelated workqueue lifecycle or scheduling changes.

Activity

  1. added a commit that references this issue on Nov 4, 2026
    562bbc5
  2. added a commit that references this issue on Nov 4, 2026
    4567e30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions