Skip to content

fix(octopus): don't re-read a rate-limited GraphQL response - #5249

Merged
springfall2008 merged 1 commit into
mainfrom
fix/octopus-rate-limit-no-retry
Sep 26, 2026
Merged

springfall2008 merged 1 commit into
mainfrom
fix/octopus-rate-limit-no-retry

Conversation

@springfall2008

Copy link
Copy Markdown
Owner

Problem

When Octopus rate-limits a GraphQL request (KT-CT-1199), async_read_response() sleeps for 5 seconds and returns None. async_read_response_retry() then treated that None as worth retrying, and re-read the same response four more times, with a 1, 2, 4 and 8 second backoff.

aiohttp caches the response body, so every re-read returns the same rate limit. The existing 401/403 handling in that loop already says so. As a result, one request to Octopus logged five warnings and held the component for about 35 seconds, before failing anyway.

Seen live on a flexPlannedDispatches poll: five Rate limit error in request warnings 6, 7, 9 and 13 seconds apart (5 s plus the backoff), followed by get-intelligent-dispatches (status 200): None.

Change

A rate limit is now definitive for the response, as 401/403 already are:

  • async_read_response() records the rate limit in last_read_rate_limited, and no longer sleeps.
  • async_read_response_retry() returns straight away when that flag is set.

Every other failure retries exactly as before.

Tests

test_octopus_read_response_retry Test 7 asserted the old retry. It mocked a second read succeeding, which can't happen on a cached body. It now drives a real rate-limit body through the real reader, and checks:

  • the body is read once;
  • nothing sleeps;
  • exactly one warning is logged;
  • the result is None, and failures_total goes up by 1.

It failed before the fix, with the response read 5 times. All 24 Octopus test modules pass, and so do ./run_all --quick and pre-commit.

Not in this PR

Deliberately scoped to the retry loop. Two related gaps, left as possible follow-ups:

  • No backoff across polls: the dispatch poll runs again 2 minutes later, whatever happened.
  • A failed dispatch fetch publishes an empty planned_dispatches list for that poll. The settings query next to it carries its last known values forward, but the dispatch query doesn't.

🤖 Generated with Claude Code

A KT-CT-1199 rate limit made async_read_response() return None after sleeping
5s. async_read_response_retry() treated that None as retryable, and re-read the
same response four more times with a 1/2/4/8s backoff. aiohttp caches the body,
so every re-read was the same rate limit: one request to Octopus logged five
warnings and stalled about 35 seconds before giving up anyway. That is how it
looked live on a flexPlannedDispatches poll (gaps of 6, 7, 9 and 13 seconds).

A rate limit is now definitive for the response, like 401/403.
async_read_response() records it in last_read_rate_limited and no longer sleeps,
and the retry loop returns straight away. Other failures retry as before.

Test 7 in test_octopus_read_response_retry asserted the old retry. It mocked a
second read succeeding, which cannot happen on a cached body. It now drives a
real rate-limit body through the real reader: one read, no sleep, one warning.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@springfall2008
springfall2008 merged commit 83ffacd into main Sep 26, 2026
2 checks passed
@springfall2008
springfall2008 deleted the fix/octopus-rate-limit-no-retry branch September 26, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant