Repository navigation
fix(octopus): don't re-read a rate-limited GraphQL response - #5249
Merged
Merged
Conversation
A KT-CT-1199 rate limit made async_read_response() return None after sleeping 5s. async_read_response_retry() treated that None as retryable, and re-read the same response four more times with a 1/2/4/8s backoff. aiohttp caches the body, so every re-read was the same rate limit: one request to Octopus logged five warnings and stalled about 35 seconds before giving up anyway. That is how it looked live on a flexPlannedDispatches poll (gaps of 6, 7, 9 and 13 seconds). A rate limit is now definitive for the response, like 401/403. async_read_response() records it in last_read_rate_limited and no longer sleeps, and the retry loop returns straight away. Other failures retry as before. Test 7 in test_octopus_read_response_retry asserted the old retry. It mocked a second read succeeding, which cannot happen on a cached body. It now drives a real rate-limit body through the real reader: one read, no sleep, one warning. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Octopus rate-limits a GraphQL request (
KT-CT-1199),async_read_response()sleeps for 5 seconds and returnsNone.async_read_response_retry()then treated thatNoneas worth retrying, and re-read the same response four more times, with a 1, 2, 4 and 8 second backoff.aiohttp caches the response body, so every re-read returns the same rate limit. The existing 401/403 handling in that loop already says so. As a result, one request to Octopus logged five warnings and held the component for about 35 seconds, before failing anyway.
Seen live on a
flexPlannedDispatchespoll: fiveRate limit error in requestwarnings 6, 7, 9 and 13 seconds apart (5 s plus the backoff), followed byget-intelligent-dispatches (status 200): None.Change
A rate limit is now definitive for the response, as 401/403 already are:
async_read_response()records the rate limit inlast_read_rate_limited, and no longer sleeps.async_read_response_retry()returns straight away when that flag is set.Every other failure retries exactly as before.
Tests
test_octopus_read_response_retryTest 7 asserted the old retry. It mocked a second read succeeding, which can't happen on a cached body. It now drives a real rate-limit body through the real reader, and checks:None, andfailures_totalgoes up by 1.It failed before the fix, with the response read 5 times. All 24 Octopus test modules pass, and so do
./run_all --quickand pre-commit.Not in this PR
Deliberately scoped to the retry loop. Two related gaps, left as possible follow-ups:
planned_dispatcheslist for that poll. The settings query next to it carries its last known values forward, but the dispatch query doesn't.🤖 Generated with Claude Code