Repository navigation
feat(upstream): honor Retry-After on 429 via transport RoundTripper - #1042
Merged
Merged
Conversation
mcp-go flattens every non-2xx upstream response into an error string, so a 429's Retry-After header was gone before any connection-state code ran and the reconnect ladder paced a rate-limited upstream on its own schedule. Capture the hint where the *http.Response still exists: a RetryAfterTransport RoundTripper installed under every HTTP/SSE client we hand to mcp-go, feeding a per-server RetryAfterRecorder. The managed client stamps the resulting deadline onto the state machine, and ConnectionInfo.ShouldAutoReconnect / ShouldRetry treat it as a floor under the existing backoff — effective delay is max(backoff, Retry-After), capped at one hour. - Both RFC 7231 forms parse (delta-seconds and HTTP-date); a malformed, absent, or already-elapsed value falls back to the existing ladder untouched. - 503 counts only when it carries a parseable hint; a bare 503 is an outage the ladder already paces. - The wrapper composes with the existing tracing transport and sits below mcp-go's OAuth handling, so the three authorize paths are unchanged. The two OAuth client constructors now receive our basic client explicitly, since OAuthConfig.HTTPClient covers only metadata/DCR/token requests. - The window is cleared on a successful connect and on Reset, so a stale hint cannot hold back a later, unrelated reconnect. Closes #1040. Refs #1013, #1039.
Deploying mcpproxy-docs with
|
| Latest commit: |
8d7b0c8
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://dc7e434d.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://fix-1040-retry-after-429.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…he OAuth ladder Cross-model review of #1042 found four real holes: - The hint only reached the state machine after a failed Connect, so a 429 answered to the health-check ping was recorded and then ignored. syncRetryAfterFromTransport is now called from every path that turns an upstream failure into Error state. - The recorder was cleared only on success, so a window from an earlier attempt could be re-attributed to a later, unrelated failure — re-parking a server the upstream had not rate-limited again (notably right after a manual reconnect reset the state machine). Each attempt now starts with a clean slate. - ShouldRetryOAuth (the monitoring loop's reconnect path) ignored the window, and its rungs can be shorter than what the upstream asked for. - Converting delta-seconds to a Duration before capping overflowed int64 nanoseconds past ~292 years, turning a huge header into a tiny or negative delay instead of the cap. The bound check now precedes the multiplication. Tests: OAuth streamable-HTTP and SSE clients now have wire-level coverage (a pre-seeded token store makes the request actually go out, so the 429 is observable) — verified to fail when either constructor loses the basic-client option; plus per-attempt window scoping at the core level, the OAuth-ladder gate, and the two overflow-shaped Retry-After inputs. Refs #1040.
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 32807774216 --repo smart-mcp-proxy/mcpproxy-go
|
… swap recorders per attempt
Round 2 of the cross-model review found the round-1 health-path fix was dead
code and the per-attempt clear was too blunt:
- performHealthCheck only reached the sync after isConnectionError, which
matches transport failures ("connection refused", "deadline exceeded") and
not HTTP statuses — so a ping answered 429 never handed its window to the
state machine. The sync now runs on any failed ping, before classification;
it does not change the health verdict, it only stops the automatic reconnect
paths from returning early. The same sync now also runs on a failed
tools/call, the other surface that can be rate-limited with no connect
attempt in sight.
- Clearing one shared recorder at the start of an attempt let a request still
in flight on a superseded transport write into the new attempt's slate (and
discarded a hint recorded moments before the clear). The core client now
holds a generation pointer and swaps in a fresh recorder per attempt, so each
transport keeps the recorder it was built with.
Tests: the health-path test is verified to fail when the sync moves back behind
isConnectionError, plus tools/call coverage.
Refs #1040.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1040 — the third and last ask from #1013 (after #1014 and #1039).
What
mcp-goflattens every non-2xx upstream response into an error string, so a 429'sRetry-Afterheader was gone before any connection-state code ran. This captures it in the one place the*http.Responsestill exists — aRetryAfterTransportRoundTripper installed under the HTTP/SSE clients we hand to mcp-go — and feeds a per-serverRetryAfterRecorder.The managed client stamps the resulting deadline onto the state machine;
ConnectionInfo.ShouldAutoReconnect(the supervisor's reconcile gate) andStateManager.ShouldRetry(ConnectAll / health) treat it as a floor under the existing ladder: effective delay ismax(backoff, Retry-After), capped at 1h.503counts only when it carries a parseable hint; a bare 503 is an outage the ladder already paces.Reset, so a stale hint can't hold back a later, unrelated reconnect.OAuth paths — coverage and limitations
The wrapper composes with the existing tracing transport and sits strictly below mcp-go's OAuth handling, so the three authorize/DCR emission paths are unchanged.
NewOAuthStreamableHttpClient/NewOAuthSSEClientbuild their own transport internally, andOAuthConfig.HTTPClientcovers only the OAuth metadata/DCR/token requests — not the MCP requests. Both constructors now receive our basic client explicitly (viaWithHTTPBasicClient/WithHTTPClient), so MCP-endpoint 429s on OAuth'd upstreams are covered.internal/oauth/config.go, which is untouched here. Those failures already land on the coarse OAuth ladder (5min → 24h), so they are paced regardless.Client.Timeouton SSE so long-lived streams still work.Tests
internal/transport/retry_after_test.go— parsing table (seconds / HTTP-date / malformed / expired / float), recorder cap + extend-only semantics, httptest round-trips for 429/503/200/401, body-passthrough guard.internal/upstream/retry_after_connect_test.go— end-to-end connect against a 429 httptest upstream across the streamable-HTTP, headers, and SSE construction branches; hour-long park, HTTP-date form, malformed fallback to the ladder, oversized value clamped.internal/upstream/types/retry_after_test.go— the gate itself (floor semantics, both directions) and the state-manager lifecycle.internal/runtime/supervisor/retry_after_test.go— reconcile plans noActionConnectinside the window and reconnects once it elapses.Verified: both-edition builds,
go test -race ./internal/upstream/... ./internal/transport/... ./internal/runtime/...,internal/server+contracts+httpapiunder the CI skip regex,golangci-lint v2 ./...clean.