Skip to content

feat(upstream): honor Retry-After on 429 via transport RoundTripper - #1042

Merged
github-actions[bot] merged 3 commits into
mainfrom
fix/1040-retry-after-429
Aug 25, 2026
Merged

github-actions[bot] merged 3 commits into
mainfrom
fix/1040-retry-after-429

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 25, 2026

Copy link
Copy Markdown
Member

Closes #1040 — the third and last ask from #1013 (after #1014 and #1039).

What

mcp-go flattens every non-2xx upstream response into an error string, so a 429's Retry-After header was gone before any connection-state code ran. This captures it in the one place the *http.Response still exists — a RetryAfterTransport RoundTripper installed under the HTTP/SSE clients we hand to mcp-go — and feeds a per-server RetryAfterRecorder.

The managed client stamps the resulting deadline onto the state machine; ConnectionInfo.ShouldAutoReconnect (the supervisor's reconcile gate) and StateManager.ShouldRetry (ConnectAll / health) treat it as a floor under the existing ladder: effective delay is max(backoff, Retry-After), capped at 1h.

  • Both RFC 7231 forms parse (delta-seconds and HTTP-date). Malformed, absent, or already-elapsed values fall back to the existing exponential ladder, untouched.
  • 503 counts only when it carries a parseable hint; a bare 503 is an outage the ladder already paces.
  • The window is cleared on a successful connect and on Reset, so a stale hint can't hold back a later, unrelated reconnect.
  • A later, shorter hint never shortens an outstanding window.

OAuth paths — coverage and limitations

The wrapper composes with the existing tracing transport and sits strictly below mcp-go's OAuth handling, so the three authorize/DCR emission paths are unchanged.

  • NewOAuthStreamableHttpClient / NewOAuthSSEClient build their own transport internally, and OAuthConfig.HTTPClient covers only the OAuth metadata/DCR/token requests — not the MCP requests. Both constructors now receive our basic client explicitly (via WithHTTPBasicClient / WithHTTPClient), so MCP-endpoint 429s on OAuth'd upstreams are covered.
  • Not covered: a 429 on the OAuth token/DCR/metadata endpoints themselves — those go through the client built in internal/oauth/config.go, which is untouched here. Those failures already land on the coarse OAuth ladder (5min → 24h), so they are paced regardless.
  • Timeout policy is preserved per-branch exactly as before: 180s for the trace and header-less streamable-HTTP clients, mcp-go's default (none) for the headers and OAuth variants, and no Client.Timeout on SSE so long-lived streams still work.

Tests

  • internal/transport/retry_after_test.go — parsing table (seconds / HTTP-date / malformed / expired / float), recorder cap + extend-only semantics, httptest round-trips for 429/503/200/401, body-passthrough guard.
  • internal/upstream/retry_after_connect_test.go — end-to-end connect against a 429 httptest upstream across the streamable-HTTP, headers, and SSE construction branches; hour-long park, HTTP-date form, malformed fallback to the ladder, oversized value clamped.
  • internal/upstream/types/retry_after_test.go — the gate itself (floor semantics, both directions) and the state-manager lifecycle.
  • internal/runtime/supervisor/retry_after_test.go — reconcile plans no ActionConnect inside the window and reconnects once it elapses.

Verified: both-edition builds, go test -race ./internal/upstream/... ./internal/transport/... ./internal/runtime/..., internal/server + contracts + httpapi under the CI skip regex, golangci-lint v2 ./... clean.

mcp-go flattens every non-2xx upstream response into an error string, so a
429's Retry-After header was gone before any connection-state code ran and the
reconnect ladder paced a rate-limited upstream on its own schedule.

Capture the hint where the *http.Response still exists: a RetryAfterTransport
RoundTripper installed under every HTTP/SSE client we hand to mcp-go, feeding a
per-server RetryAfterRecorder. The managed client stamps the resulting deadline
onto the state machine, and ConnectionInfo.ShouldAutoReconnect / ShouldRetry
treat it as a floor under the existing backoff — effective delay is
max(backoff, Retry-After), capped at one hour.

- Both RFC 7231 forms parse (delta-seconds and HTTP-date); a malformed, absent,
  or already-elapsed value falls back to the existing ladder untouched.
- 503 counts only when it carries a parseable hint; a bare 503 is an outage the
  ladder already paces.
- The wrapper composes with the existing tracing transport and sits below
  mcp-go's OAuth handling, so the three authorize paths are unchanged. The two
  OAuth client constructors now receive our basic client explicitly, since
  OAuthConfig.HTTPClient covers only metadata/DCR/token requests.
- The window is cleared on a successful connect and on Reset, so a stale hint
  cannot hold back a later, unrelated reconnect.

Closes #1040. Refs #1013, #1039.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 8d7b0c8
Status: ✅  Deploy successful!
Preview URL: https://dc7e434d.mcpproxy-docs.pages.dev
Branch Preview URL: https://fix-1040-retry-after-429.mcpproxy-docs.pages.dev

View logs

@codecov-commenter

codecov-commenter commented Aug 25, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 76.31579% with 45 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/transport/http.go 57.69% 20 Missing and 2 partials ⚠️
internal/transport/retry_after.go 89.61% 7 Missing and 1 partial ⚠️
internal/upstream/core/connection_oauth.go 0.00% 5 Missing ⚠️
internal/upstream/types/types.go 84.61% 4 Missing ⚠️
internal/upstream/managed/client.go 78.57% 2 Missing and 1 partial ⚠️
internal/upstream/core/client.go 81.81% 2 Missing ⚠️
internal/upstream/core/connection_http.go 66.66% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…he OAuth ladder

Cross-model review of #1042 found four real holes:

- The hint only reached the state machine after a failed Connect, so a 429
  answered to the health-check ping was recorded and then ignored.
  syncRetryAfterFromTransport is now called from every path that turns an
  upstream failure into Error state.
- The recorder was cleared only on success, so a window from an earlier attempt
  could be re-attributed to a later, unrelated failure — re-parking a server the
  upstream had not rate-limited again (notably right after a manual reconnect
  reset the state machine). Each attempt now starts with a clean slate.
- ShouldRetryOAuth (the monitoring loop's reconnect path) ignored the window,
  and its rungs can be shorter than what the upstream asked for.
- Converting delta-seconds to a Duration before capping overflowed int64
  nanoseconds past ~292 years, turning a huge header into a tiny or negative
  delay instead of the cap. The bound check now precedes the multiplication.

Tests: OAuth streamable-HTTP and SSE clients now have wire-level coverage (a
pre-seeded token store makes the request actually go out, so the 429 is
observable) — verified to fail when either constructor loses the basic-client
option; plus per-attempt window scoping at the core level, the OAuth-ladder
gate, and the two overflow-shaped Retry-After inputs.

Refs #1040.
@github-actions

github-actions Bot commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: fix/1040-retry-after-429

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (25 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (20 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 32807774216 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

… swap recorders per attempt

Round 2 of the cross-model review found the round-1 health-path fix was dead
code and the per-attempt clear was too blunt:

- performHealthCheck only reached the sync after isConnectionError, which
  matches transport failures ("connection refused", "deadline exceeded") and
  not HTTP statuses — so a ping answered 429 never handed its window to the
  state machine. The sync now runs on any failed ping, before classification;
  it does not change the health verdict, it only stops the automatic reconnect
  paths from returning early. The same sync now also runs on a failed
  tools/call, the other surface that can be rate-limited with no connect
  attempt in sight.

- Clearing one shared recorder at the start of an attempt let a request still
  in flight on a superseded transport write into the new attempt's slate (and
  discarded a hint recorded moments before the clear). The core client now
  holds a generation pointer and swaps in a fresh recorder per attempt, so each
  transport keeps the recorder it was built with.

Tests: the health-path test is verified to fail when the sync moves back behind
isConnectionError, plus tools/call coverage.

Refs #1040.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (Model B): Paperclip review verdicts = ACCEPT and qa-gate green at this head SHA. Arming auto-merge; GitHub merges when all required checks pass.

@github-actions
github-actions Bot merged commit 5774ba9 into main Aug 25, 2026
45 of 46 checks passed

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (Model B): Paperclip review verdicts = ACCEPT and qa-gate green at this head SHA. Arming auto-merge; GitHub merges when all required checks pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: connect retries ignore Retry-After on 429 (remaining ask from #1013)

2 participants