Repository navigation
fix(upstream): close the remaining reconnect-storm holes after #1014 - #1039
Merged
Merged
Conversation
#1014 gated the supervisor's 30s reconcile on the client's retry policy, but three holes in that gate remain — two of which leave the original storm intact: - OAuth-classified failures are recorded by SetOAuthError, which bumps OAuthRetryCount and never RetryCount. ShouldAutoReconnect read them as "RetryCount == 0 → no failures yet" and re-dialed on every tick, forever. ShouldAutoReconnect now also honors the OAuth ladder (5min → 24h), extracted from ShouldRetryOAuth into OAuthRetryBackoffDuration so both paths share one schedule. - The gate's StatePendingAuth branch was dead code: managed.Connect parked a login-blocked server with TransitionTo(StatePendingAuth) followed by SetError, and SetError forces StateError unconditionally, so the park lasted microseconds. StateManager.SetPendingAuth parks it for real; the supervisor, ConnectAll and the health check all leave it alone until a human acts. The wake paths are wired to match: Connect clears the stale core client from PendingAuth too, Manager.scanForNewTokens accepts a parked client when a token appears, PendingAuth is a valid transition source/target, health reports Sign-in required instead of falling through to healthy, the notifier emits the login prompt, and monitorConnectionStatus answers immediately instead of stalling until its timeout. - After MaxConnectionRetries the gate said "never again", so any outage longer than the ladder (~1h: sleep, VPN, overnight maintenance) left the upstream silently dead until someone reconnected it by hand — recovery the supervisor used to provide by re-dialing unconditionally. A given-up server is now probed once per GaveUpProbeInterval (30min): ~48 requests/day instead of ~2880, and it self-heals. Refs #1013, #1014
Deploying mcpproxy-docs with
|
| Latest commit: |
5b8e0a3
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9accf1a6.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://fix-reconnect-backoff-follow.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…-scan redial loop
Cross-model review of the previous commit surfaced four defects, three of them
in code it newly depended on:
- `IsOAuthPending` type-asserted instead of unwrapping, but the deferred-OAuth
error is wrapped twice on its way out of Connect ("all authentication
strategies failed, last error: %w" then "failed to connect: %w"). The check
never fired in production, so the park — and every consumer keyed to it — was
unreachable. Now `errors.As`, with the production wrapping shape pinned in a
test.
- `Manager.scanForNewTokens` triggered a reconnect whenever *any* persisted
token existed. A failing server usually still holds the expired/revoked token
it just failed with, so the 5s scan redialed it every pass (rate-limited only
to 10s/server) — a tighter loop than the supervisor's and one that walks
straight through the park. It now records a fingerprint of the token it
retried with and fires only when a login or refresh writes a different one.
- `ConnectAll` gated on `StateError && !ShouldRetry()`, so a config reload
redialed parked and OAuth-backed-off servers. It now consults the same
`ShouldAutoReconnect` policy as the supervisor. A config change for the
server itself still reconnects it via ActionReconnect.
- `SetPendingAuth` dispatched its state-change callback detached, which can be
scheduled after a later Connecting/Ready callback and surface a stale
"sign in required" prompt right after a successful login. It is synchronous
now, matching the TransitionTo that used to deliver this state.
Also replaces the reconcile test's fixed sleeps with an actionWg barrier so the
negative assertions cannot pass before an erroneous redial would have run.
Refs #1013, #1014
Round-2 review: a provider that re-issues a byte-identical access token (same refresh token, same expiry) left the scan's fingerprint unchanged, permanently suppressing the wake for a parked server — and when a CLI login cannot persist its OAuth completion event, this scan is the only wake left. The record's persisted Updated stamp is now part of the fingerprint, so any new write counts as news while an untouched token still does not. Refs #1013
The scan test's later legs ran while the reconnect triggered by the previous leg was still in flight, so the client could be Connecting — a state the scan skips for a reason the test is not about — and the assertion failed in CI. Each leg now waits for the client to settle back into PendingAuth or Error first. Refs #1013
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 32777420379 --repo smart-mcp-proxy/mcpproxy-go
|
This was referenced Aug 24, 2026
Merged
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 25, 2026
…1042) * feat(upstream): honor Retry-After on 429 via transport RoundTripper mcp-go flattens every non-2xx upstream response into an error string, so a 429's Retry-After header was gone before any connection-state code ran and the reconnect ladder paced a rate-limited upstream on its own schedule. Capture the hint where the *http.Response still exists: a RetryAfterTransport RoundTripper installed under every HTTP/SSE client we hand to mcp-go, feeding a per-server RetryAfterRecorder. The managed client stamps the resulting deadline onto the state machine, and ConnectionInfo.ShouldAutoReconnect / ShouldRetry treat it as a floor under the existing backoff — effective delay is max(backoff, Retry-After), capped at one hour. - Both RFC 7231 forms parse (delta-seconds and HTTP-date); a malformed, absent, or already-elapsed value falls back to the existing ladder untouched. - 503 counts only when it carries a parseable hint; a bare 503 is an outage the ladder already paces. - The wrapper composes with the existing tracing transport and sits below mcp-go's OAuth handling, so the three authorize paths are unchanged. The two OAuth client constructors now receive our basic client explicitly, since OAuthConfig.HTTPClient covers only metadata/DCR/token requests. - The window is cleared on a successful connect and on Reset, so a stale hint cannot hold back a later, unrelated reconnect. Closes #1040. Refs #1013, #1039. * fix(upstream): scope the Retry-After window to its attempt and gate the OAuth ladder Cross-model review of #1042 found four real holes: - The hint only reached the state machine after a failed Connect, so a 429 answered to the health-check ping was recorded and then ignored. syncRetryAfterFromTransport is now called from every path that turns an upstream failure into Error state. - The recorder was cleared only on success, so a window from an earlier attempt could be re-attributed to a later, unrelated failure — re-parking a server the upstream had not rate-limited again (notably right after a manual reconnect reset the state machine). Each attempt now starts with a clean slate. - ShouldRetryOAuth (the monitoring loop's reconnect path) ignored the window, and its rungs can be shorter than what the upstream asked for. - Converting delta-seconds to a Duration before capping overflowed int64 nanoseconds past ~292 years, turning a huge header into a tiny or negative delay instead of the cap. The bound check now precedes the multiplication. Tests: OAuth streamable-HTTP and SSE clients now have wire-level coverage (a pre-seeded token store makes the request actually go out, so the 429 is observable) — verified to fail when either constructor loses the basic-client option; plus per-attempt window scoping at the core level, the OAuth-ladder gate, and the two overflow-shaped Retry-After inputs. Refs #1040. * fix(upstream): make the 429 sync reachable from the request paths and swap recorders per attempt Round 2 of the cross-model review found the round-1 health-path fix was dead code and the per-attempt clear was too blunt: - performHealthCheck only reached the sync after isConnectionError, which matches transport failures ("connection refused", "deadline exceeded") and not HTTP statuses — so a ping answered 429 never handed its window to the state machine. The sync now runs on any failed ping, before classification; it does not change the health verdict, it only stops the automatic reconnect paths from returning early. The same sync now also runs on a failed tools/call, the other surface that can be rate-limited with no connect attempt in sight. - Clearing one shared recorder at the start of an attempt let a request still in flight on a superseded transport write into the new attempt's slate (and discarded a hint recorded moments before the clear). The core client now holds a generation pointer and swaps in a fresh recorder per attempt, so each transport keeps the recorder it was built with. Tests: the health-path test is verified to fail when the sync moves back behind isConnectionError, plus tools/call coverage. Refs #1040.
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 25, 2026
…specs 101/102 tracked (#1041) - release-qa-gate-playwright (T2): in_review -> done, pr #1030 (advisory Playwright web-UI sweep merged into the release gate). - mcp-2026-upgrade: blocked -> in_progress. The mcp-go gate cleared with v1.0.0-beta.1 (2026-08-12); spec 058 revision in review as #1033. Note rewritten: the remaining blocker is the Spec 057 cross-spec conflict, not the dependency. - tpa-db: todo -> in_review with spec: specs/101-tpa-db and pr #1028 (spec stage only; all three child tasks stay todo). - New epic schema-deferred (P1, spec 102, in_progress): deferred-schema serialization for the direct tools/list surface, spec merged in #1035 (issue #971), plan in progress. Reconnect-backoff work (#1014/#1039/#1040) is deliberately not added — the roadmap never tracked it; git history does. ROADMAP.md regenerated; gen-roadmap.py --check passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1014 (merged), which fixed the headline storm from #1013. Reviewing that fix against the issue's three asks turned up gaps in the new gate — two of which left the original 30s storm intact for a whole error class, and one of which made the fix's
PendingAuthbranch dead code.1. OAuth-classified failures were still re-dialed every 30s
SetOAuthErrorbumpsoauthRetryCountand neverretryCount.ConnectionInfo.ShouldAutoReconnectonly consultedRetryCount, so an OAuth failure read asRetryCount == 0→ "no failures yet" →ActionConnecton every reconcile tick, forever. The exact behavior #1014 set out to stop, for the error class (invalid_token,401,unauthorized,invalid_grant, …) most likely to produce it.ShouldAutoReconnectnow also honors the OAuth ladder (5min → 15min → 1h → 4h → 24h), extracted fromStateManager.ShouldRetryOAuthintoOAuthRetryBackoffDurationso the health-check path and the supervisor path share one schedule — mirroring what #1014 did withRetryBackoffDuration. Both gates apply when both counters moved; the longer wait wins.2. The park never happened — twice over
Issue ask (2) — "auth-required failures should park in an awaiting-login state rather than retry at all" — was not in effect, for two independent reasons:
IsOAuthPendingnever returned true. It type-asserted, but the deferred-OAuth error is wrapped twice on its way out:connectHTTP/connectSSEadd"all authentication strategies failed, last error: %w"andConnectadds"failed to connect: %w". Nowerrors.As, with the production wrapping shape pinned in a test.SetErrorundid the park.managed.ConnectdidTransitionTo(StatePendingAuth)immediately followed bySetError(connectErr), andSetErrorsetscurrentState = StateErrorunconditionally — the park survived microseconds.StateManager.SetPendingAuthnow parks for real: state staysPendingAuth, the error stays attached,RetryCountis deliberately not advanced (a parked server is not retrying). The supervisor,ConnectAllandperformHealthCheckall leave it alone until a human acts.Making that state reachable means the paths that assumed "OAuth-required ⇒ StateError" had to be wired to match:
managed.ConnectPendingAuthtoo, so the post-login reconnect doesn't hit "client already connected"Manager.scanForNewTokensStateError-only) — the fallback wake when a CLI login can't write its completion eventValidateTransitionConnecting/Authenticating → PendingAuthandPendingAuth → Connecting/Disconnected/Errorare legal (the old park printed "Invalid state transition" to stdout)health.CalculateHealth"pending auth"→Sign-in required(amber) /Authentication required(red for a broken stored token). Without this a parked server fell through the state switch and could report healthynotificationsPendingAuth(the oldSetErroremitted a generic server error)monitorConnectionStatusupstream_serverscaller until the monitor timeoutSpec 098's preflight already had a
RuntimeStatePendingAuthverdict ("Server is waiting for OAuth login") keyed on this state string — until now unreachable. Manual reconnect,ForceReconnect/RetryConnection,reconnect_on_useand login flows are state-agnostic and unaffected.3. The token scan was a tighter storm than the one being fixed
scanForNewTokens(every 5s) triggered a reconnect whenever any persisted token existed, rate-limited only to one per 10s per server. A failing server usually still holds the expired/revoked token it just failed with — so this redialed it ~8,600×/day on its own, out-pacing the supervisor loop and walking straight through the park.It now fingerprints the token it retried with (truncated SHA-256 over access + refresh + expiry + the record's persisted
Updatedstamp — never the raw token) and fires only when a login or refresh writes a different one. The write stamp is part of the identity so a provider re-issuing a byte-identical token still counts as news.4. "Gave up" meant never again
After
MaxConnectionRetriesthe gate returnedfalseunconditionally. The ladder is exhausted in roughly an hour, so any longer outage — laptop sleep, VPN drop, overnight upstream maintenance — left the upstream silently dead until a human noticed. #1013 itself points out that a failed upstream is invisible to the operator, and the pre-#1014 supervisor provided this recovery by re-dialing unconditionally.A given-up server is now probed once per
GaveUpProbeInterval(30min): ~48 requests/day instead of ~2880, and it self-heals. This also matches ask (1), which asked for a backoff capped at 15–30min, not a permanent give-up.ConnectAll(config-reload path) now consults the sameShouldAutoReconnectpolicy instead ofStateError && !ShouldRetry(), so a reload no longer re-dials servers their own client had paused. A config change for the server itself still reconnects it — the supervisor plansActionReconnect, which bypasses the gate.Not covered: ask (3),
Retry-Afteron 429 — #1040It cannot be done in this layer:
mcp-goflattens the response tofmt.Errorf("request failed with status %d: %s", …), so the header never reaches us. Honoring it needs aRoundTripperininternal/transportthreaded through every client-construction branch (including the twoNewOAuth*Clientpaths that build their ownhttp.Client) — a change to all upstream HTTP traffic, not just connect. Design and acceptance criteria are in #1040. With the fixes above a 429-ing upstream is already paced 1s → 5min → 30min probes, so the risk is bounded meanwhile.Verification
New/updated tests:
TestIsOAuthPending(+%w-wrapped case),TestOAuthRetryBackoffDuration,TestSetPendingAuth(park sticks, error preserved, no auto-redial, wake transition legal, consumers notified),TestConnectionInfo_ShouldAutoReconnect(+7 cases: OAuth ladder open/elapsed/long, both-ladders, give-up probe),TestTokenFingerprint,TestScanForNewTokens_OnlyOnNewToken,TestSupervisor_Reconcile_RespectsRetryBackoff(+OAuth-window and give-up-probe legs, and anactionWgbarrier replacing fixed sleeps so the negative assertions can't pass early),TestCalculateHealth_PendingAuthParked.Cross-model review (opencode / gpt-5.6-sol): 3 rounds, 6 findings, all fixed — final verdict CLEAN. Findings 1–3 above (
errors.As, the 5s token-scan loop, theConnectAllgate) came out of round 1; the callback-ordering fix inSetPendingAuth(synchronous, like theTransitionTothat used to deliver this state, so a stale "sign in required" can't land after a successful login) and the token write-stamp came out of rounds 1–2.Refs #1013, #1014, #1040