Repository navigation
fix(upstream): respect client retry backoff in supervisor reconciliation - #1014
Merged
github-actions[bot] merged 1 commit intoAug 23, 2026
Conversation
The periodic 30s reconciliation planned ActionConnect for any enabled, unconnected server regardless of its failure history, bypassing the managed client's exponential backoff and MaxConnectionRetries give-up (the ConnectAll path already gates on ShouldRetry; the per-server reconcile path did not). A dead or 401-ing remote upstream was therefore re-dialed every 30 seconds forever - roughly 3 HTTP requests per tick, ~9k requests/day against a server that cannot answer. computeReconcilePlan now consults the client's ConnectionInfo and plans ActionNone while the backoff window is open, after the client gave up, and for servers parked in PendingAuth awaiting user OAuth login. The backoff formula is extracted from StateManager.ShouldRetry into RetryBackoffDuration so both paths share one schedule. Fixes smart-mcp-proxy#1013
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
github-actions
Bot
merged commit Aug 23, 2026
3b80912
into
smart-mcp-proxy:main
36 of 37 checks passed
This was referenced Aug 24, 2026
Dumbris
added a commit
that referenced
this pull request
Aug 24, 2026
…-scan redial loop
Cross-model review of the previous commit surfaced four defects, three of them
in code it newly depended on:
- `IsOAuthPending` type-asserted instead of unwrapping, but the deferred-OAuth
error is wrapped twice on its way out of Connect ("all authentication
strategies failed, last error: %w" then "failed to connect: %w"). The check
never fired in production, so the park — and every consumer keyed to it — was
unreachable. Now `errors.As`, with the production wrapping shape pinned in a
test.
- `Manager.scanForNewTokens` triggered a reconnect whenever *any* persisted
token existed. A failing server usually still holds the expired/revoked token
it just failed with, so the 5s scan redialed it every pass (rate-limited only
to 10s/server) — a tighter loop than the supervisor's and one that walks
straight through the park. It now records a fingerprint of the token it
retried with and fires only when a login or refresh writes a different one.
- `ConnectAll` gated on `StateError && !ShouldRetry()`, so a config reload
redialed parked and OAuth-backed-off servers. It now consults the same
`ShouldAutoReconnect` policy as the supervisor. A config change for the
server itself still reconnects it via ActionReconnect.
- `SetPendingAuth` dispatched its state-change callback detached, which can be
scheduled after a later Connecting/Ready callback and surface a stale
"sign in required" prompt right after a successful login. It is synchronous
now, matching the TransitionTo that used to deliver this state.
Also replaces the reconcile test's fixed sleeps with an actionWg barrier so the
negative assertions cannot pass before an erroneous redial would have run.
Refs #1013, #1014
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 24, 2026
…1039) * fix(upstream): close the remaining reconnect-storm holes after #1014 #1014 gated the supervisor's 30s reconcile on the client's retry policy, but three holes in that gate remain — two of which leave the original storm intact: - OAuth-classified failures are recorded by SetOAuthError, which bumps OAuthRetryCount and never RetryCount. ShouldAutoReconnect read them as "RetryCount == 0 → no failures yet" and re-dialed on every tick, forever. ShouldAutoReconnect now also honors the OAuth ladder (5min → 24h), extracted from ShouldRetryOAuth into OAuthRetryBackoffDuration so both paths share one schedule. - The gate's StatePendingAuth branch was dead code: managed.Connect parked a login-blocked server with TransitionTo(StatePendingAuth) followed by SetError, and SetError forces StateError unconditionally, so the park lasted microseconds. StateManager.SetPendingAuth parks it for real; the supervisor, ConnectAll and the health check all leave it alone until a human acts. The wake paths are wired to match: Connect clears the stale core client from PendingAuth too, Manager.scanForNewTokens accepts a parked client when a token appears, PendingAuth is a valid transition source/target, health reports Sign-in required instead of falling through to healthy, the notifier emits the login prompt, and monitorConnectionStatus answers immediately instead of stalling until its timeout. - After MaxConnectionRetries the gate said "never again", so any outage longer than the ladder (~1h: sleep, VPN, overnight maintenance) left the upstream silently dead until someone reconnected it by hand — recovery the supervisor used to provide by re-dialing unconditionally. A given-up server is now probed once per GaveUpProbeInterval (30min): ~48 requests/day instead of ~2880, and it self-heals. Refs #1013, #1014 * fix(upstream): make the PendingAuth park reachable and stop the token-scan redial loop Cross-model review of the previous commit surfaced four defects, three of them in code it newly depended on: - `IsOAuthPending` type-asserted instead of unwrapping, but the deferred-OAuth error is wrapped twice on its way out of Connect ("all authentication strategies failed, last error: %w" then "failed to connect: %w"). The check never fired in production, so the park — and every consumer keyed to it — was unreachable. Now `errors.As`, with the production wrapping shape pinned in a test. - `Manager.scanForNewTokens` triggered a reconnect whenever *any* persisted token existed. A failing server usually still holds the expired/revoked token it just failed with, so the 5s scan redialed it every pass (rate-limited only to 10s/server) — a tighter loop than the supervisor's and one that walks straight through the park. It now records a fingerprint of the token it retried with and fires only when a login or refresh writes a different one. - `ConnectAll` gated on `StateError && !ShouldRetry()`, so a config reload redialed parked and OAuth-backed-off servers. It now consults the same `ShouldAutoReconnect` policy as the supervisor. A config change for the server itself still reconnects it via ActionReconnect. - `SetPendingAuth` dispatched its state-change callback detached, which can be scheduled after a later Connecting/Ready callback and surface a stale "sign in required" prompt right after a successful login. It is synchronous now, matching the TransitionTo that used to deliver this state. Also replaces the reconcile test's fixed sleeps with an actionWg barrier so the negative assertions cannot pass before an erroneous redial would have run. Refs #1013, #1014 * fix(upstream): fingerprint the token write, not just its bytes Round-2 review: a provider that re-issues a byte-identical access token (same refresh token, same expiry) left the scan's fingerprint unchanged, permanently suppressing the wake for a parked server — and when a CLI login cannot persist its OAuth completion event, this scan is the only wake left. The record's persisted Updated stamp is now part of the fingerprint, so any new write counts as news while an untouched token still does not. Refs #1013 * test(upstream): wait for the client to settle between token-scan legs The scan test's later legs ran while the reconnect triggered by the previous leg was still in flight, so the client could be Connecting — a state the scan skips for a reason the test is not about — and the assertion failed in CI. Each leg now waits for the client to settle back into PendingAuth or Error first. Refs #1013
This was referenced Aug 25, 2026
Merged
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 25, 2026
…specs 101/102 tracked (#1041) - release-qa-gate-playwright (T2): in_review -> done, pr #1030 (advisory Playwright web-UI sweep merged into the release gate). - mcp-2026-upgrade: blocked -> in_progress. The mcp-go gate cleared with v1.0.0-beta.1 (2026-08-12); spec 058 revision in review as #1033. Note rewritten: the remaining blocker is the Spec 057 cross-spec conflict, not the dependency. - tpa-db: todo -> in_review with spec: specs/101-tpa-db and pr #1028 (spec stage only; all three child tasks stay todo). - New epic schema-deferred (P1, spec 102, in_progress): deferred-schema serialization for the direct tools/list surface, spec merged in #1035 (issue #971), plan in progress. Reconnect-backoff work (#1014/#1039/#1040) is deliberately not added — the roadmap never tracked it; git history does. ROADMAP.md regenerated; gen-roadmap.py --check passes.
Contributor
Author
|
Thanks for merging this fix! Respecting backoff and retry limits should help avoid excessive requests to unavailable upstreams. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1013.
Impact
The supervisor's periodic 30s reconciliation planned
ActionConnectfor any enabled, unconnected server regardless of failure history, bypassing the managed client's exponential backoff andMaxConnectionRetriesgive-up. TheConnectAllpath ininternal/upstream/manager.goalready gates onStateError && !ShouldRetry()— the per-server reconcile path did not. A dead or 401-ing remote upstream was re-dialed every 30 seconds forever (~3 HTTP requests per attempt, ~9k requests/day). In our case this tripped a third-party MCP vendor's server alarms twice and got our account paused (full incident detail in #1013).Changes
internal/upstream/types/types.go— extracted the backoff formula fromStateManager.ShouldRetryintoRetryBackoffDuration(retryCount)(behavior unchanged: 1s → 2s → 4s … capped at 5min), and addedConnectionInfo.ShouldAutoReconnect(now): false while the backoff window is open, afterGaveUp/MaxConnectionRetries, and forStatePendingAuth(redialing cannot succeed until the user completes OAuth login; each attempt costs real requests against the upstream).internal/runtime/supervisor/supervisor.go—computeReconcilePlanconsultsactual.ConnectionInfo.ShouldAutoReconnect(...)before planningActionConnectand plansActionNone(with a debug log) when the client's retry policy says wait.Manual reconnects, login flows,
reconnect_on_use, and config-changeActionReconnectare unaffected — only the automatic periodic redial is gated. Once the backoff window elapses, the next reconcile tick reconnects as before.Verification
New tests:
TestRetryBackoffDuration— schedule values and overflow capsTestConnectionInfo_ShouldAutoReconnect— nil/fresh/ready/pending-auth/backoff-open/backoff-elapsed/gave-up casesTestSupervisor_Reconcile_RespectsRetryBackoff— reconcile skips a failed server inside its backoff window, a gave-up server, and a PendingAuth server; reconnects after backoff elapsesLog excerpt of the failure mode this prevents (identical cycle every 30s for two weeks):