Repository navigation
Description
What happened?
The supervisor reconnects failed upstreams every 30 seconds forever, with no backoff, no retry cap, and no distinction for auth failures. Each attempt fires ~3 HTTP requests at the upstream (plain initialize, a retry, then the OAuth-strategy initialize), so a single dead remote server generates ~8,600 requests/day indefinitely.
This is not theoretical — it burned a real vendor relationship twice:
- Late July 2026: a remote upstream (
pressranger.com/mcp) started returning 401 after a static bearer token expired. mcpproxy hammered it (~95k req/day across our agents' instances) until the vendor's server alarms tripped and they paused our account, emailing us to ask what was wrong. - Aug 6–20, 2026: same upstream, token revoked by the vendor — the loop ran again at ~8–9k req/day for two weeks before we noticed. A failed upstream is completely silent from the operator's side (agents just don't see its tools); the entire cost of the retry loop lands on the remote server.
The ironic part: on every failed attempt mcpproxy correctly diagnoses the situation and logs OAuth authentication required for X: login available via Web UI... — a state that cannot be fixed by retrying — and then retries 30 seconds later anyway.
Where
Confirmed present on main @ 52146a4 (post-v0.59.0), same behavior observed on v0.51.0:
internal/runtime/supervisor/supervisor.go—reconciliationLooprunsreconcileon a fixedtime.NewTicker(30 * time.Second).computeReconcilePlanplansActionConnectfor any server withEnabled && !actuallyConnected(unless quarantined or user-logged-out). It never consults the managed client'sRetryCount, last error time, or error class — so an upstream inErrorstate is re-dialed on every tick, forever.- The per-client
retry_countalready tracked inConnectionInfo/stateview is only used for display, never for pacing.
Backoff logic already exists elsewhere in the codebase (tray API client, registry httpclient, OAuth refresh manager) — the upstream reconnect path is the one place it's missing, and it's the path that hits third-party servers.
Expected behavior
- Exponential backoff per server on consecutive connect failures: e.g. 30s → 1m → 2m → ... capped at 15–30min, reset on success, config change, or explicit user action (enable/restart/login). The existing
retry_countinConnectionInfois already most of the state needed;computeReconcilePlancould skipActionConnectwhilenow < lastFailure + backoff(retryCount). - Auth-required failures (401 + OAuth-required classification) should park the server in an "awaiting login" state rather than retry at all — retrying cannot succeed until a human completes the OAuth flow, and mcpproxy already emits exactly that log line.
reconnect_on_use/ manual login can still trigger a fresh attempt. - Honor
Retry-Afterwhen the upstream answers 429 (the OAuth resource-detect path already parses hints; connect retries should too).
Steps to reproduce
- Add a
streamable-httpupstream that answers 401 toinitialize(any endpoint with a revoked/expired bearer token inheaders),"enabled": true, no valid OAuth token stored. - Run
mcpproxy serveheadless and taillogs/server-<name>.log. - Observe the identical attempt cycle every 30s with no growth in interval:
Starting connection attempt→ 2×MCP initialize JSON-RPC call failed ... authorization required→ OAuth strategy failure →Connection failed→ repeat.tcpdump/upstream access logs show ~3 requests per cycle, ~8.6k/day.
MCPProxy version
v0.51.0 linux/amd64 (incident); bug confirmed by code inspection on main @ 52146a4
Operating system
Linux
Relevant logs
2026-08-20T18:15:52.759Z | INFO | managed/client.go:239 | Starting connection attempt | {"server": "press-ranger", "transport": "http", "url": "https://pressranger.com/mcp", "protocol": "http"}
2026-08-20T18:15:53.809Z | ERROR | core/connection_http.go:191 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: authorization required"}
2026-08-20T18:15:54.523Z | ERROR | core/connection_http.go:219 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: authorization required"}
2026-08-20T18:15:58.213Z | ERROR | core/connection_oauth.go:426 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: failed to send request: no valid token available, authorization required"}
2026-08-20T18:15:58.214Z | ERROR | managed/client.go:239 | Connection failed | {"server": "press-ranger", "error": "OAuth authentication required for press-ranger: login available via Web UI, system tray menu, or 'mcpproxy auth login' CLI command"}
2026-08-20T18:16:22.758Z | INFO | managed/client.go:239 | Starting connection attempt | ... <- exactly 30s later, forever
(Identical cycle fills every rotated server-press-ranger.log from Aug 6 to Aug 20 — ~40k attempts, ~120k HTTP requests, all against an endpoint that could never succeed without human OAuth login.)
Activity
antongulin commentedon Aug 20, 2026
Root cause pinned down while writing the fix: the client-level exponential backoff and
MaxConnectionRetriesgive-up already exist (StateManager.ShouldRetry), and theConnectAllpath ininternal/upstream/manager.gocorrectly gates on them — but the supervisor's periodic per-server reconcile path (computeReconcilePlan→ActionConnect→ActorPoolSimple.ConnectServer→client.Connect()) bypasses that policy entirely, so the 30s ticker re-dials failed servers forever.StatePendingAuthservers are re-dialed too, even though the retry can't succeed until the user completes the OAuth login. Fix in #1014.Respect client retry backoff in supervisor reconciliation (#1014)
fix(upstream): make the PendingAuth park reachable and stop the token…
fix(upstream): fingerprint the token write, not just its bytes
test(upstream): wait for the client to settle between token-scan legs
fix(upstream): close the remaining reconnect-storm holes after #1014 (#…
Dumbris commentedon Aug 24, 2026
Status update on the three asks in this report, now that the follow-up work has landed.
#1014 (merged) gated the supervisor's 30s reconcile on the managed client's retry policy. Reviewing it against the asks here turned up several gaps in that gate — some of which left the original storm running — so #1039 (merged,
7b118d1) closes them:Ask 1 — exponential backoff: covered.
ShouldAutoReconnectnow also honors the OAuth ladder (5min → 15min → 1h → 4h → 24h).SetOAuthErrorbumpsoauthRetryCountand neverretryCount, so OAuth-classified failures — the very class this report is about — read as "no failures yet" and were still re-dialed every 30s after #1014.ConnectAll(config-reload path) consults the same policy now instead ofStateError && !ShouldRetry().Rather than a permanent give-up after
MaxConnectionRetries(≈1h of failures), a given-up server is probed once per 30min — matching your "capped at 15–30min", and keeping an upstream that was merely offline overnight from staying silently dead until someone notices. ~48 requests/day instead of ~2880.Ask 2 — park auth-required servers: covered.
Two independent bugs meant the park never happened:
IsOAuthPendingused a bare type assertion while the deferred-OAuth error is wrapped twice on the way out ofConnect(so it never returned true), and the park itself wasTransitionTo(StatePendingAuth)immediately followed bySetError, which forcesStateErrorunconditionally.StateManager.SetPendingAuthnow parks for real: no supervisor redial, noConnectAllredial, no health-check redial, until a human acts. The wake paths (login, manual reconnect,reconnect_on_use, persisted-token scan) are wired to it, health reports Sign-in required instead of falling through to healthy, and spec 098's preflight "waiting for OAuth login" verdict becomes reachable.A related storm turned up while wiring that:
scanForNewTokens(every 5s) triggered a reconnect whenever any persisted token existed — and a failing server usually still holds the expired/revoked token it just failed with, so on its own that path redialed ~8,600×/day, out-pacing the loop you reported. It now fires only when a login or refresh actually writes a new token.Ask 3 — honor
Retry-Afteron 429: not covered, tracked in #1040.It can't be done in the backoff layer:
mcp-goflattens the response intofmt.Errorf("request failed with status %d: %s", …), so the header never reaches us. Doing it properly needs aRoundTripperininternal/transportthreaded through every client-construction path — a change to all upstream HTTP traffic, not just connect — so it got its own issue with the design and acceptance criteria. In the meantime a 429-ing upstream is paced 1s → … → 5min, then one probe per 30min, so the worst case is ~48 requests/day rather than the ~8.6k that started this.This issue was auto-closed by #1014's
Fixeskeyword; leaving it closed since #1040 carries the remaining ask — happy to reopen if you'd rather track it here.Thanks for the unusually precise report — the code pointers in it were exactly right, and the incident detail is what made the third-party cost obvious.
feat(upstream): honor Retry-After on 429 via transport RoundTripper (#…
fix(upstream): stop retrying deterministic connection failures forever (
fix(upstream): stop network errors parking no-auth servers in Pending…
fix(upstream): reconnect remote servers as soon as the network returns (