Skip to content

[Bug]: Upstream reconnect has no backoff — supervisor re-dials failed servers every 30s forever, hammering dead remotes (~9k req/day) #1013

Closed

Description

@antongulin

What happened?

The supervisor reconnects failed upstreams every 30 seconds forever, with no backoff, no retry cap, and no distinction for auth failures. Each attempt fires ~3 HTTP requests at the upstream (plain initialize, a retry, then the OAuth-strategy initialize), so a single dead remote server generates ~8,600 requests/day indefinitely.

This is not theoretical — it burned a real vendor relationship twice:

  • Late July 2026: a remote upstream (pressranger.com/mcp) started returning 401 after a static bearer token expired. mcpproxy hammered it (~95k req/day across our agents' instances) until the vendor's server alarms tripped and they paused our account, emailing us to ask what was wrong.
  • Aug 6–20, 2026: same upstream, token revoked by the vendor — the loop ran again at ~8–9k req/day for two weeks before we noticed. A failed upstream is completely silent from the operator's side (agents just don't see its tools); the entire cost of the retry loop lands on the remote server.

The ironic part: on every failed attempt mcpproxy correctly diagnoses the situation and logs OAuth authentication required for X: login available via Web UI... — a state that cannot be fixed by retrying — and then retries 30 seconds later anyway.

Where

Confirmed present on main @ 52146a4 (post-v0.59.0), same behavior observed on v0.51.0:

  • internal/runtime/supervisor/supervisor.go — reconciliationLoop runs reconcile on a fixed time.NewTicker(30 * time.Second).
  • computeReconcilePlan plans ActionConnect for any server with Enabled && !actuallyConnected (unless quarantined or user-logged-out). It never consults the managed client's RetryCount, last error time, or error class — so an upstream in Error state is re-dialed on every tick, forever.
  • The per-client retry_count already tracked in ConnectionInfo/stateview is only used for display, never for pacing.

Backoff logic already exists elsewhere in the codebase (tray API client, registry httpclient, OAuth refresh manager) — the upstream reconnect path is the one place it's missing, and it's the path that hits third-party servers.

Expected behavior

  1. Exponential backoff per server on consecutive connect failures: e.g. 30s → 1m → 2m → ... capped at 15–30min, reset on success, config change, or explicit user action (enable/restart/login). The existing retry_count in ConnectionInfo is already most of the state needed; computeReconcilePlan could skip ActionConnect while now < lastFailure + backoff(retryCount).
  2. Auth-required failures (401 + OAuth-required classification) should park the server in an "awaiting login" state rather than retry at all — retrying cannot succeed until a human completes the OAuth flow, and mcpproxy already emits exactly that log line. reconnect_on_use / manual login can still trigger a fresh attempt.
  3. Honor Retry-After when the upstream answers 429 (the OAuth resource-detect path already parses hints; connect retries should too).

Steps to reproduce

  1. Add a streamable-http upstream that answers 401 to initialize (any endpoint with a revoked/expired bearer token in headers), "enabled": true, no valid OAuth token stored.
  2. Run mcpproxy serve headless and tail logs/server-<name>.log.
  3. Observe the identical attempt cycle every 30s with no growth in interval: Starting connection attempt → 2× MCP initialize JSON-RPC call failed ... authorization required → OAuth strategy failure → Connection failed → repeat. tcpdump/upstream access logs show ~3 requests per cycle, ~8.6k/day.

MCPProxy version

v0.51.0 linux/amd64 (incident); bug confirmed by code inspection on main @ 52146a4

Operating system

Linux

Relevant logs

2026-08-20T18:15:52.759Z | INFO  | managed/client.go:239 | Starting connection attempt | {"server": "press-ranger", "transport": "http", "url": "https://pressranger.com/mcp", "protocol": "http"}
2026-08-20T18:15:53.809Z | ERROR | core/connection_http.go:191 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: authorization required"}
2026-08-20T18:15:54.523Z | ERROR | core/connection_http.go:219 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: authorization required"}
2026-08-20T18:15:58.213Z | ERROR | core/connection_oauth.go:426 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: failed to send request: no valid token available, authorization required"}
2026-08-20T18:15:58.214Z | ERROR | managed/client.go:239 | Connection failed | {"server": "press-ranger", "error": "OAuth authentication required for press-ranger: login available via Web UI, system tray menu, or 'mcpproxy auth login' CLI command"}
2026-08-20T18:16:22.758Z | INFO  | managed/client.go:239 | Starting connection attempt | ...   <- exactly 30s later, forever

(Identical cycle fills every rotated server-press-ranger.log from Aug 6 to Aug 20 — ~40k attempts, ~120k HTTP requests, all against an endpoint that could never succeed without human OAuth login.)

Activity

  1. antongulin commented on Aug 20, 2026

    @antongulin
    ContributorAuthor

    Root cause pinned down while writing the fix: the client-level exponential backoff and MaxConnectionRetries give-up already exist (StateManager.ShouldRetry), and the ConnectAll path in internal/upstream/manager.go correctly gates on them — but the supervisor's periodic per-server reconcile path (computeReconcilePlan → ActionConnect → ActorPoolSimple.ConnectServer → client.Connect()) bypasses that policy entirely, so the 30s ticker re-dials failed servers forever. StatePendingAuth servers are re-dialed too, even though the retry can't succeed until the user completes the OAuth login. Fix in #1014.

  2. added a commit that references this issue on Aug 23, 2026
    3b80912
  3. Dumbris commented on Aug 24, 2026

    @Dumbris
    Member

    Status update on the three asks in this report, now that the follow-up work has landed.

    #1014 (merged) gated the supervisor's 30s reconcile on the managed client's retry policy. Reviewing it against the asks here turned up several gaps in that gate — some of which left the original storm running — so #1039 (merged, 7b118d1) closes them:

    Ask 1 — exponential backoff: covered.
    ShouldAutoReconnect now also honors the OAuth ladder (5min → 15min → 1h → 4h → 24h). SetOAuthError bumps oauthRetryCount and never retryCount, so OAuth-classified failures — the very class this report is about — read as "no failures yet" and were still re-dialed every 30s after #1014. ConnectAll (config-reload path) consults the same policy now instead of StateError && !ShouldRetry().

    Rather than a permanent give-up after MaxConnectionRetries (≈1h of failures), a given-up server is probed once per 30min — matching your "capped at 15–30min", and keeping an upstream that was merely offline overnight from staying silently dead until someone notices. ~48 requests/day instead of ~2880.

    Ask 2 — park auth-required servers: covered.
    Two independent bugs meant the park never happened: IsOAuthPending used a bare type assertion while the deferred-OAuth error is wrapped twice on the way out of Connect (so it never returned true), and the park itself was TransitionTo(StatePendingAuth) immediately followed by SetError, which forces StateError unconditionally. StateManager.SetPendingAuth now parks for real: no supervisor redial, no ConnectAll redial, no health-check redial, until a human acts. The wake paths (login, manual reconnect, reconnect_on_use, persisted-token scan) are wired to it, health reports Sign-in required instead of falling through to healthy, and spec 098's preflight "waiting for OAuth login" verdict becomes reachable.

    A related storm turned up while wiring that: scanForNewTokens (every 5s) triggered a reconnect whenever any persisted token existed — and a failing server usually still holds the expired/revoked token it just failed with, so on its own that path redialed ~8,600×/day, out-pacing the loop you reported. It now fires only when a login or refresh actually writes a new token.

    Ask 3 — honor Retry-After on 429: not covered, tracked in #1040.
    It can't be done in the backoff layer: mcp-go flattens the response into fmt.Errorf("request failed with status %d: %s", …), so the header never reaches us. Doing it properly needs a RoundTripper in internal/transport threaded through every client-construction path — a change to all upstream HTTP traffic, not just connect — so it got its own issue with the design and acceptance criteria. In the meantime a 429-ing upstream is paced 1s → … → 5min, then one probe per 30min, so the worst case is ~48 requests/day rather than the ~8.6k that started this.

    This issue was auto-closed by #1014's Fixes keyword; leaving it closed since #1040 carries the remaining ask — happy to reopen if you'd rather track it here.

    Thanks for the unusually precise report — the code pointers in it were exactly right, and the incident detail is what made the third-party cost obvious.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions