Skip to content

fix(upstream): respect client retry backoff in supervisor reconciliation - #1014

Merged
github-actions[bot] merged 1 commit into
smart-mcp-proxy:mainfrom
antongulin:fix/supervisor-reconnect-backoff
Aug 23, 2026
Merged

github-actions[bot] merged 1 commit into
smart-mcp-proxy:mainfrom
antongulin:fix/supervisor-reconnect-backoff

Conversation

@antongulin

Copy link
Copy Markdown
Contributor

Fixes #1013.

Impact

The supervisor's periodic 30s reconciliation planned ActionConnect for any enabled, unconnected server regardless of failure history, bypassing the managed client's exponential backoff and MaxConnectionRetries give-up. The ConnectAll path in internal/upstream/manager.go already gates on StateError && !ShouldRetry() — the per-server reconcile path did not. A dead or 401-ing remote upstream was re-dialed every 30 seconds forever (~3 HTTP requests per attempt, ~9k requests/day). In our case this tripped a third-party MCP vendor's server alarms twice and got our account paused (full incident detail in #1013).

Changes

  • internal/upstream/types/types.go — extracted the backoff formula from StateManager.ShouldRetry into RetryBackoffDuration(retryCount) (behavior unchanged: 1s → 2s → 4s … capped at 5min), and added ConnectionInfo.ShouldAutoReconnect(now): false while the backoff window is open, after GaveUp/MaxConnectionRetries, and for StatePendingAuth (redialing cannot succeed until the user completes OAuth login; each attempt costs real requests against the upstream).
  • internal/runtime/supervisor/supervisor.go — computeReconcilePlan consults actual.ConnectionInfo.ShouldAutoReconnect(...) before planning ActionConnect and plans ActionNone (with a debug log) when the client's retry policy says wait.

Manual reconnects, login flows, reconnect_on_use, and config-change ActionReconnect are unaffected — only the automatic periodic redial is gated. Once the backoff window elapses, the next reconcile tick reconnects as before.

Verification

go test ./internal/upstream/... ./internal/runtime/...   # all pass (incl. new tests)
go vet ./internal/upstream/... ./internal/runtime/supervisor/
gofmt -l <changed files>                                  # clean

New tests:

  • TestRetryBackoffDuration — schedule values and overflow caps
  • TestConnectionInfo_ShouldAutoReconnect — nil/fresh/ready/pending-auth/backoff-open/backoff-elapsed/gave-up cases
  • TestSupervisor_Reconcile_RespectsRetryBackoff — reconcile skips a failed server inside its backoff window, a gave-up server, and a PendingAuth server; reconnects after backoff elapses

Log excerpt of the failure mode this prevents (identical cycle every 30s for two weeks):

18:15:52.759 | INFO  | Starting connection attempt          | {"server": "press-ranger", ...}
18:15:53.809 | ERROR | MCP initialize JSON-RPC call failed  | {"error": "transport error: authorization required"}
18:15:58.214 | ERROR | Connection failed                    | {"error": "OAuth authentication required ..."}
18:16:22.758 | INFO  | Starting connection attempt          | ...

The periodic 30s reconciliation planned ActionConnect for any enabled,
unconnected server regardless of its failure history, bypassing the managed
client's exponential backoff and MaxConnectionRetries give-up (the ConnectAll
path already gates on ShouldRetry; the per-server reconcile path did not).
A dead or 401-ing remote upstream was therefore re-dialed every 30 seconds
forever - roughly 3 HTTP requests per tick, ~9k requests/day against a server
that cannot answer.

computeReconcilePlan now consults the client's ConnectionInfo and plans
ActionNone while the backoff window is open, after the client gave up, and for
servers parked in PendingAuth awaiting user OAuth login. The backoff formula
is extracted from StateManager.ShouldRetry into RetryBackoffDuration so both
paths share one schedule.

Fixes smart-mcp-proxy#1013
@Dumbris Dumbris changed the title Respect client retry backoff in supervisor reconciliation fix(upstream): respect client retry backoff in supervisor reconciliation Aug 23, 2026
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.87500% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/runtime/supervisor/supervisor.go 83.33% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (Model B): Paperclip review verdicts = ACCEPT and qa-gate green at this head SHA. Arming auto-merge; GitHub merges when all required checks pass.

@github-actions
github-actions Bot merged commit 3b80912 into smart-mcp-proxy:main Aug 23, 2026
36 of 37 checks passed
Dumbris added a commit that referenced this pull request Aug 24, 2026
…-scan redial loop

Cross-model review of the previous commit surfaced four defects, three of them
in code it newly depended on:

- `IsOAuthPending` type-asserted instead of unwrapping, but the deferred-OAuth
  error is wrapped twice on its way out of Connect ("all authentication
  strategies failed, last error: %w" then "failed to connect: %w"). The check
  never fired in production, so the park — and every consumer keyed to it — was
  unreachable. Now `errors.As`, with the production wrapping shape pinned in a
  test.

- `Manager.scanForNewTokens` triggered a reconnect whenever *any* persisted
  token existed. A failing server usually still holds the expired/revoked token
  it just failed with, so the 5s scan redialed it every pass (rate-limited only
  to 10s/server) — a tighter loop than the supervisor's and one that walks
  straight through the park. It now records a fingerprint of the token it
  retried with and fires only when a login or refresh writes a different one.

- `ConnectAll` gated on `StateError && !ShouldRetry()`, so a config reload
  redialed parked and OAuth-backed-off servers. It now consults the same
  `ShouldAutoReconnect` policy as the supervisor. A config change for the
  server itself still reconnects it via ActionReconnect.

- `SetPendingAuth` dispatched its state-change callback detached, which can be
  scheduled after a later Connecting/Ready callback and surface a stale
  "sign in required" prompt right after a successful login. It is synchronous
  now, matching the TransitionTo that used to deliver this state.

Also replaces the reconcile test's fixed sleeps with an actionWg barrier so the
negative assertions cannot pass before an erroneous redial would have run.

Refs #1013, #1014
github-actions Bot pushed a commit that referenced this pull request Aug 24, 2026
…1039)

* fix(upstream): close the remaining reconnect-storm holes after #1014

#1014 gated the supervisor's 30s reconcile on the client's retry policy, but
three holes in that gate remain — two of which leave the original storm intact:

- OAuth-classified failures are recorded by SetOAuthError, which bumps
  OAuthRetryCount and never RetryCount. ShouldAutoReconnect read them as
  "RetryCount == 0 → no failures yet" and re-dialed on every tick, forever.
  ShouldAutoReconnect now also honors the OAuth ladder (5min → 24h), extracted
  from ShouldRetryOAuth into OAuthRetryBackoffDuration so both paths share one
  schedule.

- The gate's StatePendingAuth branch was dead code: managed.Connect parked a
  login-blocked server with TransitionTo(StatePendingAuth) followed by
  SetError, and SetError forces StateError unconditionally, so the park lasted
  microseconds. StateManager.SetPendingAuth parks it for real; the supervisor,
  ConnectAll and the health check all leave it alone until a human acts. The
  wake paths are wired to match: Connect clears the stale core client from
  PendingAuth too, Manager.scanForNewTokens accepts a parked client when a
  token appears, PendingAuth is a valid transition source/target, health
  reports Sign-in required instead of falling through to healthy, the
  notifier emits the login prompt, and monitorConnectionStatus answers
  immediately instead of stalling until its timeout.

- After MaxConnectionRetries the gate said "never again", so any outage longer
  than the ladder (~1h: sleep, VPN, overnight maintenance) left the upstream
  silently dead until someone reconnected it by hand — recovery the supervisor
  used to provide by re-dialing unconditionally. A given-up server is now
  probed once per GaveUpProbeInterval (30min): ~48 requests/day instead of
  ~2880, and it self-heals.

Refs #1013, #1014

* fix(upstream): make the PendingAuth park reachable and stop the token-scan redial loop

Cross-model review of the previous commit surfaced four defects, three of them
in code it newly depended on:

- `IsOAuthPending` type-asserted instead of unwrapping, but the deferred-OAuth
  error is wrapped twice on its way out of Connect ("all authentication
  strategies failed, last error: %w" then "failed to connect: %w"). The check
  never fired in production, so the park — and every consumer keyed to it — was
  unreachable. Now `errors.As`, with the production wrapping shape pinned in a
  test.

- `Manager.scanForNewTokens` triggered a reconnect whenever *any* persisted
  token existed. A failing server usually still holds the expired/revoked token
  it just failed with, so the 5s scan redialed it every pass (rate-limited only
  to 10s/server) — a tighter loop than the supervisor's and one that walks
  straight through the park. It now records a fingerprint of the token it
  retried with and fires only when a login or refresh writes a different one.

- `ConnectAll` gated on `StateError && !ShouldRetry()`, so a config reload
  redialed parked and OAuth-backed-off servers. It now consults the same
  `ShouldAutoReconnect` policy as the supervisor. A config change for the
  server itself still reconnects it via ActionReconnect.

- `SetPendingAuth` dispatched its state-change callback detached, which can be
  scheduled after a later Connecting/Ready callback and surface a stale
  "sign in required" prompt right after a successful login. It is synchronous
  now, matching the TransitionTo that used to deliver this state.

Also replaces the reconcile test's fixed sleeps with an actionWg barrier so the
negative assertions cannot pass before an erroneous redial would have run.

Refs #1013, #1014

* fix(upstream): fingerprint the token write, not just its bytes

Round-2 review: a provider that re-issues a byte-identical access token (same
refresh token, same expiry) left the scan's fingerprint unchanged, permanently
suppressing the wake for a parked server — and when a CLI login cannot persist
its OAuth completion event, this scan is the only wake left. The record's
persisted Updated stamp is now part of the fingerprint, so any new write counts
as news while an untouched token still does not.

Refs #1013

* test(upstream): wait for the client to settle between token-scan legs

The scan test's later legs ran while the reconnect triggered by the previous leg
was still in flight, so the client could be Connecting — a state the scan skips
for a reason the test is not about — and the assertion failed in CI. Each leg now
waits for the client to settle back into PendingAuth or Error first.

Refs #1013
github-actions Bot pushed a commit that referenced this pull request Aug 25, 2026
…specs 101/102 tracked (#1041)

- release-qa-gate-playwright (T2): in_review -> done, pr #1030 (advisory
  Playwright web-UI sweep merged into the release gate).
- mcp-2026-upgrade: blocked -> in_progress. The mcp-go gate cleared with
  v1.0.0-beta.1 (2026-08-12); spec 058 revision in review as #1033. Note
  rewritten: the remaining blocker is the Spec 057 cross-spec conflict,
  not the dependency.
- tpa-db: todo -> in_review with spec: specs/101-tpa-db and pr #1028
  (spec stage only; all three child tasks stay todo).
- New epic schema-deferred (P1, spec 102, in_progress): deferred-schema
  serialization for the direct tools/list surface, spec merged in #1035
  (issue #971), plan in progress.

Reconnect-backoff work (#1014/#1039/#1040) is deliberately not added —
the roadmap never tracked it; git history does.

ROADMAP.md regenerated; gen-roadmap.py --check passes.
@antongulin

Copy link
Copy Markdown
Contributor Author

Thanks for merging this fix! Respecting backoff and retry limits should help avoid excessive requests to unavailable upstreams.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Upstream reconnect has no backoff — supervisor re-dials failed servers every 30s forever, hammering dead remotes (~9k req/day)

2 participants