Skip to content

fix(upstream): close the remaining reconnect-storm holes after #1014 - #1039

Merged
github-actions[bot] merged 4 commits into
mainfrom
fix/reconnect-backoff-followups
Aug 24, 2026
Merged

github-actions[bot] merged 4 commits into
mainfrom
fix/reconnect-backoff-followups

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 24, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #1014 (merged), which fixed the headline storm from #1013. Reviewing that fix against the issue's three asks turned up gaps in the new gate — two of which left the original 30s storm intact for a whole error class, and one of which made the fix's PendingAuth branch dead code.

1. OAuth-classified failures were still re-dialed every 30s

SetOAuthError bumps oauthRetryCount and never retryCount. ConnectionInfo.ShouldAutoReconnect only consulted RetryCount, so an OAuth failure read as RetryCount == 0 → "no failures yet" → ActionConnect on every reconcile tick, forever. The exact behavior #1014 set out to stop, for the error class (invalid_token, 401, unauthorized, invalid_grant, …) most likely to produce it.

ShouldAutoReconnect now also honors the OAuth ladder (5min → 15min → 1h → 4h → 24h), extracted from StateManager.ShouldRetryOAuth into OAuthRetryBackoffDuration so the health-check path and the supervisor path share one schedule — mirroring what #1014 did with RetryBackoffDuration. Both gates apply when both counters moved; the longer wait wins.

2. The park never happened — twice over

Issue ask (2) — "auth-required failures should park in an awaiting-login state rather than retry at all" — was not in effect, for two independent reasons:

  • IsOAuthPending never returned true. It type-asserted, but the deferred-OAuth error is wrapped twice on its way out: connectHTTP/connectSSE add "all authentication strategies failed, last error: %w" and Connect adds "failed to connect: %w". Now errors.As, with the production wrapping shape pinned in a test.
  • SetError undid the park. managed.Connect did TransitionTo(StatePendingAuth) immediately followed by SetError(connectErr), and SetError sets currentState = StateError unconditionally — the park survived microseconds. StateManager.SetPendingAuth now parks for real: state stays PendingAuth, the error stays attached, RetryCount is deliberately not advanced (a parked server is not retrying). The supervisor, ConnectAll and performHealthCheck all leave it alone until a human acts.

Making that state reachable means the paths that assumed "OAuth-required ⇒ StateError" had to be wired to match:

Path Change
managed.Connect clears the stale core client from PendingAuth too, so the post-login reconnect doesn't hit "client already connected"
Manager.scanForNewTokens a persisted token now also wakes a parked client (was StateError-only) — the fallback wake when a CLI login can't write its completion event
ValidateTransition Connecting/Authenticating → PendingAuth and PendingAuth → Connecting/Disconnected/Error are legal (the old park printed "Invalid state transition" to stdout)
health.CalculateHealth "pending auth" → Sign-in required (amber) / Authentication required (red for a broken stored token). Without this a parked server fell through the state switch and could report healthy
notifications emits the OAuth-required prompt on entering PendingAuth (the old SetError emitted a generic server error)
monitorConnectionStatus answers immediately for a parked server instead of stalling the upstream_servers caller until the monitor timeout

Spec 098's preflight already had a RuntimeStatePendingAuth verdict ("Server is waiting for OAuth login") keyed on this state string — until now unreachable. Manual reconnect, ForceReconnect/RetryConnection, reconnect_on_use and login flows are state-agnostic and unaffected.

3. The token scan was a tighter storm than the one being fixed

scanForNewTokens (every 5s) triggered a reconnect whenever any persisted token existed, rate-limited only to one per 10s per server. A failing server usually still holds the expired/revoked token it just failed with — so this redialed it ~8,600×/day on its own, out-pacing the supervisor loop and walking straight through the park.

It now fingerprints the token it retried with (truncated SHA-256 over access + refresh + expiry + the record's persisted Updated stamp — never the raw token) and fires only when a login or refresh writes a different one. The write stamp is part of the identity so a provider re-issuing a byte-identical token still counts as news.

4. "Gave up" meant never again

After MaxConnectionRetries the gate returned false unconditionally. The ladder is exhausted in roughly an hour, so any longer outage — laptop sleep, VPN drop, overnight upstream maintenance — left the upstream silently dead until a human noticed. #1013 itself points out that a failed upstream is invisible to the operator, and the pre-#1014 supervisor provided this recovery by re-dialing unconditionally.

A given-up server is now probed once per GaveUpProbeInterval (30min): ~48 requests/day instead of ~2880, and it self-heals. This also matches ask (1), which asked for a backoff capped at 15–30min, not a permanent give-up.

ConnectAll (config-reload path) now consults the same ShouldAutoReconnect policy instead of StateError && !ShouldRetry(), so a reload no longer re-dials servers their own client had paused. A config change for the server itself still reconnects it — the supervisor plans ActionReconnect, which bypasses the gate.

Not covered: ask (3), Retry-After on 429 — #1040

It cannot be done in this layer: mcp-go flattens the response to fmt.Errorf("request failed with status %d: %s", …), so the header never reaches us. Honoring it needs a RoundTripper in internal/transport threaded through every client-construction branch (including the two NewOAuth*Client paths that build their own http.Client) — a change to all upstream HTTP traffic, not just connect. Design and acceptance criteria are in #1040. With the fixes above a 429-ing upstream is already paced 1s → 5min → 30min probes, so the risk is bounded meanwhile.

Verification

go build -o /dev/null ./cmd/mcpproxy                                    # personal
go build -tags server -o /dev/null ./cmd/mcpproxy                       # server edition
go test -race ./internal/runtime/... ./internal/upstream/... ./internal/health/...
go test -race -skip='E2E|Binary|MCPProtocol|...' ./internal/server/...   # CI skip set
golangci-lint run --config .github/.golangci.yml   # v2, 0 issues

New/updated tests: TestIsOAuthPending (+%w-wrapped case), TestOAuthRetryBackoffDuration, TestSetPendingAuth (park sticks, error preserved, no auto-redial, wake transition legal, consumers notified), TestConnectionInfo_ShouldAutoReconnect (+7 cases: OAuth ladder open/elapsed/long, both-ladders, give-up probe), TestTokenFingerprint, TestScanForNewTokens_OnlyOnNewToken, TestSupervisor_Reconcile_RespectsRetryBackoff (+OAuth-window and give-up-probe legs, and an actionWg barrier replacing fixed sleeps so the negative assertions can't pass early), TestCalculateHealth_PendingAuthParked.

Cross-model review (opencode / gpt-5.6-sol): 3 rounds, 6 findings, all fixed — final verdict CLEAN. Findings 1–3 above (errors.As, the 5s token-scan loop, the ConnectAll gate) came out of round 1; the callback-ordering fix in SetPendingAuth (synchronous, like the TransitionTo that used to deliver this state, so a stale "sign in required" can't land after a successful login) and the token write-stamp came out of rounds 1–2.

Refs #1013, #1014, #1040

#1014 gated the supervisor's 30s reconcile on the client's retry policy, but
three holes in that gate remain — two of which leave the original storm intact:

- OAuth-classified failures are recorded by SetOAuthError, which bumps
  OAuthRetryCount and never RetryCount. ShouldAutoReconnect read them as
  "RetryCount == 0 → no failures yet" and re-dialed on every tick, forever.
  ShouldAutoReconnect now also honors the OAuth ladder (5min → 24h), extracted
  from ShouldRetryOAuth into OAuthRetryBackoffDuration so both paths share one
  schedule.

- The gate's StatePendingAuth branch was dead code: managed.Connect parked a
  login-blocked server with TransitionTo(StatePendingAuth) followed by
  SetError, and SetError forces StateError unconditionally, so the park lasted
  microseconds. StateManager.SetPendingAuth parks it for real; the supervisor,
  ConnectAll and the health check all leave it alone until a human acts. The
  wake paths are wired to match: Connect clears the stale core client from
  PendingAuth too, Manager.scanForNewTokens accepts a parked client when a
  token appears, PendingAuth is a valid transition source/target, health
  reports Sign-in required instead of falling through to healthy, the
  notifier emits the login prompt, and monitorConnectionStatus answers
  immediately instead of stalling until its timeout.

- After MaxConnectionRetries the gate said "never again", so any outage longer
  than the ladder (~1h: sleep, VPN, overnight maintenance) left the upstream
  silently dead until someone reconnected it by hand — recovery the supervisor
  used to provide by re-dialing unconditionally. A given-up server is now
  probed once per GaveUpProbeInterval (30min): ~48 requests/day instead of
  ~2880, and it self-heals.

Refs #1013, #1014
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 24, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 5b8e0a3
Status: ✅  Deploy successful!
Preview URL: https://9accf1a6.mcpproxy-docs.pages.dev
Branch Preview URL: https://fix-reconnect-backoff-follow.mcpproxy-docs.pages.dev

View logs

@codecov-commenter

codecov-commenter commented Aug 24, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 89.36170% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/upstream/manager.go 87.87% 3 Missing and 1 partial ⚠️
internal/upstream/notifications.go 0.00% 3 Missing ⚠️
internal/server/mcp.go 0.00% 2 Missing ⚠️
internal/upstream/managed/client.go 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…-scan redial loop

Cross-model review of the previous commit surfaced four defects, three of them
in code it newly depended on:

- `IsOAuthPending` type-asserted instead of unwrapping, but the deferred-OAuth
  error is wrapped twice on its way out of Connect ("all authentication
  strategies failed, last error: %w" then "failed to connect: %w"). The check
  never fired in production, so the park — and every consumer keyed to it — was
  unreachable. Now `errors.As`, with the production wrapping shape pinned in a
  test.

- `Manager.scanForNewTokens` triggered a reconnect whenever *any* persisted
  token existed. A failing server usually still holds the expired/revoked token
  it just failed with, so the 5s scan redialed it every pass (rate-limited only
  to 10s/server) — a tighter loop than the supervisor's and one that walks
  straight through the park. It now records a fingerprint of the token it
  retried with and fires only when a login or refresh writes a different one.

- `ConnectAll` gated on `StateError && !ShouldRetry()`, so a config reload
  redialed parked and OAuth-backed-off servers. It now consults the same
  `ShouldAutoReconnect` policy as the supervisor. A config change for the
  server itself still reconnects it via ActionReconnect.

- `SetPendingAuth` dispatched its state-change callback detached, which can be
  scheduled after a later Connecting/Ready callback and surface a stale
  "sign in required" prompt right after a successful login. It is synchronous
  now, matching the TransitionTo that used to deliver this state.

Also replaces the reconcile test's fixed sleeps with an actionWg barrier so the
negative assertions cannot pass before an erroneous redial would have run.

Refs #1013, #1014
Round-2 review: a provider that re-issues a byte-identical access token (same
refresh token, same expiry) left the scan's fingerprint unchanged, permanently
suppressing the wake for a parked server — and when a CLI login cannot persist
its OAuth completion event, this scan is the only wake left. The record's
persisted Updated stamp is now part of the fingerprint, so any new write counts
as news while an untouched token still does not.

Refs #1013
The scan test's later legs ran while the reconnect triggered by the previous leg
was still in flight, so the client could be Connecting — a state the scan skips
for a reason the test is not about — and the assertion failed in CI. Each leg now
waits for the client to settle back into PendingAuth or Error first.

Refs #1013
@github-actions

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: fix/reconnect-backoff-followups

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (25 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (20 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 32777420379 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (Model B): Paperclip review verdicts = ACCEPT and qa-gate green at this head SHA. Arming auto-merge; GitHub merges when all required checks pass.

@github-actions
github-actions Bot merged commit 7b118d1 into main Aug 24, 2026
40 checks passed
github-actions Bot pushed a commit that referenced this pull request Aug 25, 2026
…1042)

* feat(upstream): honor Retry-After on 429 via transport RoundTripper

mcp-go flattens every non-2xx upstream response into an error string, so a
429's Retry-After header was gone before any connection-state code ran and the
reconnect ladder paced a rate-limited upstream on its own schedule.

Capture the hint where the *http.Response still exists: a RetryAfterTransport
RoundTripper installed under every HTTP/SSE client we hand to mcp-go, feeding a
per-server RetryAfterRecorder. The managed client stamps the resulting deadline
onto the state machine, and ConnectionInfo.ShouldAutoReconnect / ShouldRetry
treat it as a floor under the existing backoff — effective delay is
max(backoff, Retry-After), capped at one hour.

- Both RFC 7231 forms parse (delta-seconds and HTTP-date); a malformed, absent,
  or already-elapsed value falls back to the existing ladder untouched.
- 503 counts only when it carries a parseable hint; a bare 503 is an outage the
  ladder already paces.
- The wrapper composes with the existing tracing transport and sits below
  mcp-go's OAuth handling, so the three authorize paths are unchanged. The two
  OAuth client constructors now receive our basic client explicitly, since
  OAuthConfig.HTTPClient covers only metadata/DCR/token requests.
- The window is cleared on a successful connect and on Reset, so a stale hint
  cannot hold back a later, unrelated reconnect.

Closes #1040. Refs #1013, #1039.

* fix(upstream): scope the Retry-After window to its attempt and gate the OAuth ladder

Cross-model review of #1042 found four real holes:

- The hint only reached the state machine after a failed Connect, so a 429
  answered to the health-check ping was recorded and then ignored.
  syncRetryAfterFromTransport is now called from every path that turns an
  upstream failure into Error state.
- The recorder was cleared only on success, so a window from an earlier attempt
  could be re-attributed to a later, unrelated failure — re-parking a server the
  upstream had not rate-limited again (notably right after a manual reconnect
  reset the state machine). Each attempt now starts with a clean slate.
- ShouldRetryOAuth (the monitoring loop's reconnect path) ignored the window,
  and its rungs can be shorter than what the upstream asked for.
- Converting delta-seconds to a Duration before capping overflowed int64
  nanoseconds past ~292 years, turning a huge header into a tiny or negative
  delay instead of the cap. The bound check now precedes the multiplication.

Tests: OAuth streamable-HTTP and SSE clients now have wire-level coverage (a
pre-seeded token store makes the request actually go out, so the 429 is
observable) — verified to fail when either constructor loses the basic-client
option; plus per-attempt window scoping at the core level, the OAuth-ladder
gate, and the two overflow-shaped Retry-After inputs.

Refs #1040.

* fix(upstream): make the 429 sync reachable from the request paths and swap recorders per attempt

Round 2 of the cross-model review found the round-1 health-path fix was dead
code and the per-attempt clear was too blunt:

- performHealthCheck only reached the sync after isConnectionError, which
  matches transport failures ("connection refused", "deadline exceeded") and
  not HTTP statuses — so a ping answered 429 never handed its window to the
  state machine. The sync now runs on any failed ping, before classification;
  it does not change the health verdict, it only stops the automatic reconnect
  paths from returning early. The same sync now also runs on a failed
  tools/call, the other surface that can be rate-limited with no connect
  attempt in sight.

- Clearing one shared recorder at the start of an attempt let a request still
  in flight on a superseded transport write into the new attempt's slate (and
  discarded a hint recorded moments before the clear). The core client now
  holds a generation pointer and swaps in a fresh recorder per attempt, so each
  transport keeps the recorder it was built with.

Tests: the health-path test is verified to fail when the sync moves back behind
isConnectionError, plus tools/call coverage.

Refs #1040.
github-actions Bot pushed a commit that referenced this pull request Aug 25, 2026
…specs 101/102 tracked (#1041)

- release-qa-gate-playwright (T2): in_review -> done, pr #1030 (advisory
  Playwright web-UI sweep merged into the release gate).
- mcp-2026-upgrade: blocked -> in_progress. The mcp-go gate cleared with
  v1.0.0-beta.1 (2026-08-12); spec 058 revision in review as #1033. Note
  rewritten: the remaining blocker is the Spec 057 cross-spec conflict,
  not the dependency.
- tpa-db: todo -> in_review with spec: specs/101-tpa-db and pr #1028
  (spec stage only; all three child tasks stay todo).
- New epic schema-deferred (P1, spec 102, in_progress): deferred-schema
  serialization for the direct tools/list surface, spec merged in #1035
  (issue #971), plan in progress.

Reconnect-backoff work (#1014/#1039/#1040) is deliberately not added —
the roadmap never tracked it; git history does.

ROADMAP.md regenerated; gen-roadmap.py --check passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants