Skip to content

HTTP upstreams stay in "Error" for up to ~30 min after macOS wake; transient network failures shown as hard errors #1929

Open

Description

@algis-dumbris

Summary

After a laptop sleeps overnight, remote HTTP upstreams (e.g. https://mcp.atlassian.com/v2/mcp, https://api.githubcopilot.com/mcp/) end up in Error state and stay there after the machine wakes and the network is back. The retry backoff has grown to ~30 min, nothing triggers an earlier retry, and the UI shows a red, misleading error. A manual Restart connects instantly (OAuth token refresh works fine), so the server was reachable the whole time — mcpproxy just didn't try.

Environment

  • MCPProxy v0.73.0 (personal), darwin/arm64, tray app (mcpproxy serve running for ~7 days)
  • macOS 26 (Darwin 25.5), on AC power, normal sleep with periodic DarkWake / Maintenance Sleep
  • Upstreams affected: 2 × protocol: http (one OAuth/DCR, one static Authorization header). Stdio/docker upstreams were unaffected.

Timeline (from server-<name>.log, main.log, pmset -g log; local time UTC+2)

Time Event
Day 1 09:18 Upstream connects OK (Successfully connected and initialized)
Day 1 ~17:00 → Day 2 04:00 Machine sleeping / dark-waking. Errors: dial tcp: lookup mcp.atlassian.com: no such host, ... i/o timeout, Post ...: context deadline exceeded
Day 2 04:00 → 07:23 Retries continue, but spaced out: 04:58, 05:04, 05:27, 05:33, 06:04, 06:51, 07:22 (backoff ≈ 30 min). retry_count reaches 23 (38 for the second upstream)
Day 2 07:23 Last automatic attempt — fails (machine still in DarkWake)
Day 2 07:28 pmset: DarkWake to FullWake ... due to UserActivity — user is back, network is up
Day 2 07:34 curl -X POST to both MCP URLs from the same host: HTTP 401 in ~0.3 s (i.e. reachable)
Day 2 07:35 Manual Restart → token refreshed from persistent store → MCP initialization completed successfully in < 1 s

Without the manual restart, the next automatic attempt would have been ~07:53, i.e. ~25 min of the server being unusable after wake while the UI showed it as broken.

Problems

  1. No reconnect trigger on wake / network change. Recovery depends solely on the backoff timer, which is at its maximum exactly when the user returns.
  2. Retries during sleep inflate the backoff. DarkWake windows have no/flaky network, so every attempt fails and pushes the backoff higher.
  3. Transient network errors are presented as hard failures. The tray shows red failed to connect: MCP initialize failed during no-auth strategy: server did not respond to MCP initialize within 30s and produced no stderr output (check that the command starts an MCP server and not a help banner). For an HTTP upstream the "stderr"/"help banner" hint doesn't apply, and the real cause (DNS failure / timeout) is hidden in the log.
  4. Tool calls don't trigger a reconnect by default. reconnect_on_use (Add optional on-demand reconnect when a tool is invoked for a disconnected upstream #354) exists but is opt-in per server, so a typical remote HTTP upstream fails fast instead of attempting a just-in-time connection.

Related: #1013 / #1014 / #1039 added the backoff on purpose (to stop hammering dead remotes) — this issue is not asking to remove it, only to reset it when the local network situation demonstrably changes.

Proposed behavior

In rough order of impact:

  1. Reset backoff and retry immediately on system wake and on network-path change (macOS: NSWorkspaceDidWakeNotification / NWPathMonitor; Linux: NetworkManager/systemd-logind signals; Windows: power/network events). This alone would have recovered both upstreams within seconds of 07:28. It's compatible with [Bug]: Upstream reconnect has no backoff — supervisor re-dials failed servers every 30s forever, hammering dead remotes (~9k req/day) #1013: one attempt per wake/network event, then the normal backoff resumes; failures caused by local network unavailability arguably shouldn't advance the backoff at all.
  2. Default reconnect_on_use to on for HTTP/SSE upstreams (or globally), and make it also apply to retrieve_tools, so a client using an Error-state upstream gets one bounded connection attempt before an error.
  3. Classify errors: DNS failure, connect/dial timeout, connection refused, context deadline exceeded on HTTP → transient → show as "Reconnecting…" (neutral/amber), not red Error. Keep red for auth failures, 4xx config problems, or persistent failure while other upstreams/hosts are reachable.
  4. Pause the retry loop while offline (no default route / path unsatisfied) instead of burning attempts and growing backoff.
  5. Lower / configurable max backoff for HTTP upstreams (e.g. max_reconnect_backoff, default ~5 min). Stdio servers may warrant a different policy.
  6. Transport-specific error text: for HTTP, e.g. mcp.atlassian.com unreachable (DNS lookup timed out); will retry, without the stdio-only hint.

Workaround

Manual Restart from tray / upstream_servers restart, or an external wake hook (sleepwatcher / launchd) that restarts Error-state upstreams.

Log excerpts

ERROR core/connection_http.go:227 MCP initialize JSON-RPC call failed {"server": "<atlassian>", "error": "transport error: failed to send request: failed to send request: Post \"https://mcp.atlassian.com/v2/mcp?tools=all\": dial tcp: lookup mcp.atlassian.com: i/o timeout"}
ERROR core/connection_http.go:227 MCP initialize JSON-RPC call failed {"server": "<atlassian>", "error": "transport error: ... Post \"https://mcp.atlassian.com/v2/mcp?tools=all\": context deadline exceeded"}
ERROR core/connection.go:345 Connection failed {"server": "<atlassian>", "transport": "http", "error": "MCP initialize failed during no-auth strategy: server did not respond to MCP initialize within 30s and produced no stderr output (check that the command starts an MCP server and not a help banner)"}
ERROR upstream/manager_generation.go:392 Connection error {"component": "managed_client", "server": "<atlassian>", "error": "failed to connect: ...", "retry_count": 23}

# after manual restart at 07:35
INFO oauth ✅ Found existing token in persistent storage {"expired": true}
INFO core/connection_oauth.go:476 MCP initialization completed successfully {"server_name": "atlassian-mcp-server", "protocol_version": "2025-11-25"}
INFO managed/client.go:623 Successfully connected and initialized

Error counts in the affected upstream's log over the period: ~324 transport errors (deadline exceeded / DNS i/o timeout / no such host), 0 non-network errors after the initial OAuth setup.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions