Skip to content

fix(upstream): stop retrying deterministic connection failures forever - #1155

Merged
Dumbris merged 3 commits into
mainfrom
fix/1145-retry-permanent
Sep 1, 2026
Merged

Dumbris merged 3 commits into
mainfrom
fix/1145-retry-permanent

Conversation

@Dumbris

@Dumbris Dumbris commented Sep 1, 2026

Copy link
Copy Markdown
Member

Fixes #1145.

The problem

A server whose failure is deterministic and can never self-heal was retried indefinitely: 55 failed attempts over ~19 hours, retry_count past 26, each one a full container spawn plus a failed package install, all producing byte-identical stderr. It burned CPU, disk and network on a guaranteed failure and flooded main.log — 359 lines for a single server, drowning out signal for anything else.

What this changes

Once a failure is proven permanent, the server is parked with a visible reason instead of looping forever. Transient failures keep the existing backoff untouched.

The part that matters most: permanence has to be provable

The first implementation used diagnostics.Classify as the permanence signal, and adversarial review caught that this weaponises the classifier's deliberately-broad substring fallback — which matches against the child process's captured stderr. A transient crash whose stderr merely contained "permission denied" or "no such file or directory" would have been parked forever. That is a far worse failure than the bug being fixed.

So permanence now requires two independent conditions, via ParkableCode:

  1. the code is declared Retry: RetryPermanent in its existing register(...) call, and
  2. the classification is derived from typed/structured evidence — a real syscall errno, an *exec.Error, an explicitly WrapError-attached code — never from a stderr substring match.

Verified against the real code:

parkable=false  code=MCPX_DOCKER_EXEC_NOT_FOUND   :: stderr "warn: /etc/foo: no such file or directory"
parkable=false  code=MCPX_UNKNOWN_UNCLASSIFIED    :: stderr "error: permission denied while reading cache"
parkable=false  code=MCPX_DOCKER_EXEC_NOT_FOUND   :: stderr "sh: 1: helper: command not found"
parkable=false  code=MCPX_UNKNOWN_UNCLASSIFIED    :: handshake timeout

Note rows 1 and 3: the code is one declared permanent, and the gate still refuses to park, because the evidence was only text. That is the property under test.

MCPX_UNKNOWN_UNCLASSIFIED stays retryable by construction — failing to recognise an error is not evidence that retrying it is futile.

Recovery is not a trap

A parked server becomes retryable again when its config changes or the user restarts/enables it, so someone who fixes the problem sees recovery rather than a permanently dead entry. Covered by test.

Also fixed during review

  • MCPX_HTTP_LEGACY_SSE was initially declared permanent, but mcp-go returns that sentinel for any 4xx on the initialize POST except 401 — a 429 (the textbook transient) would have parked a server forever.
  • The post-give-up probe backoff was initially escalated to a 6-hour ceiling for all failures, weakening the [Bug]: Upstream reconnect has no backoff — supervisor re-dials failed servers every 30s forever, hammering dead remotes (~9k req/day) #1013 self-heal guarantee from 30 minutes for genuinely transient outages. Reverted for the transient path.
  • specs/044-diagnostics-taxonomy/contracts/catalog-schema.json sets additionalProperties:false, so the new retry field on CatalogEntry needed a deliberate contract update — otherwise mcpproxy doctor list-codes -o json would emit a document failing its own schema.

Known nuance

MCPX_DOCKER_EXEC_NOT_FOUND and MCPX_DOCKER_OCI_RUNTIME are declared permanent but are currently unreachable through the evidence gate (no producer attributes them via WrapError, and neither is derivable from an *exec.Error). The declarations are inert rather than wrong — they become live the moment a producer attaches typed evidence. Flagging it so the next reader doesn't assume they're active.

Verification

go build ./..., -tags server, go test ./internal/... -race, golangci-lint v2 → 0 issues.

Two fix→re-review rounds under three adversarial lenses. Final verdict: clean.

#1145)

A server whose connect failure can never succeed was retried 55 times over
19 hours: every attempt a full container spawn plus a failed package
install, every stderr byte-identical, 359 log lines for one server.

The suspected root cause -- the supervisor reconcile loop bypassing
ShouldRetry -- is no longer true; #1013/#1040 gated it, and both that gate
and Manager.ConnectAll now share one policy. The real cause is the tail of
that policy in ConnectionInfo.ShouldAutoReconnect: after
MaxConnectionRetries (20, ~1.2h of ladder) it falls back to
GaveUpProbeInterval, a flat 30-minute probe that runs forever. 20 + 35 is
exactly the 55 attempts in the report, and it also explains retry_count
climbing past 26, since SetError bumps the counter on every probe. That
probe exists to self-heal a long outage (laptop sleep, VPN, maintenance)
and cannot tell one from a config error that will never fix itself.

Permanence vocabulary. CatalogEntry gains a Retry class and the catalog
gains IsPermanent(). Nine config/toolchain codes are declared permanent;
everything that can clear on its own stays retryable -- daemon down, image
pull, Docker permissions, a secret that may yet appear in the keyring, the
generic "process died" bucket, and every OAuth/HTTP/network code, each
with the reason recorded next to it. An unregistered or unclassified code
defaults to retryable, so MCPX_UNKNOWN_UNCLASSIFIED keeps retrying;
tests pin both the permanent set and that anti-set. Declaring a new code
permanent is one line in its existing register(...) call.

Terminal state. ConnectionInfo gains Terminal/TerminalCode (plain string,
so the package stays a leaf) and StateManager gains SetTerminalError.
Both retry gates stop once PermanentFailureAttempts (2) is spent: the
second attempt is a confirmation, so a classifier false positive costs one
retry rather than the server. managed.Client.Connect classifies its own
failure and parks when the code is permanent -- 55 attempts become 2.

Recovery, which is what makes parking safe. Plain SetError clears the
park (an unclassified follow-up failure is not evidence of permanence), as
do a successful connection, Reset (so Disconnect and ForceReconnect
un-park), and client recreation. The supervisor's own escape is
ActionReconnect, computed before the backoff gate -- but configChanged
compared five fields and missed args, env, headers, working_dir,
isolation and the timeouts. Now that a parked server has no other way
back, a user fixing their args or isolation.image would have seen nothing
happen, so it delegates to config.ConnectionEquivalent. Created/Updated
are deliberately excluded: Updated churns on unrelated writes and would
reconnect every server on every tick.

One classifier, not two. internal/diagnostics/hints builds ClassifierHints
for both the supervisor and the managed client, so they cannot disagree
about whether a server was Docker-launched. That drift was only a wrong
help message in #1142; it now decides retry behaviour.

The unclassified tail. For failures we cannot prove permanent, the
post-give-up probe escalates 30m, 1h, 2h, 4h, 6h instead of repeating at
30m. The first probe is unchanged, so #1013's self-healing still holds for
any outage that recovers promptly.

Nothing is swallowed. stateview, the REST /servers payload, the MCP
upstream list and the health calculator all report retry_stopped with the
stable code and the catalog reason, at unhealthy with a Restart CTA --
restart being exactly what un-parks a server once its config is fixed.

Known limitation, worth stating plainly: the exact server in the report
may still not be parked by classification. A failed package install inside
a container most likely reads as MCPX_STDIO_EXIT_BEFORE_INITIALIZE or
UNKNOWN, neither of which is permanent. The escalating probe covers it
(35 probes over 19h down to ~7) until a code for the missing-toolchain
case lands and flips one line.
Review of #1145 found the permanence signal was not trustworthy enough to
be load-bearing, and that two side effects reached further than intended.

Child stderr could park a server forever. Classify's stdio and docker
fallbacks are deliberately broad substring matches, and the string they
match is not ours: core.enrichTransportClosedError folds the CHILD
PROCESS'S captured stderr into the error before it reaches the
classifier. Marking STDIO_SPAWN_ENOENT/EACCES/EXEC_FORMAT and
DOCKER_EXEC_NOT_FOUND permanent therefore parked any server that crashed
transiently having printed "no such file or directory", "permission
denied" or "command not found" — text it wrote about itself.
TestConnect_ChildStderrNeverParks drives the real connect path with a
child that prints exactly that and fails on the unfixed tree for all
three phrasings.

The fix separates the message from the decision rather than narrowing
the classifier. diagnostics.ParkableCode returns Classify's code
unchanged — the user still reads the most useful diagnosis we have — plus
a boolean that is true only when the SAME code is reachable from a typed
signal alone: a code a producer attached with WrapError, or a real
*exec.Error errno from our own spawn syscall. Broad matching is right for
a help message and wrong for a decision that stops automatic recovery.

That costs coverage, deliberately. A non-Docker stdio command is spawned
through a login shell so it inherits the user's PATH, so a genuinely
missing binary reaches us as shell text and is no longer parked; it
retries on the normal ladder as before #1145. To keep the feature alive
on a signal we own, core.validateStdioConfig now attributes its
pre-spawn verdict with WrapError(ConfigInvalidCommand) — no subprocess
has run at that point, so nothing there can be a child's stderr.

MCPX_HTTP_LEGACY_SSE is no longer permanent. mcp-go returns its
ErrLegacySSEServer sentinel for ANY 4xx on the initialize POST except
401, so a 429 from a rate-limited upstream — or a 403/404/408 during a
deploy — arrived as that code and parked the server after two attempts.

The post-give-up probe goes back to a flat 30 minutes. The escalating
30m/1h/2h/4h/6h ladder applied to the whole give-up tail, including
failures this same change classifies as transient, stretching unattended
recovery from #1013's documented 30 minutes to as much as six hours with
nothing in-tree to shorten it again. Parking is the mechanism for
failures we can prove; the tail keeps its guarantee.

Contract. CatalogEntry.retry is added to the published catalog schema,
which sets additionalProperties:false — `doctor list-codes -o json` was
emitting a document that failed its own contract. Nothing validated that
contract anywhere, so TestCatalog_MatchesPublishedContract now does, and
it also surfaced a pre-existing drift: the UPDATE domain was missing from
the code pattern.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: ee1cdc3
Status: ✅  Deploy successful!
Preview URL: https://0a54a0e3.mcpproxy-docs.pages.dev
Branch Preview URL: https://fix-1145-retry-permanent.mcpproxy-docs.pages.dev

View logs

@codecov-commenter

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: fix/1145-retry-permanent

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (26 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (21 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 33471406817 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@Dumbris
Dumbris merged commit e8d2ba3 into main Sep 1, 2026
40 checks passed
@Dumbris
Dumbris deleted the fix/1145-retry-permanent branch September 1, 2026 06:49
Dumbris added a commit that referenced this pull request Sep 1, 2026
…anence gate

main extracted classifier-hint building into internal/diagnostics/hints,
which already covers the Docker enrichment this branch was adding inline —
so the conflict resolves to main's hints.For call, with DockerArgs added
there because the #1143 git-dependency remediation reads it.

#1155's TestParkableCode_ChildStderrNeverProvesPermanence pinned the OLD
classifications to isolate its permanence gate; this branch fixes those
classifications, so the expected codes move. The gate's invariant is
unchanged and still asserted: two stderr-only rows that still reach a
PERMANENT code are added so it is never left untested.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deterministic, unrecoverable connection failures retry forever (55 attempts / 19h on one server)

2 participants