Repository navigation
fix(upstream): stop retrying deterministic connection failures forever - #1155
Merged
Merged
Conversation
#1145) A server whose connect failure can never succeed was retried 55 times over 19 hours: every attempt a full container spawn plus a failed package install, every stderr byte-identical, 359 log lines for one server. The suspected root cause -- the supervisor reconcile loop bypassing ShouldRetry -- is no longer true; #1013/#1040 gated it, and both that gate and Manager.ConnectAll now share one policy. The real cause is the tail of that policy in ConnectionInfo.ShouldAutoReconnect: after MaxConnectionRetries (20, ~1.2h of ladder) it falls back to GaveUpProbeInterval, a flat 30-minute probe that runs forever. 20 + 35 is exactly the 55 attempts in the report, and it also explains retry_count climbing past 26, since SetError bumps the counter on every probe. That probe exists to self-heal a long outage (laptop sleep, VPN, maintenance) and cannot tell one from a config error that will never fix itself. Permanence vocabulary. CatalogEntry gains a Retry class and the catalog gains IsPermanent(). Nine config/toolchain codes are declared permanent; everything that can clear on its own stays retryable -- daemon down, image pull, Docker permissions, a secret that may yet appear in the keyring, the generic "process died" bucket, and every OAuth/HTTP/network code, each with the reason recorded next to it. An unregistered or unclassified code defaults to retryable, so MCPX_UNKNOWN_UNCLASSIFIED keeps retrying; tests pin both the permanent set and that anti-set. Declaring a new code permanent is one line in its existing register(...) call. Terminal state. ConnectionInfo gains Terminal/TerminalCode (plain string, so the package stays a leaf) and StateManager gains SetTerminalError. Both retry gates stop once PermanentFailureAttempts (2) is spent: the second attempt is a confirmation, so a classifier false positive costs one retry rather than the server. managed.Client.Connect classifies its own failure and parks when the code is permanent -- 55 attempts become 2. Recovery, which is what makes parking safe. Plain SetError clears the park (an unclassified follow-up failure is not evidence of permanence), as do a successful connection, Reset (so Disconnect and ForceReconnect un-park), and client recreation. The supervisor's own escape is ActionReconnect, computed before the backoff gate -- but configChanged compared five fields and missed args, env, headers, working_dir, isolation and the timeouts. Now that a parked server has no other way back, a user fixing their args or isolation.image would have seen nothing happen, so it delegates to config.ConnectionEquivalent. Created/Updated are deliberately excluded: Updated churns on unrelated writes and would reconnect every server on every tick. One classifier, not two. internal/diagnostics/hints builds ClassifierHints for both the supervisor and the managed client, so they cannot disagree about whether a server was Docker-launched. That drift was only a wrong help message in #1142; it now decides retry behaviour. The unclassified tail. For failures we cannot prove permanent, the post-give-up probe escalates 30m, 1h, 2h, 4h, 6h instead of repeating at 30m. The first probe is unchanged, so #1013's self-healing still holds for any outage that recovers promptly. Nothing is swallowed. stateview, the REST /servers payload, the MCP upstream list and the health calculator all report retry_stopped with the stable code and the catalog reason, at unhealthy with a Restart CTA -- restart being exactly what un-parks a server once its config is fixed. Known limitation, worth stating plainly: the exact server in the report may still not be parked by classification. A failed package install inside a container most likely reads as MCPX_STDIO_EXIT_BEFORE_INITIALIZE or UNKNOWN, neither of which is permanent. The escalating probe covers it (35 probes over 19h down to ~7) until a code for the missing-toolchain case lands and flips one line.
Review of #1145 found the permanence signal was not trustworthy enough to be load-bearing, and that two side effects reached further than intended. Child stderr could park a server forever. Classify's stdio and docker fallbacks are deliberately broad substring matches, and the string they match is not ours: core.enrichTransportClosedError folds the CHILD PROCESS'S captured stderr into the error before it reaches the classifier. Marking STDIO_SPAWN_ENOENT/EACCES/EXEC_FORMAT and DOCKER_EXEC_NOT_FOUND permanent therefore parked any server that crashed transiently having printed "no such file or directory", "permission denied" or "command not found" — text it wrote about itself. TestConnect_ChildStderrNeverParks drives the real connect path with a child that prints exactly that and fails on the unfixed tree for all three phrasings. The fix separates the message from the decision rather than narrowing the classifier. diagnostics.ParkableCode returns Classify's code unchanged — the user still reads the most useful diagnosis we have — plus a boolean that is true only when the SAME code is reachable from a typed signal alone: a code a producer attached with WrapError, or a real *exec.Error errno from our own spawn syscall. Broad matching is right for a help message and wrong for a decision that stops automatic recovery. That costs coverage, deliberately. A non-Docker stdio command is spawned through a login shell so it inherits the user's PATH, so a genuinely missing binary reaches us as shell text and is no longer parked; it retries on the normal ladder as before #1145. To keep the feature alive on a signal we own, core.validateStdioConfig now attributes its pre-spawn verdict with WrapError(ConfigInvalidCommand) — no subprocess has run at that point, so nothing there can be a child's stderr. MCPX_HTTP_LEGACY_SSE is no longer permanent. mcp-go returns its ErrLegacySSEServer sentinel for ANY 4xx on the initialize POST except 401, so a 429 from a rate-limited upstream — or a 403/404/408 during a deploy — arrived as that code and parked the server after two attempts. The post-give-up probe goes back to a flat 30 minutes. The escalating 30m/1h/2h/4h/6h ladder applied to the whole give-up tail, including failures this same change classifies as transient, stretching unattended recovery from #1013's documented 30 minutes to as much as six hours with nothing in-tree to shorten it again. Parking is the mechanism for failures we can prove; the tail keeps its guarantee. Contract. CatalogEntry.retry is added to the published catalog schema, which sets additionalProperties:false — `doctor list-codes -o json` was emitting a document that failed its own contract. Nothing validated that contract anywhere, so TestCatalog_MatchesPublishedContract now does, and it also surfaced a pre-existing drift: the UPDATE domain was missing from the code pattern.
Deploying mcpproxy-docs with
|
| Latest commit: |
ee1cdc3
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://0a54a0e3.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://fix-1145-retry-permanent.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 33471406817 --repo smart-mcp-proxy/mcpproxy-go
|
Dumbris
added a commit
that referenced
this pull request
Sep 1, 2026
…anence gate main extracted classifier-hint building into internal/diagnostics/hints, which already covers the Docker enrichment this branch was adding inline — so the conflict resolves to main's hints.For call, with DockerArgs added there because the #1143 git-dependency remediation reads it. #1155's TestParkableCode_ChildStderrNeverProvesPermanence pinned the OLD classifications to isolate its permanence gate; this branch fixes those classifications, so the expected codes move. The gate's invariant is unchanged and still asserted: two stderr-only rows that still reach a PERMANENT code are added so it is never left untested.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1145.
The problem
A server whose failure is deterministic and can never self-heal was retried indefinitely: 55 failed attempts over ~19 hours,
retry_countpast 26, each one a full container spawn plus a failed package install, all producing byte-identical stderr. It burned CPU, disk and network on a guaranteed failure and floodedmain.log— 359 lines for a single server, drowning out signal for anything else.What this changes
Once a failure is proven permanent, the server is parked with a visible reason instead of looping forever. Transient failures keep the existing backoff untouched.
The part that matters most: permanence has to be provable
The first implementation used
diagnostics.Classifyas the permanence signal, and adversarial review caught that this weaponises the classifier's deliberately-broad substring fallback — which matches against the child process's captured stderr. A transient crash whose stderr merely contained "permission denied" or "no such file or directory" would have been parked forever. That is a far worse failure than the bug being fixed.So permanence now requires two independent conditions, via
ParkableCode:Retry: RetryPermanentin its existingregister(...)call, and*exec.Error, an explicitlyWrapError-attached code — never from a stderr substring match.Verified against the real code:
Note rows 1 and 3: the code is one declared permanent, and the gate still refuses to park, because the evidence was only text. That is the property under test.
MCPX_UNKNOWN_UNCLASSIFIEDstays retryable by construction — failing to recognise an error is not evidence that retrying it is futile.Recovery is not a trap
A parked server becomes retryable again when its config changes or the user restarts/enables it, so someone who fixes the problem sees recovery rather than a permanently dead entry. Covered by test.
Also fixed during review
MCPX_HTTP_LEGACY_SSEwas initially declared permanent, but mcp-go returns that sentinel for any 4xx on the initialize POST except 401 — a 429 (the textbook transient) would have parked a server forever.specs/044-diagnostics-taxonomy/contracts/catalog-schema.jsonsetsadditionalProperties:false, so the newretryfield onCatalogEntryneeded a deliberate contract update — otherwisemcpproxy doctor list-codes -o jsonwould emit a document failing its own schema.Known nuance
MCPX_DOCKER_EXEC_NOT_FOUNDandMCPX_DOCKER_OCI_RUNTIMEare declared permanent but are currently unreachable through the evidence gate (no producer attributes them viaWrapError, and neither is derivable from an*exec.Error). The declarations are inert rather than wrong — they become live the moment a producer attaches typed evidence. Flagging it so the next reader doesn't assume they're active.Verification
go build ./...,-tags server,go test ./internal/... -race, golangci-lint v2 → 0 issues.Two fix→re-review rounds under three adversarial lenses. Final verdict: clean.