Repository navigation
Commit 48ad26e
authored
fix(upstream): detect a dead stdio transport and respawn the process (RC4-STDIO-001) (#1489)
# Pull Request
## Description
Fixes **RC4-STDIO-001**, from the codex QA of v0.70.0-rc.4. The bug also
exists in rc.2. When a stdio upstream's child process dies, every call
fails with `transport error: transport closed`, but REST keeps reporting
the server connected, healthy and usable, and it is never respawned.
Only a manual restart recovered it.
**Cause.** After the child dies, mcp-go's stdio reader hits EOF and
every request returns `transport.ErrTransportClosed`.
`isConnectionError` in `internal/upstream/managed/client.go` had no
marker for that error, so:
- the health loop logged it as "timeout (high activity), ignoring";
- a failed `tools/call` took the not-a-connection-error branch.
`SetError` was therefore never called, and the backoff reconnect, which
only runs from `StateError`, never fired.
**Fix** (managed layer only):
- **`isDeadTransportError`:** matches `ErrTransportClosed`,
`io.ErrClosedPipe` and `os.ErrClosed` through the error chain. Its text
fallback counts only when the text sits inside a wrapped
`transport.Error`, so a healthy server's JSON-RPC error text can't
trigger it. `io.EOF`/`io.ErrUnexpectedEOF` are deliberately excluded,
because HTTP/SSE transports wrap request EOFs that way on live
connections.
- **Effect:** a dead transport is now a hard connection error and is not
treated as a transient health-check error. The first failed ping or call
moves the server to Error, and the existing `ShouldRetry` backoff and
`tryReconnect` respawn the process.
- **Generation guard:** both the call path (`recordDeadConnection`) and
the health path record the error only while the client is still Ready on
the same connection generation, serialized under `epochMu`. A ping or
call that races a deliberate `Disconnect` can't flip a reset client to
Error, and a burst of calls on one dead transport costs only one retry.
- **No new risks:** the reconnect path is unchanged, the managed
`Disconnect` is never called from the health goroutine (no self-join),
and backoff and the supervisor behave as before.
## Testing
- [x] I have tested these changes locally
- [x] I have added/updated tests that prove my fix is effective or my
feature works
- [x] All existing tests pass
- **Tests:**
- New: `dead_transport_test.go` covers the classification table
(including negatives for JSON-RPC text and HTTP EOF), health check flips
to Error, a call marks Error, a burst counts one retry, and a call after
Disconnect stays Disconnected.
- New: `dead_transport_fixture_test.go` uses a real `mcpfixture` child:
SIGKILL, assert it leaves Ready, assert it respawns to a new
`instance_id`.
- All of these failed before the fix.
- **Race suites:** `go test -race` passes for `./internal/upstream/...`
and `./internal/runtime/supervisor/...`. The fixture test passed with
`-count=3` and `-count=5`.
- **Live probe on a real core:** within 5s of SIGKILLing the stdio
child, REST shows `connected=false, unhealthy`. By 30s a new child was
spawned and the server was healthy again.
- **Review:** zcode round 1 raised three mediums (HTTP EOF, JSON-RPC
text, health-path epoch). All are fixed in `603614463`.
- **Known low follow-ups, not in this PR:**
- `runListToolsAsLeader`, `GetCachedToolCount` and `GetPrompt` still
call `SetError` on connection errors with no epoch guard.
- The epoch branch of the `recordDeadConnection` guard has no dedicated
test.
🤖 Generated with [Claude Code](https://claude.com/claude-code)1 parent a010eca commit 48ad26e
4 files changed
Lines changed: 485 additions & 11 deletions
File tree
- internal/upstream/managed
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
| 10 | + | |
9 | 11 | | |
10 | 12 | | |
11 | 13 | | |
| |||
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| 27 | + | |
25 | 28 | | |
26 | 29 | | |
27 | 30 | | |
| |||
1058 | 1061 | | |
1059 | 1062 | | |
1060 | 1063 | | |
| 1064 | + | |
1061 | 1065 | | |
1062 | 1066 | | |
1063 | 1067 | | |
| |||
1074 | 1078 | | |
1075 | 1079 | | |
1076 | 1080 | | |
1077 | | - | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
1078 | 1084 | | |
1079 | 1085 | | |
1080 | 1086 | | |
| |||
1138 | 1144 | | |
1139 | 1145 | | |
1140 | 1146 | | |
1141 | | - | |
1142 | 1147 | | |
1143 | 1148 | | |
1144 | 1149 | | |
| |||
1192 | 1197 | | |
1193 | 1198 | | |
1194 | 1199 | | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
1195 | 1204 | | |
1196 | 1205 | | |
1197 | 1206 | | |
| |||
1247 | 1256 | | |
1248 | 1257 | | |
1249 | 1258 | | |
1250 | | - | |
| 1259 | + | |
1251 | 1260 | | |
1252 | 1261 | | |
1253 | 1262 | | |
| |||
1262 | 1271 | | |
1263 | 1272 | | |
1264 | 1273 | | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
1265 | 1315 | | |
1266 | 1316 | | |
1267 | 1317 | | |
| |||
1698 | 1748 | | |
1699 | 1749 | | |
1700 | 1750 | | |
| 1751 | + | |
| 1752 | + | |
| 1753 | + | |
| 1754 | + | |
1701 | 1755 | | |
1702 | 1756 | | |
1703 | 1757 | | |
| |||
1747 | 1801 | | |
1748 | 1802 | | |
1749 | 1803 | | |
1750 | | - | |
1751 | | - | |
1752 | | - | |
1753 | | - | |
1754 | | - | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
1755 | 1814 | | |
1756 | 1815 | | |
1757 | 1816 | | |
| |||
1836 | 1895 | | |
1837 | 1896 | | |
1838 | 1897 | | |
| 1898 | + | |
| 1899 | + | |
| 1900 | + | |
| 1901 | + | |
| 1902 | + | |
| 1903 | + | |
1839 | 1904 | | |
1840 | 1905 | | |
1841 | 1906 | | |
| |||
2133 | 2198 | | |
2134 | 2199 | | |
2135 | 2200 | | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
| 2205 | + | |
| 2206 | + | |
| 2207 | + | |
| 2208 | + | |
| 2209 | + | |
| 2210 | + | |
| 2211 | + | |
| 2212 | + | |
| 2213 | + | |
| 2214 | + | |
| 2215 | + | |
| 2216 | + | |
| 2217 | + | |
| 2218 | + | |
| 2219 | + | |
| 2220 | + | |
| 2221 | + | |
| 2222 | + | |
| 2223 | + | |
| 2224 | + | |
| 2225 | + | |
| 2226 | + | |
| 2227 | + | |
| 2228 | + | |
| 2229 | + | |
| 2230 | + | |
| 2231 | + | |
| 2232 | + | |
| 2233 | + | |
| 2234 | + | |
| 2235 | + | |
| 2236 | + | |
| 2237 | + | |
| 2238 | + | |
| 2239 | + | |
| 2240 | + | |
| 2241 | + | |
| 2242 | + | |
| 2243 | + | |
| 2244 | + | |
| 2245 | + | |
| 2246 | + | |
| 2247 | + | |
| 2248 | + | |
| 2249 | + | |
| 2250 | + | |
| 2251 | + | |
| 2252 | + | |
| 2253 | + | |
| 2254 | + | |
| 2255 | + | |
| 2256 | + | |
| 2257 | + | |
| 2258 | + | |
| 2259 | + | |
2136 | 2260 | | |
2137 | 2261 | | |
2138 | 2262 | | |
2139 | 2263 | | |
2140 | 2264 | | |
| 2265 | + | |
| 2266 | + | |
| 2267 | + | |
2141 | 2268 | | |
2142 | 2269 | | |
2143 | 2270 | | |
| |||
2352 | 2479 | | |
2353 | 2480 | | |
2354 | 2481 | | |
| 2482 | + | |
2355 | 2483 | | |
2356 | 2484 | | |
2357 | 2485 | | |
| |||
2360 | 2488 | | |
2361 | 2489 | | |
2362 | 2490 | | |
2363 | | - | |
| 2491 | + | |
| 2492 | + | |
2364 | 2493 | | |
2365 | | - | |
| 2494 | + | |
2366 | 2495 | | |
2367 | 2496 | | |
2368 | 2497 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
0 commit comments