Repository navigation
Commit a13b4a3
committed
fix(daemon): re-issue the interrupted request against a replacement daemon
When the shared daemon dies under a live MCP frontend (#2796), the worker's
daemon exchange fails and the current failure path calls _Exit(EXIT_FAILURE)
— "a failed daemon transport cannot wake a thread blocked in stdio
portably". The agent then sees server EOF and every pooled connection to the
same daemon breaks at once. On Windows this happens whenever a client reaps
its spawned endpoint tree: DETACHED_PROCESS detaches the console, not the
process tree, so the shared daemon dies with the endpoint that spawned it.
Keep the reader untouched and recover on the worker instead: when
cbm_daemon_application_client_mcp_tagged reports a transport failure and the
caller supplied a replacement supplier (new daemon_lost callback on mcp_run),
swap in the replacement client, re-mint the request token (tokens are scoped
to the client that reserved them), and re-issue the very request whose
exchange was interrupted. Only transport errors retry: BUSY, UNAVAILABLE,
REJECTED, HANDLER_ERROR and CANCELLED are daemon answers that keep their
existing meaning, so a cancellation is still answered as a cancellation.
Bounded by FRONTEND_DAEMON_RECONNECT_MAX; without a callback or once the
budget is spent the old immediate-exit path stays. A replayed request may
already have executed in the dead daemon, so the interrupted call has
at-least-once semantics; frontend.h states this explicitly.
The replaced client stays owned by the frontend and is only close_begin'd at
swap time: the stdin reader and the maintenance monitor can still hold a
pointer read before the swap, so close_finish runs in the run tail after the
worker is joined and the monitor is stopped. All non-worker client reads are
snapshotted under the state mutex. A supplier that returns the current or an
already-retired client is rejected instead of being double-freed.
Cancellation survives the reconnect: the published token is invalidated while
the callback runs, a matching cancel is remembered in cancel_pending, and the
retry delivers it against the freshly minted token. The retry reserves and
publishes that token under one lock, and CANCEL_ERROR no longer tears down the
reader — the in-flight exchange observes the same failure and the worker owns
recovery. A maintenance stop that lands in the reconnect window now takes the
monitor's bounded grace instead of exiting immediately.
The EOF drain derives its deadline each iteration from the last completed
item, grants a bounded 120 s grace per reconnect attempt (covering the default
connect/startup/upgrade-drain worst case and tracked by an atomic attempt
counter, so a sub-poll failure still resets it), gives the retried exchange a
fresh stuck window, and re-samples the reconnect flag before stopping.
main() supplies the callback: it re-runs the connect-or-spawn leg plus session
context and returns the replacement. It returns NULL immediately once
g_shutdown is set (SIGTERM/SIGINT), so shutdown never cold-starts a daemon. It
drops its global handle before the run so ownership follows the code, logs the
reconnect through cbm_log_warn, and reports failures on stderr only —
main_report_client_bootstrap_failure() would inject an unsolicited id:null
JSON-RPC frame into the live session's stdout; an endpoint failure includes
the validation detail. The POSIX spawn child prefers close_range/closefrom
(with the 436 syscall fallback for older headers) and only then a getrlimit
sweep, keeping the fork child free of libc allocation because the callback
runs on a worker thread. UI flags are not re-applied; the daemon persists its
UI configuration.
Verified on WSL/gcc 13.3: scripts/test.sh --suites daemon_frontend reports 19
passed, including a transport-failure reconnect, a two-replacement retry
(exercising the budget loop and repeated retirement), a CANCELLED-no-retry
case that fails when the retry gate is widened, a NULL supplier that fail-stops
with a marker proving the callback ran, and a supplier returning the dead
client that is rejected without a double close. A real daemon kill -9
mid-session is served by a re-spawned daemon with one daemon_lost_reconnect
warning; pre-fix the same scenario exits 1 and the request is lost.
Fixes #2796
Signed-off-by: Minervaowl7 <127482476+Minervaowl7@users.noreply.github.com>1 parent 10ab613 commit a13b4a3
5 files changed
Lines changed: 777 additions & 52 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
36 | 45 | | |
37 | 46 | | |
38 | 47 | | |
| |||
1302 | 1311 | | |
1303 | 1312 | | |
1304 | 1313 | | |
1305 | | - | |
1306 | | - | |
1307 | | - | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
1308 | 1322 | | |
1309 | | - | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
1310 | 1336 | | |
1311 | 1337 | | |
1312 | 1338 | | |
| |||
0 commit comments