Repository navigation
Commit e577e43
committed
fix(daemon): re-issue the interrupted request against a replacement daemon
When the shared daemon dies under a live MCP frontend (#2796), the worker's
daemon exchange fails and the current failure path calls _Exit(EXIT_FAILURE)
— "a failed daemon transport cannot wake a thread blocked in stdio
portably". The agent then sees server EOF and every pooled connection to the
same daemon breaks at once. On Windows this happens whenever a client reaps
its spawned endpoint tree: DETACHED_PROCESS detaches the console, not the
process tree, so the shared daemon dies with the endpoint that spawned it.
Keep the reader untouched and recover on the worker instead: when
cbm_daemon_application_client_mcp_tagged reports a transport failure and the
caller supplied a replacement supplier (new daemon_lost callback on mcp_run),
swap in the replacement client, re-mint the request token (tokens are scoped
to the client that reserved them), and re-issue the very request whose
exchange was interrupted. Only transport errors retry: BUSY, UNAVAILABLE,
REJECTED, HANDLER_ERROR and CANCELLED are daemon answers that keep their
existing meaning, so a cancellation is still answered as a cancellation.
Bounded by FRONTEND_DAEMON_RECONNECT_MAX; without a callback or once the
budget is spent the old immediate-exit path stays. A replayed request may
already have executed in the dead daemon, so the interrupted call has
at-least-once semantics; frontend.h states this explicitly.
The replaced client stays owned by the frontend and is only close_begin'd at
swap time: the stdin reader and the maintenance monitor can still hold a
pointer read before the swap, so close_finish runs in the run tail after the
worker is joined and the monitor is stopped. All non-worker client reads are
snapshotted under the state mutex. A supplier that returns the current or an
already-retired client is rejected instead of being double-freed.
Cancellation survives the reconnect: the published token is invalidated while
the callback runs, a matching cancel is remembered in cancel_pending, and the
retry delivers it against the freshly minted token. The retry reserves and
publishes that token under one lock, and CANCEL_ERROR no longer tears down the
reader — the in-flight exchange observes the same failure and the worker owns
recovery. A maintenance stop that lands in the reconnect window now takes the
monitor's bounded grace instead of exiting immediately.
The EOF drain derives its deadline each iteration from the last completed
item, grants a bounded FRONTEND_EOF_RECONNECT_GRACE_MS grace per reconnect
attempt (tracked by an atomic attempt counter, so a sub-poll failure still
resets it), gives the retried exchange a fresh stuck window, and re-samples
the reconnect flag before stopping.
main() supplies the callback: it re-runs the connect-or-spawn leg plus session
context and returns the replacement. It drops its global handle before the run
so ownership follows the code, logs the reconnect through cbm_log_warn, and
reports failures on stderr only — main_report_client_bootstrap_failure() would
inject an unsolicited id:null JSON-RPC frame into the live session's stdout.
The POSIX spawn child prefers close_range/closefrom and keeps only
async-signal-safe calls, because the callback runs on a worker thread while
the reader and maintenance monitor are live. UI flags are not re-applied; the
daemon persists its UI configuration.
Verified on WSL/gcc 13.3: scripts/test.sh --suites daemon_frontend reports 19
passed, including a transport-failure reconnect, a two-replacement retry
(exercising the budget loop and repeated retirement), a CANCELLED-no-retry
case that fails when the retry gate is widened, a NULL supplier that fail-stops
with a marker proving the callback ran, and a supplier returning the dead
client that is rejected without a double close. A real daemon kill -9
mid-session is served by a re-spawned daemon with one daemon_lost_reconnect
warning; pre-fix the same scenario exits 1 and the request is lost.
Fixes #2796
Signed-off-by: Minervaowl7 <127482476+Minervaowl7@users.noreply.github.com>1 parent 10ab613 commit e577e43
5 files changed
Lines changed: 755 additions & 52 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
36 | 39 | | |
37 | 40 | | |
38 | 41 | | |
| |||
1302 | 1305 | | |
1303 | 1306 | | |
1304 | 1307 | | |
1305 | | - | |
1306 | | - | |
1307 | | - | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
1308 | 1316 | | |
1309 | | - | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
1310 | 1323 | | |
1311 | 1324 | | |
1312 | 1325 | | |
| |||
0 commit comments