Skip to content

Commit 5e23839

Browse files
committed
fix(daemon): make request completion transactional and EOF drain-safe
Back-to-back requests on one frontend<->daemon connection failed roughly half the time on loaded Windows hosts, surfacing as wandering MCP session drops (smoke Phase 5/5e/5f "no tools/list response", "timed out waiting for id=2"). Root cause, confirmed on real Windows: the application worker stored its done flag AFTER writing the response, so a client that pipelines its next request the instant it sees a response raced the previous worker's final instructions and its join, and admission rejected the new request with BUSY. On EOF that BUSY was then classified as an expected shutdown and the response was silently dropped with a success exit; mid-session it exited non-zero. A timing-sensitive heisenbug — any added stderr shifted the window and it passed. - runtime_application_worker publishes the done flag BEFORE sending the response, so completion is one atomic transition from the client's view: by the time any client can react, the slot already reads reusable. The admission side joins the thread after observing the flag, which absorbs the send tail. No timing constants. - frontend_should_stop's input-closed leg now also requires no request in flight, so a daemon-side failure of the final post-EOF item is loud, never swallowed as an expected stop. - The EOF drain is now progress-aware: it resets its window on each completed item, bounding stuckness rather than total batch length, so a slow cold daemon spawn no longer abandons queued tail responses. - Latch the never-connected shutdown window on a monotonic admitted-total counter instead of sampling the live client count, which can read zero between two short-lived sessions and stop a daemon mid-conversation. - Add structured lifecycle diagnostics (daemon.runtime_stopping reason=, lifetime_end reason=, connection_end) so this class is observable in the logs instead of being inferred. Validated: real-Windows loop 12/12 (was ~50% loss), macOS daemon suites 143/143. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent aa90491 commit 5e23839

4 files changed

Lines changed: 108 additions & 17 deletions

File tree

‎src/daemon/frontend.c‎

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,18 @@ enum {
2727
* close stdin before the worker is scheduled. Give already-accepted input
2828
* a short FIFO drain window so EOF cannot silently discard it. A genuinely
2929
* active long operation is still cancelled at this deadline. */
30-
FRONTEND_EOF_DRAIN_MS = 1000,
3130
FRONTEND_WAIT_US = 1000,
3231
/* An idle thin frontend owns no work and only needs to notice the next
3332
* queue item promptly. Ten milliseconds avoids a 1 kHz wake-up loop per
3433
* connected coding-agent session without perceptible request latency. */
3534
FRONTEND_IDLE_WAIT_US = 10 * 1000,
35+
/* EOF-drain progress window. Sized to cover a cold daemon spawn plus one
36+
* request round-trip on the slowest platform (several seconds on Windows
37+
* under CI load): the drain aborts only after this long with NO item
38+
* completing, so batch clients that close stdin after writing keep
39+
* receiving every response while a genuinely stuck request still gets
40+
* cancelled and the frontend exits cleanly. */
41+
FRONTEND_EOF_DRAIN_MS = 15000,
3642
FRONTEND_MAINTENANCE_POLL_MS = 10,
3743
/* The owner thread may be draining a supervised process tree. Preserve the
3844
* supervisor's complete graceful + forced-settle window before the monitor
@@ -71,6 +77,10 @@ typedef struct {
7177
const char *active_id_str;
7278
cbm_daemon_runtime_application_token_t active_request_token;
7379
bool failed;
80+
/* Monotonic count of fully processed queue items (responses written or
81+
* cancellations acknowledged). The EOF drain below watches it to tell a
82+
* batch that is still making progress from a genuinely stuck request. */
83+
atomic_uint_fast64_t completed_items;
7484
} frontend_state_t;
7585

7686
typedef struct {
@@ -190,7 +200,15 @@ static void frontend_item_free(frontend_item_t *item) {
190200

191201
static bool frontend_should_stop(frontend_state_t *state) {
192202
cbm_mutex_lock(&state->mutex);
193-
bool stopping = state->stopping || (state->input_closed && state->count == 0);
203+
/* The input-closed leg must also require that no item is IN FLIGHT: the
204+
* worker consults this while deciding whether a request failure was an
205+
* expected shutdown, and the final queued item after EOF has count == 0
206+
* precisely while its own forward is outstanding. Without the in_request
207+
* term, any daemon-side failure of that last item was classified as an
208+
* expected stop and its response silently dropped with a success exit —
209+
* an unanswerable failure mode for the client. */
210+
bool stopping =
211+
state->stopping || (state->input_closed && state->count == 0 && !state->in_request);
194212
cbm_mutex_unlock(&state->mutex);
195213
return stopping;
196214
}
@@ -438,6 +456,7 @@ static void *frontend_worker(void *opaque) {
438456
bool expected_stop = failed && frontend_should_stop(state);
439457
frontend_end_request(state, failed && !expected_stop);
440458
frontend_item_free(&item);
459+
atomic_fetch_add_explicit(&state->completed_items, 1, memory_order_release);
441460
if (failed) {
442461
if (!expected_stop) {
443462
/* A failed daemon transport cannot wake a thread blocked in
@@ -540,6 +559,7 @@ int cbm_daemon_frontend_mcp_run(cbm_daemon_runtime_client_t *client,
540559
.out = out,
541560
};
542561
cbm_mutex_init(&state.mutex);
562+
atomic_init(&state.completed_items, 0);
543563
cbm_daemon_maintenance_monitor_t *maintenance_monitor = cbm_daemon_maintenance_monitor_start(
544564
cohort_manager, frontend_cancel_for_maintenance, &state, EXIT_SUCCESS, "MCP frontend");
545565
if (!maintenance_monitor) {
@@ -598,11 +618,28 @@ int cbm_daemon_frontend_mcp_run(cbm_daemon_runtime_client_t *client,
598618

599619
if (clean_eof && !close_begun) {
600620
frontend_input_closed(&state);
601-
uint64_t now = cbm_now_ms();
602-
uint64_t drain_deadline =
603-
now > UINT64_MAX - FRONTEND_EOF_DRAIN_MS ? UINT64_MAX : now + FRONTEND_EOF_DRAIN_MS;
621+
/* EOF ends INPUT, not accepted work: as long as queued items keep
622+
* completing, every already-enqueued request still receives its
623+
* response, exactly like the pre-daemon stdio server, which
624+
* processed the whole stream before exiting. The window resets on
625+
* each completed item, so it bounds STUCKNESS, not batch length: a
626+
* request making no progress for the full window is cancelled by the
627+
* stop path below and the frontend still exits cleanly. The previous
628+
* fixed 1s total window abandoned queued requests whenever the
629+
* daemon cold spawn plus the first request outlasted it (several
630+
* seconds on Windows), so batch clients that close stdin right after
631+
* writing — file redirects, piped harnesses, hook one-shots —
632+
* intermittently lost their tail responses. */
633+
uint64_t drained = atomic_load_explicit(&state.completed_items, memory_order_acquire);
634+
uint64_t drain_deadline = cbm_now_ms() + FRONTEND_EOF_DRAIN_MS;
604635
while (!frontend_worker_is_done(&state) && cbm_now_ms() < drain_deadline) {
605636
cbm_usleep(FRONTEND_WAIT_US);
637+
uint64_t now_drained =
638+
atomic_load_explicit(&state.completed_items, memory_order_acquire);
639+
if (now_drained != drained) {
640+
drained = now_drained;
641+
drain_deadline = cbm_now_ms() + FRONTEND_EOF_DRAIN_MS;
642+
}
606643
}
607644
}
608645
if (!close_begun) {

‎src/daemon/host.c‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -679,11 +679,11 @@ static bool host_background_start(host_state_t *host) {
679679
static bool host_wait_for_lifetime(cbm_daemon_runtime_service_t *service,
680680
atomic_int *stop_requested, host_state_t *host) {
681681
uint64_t initial_deadline = cbm_now_ms() + HOST_INITIAL_CLIENT_TIMEOUT_MS;
682-
bool admitted = false;
683682
uint64_t stopping_deadline = 0;
684683
for (;;) {
685684
cbm_daemon_runtime_service_state_t state = cbm_daemon_runtime_service_state(service);
686685
if (state == CBM_DAEMON_RUNTIME_SERVICE_EXITED) {
686+
cbm_log_info("daemon.lifetime_end", "reason", "runtime_exited");
687687
return true;
688688
}
689689
if (state == CBM_DAEMON_RUNTIME_SERVICE_STOPPING) {
@@ -696,11 +696,16 @@ static bool host_wait_for_lifetime(cbm_daemon_runtime_service_t *service,
696696
(void)cbm_daemon_runtime_service_wait_exited(service, HOST_WAIT_TICK_MS);
697697
continue;
698698
}
699-
if (cbm_daemon_runtime_service_active_clients(service) > 0) {
700-
admitted = true;
701-
}
699+
/* The nobody-ever-connected window latches on the monotonic admission
700+
* total, not the live client count: a short first session can begin
701+
* and end entirely between two polls of this loop, and a sampled
702+
* count then reads zero forever — stopping a daemon whose client is
703+
* mid-conversation between per-request connections. */
704+
bool admitted = cbm_daemon_runtime_service_clients_admitted_total(service) > 0;
702705
bool stop = stop_requested && atomic_load(stop_requested);
703706
if (stop || (!admitted && cbm_now_ms() >= initial_deadline)) {
707+
cbm_log_info("daemon.lifetime_end", "reason",
708+
stop ? "stop_requested" : "initial_window_expired");
704709
return cbm_daemon_runtime_service_stop(service, HOST_RUNTIME_SHUTDOWN_MS);
705710
}
706711
host_http_reconcile_at(host, cbm_now_ms(), false);

‎src/daemon/runtime.c‎

Lines changed: 53 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,12 @@ struct cbm_daemon_runtime_service {
142142
size_t worker_mutexes_initialized;
143143
size_t active_connections;
144144
size_t committed_clients;
145+
/* Monotonic count of every admission since service start. The host's
146+
* nobody-ever-connected window latches on this instead of sampling the
147+
* live client count: a short-lived first session can begin and end
148+
* entirely between two host polls, and a sampled count then reads zero
149+
* forever, stopping a daemon whose client is mid-conversation. */
150+
uint64_t admitted_total;
145151

146152
cbm_thread_t accept_thread;
147153
bool accept_thread_started;
@@ -1019,10 +1025,16 @@ static void runtime_result_rejected(cbm_daemon_runtime_connect_result_t *result,
10191025
}
10201026

10211027
static void runtime_service_begin_stopping_locked(cbm_daemon_runtime_service_t *service,
1022-
uint64_t deadline, bool emergency) {
1028+
uint64_t deadline, bool emergency,
1029+
const char *reason) {
10231030
if (service->state == CBM_DAEMON_RUNTIME_SERVICE_RUNNING) {
10241031
service->state = CBM_DAEMON_RUNTIME_SERVICE_STOPPING;
10251032
service->stop_deadline_ms = deadline;
1033+
/* The generation's fate is decided here by one of several owners
1034+
* (last-client exit, coordinator stop, activation drain, external
1035+
* stop). Sessions dropped by the losing side of a race are
1036+
* undiagnosable without naming which owner pulled the trigger. */
1037+
cbm_log_info("daemon.runtime_stopping", "reason", reason ? reason : "unspecified");
10261038
} else if (service->state == CBM_DAEMON_RUNTIME_SERVICE_STOPPING &&
10271039
(service->stop_deadline_ms == 0 || deadline < service->stop_deadline_ms)) {
10281040
service->stop_deadline_ms = deadline;
@@ -1033,10 +1045,11 @@ static void runtime_service_begin_stopping_locked(cbm_daemon_runtime_service_t *
10331045
}
10341046

10351047
static void runtime_service_begin_stopping(cbm_daemon_runtime_service_t *service,
1036-
uint32_t timeout_ms, bool emergency) {
1048+
uint32_t timeout_ms, bool emergency,
1049+
const char *reason) {
10371050
uint64_t deadline = runtime_deadline_after(timeout_ms);
10381051
cbm_mutex_lock(&service->mutex);
1039-
runtime_service_begin_stopping_locked(service, deadline, emergency);
1052+
runtime_service_begin_stopping_locked(service, deadline, emergency, reason);
10401053
cbm_mutex_unlock(&service->mutex);
10411054
}
10421055

@@ -1084,7 +1097,8 @@ static void runtime_worker_disconnect(cbm_daemon_runtime_worker_t *worker) {
10841097
/* A HELLO whose application session is still opening is only a
10851098
* provisional coordinator client. It cannot keep the generation
10861099
* alive after the final fully committed frontend disconnects. */
1087-
runtime_service_begin_stopping_locked(service, shutdown_deadline, false);
1100+
runtime_service_begin_stopping_locked(service, shutdown_deadline, false,
1101+
"last_committed_client_disconnected");
10881102
}
10891103
}
10901104
cbm_mutex_unlock(&service->mutex);
@@ -1098,7 +1112,8 @@ static void runtime_worker_disconnect(cbm_daemon_runtime_worker_t *worker) {
10981112
worker->application_session);
10991113
}
11001114
if (cbm_daemon_coordinator_state(service->coordinator) == CBM_DAEMON_COORDINATOR_STOPPING) {
1101-
runtime_service_begin_stopping(service, service->shutdown_timeout_ms, false);
1115+
runtime_service_begin_stopping(service, service->shutdown_timeout_ms, false,
1116+
"coordinator_stopping");
11021117
}
11031118
}
11041119

@@ -1121,6 +1136,7 @@ static bool runtime_worker_admit(cbm_daemon_runtime_worker_t *worker,
11211136
worker->client_id = client_id;
11221137
worker->admitted = true;
11231138
worker->admission_committed = false;
1139+
service->admitted_total++;
11241140
}
11251141
}
11261142
cbm_mutex_unlock(&service->mutex);
@@ -1202,13 +1218,23 @@ static void *runtime_application_worker(void *opaque) {
12021218
status = CBM_DAEMON_RUNTIME_APPLICATION_HANDLER_ERROR;
12031219
}
12041220

1221+
/* Completion must be one atomic transition from the client's point of
1222+
* view: publish the done flag BEFORE the response bytes leave, so by the
1223+
* time any client can react to the response, admission already observes
1224+
* this slot as reusable. With the store after the send, a client that
1225+
* pipelines its next request the moment it sees a response raced this
1226+
* thread's final instructions and was rejected BUSY — on loaded Windows
1227+
* hosts roughly half of all back-to-back requests on one connection,
1228+
* surfacing as the wandering Phase 5 session drops. The admission side
1229+
* joins this thread after observing the flag, which safely absorbs the
1230+
* tail of the send. */
1231+
atomic_store_explicit(&worker->application_thread_done, true, memory_order_release);
12051232
bool sent = runtime_worker_send_application_response(worker, worker->application_request_token,
12061233
status, response, response_length, true);
12071234
free(response);
12081235
if (!sent && !atomic_load_explicit(&worker->disconnecting, memory_order_acquire)) {
12091236
cbm_daemon_ipc_connection_interrupt(worker->connection);
12101237
}
1211-
atomic_store_explicit(&worker->application_thread_done, true, memory_order_release);
12121238
return NULL;
12131239
}
12141240

@@ -1383,7 +1409,7 @@ static void runtime_worker_handle_activation_shutdown(cbm_daemon_runtime_worker_
13831409
result.active_connections =
13841410
service->active_connections > 0 ? (uint64_t)(service->active_connections - 1U) : 0;
13851411
if (service->state == CBM_DAEMON_RUNTIME_SERVICE_RUNNING) {
1386-
runtime_service_begin_stopping_locked(service, deadline, false);
1412+
runtime_service_begin_stopping_locked(service, deadline, false, "activation_shutdown");
13871413
service->activation_shutdown_requested = true;
13881414
service->activation_response_inflight = true;
13891415
worker->final_response_inflight = true;
@@ -1569,6 +1595,15 @@ static void *runtime_connection_worker(void *opaque) {
15691595
received = cbm_daemon_ipc_receive_frame(worker->connection, CBM_DAEMON_IPC_WAIT_FOREVER,
15701596
&frame, &payload);
15711597
if (received != 1 || frame.type != CBM_DAEMON_FRAME_REQUEST) {
1598+
/* An admitted frontend disconnecting decides the daemon's fate
1599+
* (last committed client ends the generation), so the transport
1600+
* cause must be visible: 0 = orderly EOF, negative = transport
1601+
* error. A silent break here previously made session-drop races
1602+
* undiagnosable across three platforms. */
1603+
char received_text[16];
1604+
(void)snprintf(received_text, sizeof(received_text), "%d", received);
1605+
cbm_log_info("daemon.connection_end", "received", received_text, "frame_type",
1606+
received == 1 ? "non_request" : "none");
15721607
break;
15731608
}
15741609
switch (frame.flags) {
@@ -2026,6 +2061,16 @@ cbm_daemon_runtime_service_state_t cbm_daemon_runtime_service_state(
20262061
return state;
20272062
}
20282063

2064+
uint64_t cbm_daemon_runtime_service_clients_admitted_total(cbm_daemon_runtime_service_t *service) {
2065+
if (!service) {
2066+
return 0;
2067+
}
2068+
cbm_mutex_lock(&service->mutex);
2069+
uint64_t total = service->admitted_total;
2070+
cbm_mutex_unlock(&service->mutex);
2071+
return total;
2072+
}
2073+
20292074
size_t cbm_daemon_runtime_service_active_clients(cbm_daemon_runtime_service_t *service) {
20302075
if (!service) {
20312076
return 0;
@@ -2129,7 +2174,7 @@ bool cbm_daemon_runtime_service_stop(cbm_daemon_runtime_service_t *service, uint
21292174
if (cbm_daemon_runtime_service_state(service) == CBM_DAEMON_RUNTIME_SERVICE_EXITED) {
21302175
return cbm_daemon_runtime_service_wait_exited(service, timeout_ms);
21312176
}
2132-
runtime_service_begin_stopping(service, timeout_ms, true);
2177+
runtime_service_begin_stopping(service, timeout_ms, true, "service_stop");
21332178
runtime_service_interrupt_connections(service);
21342179
return cbm_daemon_runtime_service_wait_exited(service, timeout_ms);
21352180
}

‎src/daemon/runtime.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,10 @@ cbm_daemon_runtime_service_t *cbm_daemon_runtime_service_start_reserved(
248248
cbm_daemon_runtime_service_state_t cbm_daemon_runtime_service_state(
249249
cbm_daemon_runtime_service_t *service);
250250
size_t cbm_daemon_runtime_service_active_clients(cbm_daemon_runtime_service_t *service);
251+
/* Monotonic count of every admission since service start; never decremented.
252+
* Use for "has any client ever connected" decisions — the live count above
253+
* can read zero between two short-lived sessions. */
254+
uint64_t cbm_daemon_runtime_service_clients_admitted_total(cbm_daemon_runtime_service_t *service);
251255
/* Includes accepted connections still waiting for HELLO. Never exceeds the
252256
* configured max_clients; over-cap peers receive REJECTED before close. */
253257
size_t cbm_daemon_runtime_service_active_connections(cbm_daemon_runtime_service_t *service);

0 commit comments

Comments
 (0)