Intermittent cold-start race: version_cohort.claimed_unheld -> daemon.ipc.listen_failed stage=reservation_validation kills the stdio MCP server for the whole host session #2162

Version: codebase-memory-mcp 0.10.8
Platform: macOS (Apple Silicon)
Install channel: GitHub release archive / install.sh / install.ps1
Binary variant: ui

What happened, and what did you expect?

Some cold daemon starts fail during listener handoff. The client then prints
CBM daemon could not start within 30000 ms and exits, closing stdio. MCP hosts do
not respawn a crashed stdio server, so that host session loses the graph permanently —
Codex reports every subsequent tool call as Transport closed.

Rate: with a deliberate cold-start probe (below) I hit it 2 times in ~30 cold starts.
My cbm-daemon.log also carries 1 occurrence I did not cause, against 83 successful
daemon starts in the same file. I cannot attribute any specific host-session outage to
this race — I am reporting a reproducible fault, not a measured production rate.

Two things make the blast radius much larger than the raw rate suggests:

1. The daemon exits the instant its last client disconnects
   (daemon.runtime_stopping reason=last_committed_client_disconnected) and tears down
   the whole rendezvous directory. So every new host session is a fresh cold start
   and a fresh roll of the dice. This is the same teardown behaviour reported in hook-augment restarts full daemon on every invocation, causing ~3s latency per call and hook timeouts #2058.
2. hook-augment runs as a short-lived client on nearly every turn in both Claude Code
   and Codex, with a 2000 ms deadline. Cold start costs ~2 s, so those clients routinely
   abandon a start mid-flight — hook-augment-timeouts.log logged 1183
   deadline_exceeded ms=2000 entries in a single day on this machine. That is a lot
   of start/teardown churn feeding the race. (cf. hook-augment: needs a warm daemon to ever emit context; install leaves it timing out on every Grep/Glob (follow-up to #858) #1335, hook-augment restarts full daemon on every invocation, causing ~3s latency per call and hook timeouts #2058, perf(daemon): hash the executable image once per process; announce missed hook deadlines #1767)

Expected: a cold start that loses a reservation race should retry or wait for the
incumbent, not hard-fail. And a hard failure should not be reported as a 30 s timeout —
the call returns in 0.8–2.1 s.

Reproduction

No project or indexed code required — this is purely daemon lifecycle. Run from any
directory; a public repo works fine as cwd.

# repro.py — cold-start the stdio client repeatedly and count handshake failures.
# Each iteration attaches, handshakes, detaches, and waits long enough for the
# daemon to reach last_committed_client_disconnected and exit, so the next
# iteration is a genuine cold start.
import json, subprocess, sys, time

BIN = "codebase-memory-mcp"   # or an absolute path to the binary
post, n = float(sys.argv[1]), int(sys.argv[2])
res = []
for i in range(n):
    p = subprocess.Popen([BIN], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                         stderr=subprocess.DEVNULL, text=True, bufsize=1)
    p.stdin.write(json.dumps({"jsonrpc": "2.0", "id": 1, "method": "initialize",
                              "params": {"protocolVersion": "2025-06-18",
                                         "capabilities": {},
                                         "clientInfo": {"name": "probe", "version": "1"}}}) + "\n")
    p.stdin.flush()
    line = p.stdout.readline()
    res.append("error" not in line)
    p.kill(); p.wait()
    time.sleep(post)
print(f"runs={n} ok={sum(res)} fail={n-sum(res)}",
      "".join("." if r else "X" for r in res))

$ python3 repro.py 5 12
runs=12 ok=11 fail=1 ...X........

The failing iteration returns, in ~0.8 s:

{"jsonrpc":"2.0","id":null,"error":{"code":-32001,"message":"CBM daemon could not start within 30000 ms"}}

It is intermittent — a single pass can come up clean. Two failures in ~30 cold
iterations here, across a 6-iteration run at a 0.2 s gap and a 12-iteration run at a 5 s
gap. Counter-check: pinning one initialized client so the daemon never exits drives
new cold starts to zero, and 20 further attach/detach cycles then produced 0 daemon
starts and 0 failures — consistent with the race living entirely in the cold-start path.

Logs

~/.cache/codebase-memory-mcp/logs/cbm-daemon.log, the complete failing sequence:

level=info  msg=version_cohort.claimed_unheld build=996bad5fe6fb89c0c50363d87f03dc78f6e536aeea746c06679a294d14eca435
level=info  msg=mem.allocator.bound_populations_only owned_classes=0/6 populations=sqlite,tree_sitter
level=info  msg=mem.init budget_mb=11468 total_ram_mb=32768 source=ram_fraction
level=error msg=daemon.ipc.listen_failed    stage=reservation_validation
level=error msg=daemon.runtime.start_failed stage=listener_handoff
level=error msg=daemon.start_failed         component=runtime

and on the client's stderr:

codebase-memory-mcp: CBM daemon could not start within 30000 ms

Notes on that trace:

• version_cohort.claimed_unheld precedes all 3 listen_failed entries in my log,
  but is not sufficient on its own — it also appears before 2 starts that went on to
  succeed. It looks like the marker of the racy path rather than the fault itself.
• build=996bad5f… is the SHA-256 of the 295 MB executable. shasum -a 256 on the same
  binary reproduces it in 0.56 s on the hardware path; the in-process software path took
  1.31 s in these traces. That is the cost perf(daemon): hash the executable image once per process; announce missed hook deadlines #1767 is caching away, and it is most of the
  cold-start window in which this race can fire.
• Only one build/cohort is present the whole time — a single
  cbm-version-cohort-lifetime-v1.lock holding 0.10.8 / build 996bad5f… / cache
  fingerprint 39d39f5…. This is not a mixed-version cohort.

Relationship to existing issues

Searched before filing; this is a distinct cause from the near neighbours:

• Daemon start fails with daemon.ipc.listen_failed stage=pending_publication when /tmp is full; clients get a misleading "active or starting" 30s timeout and MCP hosts silently drop the server #1828 — same misleading 30 s message, but stage=pending_publication from a full
  /tmp. Disk here had ample space and the stage differs.
• Orphaned socket with no .sock.identity sidecar wedges every client permanently; no daemon log, misleading 30s timeout #2107 — same misleading 30 s message, but produces zero daemon-log lines and
  wedges permanently. This one logs the full listen_failed/start_failed trio and
  self-heals on the next attempt.
• hook-augment restarts full daemon on every invocation, causing ~3s latency per call and hook timeouts #2058 / hook-augment: needs a warm daemon to ever emit context; install leaves it timing out on every Grep/Glob (follow-up to #858) #1335 / perf(daemon): hash the executable image once per process; announce missed hook deadlines #1767 — the churn and cold-start-cost side. They make this race
  fire far more often, but none of them describes the reservation_validation failure.

No open or closed issue mentions reservation_validation or claimed_unheld.

Suggested direction

1. Treat a lost reservation as retryable. Losing the socket reservation between reserve
   and validate is an expected outcome of two clients starting at once; back off and
   re-attach to the incumbent rather than failing the whole start.
2. Fix the error message. It is not a timeout and it names no cause or path. Something
   like reservation for /tmp/cbm-daemon-501/cbm-<hash>.sock was invalidated during listener handoff (another daemon started or exited concurrently) would have made
   this a five-minute diagnosis.
3. Consider a short idle linger before the daemon exits on
   last_committed_client_disconnected. Exiting immediately makes every host session
   pay a cold start, which is what turns a rare race into a daily outage.

Project scale

Not relevant — reproduces with no project indexed. (The graphs in play here are 2–50 MB
SQLite DBs, ~24k nodes on the largest.)

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository),
  not proprietary code.