A live daemon that is not holding the version-cohort marker poisons every index worker it forks, for its entire lifetime — no self-heal, and the conflict log names a build that does not exist

Summary

If a daemon is alive but is not holding the VERSION_COHORT_DAEMON_FILE marker lock, every index
worker that same daemon forks rejects itself at startup and exits 1:

CBM index worker could not start: active daemon coordination could not be verified safely

The condition never clears on its own. The daemon keeps running, keeps forking workers, and every one
of them fails the same way until the daemon is killed by hand. On 2026-09-11 this produced 370
conflict records over 427 minutes (~52/hour) on my machine, and it stopped at the exact second I
killed the daemon.

This is why the workers fail. #2015 is about the cadence at which the watcher re-forks them once
they do. They are independent: I am running the #2075 backoff, it worked, and the wedge still produced
370 dead workers — see Impact #3.

Mechanism

version_cohort_active_daemon_presence() (src/daemon/version_cohort.c:805-839) classifies via the
daemon marker lock:

`cbm_private_file_lock_try_acquire(VERSION_COHORT_DAEMON_FILE, EX)`	lifetime probe	presence
`BUSY` (someone holds it)	—	`COORDINATED` ✅
`OK` (nobody holds it)	`0`	`ABSENT` ✅
`OK` (nobody holds it)	`1` (a daemon IS alive)	`UNCOORDINATED` ❌

The worker then refuses in src/main.c:2787-2798:

cbm_version_cohort_daemon_presence_t worker_daemon_presence =
    cbm_version_cohort_daemon_presence_under_transition(worker_cohort_manager,
                                                        worker_endpoint, worker_transition);
if (worker_daemon_presence != CBM_VERSION_COHORT_DAEMON_ABSENT &&
    worker_daemon_presence != CBM_VERSION_COHORT_DAEMON_COORDINATED) {
    if (worker_daemon_presence == CBM_VERSION_COHORT_DAEMON_UNCOORDINATED) {
        (void)cbm_version_cohort_log_uncoordinated_daemon(&identity);
    }
    (void)fprintf(stderr, "CBM index worker could not start: active daemon coordination "
                          "could not be verified safely\n");
    goto worker_cleanup;
}

So the bottom row is the wedge: a daemon is alive, and it is not in the cohort. Since that daemon
is also the thing forking the workers, it is rejecting its own children — and because nothing makes
the daemon re-acquire the marker, the state is absorbing.

The conflict log actively misleads

cbm_version_cohort_log_uncoordinated_daemon() (src/daemon/version_cohort.c:963-980) writes a
hardcoded sentinel for the active side:

(void)snprintf(conflict.active_version, sizeof(conflict.active_version), "%s", "pre-cohort/unknown");
memset(conflict.active_build_fingerprint, '0', sizeof(conflict.active_build_fingerprint) - 1);

which lands in daemon-conflicts.ndjson as:

{"event":"daemon.version_conflict","timestamp_unix_s":1789149612,"reason":"build",
 "active_version":"pre-cohort/unknown",
 "active_build":"0000000000000000000000000000000000000000000000000000000000000000",
 "requested_version":"dev",
 "requested_build":"273d3ffe8585bb5f98bb5ed99abc1be6fece06fbe50f7fda849450d216ac4e99"}

The name asserts a diagnosis — an old pre-cohort binary is running — that is not true here. The
only runnable CBM binary on this machine is the 273d3ffe build in the requested_build field. (There
is one other copy on disk, a March nix build in a repo working tree, but it aborts in dyld on launch
and contains no watcher log keys at all, so it cannot have been the daemon.) The active daemon was
almost certainly the same build as the requester — it just wasn't holding the marker.

Reporting a zero fingerprint and a version string that no binary reports sends you looking for a stale
install that isn't there. A record saying "a live daemon holds no cohort marker; identity unknown"
would point at the actual fault.

Impact / diagnosability

No self-heal. Only kill <daemon-pid> clears it. Nothing in the daemon log names the
condition; index_repository and index_status returned status=error throughout without
surfacing the reason.
The reason is invisible from the daemon side. The daemon log records only:
```
level=info msg=index.supervisor.reap outcome=exit_nonzero exit_code=1 signal=0
level=warn msg=index.supervisor.worker_failed outcome=exit_nonzero exit_code=1 log=…/.worker-log-y4KhDv
```
The actual sentence is written to a per-worker .worker-log-XXXXXX temp file, never promoted. You
have to know to go read a randomly-named dotfile in the log directory.
Backoff bounds the rate but not the total. This machine is running the fix(watcher): back off re-forking a hard-failing index worker #2075 branch, so the
rc < 0 backoff was active throughout and behaved exactly as designed — retries decayed to the
INDEX_FAIL_CEILING_MS ceiling of one per 5 min per project, and watcher.index.sustained_failure
fired at consecutive=10 for six projects. Over a 427-minute wedge that ceiling still permits ~85
retries per project; liverpool-cleanup reached consecutive=81, i.e. it sat at the cap the whole
time. Because the wedge never clears, dead workers accumulate linearly with daemon uptime no
matter how good the backoff is — 370 conflicts across 8 watched projects in 7 hours. Capping the
cadence was the right fix for Watcher re-forks a permanently-failing index worker forever — no backoff on rc < 0 (2233 workers in 4h43m); gap in #937 #2015 and is not a fix for this.
Those worker-log files are never pruned — 2618 of them here going back to 2026-09-01, every one
a startup failure. Minor, but it is how I found the pattern.

Field data

~/.cache/codebase-memory-mcp/logs/daemon-conflicts.ndjson, 387 records, 386 of a single shape
(pre-cohort/unknown / all-zero → dev/273d3ffe):

date	conflicts	note
2026-09-04	1
2026-09-06	16
2026-09-11	370	10:52:48Z → 18:00:12Z, 427 min, ended at the daemon kill

The single remaining record is a genuine mixed-build conflict of the normal kind
(version | active=dev/273d3ffe → requested=0.10.8/2412e017), which is the mechanism working
as designed and is not what this issue is about.

The same cbm-daemon.log happens to contain a clean before/after for #2075, because the
watcher.index.err line only carries rc / consecutive on the patched build:

build	`watcher.index.err` lines	projects	max streak
unpatched (no `rc=` field)	2226	1 (`add-translate-buttons`)	n/a — unthrottled
patched (`rc=` present)	~560	8	81, at the 5-min ceiling

There is a sibling arm one branch earlier, src/main.c:2778-2786
(cbm_daemon_ipc_local_transition_seal_legacy() != 1), which prints "a pre-coordination or unverified
CBM generation is active". It produced a 2233-worker wave on 2026-09-01 — the same one quoted in
#2015 — and also calls cbm_version_cohort_log_uncoordinated_daemon(). If these share a cause, both
error strings should probably move with the fix.

Environment

macOS 15 (Darwin 25.6.0), arm64
codebase-memory-mcp dev, build 273d3ffe8585bb5f98bb5ed99abc1be6fece06fbe50f7fda849450d216ac4e99
(local build of main at v0.10.8, plus the fix(watcher): back off re-forking a hard-failing index worker #2075 branch)
Several concurrent MCP frontend clients (long-lived Claude Code sessions), watcher active on ~8 projects

Repro

I do not have a deterministic repro — it has appeared three times in eight days with no action on my
part that I can correlate. What I can say is that it is fully observable after the fact, and that
kill <daemon-pid> is a reliable recovery. If it would help, I am happy to run an instrumented build
that logs the marker-lock acquire/release path on the daemon side and report back the next time it
trips.

Questions for maintainers

Is the daemon expected to hold VERSION_COHORT_DAEMON_FILE for its whole lifetime? If so, is there
a path that releases it while the daemon stays up — and should a daemon that finds itself outside
the cohort re-acquire, or fail loudly and exit rather than keep serving?
Should a worker forked by a daemon consult the marker at all, versus inheriting a coordination
token from the parent that forked it? The current check treats the parent as an untrusted stranger.
Would you take a PR that (a) replaces the pre-cohort/unknown sentinel with a record stating what
was actually observed, and (b) promotes the worker's failure sentence into the daemon log so
worker_failed carries the reason rather than a path to a temp file? Those are small and
independent of whatever fixes the underlying coordination loss.

@LinRds

Root cause found: macOS `tmp_cleaner` unlinks the held cohort lock files

It happened again this morning, and this time the filesystem kept the evidence. The marker lock is not being released. The file it locks is deleted while the daemon still holds the lock, and the next client creates a new, unlocked file at the same path.

Mechanism

macOS ships com.apple.tmp_cleaner (/System/Library/LaunchDaemons/com.apple.tmp_cleaner.plist). It runs daily at midnight, or on the next wake if the machine was asleep. It runs:

# /usr/libexec/tmp_cleaner, daily_clean_tmps_dirs="/tmp", daily_clean_tmps_days="3"
find -dx . -fstype local -type f -atime +3 -mtime +3 -ctime +3 -delete -print

cbm-version-cohort-daemon-v1.lock and cbm-version-cohort-admission-v1.lock are zero-byte files. They are created once and never written. flock does not update atime, mtime or ctime. After three days they qualify and get unlinked, and the daemon's flock stays on the orphaned inode. The next cbm_private_file_lock_try_acquire() goes through O_CREAT|O_EXCL, which creates a new inode that nobody holds. From then on, version_cohort_active_daemon_presence() sees marker OK and lifetime probe 1, which is UNCOORDINATED, and every worker refuses to start. Nothing in the daemon re-checks its claim, so the state never clears.

The lock code does not cause this. private_file_lock.c never unlinks anything, and private_file_revalidate() already compares path inode to fd inode, but only while acquiring.

Evidence (2026-09-16, times CDT)

time	event	source
09-15 17:45:36	daemon pid 93629 starts	`daemon.start`, `cbm-version-cohort-lifetime-v1.lock` mtime
09-16 06:47:07	lid-open wake; the midnight `tmp_cleaner` run was missed	`pmset -g log`
06:50:37	birth time of both `cbm-version-cohort-daemon-v1.lock` and `cbm-version-cohort-admission-v1.lock`, while pid 93629 was still running	`stat -f %SB`
06:50:37	a new MCP client connects to the daemon (`initialize`, `watcher.watch`) and recreates both files	`cbm-daemon.log`
06:50:54	first worker log: `active daemon coordination could not be verified safely`	`.worker-log-*` birth time
06:50:56	first `pre-cohort/unknown` record today	`daemon-conflicts.ndjson`

The files that were not deleted fit the rule. cbm-version-cohort-lifetime-v1.lock was rewritten when the daemon started (09-15), and cbm-version-cohort-maintenance-v1.lock was born 09-15, so neither was more than three days old. All four onsets on this machine (09-04 07:30, 09-06 10:52, 09-11 05:52, 09-16 06:50) were in the morning, which is when a missed midnight job runs on wake. The irregular gaps between them fit a rule based on file age.

Probably also behind #1757

tmp_cleaner uses -type f, so it deletes regular files and leaves sockets alone. In that same runtime directory, .sock and .anc are sockets, and .sock.identity is a regular file written once at bind. After the daemon has been up for three days, one cleaner pass leaves exactly the "markerless" shape from #1757: a hard-linked .sock/.anc pair with no .sock.identity. In #1757, @LinRds's listing shows that shape together with 4882 pre-cohort/unknown conflicts in the same window, which is this issue's signature. @alekseysotnikov describes the same pairing as "re-entrant across generations". #1894 makes startup recover from the markerless pair, which fixes the symptom. The file deletion that produces the pair is still there, and nothing recovers a live daemon whose marker files were deleted.

I have not reproduced #1757 this way. This is an inference from the cleaner rule and the listings in that thread.

Suggested fix

Keep held coordination files fresh. On a timer well under 3 days (hourly is plenty), the daemon calls futimens(fd, NULL) on every private lock and identity file it holds or owns. ctime moves with it, so find -ctime +3 never matches. This is a few lines in the existing daemon loop and needs no new files or protocol.
Detect the loss anyway. On the same timer, run private_file_revalidate() for the daemon claim marker (path inode == held fd inode) and for .sock.identity. On a mismatch, log daemon.claim_lost and do a clean daemon.stop so the next client starts a properly coordinated generation. Re-claiming in place also works, but exiting is simpler and cannot race a replacement. This also covers deletion by anything other than tmp_cleaner.
(Optional) A worker whose parent pid is the daemon's recorded pid could still refuse, but with an accurate record instead of the pre-cohort/unknown / all-zero placeholder. Point (3a) from the original report still applies.

(1) alone stops this on stock macOS. (2) turns any future loss into a single restart instead of an endless failure loop.

Checked against upstream/main 59a05eb (v0.11.0): nothing touches the held files' timestamps or re-checks the claim after acquire. #2046 / 9104feb6 retries CONFLICT rather than UNCOORDINATED, and #1894 only reclaims a markerless pair at startup.

Workaround until then: kill <--cbm-daemon-internal pid>. Index data is not affected.

Linux data point: the same failure class on stock Ubuntu, caused by systemd-tmpfiles. It hit a different held file and produced a different refusal message.

Setup: 0.10.8 (build cb6dc545), Ubuntu with systemd, permanent daemon (daemon start), runtime directory /tmp/cbm-daemon-<uid>.

Cleaner: /usr/lib/tmpfiles.d/tmp.conf ships q /tmp 1777 root root 10d, and systemd-tmpfiles-clean.timer runs daily. Like tmp_cleaner, it removes entries whose atime, mtime and ctime are all past the age.

time (UTC+10)	event
09-13 19:55	The daemon starts. It creates `cbm-<endpoint>.lifetime.lock` once and never touches it again.
09-24 20:09:34	`systemd-tmpfiles-clean.service` runs.
09-24 20:12	A new CLI call recreates `cbm-<endpoint>.lifetime.lock` (mtime 20:12) and refuses. The first refusal in our agents' logs is at 20:12:30.
since then	Every `codebase-memory-mcp cli …` call fails with `CBM CLI could not start because a pre-coordination or unverified CBM generation is active; close all CBM sessions and commands, then retry`.

/proc/<daemon pid>/fd still shows the daemon's lock on …/cbm-<endpoint>.lifetime.lock (deleted), while the path now holds a different, unheld inode. So the lost file here was the per-endpoint lifetime reservation probed in cbm_daemon_ipc_local_transition_seal_legacy(), not the cohort daemon marker. The resulting message is the seal_legacy one, as in #2271.

cbm-<endpoint>.startup-v2.lock and cbm-version-cohort-maintenance-v1.lock were also 12 days old but survived. Clients open them often enough to keep their atime recent. Only a file that one long-lived process holds and nobody reopens is exposed.

Your suggested fix (1) holds on Linux too. As a stopgap we run a daily user timer that calls utime() on every file of the live daemon's endpoint and on the cbm-version-cohort-* locks. It cannot undo an unlink that already happened, so this daemon stays wedged until it restarts. On Linux, a runtime directory under $XDG_RUNTIME_DIR (a per-user tmpfs that /tmp aging does not touch) would also sidestep it.

@balaji-dutt

On AI assistance: Claude (Opus 5 / 1M context) helped me narrow down the behavior and draft the wording below, including the repro steps. The investigation, the actual issue, and the workflow are mine. — @balaji-dutt

Another manifestation of this on macOS, from the CLI side rather than an index worker.

Environment: v0.10.8, macOS 26.6.2 (build 25G83, Darwin 25.6.0), arm64, installed via mise.

The user-visible symptom is the local CLI refusing to start at all, for any subcommand, while a daemon is live:

codebase-memory-mcp: CBM CLI could not start because an active pre-coordination or unverified CBM daemon is running. Close all CBM sessions and commands, then retry.

This broke an unrelated chezmoi apply hook that only ever runs config get auto_index. The value it reads was already correct; the CLI simply would not start.

daemon-conflicts.ndjson shows the sentinel described above, with the daemon and the CLI on the same build:

{"event":"daemon.version_conflict","reason":"build",
 "active_version":"pre-cohort/unknown",
 "active_build":"0000000000000000000000000000000000000000000000000000000000000000",
 "requested_version":"0.10.8","requested_build":"2412e017…"}

The live daemon really was 0.10.8 — cbm-daemon.log records daemon.start version=0.10.8 for that pid, and the cohort lifetime lock held CBMCOH 0.10.8 2412e017…, matching requested_build exactly. Only one version was installed on the machine.

Direct evidence of the transition you describe: an initial lsof on the daemon pid showed it holding cbm-version-cohort-daemon-v1.lock; a later lsof +D on the same live pid showed it no longer held that file, and it never re-acquired it. The wedge persisted until the daemon was replaced.

Rate on this machine, from daemon-conflicts.ndjson — 226 records total, of which one hour accounts for most:

2026-09-24 14:00     23
2026-09-24 15:00    198

Every record carries active_version: "pre-cohort/unknown", matching the hardcoded sentinel rather than any real reading.

Two notes that may help with scoping. First, the CLI refusal text differs from the index-worker message in the issue body, so the UNCOORDINATED classification is reachable from at least two callers. Second, #2047 in v0.11.0 looks like it will delay rather than remove this refusal: the local CLI acquires with main_deadline_after(MAIN_STARTUP_TIMEOUT_MS) (src/main.c:2867, 10000 ms), and a daemon wedged this way is a "peer that stays", so the conflict should now surface only after the full deadline.

@balaji-dutt

@balaji-dutt did you try testing against the PR #2289 to see if it resolves your issue? I am running it locally and have not seen this issue recently.

style(tests): drop unrelated reflow hunks

I hit this on macOS (Darwin 27, arm64, v0.10.8) and found what removes the marker. Short version: macOS deletes it.

Mechanism

/System/Library/LaunchDaemons/com.apple.tmp_cleaner.plist runs /usr/libexec/tmp_cleaner every day at 00:00. It does:
```
cd /tmp && find -dx . -fstype local -type f -atime +3 -mtime +3 -ctime +3 -delete
```
So any file under /tmp whose atime, mtime and ctime are all at least 4 whole days old is unlinked. The thresholds are hardcoded in the script; there is no periodic.conf override on current macOS.
The version-cohort markers (cbm-version-cohort-{daemon,lifetime,admission,maintenance}-v1.lock, cbm-<key>.lifetime.lock) are opened through cbm_private_file_lock_try_acquire(), which calls fchmod only when it creates the file (src/foundation/private_file_lock.c:316-331, unchanged in v0.11.0 and on main at 279cd73). Holding an flock changes no timestamp. After four days, a marker held by a live daemon qualifies, gets unlinked at midnight, and the next process recreates it with a new inode. The daemon keeps its lock on the deleted inode. That is exactly the bottom row of your table: alive, but not holding the marker anyone can see.
The endpoint locks (cbm-<key>.lock, .startup-v2.lock) survive, because src/daemon/ipc.c (around 1046-1054) calls fchmod(fd, 0600) on every open, so each new client refreshes their ctime.

Evidence from one machine

lsof -p <daemon> showed flocks on inodes that no longer existed in /tmp/cbm-daemon-<uid>/, while the files at those names had new inodes.
Markers born 01:32 on day 0 were kept at 00:00 on day 4 (3d22h old). At 00:00 on day 5 (4d22h old) they were deleted, and recreated at 00:00:04 / 00:01:26.
Lifetime markers last written on day 1 were deleted at 00:00 on day 6 and recreated at 00:00:11.
launchctl print system/com.apple.tmp_cleaner showed runs = 6, matching the days since boot.
The trigger is a daemon that stays alive for more than 4 days. Here a long-lived editor (the Codex app) kept its stdio clients attached the whole time. That would explain why it looks random and recurs every few days.

Your report is on macOS 15. tmp_cleaner with the same rule has been part of macOS for a long time (it used to run as periodic daily), so I would expect the same cause there.

Recovery that worked
Killing the daemon alone was not enough. The leftover v0.10.8 stdio clients still held the deleted lifetime inode, and a fresh client then logged version_cohort.claimed_unheld and never started a daemon. Recovery needed every cbm process gone, then the runtime dir moved aside.

Possible fixes

Refresh the markers the way the endpoint locks already are: fchmod/futimens on every open, plus a periodic touch from the daemon while it holds them.
Have the daemon notice that a marker it holds has been unlinked (compare fstat of its fd with fstatat of the name), then re-acquire or exit cleanly instead of rejecting its own workers forever.
On macOS, default the rendezvous to a directory that tmp_cleaner does not sweep.

Workaround
Set CBM_RUNTIME_DIR to a short, private directory outside /tmp for every cbm process. Wrapping the binary in a launcher that exports it is the easiest way to guarantee that.

Thank you for the subsequent inode evidence from macOS and Linux and for separating different held marker files. That adds useful detail beyond the original refusal message. The recovery/claim-loss policy and the status of #2289 still need review; I cannot confirm the proposed cleanup workarounds or a completed fix.

Thank you for the detailed macOS and Linux inode evidence. Claim-loss recovery and the linked proposal in #2289 still need maintainer review. Sorry for the delay; I’ll follow up soon.

@halindrome

Fixed on main in 5e9f146 (#2289 - thank you, @halindrome). The daemon now refreshes its held runtime lock and marker files every 60 seconds through the descriptors it already holds, so temp cleaners no longer age them out, and it stops cleanly only when a held file is provably gone. Please reopen if you still see the daemon lose its runtime files after upgrading.

fix(daemon): keep held runtime files fresh and exit when one is lost

Navigation Menu

Repository navigation

A live daemon that is not holding the version-cohort marker poisons every index worker it forks, for its entire lifetime — no self-heal, and the conflict log names a build that does not exist #2178

Summary

Mechanism

The conflict log actively misleads

Impact / diagnosability

Field data

Environment

Repro

Questions for maintainers

Root cause found: macOS `tmp_cleaner` unlinks the held cohort lock files

Mechanism

Evidence (2026-09-16, times CDT)

Probably also behind #1757

Suggested fix

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

A live daemon that is not holding the version-cohort marker poisons every index worker it forks, for its entire lifetime — no self-heal, and the conflict log names a build that does not exist #2178

Description

Summary

Mechanism

The conflict log actively misleads

Impact / diagnosability

Field data

Environment

Repro

Questions for maintainers

Activity

halindrome commented on Sep 16, 2026

Root cause found: macOS tmp_cleaner unlinks the held cohort lock files

Mechanism

Evidence (2026-09-16, times CDT)

Probably also behind #1757

Suggested fix

ptfff commented on Sep 25, 2026

balaji-dutt commented on Sep 25, 2026

halindrome commented on Sep 25, 2026

MarcelDuchamp commented on Oct 7, 2026

DeusData commented on Oct 19, 2026

DeusData commented on Oct 25, 2026

DeusData commented on Oct 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Root cause found: macOS `tmp_cleaner` unlinks the held cohort lock files