feat(watcher): opt-in polling of non-git project roots (#1948)

A project indexed from a directory that is not a git repository was never
refreshed after its first index: init_baseline classified it is_git=false
and poll_project/check_changes returned early for it forever, so the graph
silently went stale. The comment in init_baseline even claimed such folders
were "polled as a plain directory", which was never true.

Add an opt-in config key, watch_non_git (default false, read once at daemon
startup like watcher_enabled). When on, a non-git root is polled on the
existing adaptive cadence (cbm_watcher_poll_interval_ms) by a tree
signature: the indexer's own cbm_discover walk (same skip lists, .gitignore
stack and .cbmignore), sorted by path and folded over (relative path, size,
mtime). A changed signature takes the existing reindex path with the same
commit-after-success discipline as the #937 dirty signature. The baseline
leaves the signature "unknown", so the first poll reindexes once
(at-least-once, covering edits made while the daemon was down).

Runaway guard: only files discovery would index enter the signature, so
cbm's own .codebase-memory output (built-in skip) and the cache directory
(pruned by absolute path) can never make a finished reindex look like a
new change; a scratch folder under an unrelated dirty repository is polled
on its own files and never inherits the ancestor's dirty state.

cbm_file_info_t gains mtime_ns, filled from the stat discovery already
performs, so the signature needs no second stat per file and stays
UTF-8-correct on Windows (discovery's wide stat).

Docs: README and docs/CONFIGURATION.md now say non-git roots are not
watched by default and document watch_non_git. The staleness signal in
index_status is left to #1561's freshness object. Refs #1948.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

Author: DeusData

README.md

@@ -155,6 +155,8 @@ Watcher registration is controlled separately by `auto_watch` (default `true`).
 
 To turn the watcher off entirely, set `config set watcher_enabled false` (default `true`): the background poll thread never starts and no project is registered, while `auto_index` and manual `index_repository` keep working. Unlike `auto_watch` — which is consulted per session — `watcher_enabled` is read once when the background daemon starts, so run `codebase-memory-mcp daemon stop` after changing it; reconnecting your MCP client alone will not restart the daemon. See [docs/CONFIGURATION.md](docs/CONFIGURATION.md#2-cli-managed-runtime-settings).
 
+The watcher follows **git** projects only by default: a project indexed from a directory that is not a git repository is not watched and stays at its last index until you run `index_repository` again. To have such roots polled too, set `config set watch_non_git true` (default `false`, read when the daemon starts): they are then checked on the same cadence with a file-tree scan that uses the indexer's own ignore rules.
+
 ### Keeping Up to Date
 
 **Updates run from the install script on every platform, not from inside the running binary.** `codebase-memory-mcp update` validates your flags and then prints the exact command to run:
@@ -736,6 +738,7 @@ codebase-memory-mcp config set auto_index true           # auto-index on session
 codebase-memory-mcp config set auto_index_limit 50000    # max files for auto-index
 codebase-memory-mcp config set auto_watch false          # don't register background git watcher (default: true)
 codebase-memory-mcp config set watcher_enabled false     # stop the watcher thread entirely (default: true)
+codebase-memory-mcp config set watch_non_git true        # also poll non-git project roots (default: false)
 codebase-memory-mcp config set index_max_files 250000    # optional per-index source-file limit
 codebase-memory-mcp config set index_max_source_mb 16384 # optional per-index source-size limit
 codebase-memory-mcp config reset auto_index              # reset to default

docs/CONFIGURATION.md

@@ -89,6 +89,7 @@ Current keys:
 | `auto_index_limit` | `50000` | Maximum file count allowed for automatic indexing of a new project. |
 | `auto_watch` | `true` | Register the session's project with the background git watcher on connect. Set `false` to keep a session from registering its project (the watcher still runs for other projects). |
 | `watcher_enabled` | `true` | Master switch for the background watcher subsystem. Set `false` to stop the watcher from starting at all — no poll thread and no project registration. Reindex manually with `index_repository` when disabled. |
+| `watch_non_git` | `false` | Also poll project roots that are **not git repositories**. By default the watcher only follows git projects, so a project indexed from a plain directory is never refreshed after its first index — reindex it manually with `index_repository`. Set `true` to poll such roots on the same adaptive cadence with a file-tree scan: the indexer's own discovery walk (same skip lists, `.gitignore` and `.cbmignore` rules) hashed over each file's path, size and mtime. Any change reindexes once; paths the indexer skips (including cbm's own `.codebase-memory/` output) never trigger. The first poll after the daemon starts reindexes each such project once, since nothing records which tree state the index holds. The scan walks the whole tree every poll, so it costs more than git polling on very large trees. Read once when the daemon starts, like `watcher_enabled`. |
 | `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |
 | `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
 

src/cli/cli.c

@@ -7404,6 +7404,11 @@ bool cbm_config_watcher_enabled(cbm_config_t *cfg) {
     return cbm_config_get_bool(cfg, CBM_CONFIG_WATCHER_ENABLED, true);
 }
 
+/* Opt-in tree polling of non-git roots (#1948); see cli.h. */
+bool cbm_config_watch_non_git(cbm_config_t *cfg) {
+    return cbm_config_get_bool(cfg, CBM_CONFIG_WATCH_NON_GIT, false);
+}
+
 bool cbm_config_load_index_policy(cbm_config_t *cfg, cbm_index_resource_policy_t *policy,
                                   char *error, size_t error_size) {
     if (!cfg || !policy) {
@@ -7444,6 +7449,8 @@ static const config_key_def_t CONFIG_KEYS[] = {
     {CBM_CONFIG_AUTO_WATCH, "true", "Register background git watcher on session connect"},
     {CBM_CONFIG_WATCHER_ENABLED, "true",
      "Run the background watcher thread (auto-reindex); false to disable"},
+    {CBM_CONFIG_WATCH_NON_GIT, "false",
+     "Also poll non-git project roots for changes (file-tree scan); off by default"},
     {CBM_CONFIG_UI_LANG, "auto", "Pin graph UI language: en, zh, or auto"},
     {CBM_CONFIG_UI_ENABLED, "false", "Serve the graph UI on a loopback HTTP port"},
     {CBM_CONFIG_UI_PORT, "9749", "Port for the graph UI listener when enabled"},

src/cli/cli.h

@@ -445,6 +445,7 @@ bool cbm_config_load_index_policy(cbm_config_t *cfg, cbm_index_resource_policy_t
 #define CBM_CONFIG_AUTO_WATCH "auto_watch"
 #define CBM_CONFIG_UI_LANG "ui-lang"
 #define CBM_CONFIG_WATCHER_ENABLED "watcher_enabled"
+#define CBM_CONFIG_WATCH_NON_GIT "watch_non_git"
 /* #1558: the graph UI's loopback listener. Stored in the UI config file rather
  * than the key-value store, but surfaced through `config` so it is findable. */
 #define CBM_CONFIG_UI_ENABLED "ui_enabled"
@@ -459,6 +460,11 @@ bool cbm_config_load_index_policy(cbm_config_t *cfg, cbm_index_resource_policy_t
  * the watcher. */
 bool cbm_config_watcher_enabled(cbm_config_t *cfg);
 
+/* Whether the watcher also polls NON-GIT project roots (default false, #1948).
+ * Non-git roots are otherwise never refreshed after their first index. Read
+ * once at daemon startup, like watcher_enabled. NULL-safe (NULL → false). */
+bool cbm_config_watch_non_git(cbm_config_t *cfg);
+
 /* ── Binary activation safety ─────────────────────────────────── */
 
 typedef void *cbm_cli_activation_lock_t;

src/daemon/host.c

@@ -609,6 +609,12 @@ static bool host_state_prepare(host_state_t *host, const cbm_daemon_ipc_endpoint
     if (watcher_enabled) {
         host->watch_store = cbm_store_open_memory();
         host->watcher = cbm_watcher_new(host->watch_store, host_watcher_index, host);
+        /* #1948: non-git roots stay unwatched unless watch_non_git opts in.
+         * Same read-once-at-startup contract as watcher_enabled. */
+        if (host->watcher && cbm_config_watch_non_git(host->runtime_config)) {
+            cbm_watcher_set_poll_non_git(host->watcher, true);
+            cbm_log_info("watcher.non_git", "mode", "tree_poll");
+        }
     } else {
         cbm_log_info("watcher.disabled", "reason", "config");
     }

src/discover/discover.c

@@ -518,8 +518,23 @@ static void file_list_add_ignored(file_list_t *fl, const char *rel_path, const c
     fl->ignored_count++;
 }
 
+/* Nanosecond mtime from a stat result, per platform. Windows' wide_stat only
+ * carries whole seconds, so the value is seconds scaled to ns there. */
+#define DISCOVER_NS_PER_SEC 1000000000LL
+static int64_t discover_stat_mtime_ns(const struct stat *st) {
+#ifdef __APPLE__
+    return ((int64_t)st->st_mtimespec.tv_sec * DISCOVER_NS_PER_SEC) +
+           (int64_t)st->st_mtimespec.tv_nsec;
+#elif defined(_WIN32)
+    return (int64_t)st->st_mtime * DISCOVER_NS_PER_SEC;
+#else
+    return ((int64_t)st->st_mtim.tv_sec * DISCOVER_NS_PER_SEC) + (int64_t)st->st_mtim.tv_nsec;
+#endif
+}
+
 static void fl_add(file_list_t *fl, const char *abs_path, const char *rel_path, CBMLanguage lang,
-                   int64_t size) {
+                   const struct stat *st) {
+    int64_t size = (int64_t)st->st_size;
     if (fl->max_files >= 0 && fl->count >= fl->max_files) {
         fl->limit_exceeded = true;
         return;
@@ -574,6 +589,7 @@ static void fl_add(file_list_t *fl, const char *abs_path, const char *rel_path,
     fi->rel_path = relative_copy;
     fi->language = lang;
     fi->size = size;
+    fi->mtime_ns = discover_stat_mtime_ns(st);
 }
 
 /* ── Recursive walk ─────────────────────────────── */
@@ -878,9 +894,10 @@ static void walk_dir_process_file(const char *abs_path, const char *rel_path, co
                                   const cbm_discover_opts_t *opts,
                                   const gitignore_link_t *ignore_chain,
                                   const cbm_gitignore_t *global_gi,
-                                  const cbm_gitignore_t *cbmignore, off_t size, file_list_t *out) {
+                                  const cbm_gitignore_t *cbmignore, const struct stat *st,
+                                  file_list_t *out) {
     const char *skip_reason =
-        file_skip_reason(name, rel_path, opts, ignore_chain, global_gi, cbmignore, size);
+        file_skip_reason(name, rel_path, opts, ignore_chain, global_gi, cbmignore, st->st_size);
     if (skip_reason) {
         /* Deliberately not indexed (#963) — record so callers can surface it.
          * Unsupported-language files below are NOT recorded: "no grammar for
@@ -893,7 +910,7 @@ static void walk_dir_process_file(const char *abs_path, const char *rel_path, co
     if (lang == CBM_LANG_COUNT) {
         return;
     }
-    fl_add(out, abs_path, rel_path, lang, size);
+    fl_add(out, abs_path, rel_path, lang, st);
 }
 
 typedef struct {
@@ -1005,7 +1022,7 @@ static void walk_dir_process_entry(cbm_dirent_t *entry, const walk_frame_t *fram
         }
     } else if (S_ISREG(st.st_mode)) {
         walk_dir_process_file(abs_path, rel_path, entry->name, opts, frame->ignore_chain, global_gi,
-                              cbmignore, st.st_size, out);
+                              cbmignore, &st, out);
     }
 }
 

src/discover/discover.h

@@ -141,6 +141,7 @@ typedef struct {
     char *rel_path;       /* relative to repo root (heap-allocated) */
     CBMLanguage language; /* detected language */
     int64_t size;         /* file size in bytes */
+    int64_t mtime_ns;     /* modification time, ns (whole seconds on Windows) */
 } cbm_file_info_t;
 
 typedef struct {