feat(index): enforce opt-in storage resource limits

An index that fits in memory and finishes in time can still fill the disk.
Publication needs room for the staging artifacts and the final database at
the same time, and running out of space during publication is the one
failure that can cost a working index.

Add index_cache_max_mb and index_min_free_disk_mb, measured before staging
and again before publication, together with internal ceilings on the final
database, the staging artifacts and the task temporary directory. Those three
have no public keys because they are only meaningful as part of one composed
decision. Crossing any of them fails the attempt before the old database is
touched, so atomic publication is unchanged and the previous index keeps
serving.

Staging cleanup is scoped by a private per-task token, so a worker removes
only the artifacts it created and never a concurrent run's. An old database
is treated as replaceable only after an integrity verdict distinguishes real
corruption from a transient busy error. A probe that cannot complete fails
closed. Both public keys default to off.

The shell fixture that stands in for the supervisor names them too, so the
worker still recognises the policy it is handed.

Signed-off-by: 刘冲 <mail@liuchong.dev>

Author: liuchong

README.md

@@ -737,12 +737,16 @@ codebase-memory-mcp config set index_max_files 250000    # optional per-index so
 codebase-memory-mcp config set index_max_source_mb 16384 # optional per-index source-size limit
 codebase-memory-mcp config set index_max_rss_mb 8192     # optional worker-tree current RSS limit
 codebase-memory-mcp config set index_max_duration_seconds 3600 # optional total worker duration
+codebase-memory-mcp config set index_cache_max_mb 32768  # optional projected cache-size limit
+codebase-memory-mcp config set index_min_free_disk_mb 4096 # optional free-space reserve
 codebase-memory-mcp config reset auto_index              # reset to default
 ```
 
-The four `index_max_*` settings default to `off`. Exceeding one fails the complete
-index attempt rather than publishing a partial graph; an existing serving index
-is preserved. See [Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
+The six index resource settings default to `off`. Exceeding one fails the
+complete index attempt rather than publishing a partial graph; an existing
+serving index is preserved. Storage probes also fail closed when an enabled
+measurement cannot be completed. See
+[Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
 
 ### Environment Variables
 

docs/CONFIGURATION.md

@@ -93,6 +93,8 @@ Current keys:
 | `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
 | `index_max_rss_mb` | `off` | Optional maximum current RSS in MiB for the complete contained index-worker process tree (`64..1048576`). |
 | `index_max_duration_seconds` | `off` | Optional maximum total worker duration in seconds (`1..86400`). |
+| `index_cache_max_mb` | `off` | Optional maximum projected cache size in MiB after publication. |
+| `index_min_free_disk_mb` | `off` | Optional minimum free MiB reserved on the cache filesystem while indexing. |
 
 > **`watcher_enabled` vs `auto_watch`.** `watcher_enabled` controls whether the
 > watcher *subsystem* starts at all (the background poll thread). `auto_watch` is
@@ -120,13 +122,14 @@ Current keys:
 > `auto_index` still runs, and `index_repository` stays available for manual
 > reindexing.
 
-The four `index_max_*` settings are independent and disabled by default. They
-apply to explicit indexing, automatic indexing, and watcher re-indexing, but not
-to `cross-repo-intelligence`, which does not scan repository source files.
-Equality is allowed; exceeding any setting fails the complete index request and
-preserves any previously serving database. Worker RSS covers descendants and is
-not the same as the internal `CBM_MEM_BUDGET_MB` allocation budget. Total
-duration is independent of the existing 15-minute no-log-progress timeout. See
+The six index resource settings are independent and disabled by default.
+They apply to explicit indexing, automatic indexing, and watcher re-indexing,
+but not to `cross-repo-intelligence`, which does not scan or publish repository
+source indexes. Equality is allowed; exceeding a setting fails the complete
+index request and preserves any previously serving database. Worker RSS covers
+descendants and is not the same as the internal `CBM_MEM_BUDGET_MB` allocation
+budget. Total duration is independent of the existing 15-minute no-log-progress
+timeout. See
 [Index resource limits](INDEX_RESOURCE_LIMITS.md) for counting, validation, and
 error-response details.
 

docs/INDEX_RESOURCE_LIMITS.md

@@ -94,6 +94,31 @@ Worker limit failures use the same shape with `stage=worker`,
 and omit `observed`, `limit`, and `unit` because no trustworthy observation was
 available.
 
+## Storage settings
+
+| Key | Default | Accepted value | Protects |
+|---|---:|---:|---|
+| `index_cache_max_mb` | `off` | `off` or `1..1048576` | Projected cache bytes after replacement |
+| `index_min_free_disk_mb` | `off` | `off` or `1..1048576` | Free bytes reserved on the cache filesystem |
+
+Set or reset these keys through the same `config set` and `config reset`
+commands. With both keys `off`, indexing does not scan the cache tree or probe
+filesystem capacity.
+
+Projected cache usage is the current cache size, minus the old project
+database and SQLite sidecars only when that generation is confirmed valid and
+replaceable, plus the current operation's staging artifacts. Other projects
+and unrelated files always count toward the limit and are never evicted.
+
+Free space is checked before staging, after the staged build completes, and
+immediately before atomic publication. Equality is allowed. An enabled
+measurement that cannot be completed fails closed with
+`code: "resource_probe_failed"` and
+`stage: "storage"`. A limit breach uses `code: "resource_limit_exceeded"`.
+Both cases preserve the old serving database. Staging files created by a
+terminated supervised worker are tagged with a private task token and removed
+after its process tree is quiescent; cleanup cannot match another attempt.
+
 ## Trust and compatibility
 
 Limits are read from the CLI-managed `_config.db`; they are not MCP request
@@ -104,5 +129,5 @@ parent policy.
 These settings do not replace or increase `auto_index_limit`, change the 512 MiB
 single-file cap, alter workspace-root authorization, or affect
 `cross-repo-intelligence`. With all settings `off`, discovery follows the
-existing path and the supervisor performs no periodic RSS probe or total-duration
-termination.
+existing path, the supervisor performs no periodic RSS probe or total-duration
+termination, and the pipeline performs no cache-tree or free-space probe.

scripts/test-runtime.sh

@@ -113,7 +113,8 @@ _cbm_test_runtime_daemon() {
 # nothing else.
 cbm_test_index_worker_policy_json() {
     printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"'
-    printf '%s' ',"index_max_rss_mb":"off","index_max_duration_seconds":"off"}'
+    printf '%s' ',"index_max_rss_mb":"off","index_max_duration_seconds":"off"'
+    printf '%s' ',"index_cache_max_mb":"off","index_min_free_disk_mb":"off"}'
 }
 
 cbm_test_runtime_cleanup() {

src/cli/cli.c

@@ -7164,6 +7164,9 @@ static const config_key_def_t CONFIG_KEYS[] = {
     {CBM_INDEX_CONFIG_MAX_SOURCE_MB, "off", "Max accepted source MiB per index, or off"},
     {CBM_INDEX_CONFIG_MAX_RSS_MB, "off", "Max worker process-tree RSS MiB, or off"},
     {CBM_INDEX_CONFIG_MAX_DURATION_SECONDS, "off", "Max worker duration in seconds, or off"},
+    {CBM_INDEX_CONFIG_CACHE_MAX_MB, "off", "Max projected cache MiB after publish, or off"},
+    {CBM_INDEX_CONFIG_MIN_FREE_DISK_MB, "off",
+     "Minimum free cache-filesystem MiB during indexing, or off"},
 };
 
 /* #1558: ui_enabled and ui_port were reachable ONLY by hand-editing

src/daemon/application.c

@@ -307,16 +307,19 @@ static int application_worker_start_default(void *context, const char *args_json
     (void)context;
     cbm_index_resource_policy_t resource_policy;
     char error[CBM_SZ_256] = {0};
+    char task_db_path[CBM_SZ_1K];
     if (!cbm_mcp_index_policy_from_internal_args(args_json, &resource_policy, error,
-                                                 sizeof(error))) {
-        cbm_log_error("daemon.index.policy", "error", error);
+                                                 sizeof(error)) ||
+        !cbm_mcp_index_task_db_path(args_json, task_db_path, sizeof(task_db_path))) {
+        cbm_log_error("daemon.index.worker_policy", "error",
+                      error[0] ? error : "could not resolve worker index path");
         *worker_out = NULL;
         return -1;
     }
     cbm_index_worker_handle_t *worker = NULL;
-    int result =
-        cbm_index_worker_start_with_policy(args_json, memory_budget_bytes, &resource_policy, false,
-                                           marker_file, quarantine_file, &worker);
+    int result = cbm_index_worker_start_with_storage_policy(
+        args_json, memory_budget_bytes, &resource_policy, task_db_path, false, marker_file,
+        quarantine_file, NULL, NULL, &worker);
     *worker_out = worker;
     return result;
 }

src/daemon/bootstrap.c

@@ -95,6 +95,21 @@ static bool bootstrap_worker_budget_valid(const char *text) {
     return value > 0;
 }
 
+static bool bootstrap_worker_stage_token_valid(const char *token) {
+    size_t length = token ? strlen(token) : 0;
+    if (length < 6U || length >= 64U) {
+        return false;
+    }
+    for (size_t index = 0; index < length; index++) {
+        unsigned char ch = (unsigned char)token[index];
+        if (!((ch >= '0' && ch <= '9') || (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') ||
+              ch == '-' || ch == '_')) {
+            return false;
+        }
+    }
+    return true;
+}
+
 /* Keep the bootstrap role boundary exact and fail closed before any client or
  * worker state is initialized. index_supervisor owns the matching builder and
  * performs the captured-build comparison after this syntax-only classification. */
@@ -129,6 +144,12 @@ static bool bootstrap_worker_argv_exact(int argc, char *const argv[]) {
         }
         next += 2;
     }
+    if (next < argc && bootstrap_arg_is(argv[next], "--index-worker-stage-token")) {
+        if (next + 1 >= argc || !bootstrap_worker_stage_token_valid(argv[next + 1])) {
+            return false;
+        }
+        next += 2;
+    }
     return next == argc;
 }