Skip to content

Commit 21ae021

Browse files
committed
feat(index): add opt-in discovery resource limits
Indexing accepts whatever a repository contains. A tree carrying a vendored monorepo, a generated dump, or a runaway build directory is discovered in full, and the first sign of trouble is a host under memory pressure with nothing that attributes it to indexing. Add two opt-in limits evaluated during discovery against accepted source files only: index_max_files and index_max_source_mb. Both default to off, so nothing changes until an operator sets one. Crossing a limit fails the whole attempt with a structured resource_limit_exceeded result naming the resource, the observed value and the limit; no partial graph is published, and an existing serving index keeps answering. Limits are read from the CLI-managed _config.db and are not MCP request arguments. A supervised parent replaces any caller-supplied policy before spawning its worker, and the worker rejects a missing or incomplete contract, so the CLI, the daemon and the supervised worker all enforce the same decision. Both keys reach an operator through the existing config get/set/list/reset with no new subcommand. `set` suppresses its own generic message for them because the policy writer names the precise reason -- so that writer speaks on every failure it can return, including a validated value whose write then fails on a database that cannot be written. Exiting non-zero in silence is not an acceptable answer from a CLI. The two shell regressions that hand-roll the supervisor's worker argv carry that contract as well. Without it the worker exits before either guard can observe anything, and the guard would go quietly vacuous. Signed-off-by: 刘冲 <mail@liuchong.dev>
1 parent 1db8bac commit 21ae021

25 files changed

Lines changed: 1371 additions & 44 deletions

‎Makefile.cbm‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,7 @@ FOUNDATION_SRCS = \
251251
src/foundation/profile.c \
252252
src/foundation/dump_verify.c \
253253
src/foundation/limits.c \
254+
src/foundation/index_policy.c \
254255
src/foundation/subprocess.c \
255256
src/foundation/sha256.c \
256257
src/foundation/secure_random.c \
@@ -532,6 +533,7 @@ TEST_FOUNDATION_SRCS = \
532533
tests/test_str_intern.c \
533534
tests/test_log.c \
534535
tests/test_str_util.c \
536+
tests/test_index_policy.c \
535537
tests/test_workspace.c \
536538
tests/test_platform.c \
537539
tests/test_diagnostics.c \

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -733,9 +733,15 @@ codebase-memory-mcp config set auto_index true # auto-index on session
733733
codebase-memory-mcp config set auto_index_limit 50000 # max files for auto-index
734734
codebase-memory-mcp config set auto_watch false # don't register background git watcher (default: true)
735735
codebase-memory-mcp config set watcher_enabled false # stop the watcher thread entirely (default: true)
736+
codebase-memory-mcp config set index_max_files 250000 # optional per-index source-file limit
737+
codebase-memory-mcp config set index_max_source_mb 16384 # optional per-index source-size limit
736738
codebase-memory-mcp config reset auto_index # reset to default
737739
```
738740

741+
The two `index_max_*` settings default to `off`. Exceeding one fails the complete
742+
index attempt rather than publishing a partial graph; an existing serving index
743+
is preserved. See [Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
744+
739745
### Environment Variables
740746

741747
| Variable | Default | Description |

‎docs/CONFIGURATION.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@ Current keys:
8989
| `auto_index_limit` | `50000` | Maximum file count allowed for automatic indexing of a new project. |
9090
| `auto_watch` | `true` | Register the session's project with the background git watcher on connect. Set `false` to keep a session from registering its project (the watcher still runs for other projects). |
9191
| `watcher_enabled` | `true` | Master switch for the background watcher subsystem. Set `false` to stop the watcher from starting at all — no poll thread and no project registration. Reindex manually with `index_repository` when disabled. |
92+
| `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |
93+
| `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
9294

9395
> **`watcher_enabled` vs `auto_watch`.** `watcher_enabled` controls whether the
9496
> watcher *subsystem* starts at all (the background poll thread). `auto_watch` is
@@ -116,6 +118,14 @@ Current keys:
116118
> `auto_index` still runs, and `index_repository` stays available for manual
117119
> reindexing.
118120
121+
The two `index_max_*` settings are independent and disabled by default. They
122+
apply to explicit indexing, automatic indexing, and watcher re-indexing, but not
123+
to `cross-repo-intelligence`, which does not scan repository source files.
124+
Equality is allowed; exceeding either setting fails the complete index request
125+
and preserves any previously serving database. See
126+
[Index resource limits](INDEX_RESOURCE_LIMITS.md) for counting, validation, and
127+
error-response details.
128+
119129
## 3. UI Settings
120130
121131
The optional built-in graph UI stores its settings in:

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Index resource limits
2+
3+
Index resource limits are optional operator controls for repositories whose
4+
discovery breadth is not known in advance. They are disabled by default so
5+
existing large-repository workloads retain their current behavior.
6+
7+
## Discovery settings
8+
9+
| Key | Default | Accepted value | Protects |
10+
|---|---:|---:|---|
11+
| `index_max_files` | `off` | `off` or `1..10000000` | Accepted source-file count |
12+
| `index_max_source_mb` | `off` | `off` or `1..1048576` | Accepted source-file bytes |
13+
14+
Set or reset them with the normal configuration command:
15+
16+
```bash
17+
codebase-memory-mcp config set index_max_files 250000
18+
codebase-memory-mcp config set index_max_source_mb 16384
19+
codebase-memory-mcp config reset index_max_files
20+
```
21+
22+
Values use base-10 integers. MiB means 1,048,576 bytes. Empty values, zero,
23+
negative values, suffixes, trailing characters, and values outside the stated
24+
ranges are rejected without changing the stored value.
25+
26+
## Counting and failure semantics
27+
28+
`index_max_files` counts a file only after it passes directory pruning, ignore
29+
rules, filename and suffix filters, language detection, and the existing
30+
per-file size rule. `index_max_source_mb` sums the filesystem sizes of that same
31+
accepted set.
32+
33+
Equality is allowed. The first file or byte that makes an observed value greater
34+
than its limit stops discovery. CBM discards the partial file list and does not
35+
publish a partial graph as a complete index.
36+
37+
For an explicit MCP request the error payload contains:
38+
39+
```json
40+
{
41+
"status": "error",
42+
"code": "resource_limit_exceeded",
43+
"stage": "discovery",
44+
"resource": "files",
45+
"observed": 250001,
46+
"limit": 250000,
47+
"unit": "files",
48+
"retryable": true,
49+
"serving_index_preserved": true,
50+
"message": "Index discovery exceeded index_max_files"
51+
}
52+
```
53+
54+
The previous database remains available because publication occurs only after a
55+
complete discovery and successful staged build. If no previous database exists,
56+
`serving_index_preserved` is false.
57+
58+
## Trust and compatibility
59+
60+
Limits are read from the CLI-managed `_config.db`; they are not MCP request
61+
arguments. A supervised parent replaces any caller-supplied internal policy
62+
before spawning its worker, and the worker rejects a missing or incomplete
63+
parent policy.
64+
65+
These settings do not replace or increase `auto_index_limit`, change the 512 MiB
66+
single-file cap, alter workspace-root authorization, or affect
67+
`cross-repo-intelligence`. With both settings `off`, discovery follows the
68+
existing unbounded path.

‎scripts/test-runtime.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,16 @@ _cbm_test_runtime_daemon() {
105105
CBM_CACHE_DIR="$_CBM_TEST_RUNTIME_PRODUCT_CACHE" "$1" daemon "$2"
106106
}
107107

108+
# The supervisor resolves one resource policy per index and hands it to the
109+
# worker in argv; a worker that finds no complete policy refuses to start rather
110+
# than index unbounded. A shell test that spawns `cli --index-worker` itself
111+
# stands in for the supervisor and owes the worker the same object. Mirrors
112+
# cbm_mcp_index_policy_add_to_args: every key in cbm_index_policy_key_at, and
113+
# nothing else.
114+
cbm_test_index_worker_policy_json() {
115+
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"}'
116+
}
117+
108118
cbm_test_runtime_cleanup() {
109119
local binary="${1:-}" root="$_CBM_TEST_RUNTIME_CREATED_ROOT"
110120
local name="${_CBM_TEST_RUNTIME_CREATED_ROOT##*/}" runtime_entry="" active=0

‎src/cli/cli.c‎

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7117,6 +7117,25 @@ bool cbm_config_watcher_enabled(cbm_config_t *cfg) {
71177117
return cbm_config_get_bool(cfg, CBM_CONFIG_WATCHER_ENABLED, true);
71187118
}
71197119

7120+
bool cbm_config_load_index_policy(cbm_config_t *cfg, cbm_index_resource_policy_t *policy,
7121+
char *error, size_t error_size) {
7122+
if (!cfg || !policy) {
7123+
if (error && error_size > 0) {
7124+
(void)snprintf(error, error_size, "index resource configuration is unavailable");
7125+
}
7126+
return false;
7127+
}
7128+
cbm_index_policy_init(policy);
7129+
for (size_t index = 0; index < cbm_index_policy_key_count(); index++) {
7130+
const char *key = cbm_index_policy_key_at(index);
7131+
const char *value = cbm_config_get(cfg, key, cbm_index_policy_default_value(key));
7132+
if (!cbm_index_policy_set(policy, key, value, error, error_size)) {
7133+
return false;
7134+
}
7135+
}
7136+
return true;
7137+
}
7138+
71207139
/* ── Config CLI subcommand ────────────────────────────────────── */
71217140

71227141
/* THE config-key table. list, get, help, and key validation all read this one
@@ -7141,6 +7160,8 @@ static const config_key_def_t CONFIG_KEYS[] = {
71417160
{CBM_CONFIG_UI_LANG, "auto", "Pin graph UI language: en, zh, or auto"},
71427161
{CBM_CONFIG_UI_ENABLED, "false", "Serve the graph UI on a loopback HTTP port"},
71437162
{CBM_CONFIG_UI_PORT, "9749", "Port for the graph UI listener when enabled"},
7163+
{CBM_INDEX_CONFIG_MAX_FILES, "off", "Max accepted source files per index, or off"},
7164+
{CBM_INDEX_CONFIG_MAX_SOURCE_MB, "off", "Max accepted source MiB per index, or off"},
71447165
};
71457166

71467167
/* #1558: ui_enabled and ui_port were reachable ONLY by hand-editing
@@ -7166,6 +7187,31 @@ static bool config_key_is_ui(const char *key) {
71667187
return key && (strcmp(key, CBM_CONFIG_UI_ENABLED) == 0 || strcmp(key, CBM_CONFIG_UI_PORT) == 0);
71677188
}
71687189

7190+
static bool config_key_is_index_policy(const char *key) {
7191+
return key && (strcmp(key, CBM_INDEX_CONFIG_MAX_FILES) == 0 ||
7192+
strcmp(key, CBM_INDEX_CONFIG_MAX_SOURCE_MB) == 0);
7193+
}
7194+
7195+
static int config_index_policy_write(cbm_config_t *config, const char *key, const char *value) {
7196+
cbm_index_resource_policy_t candidate;
7197+
cbm_index_policy_init(&candidate);
7198+
char error[CLI_BUF_256];
7199+
if (!cbm_index_policy_set(&candidate, key, value, error, sizeof(error))) {
7200+
(void)fprintf(stderr, "error: %s\n", error);
7201+
return CLI_ERR;
7202+
}
7203+
int rc = cbm_config_set(config, key, value);
7204+
if (rc != 0) {
7205+
/* The caller suppresses its own message for policy keys because this
7206+
* helper names the precise reason. That is only true if the helper
7207+
* speaks on every failure it can return: a validated value whose write
7208+
* then fails -- a locked or read-only _config.db -- used to exit
7209+
* non-zero having printed nothing at all. */
7210+
(void)fprintf(stderr, "error: failed to set %s\n", key);
7211+
}
7212+
return rc;
7213+
}
7214+
71697215
static void config_ui_read(const char *key, char *out, size_t out_sz) {
71707216
cbm_ui_config_t ui;
71717217
cbm_ui_config_load(&ui);
@@ -7303,10 +7349,16 @@ int cbm_cmd_config(int argc, char **argv) {
73037349
rc = CLI_TRUE;
73047350
}
73057351
} else {
7306-
if (cbm_config_set(cfg, argv[CLI_SKIP_ONE], argv[CLI_PAIR_LEN]) == 0) {
7352+
int set_rc =
7353+
config_key_is_index_policy(argv[CLI_SKIP_ONE])
7354+
? config_index_policy_write(cfg, argv[CLI_SKIP_ONE], argv[CLI_PAIR_LEN])
7355+
: cbm_config_set(cfg, argv[CLI_SKIP_ONE], argv[CLI_PAIR_LEN]);
7356+
if (set_rc == 0) {
73077357
printf("%s = %s\n", argv[CLI_SKIP_ONE], argv[CLI_PAIR_LEN]);
73087358
} else {
7309-
(void)fprintf(stderr, "error: failed to set %s\n", argv[CLI_SKIP_ONE]);
7359+
if (!config_key_is_index_policy(argv[CLI_SKIP_ONE])) {
7360+
(void)fprintf(stderr, "error: failed to set %s\n", argv[CLI_SKIP_ONE]);
7361+
}
73107362
rc = CLI_TRUE;
73117363
}
73127364
}

‎src/cli/cli.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@
1313
#include <stddef.h>
1414
#include <stdint.h>
1515

16+
#include "foundation/index_policy.h"
17+
1618
typedef struct cbm_mcp_server cbm_mcp_server_t;
1719

1820
/* ── Version ──────────────────────────────────────────────────── */
@@ -420,6 +422,11 @@ int cbm_config_set(cbm_config_t *cfg, const char *key, const char *value);
420422
/* Delete a config key. Returns 0 on success. */
421423
int cbm_config_delete(cbm_config_t *cfg, const char *key);
422424

425+
/* Load and validate the operator-controlled discovery policy. Invalid stored
426+
* values fail closed instead of silently disabling a guard. */
427+
bool cbm_config_load_index_policy(cbm_config_t *cfg, cbm_index_resource_policy_t *policy,
428+
char *error, size_t error_size);
429+
423430
/* Well-known config keys */
424431
#define CBM_CONFIG_AUTO_INDEX "auto_index"
425432
#define CBM_CONFIG_AUTO_INDEX_LIMIT "auto_index_limit"

‎src/daemon/application.c‎

Lines changed: 29 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,8 @@ static bool application_unique_recovery_file(char out[APPLICATION_PATH_CAP], con
208208
static bool application_update_reap(cbm_daemon_application_t *application, bool wait,
209209
uint32_t timeout_ms);
210210
static void *application_job_thread(void *opaque);
211-
static char *application_auto_index_args(const char *root_path);
211+
static char *application_auto_index_args(cbm_daemon_application_t *application,
212+
const char *root_path);
212213
static cbm_daemon_application_job_t *application_job_subscribe_locked(
213214
cbm_daemon_application_t *application, const char *project_key, const char *root_path,
214215
const char *args_json, application_job_subscribe_status_t *status_out);
@@ -1454,7 +1455,7 @@ static void application_auto_index_retry_pending_locked(cbm_daemon_application_t
14541455
application_refresh_watch_locked(session);
14551456
continue;
14561457
}
1457-
char *args = application_auto_index_args(root_path);
1458+
char *args = application_auto_index_args(application, root_path);
14581459
if (!args) {
14591460
continue;
14601461
}
@@ -2004,15 +2005,36 @@ static bool application_update_reap(cbm_daemon_application_t *application, bool
20042005
}
20052006
}
20062007

2007-
static char *application_auto_index_args(const char *root_path) {
2008+
static bool application_index_args_add_policy(cbm_daemon_application_t *application,
2009+
yyjson_mut_doc *document, yyjson_mut_val *root) {
2010+
cbm_config_t *owned_config = NULL;
2011+
cbm_config_t *config = application ? application->config : NULL;
2012+
if (!config) {
2013+
owned_config = cbm_config_open(cbm_resolve_cache_dir());
2014+
config = owned_config;
2015+
}
2016+
cbm_index_resource_policy_t policy;
2017+
char error[CBM_SZ_256] = {0};
2018+
bool loaded = cbm_config_load_index_policy(config, &policy, error, sizeof(error));
2019+
cbm_config_close(owned_config);
2020+
if (!loaded) {
2021+
cbm_log_error("daemon.index.policy", "error", error);
2022+
return false;
2023+
}
2024+
return cbm_mcp_index_policy_add_to_args(document, root, &policy);
2025+
}
2026+
2027+
static char *application_auto_index_args(cbm_daemon_application_t *application,
2028+
const char *root_path) {
20082029
yyjson_mut_doc *document = yyjson_mut_doc_new(NULL);
20092030
yyjson_mut_val *root = document ? yyjson_mut_obj(document) : NULL;
20102031
if (!document || !root) {
20112032
yyjson_mut_doc_free(document);
20122033
return NULL;
20132034
}
20142035
yyjson_mut_doc_set_root(document, root);
2015-
char *args = yyjson_mut_obj_add_strcpy(document, root, "repo_path", root_path)
2036+
char *args = yyjson_mut_obj_add_strcpy(document, root, "repo_path", root_path) &&
2037+
application_index_args_add_policy(application, document, root)
20162038
? yyjson_mut_write(document, 0, NULL)
20172039
: NULL;
20182040
yyjson_mut_doc_free(document);
@@ -2059,7 +2081,7 @@ static void application_background_initialize_impl(cbm_daemon_application_sessio
20592081
files);
20602082
}
20612083
bool args_required = auto_index_candidate && within_auto_index_limit;
2062-
char *args = args_required ? application_auto_index_args(root_path) : NULL;
2084+
char *args = args_required ? application_auto_index_args(application, root_path) : NULL;
20632085
application_jobs_reap_completed(application);
20642086
cbm_mutex_lock(&application->mutex);
20652087
if (application->stopping || application_request_cancelled_locked(session)) {
@@ -3457,7 +3479,8 @@ static int application_background_index(cbm_daemon_application_t *application,
34573479
return -1;
34583480
}
34593481
yyjson_mut_doc_set_root(document, root);
3460-
bool encoded = yyjson_mut_obj_add_strcpy(document, root, "repo_path", canonical_root);
3482+
bool encoded = yyjson_mut_obj_add_strcpy(document, root, "repo_path", canonical_root) &&
3483+
application_index_args_add_policy(application, document, root);
34613484
char *default_project = cbm_project_name_from_path(canonical_root);
34623485
bool custom_project =
34633486
project_name[0] && (!default_project || strcmp(default_project, project_name) != 0);

0 commit comments

Comments
 (0)