Skip to content

Commit 8a3b8d4

Browse files
committed
feat(index): add opt-in index resource profiles
Six independent keys are an honest interface and a poor default. An operator who only wants indexing to stay within reason has to learn all six, choose a value for each, and keep them consistent as the machine changes. Add index_resource_profile, accepting off, balanced or strict. Balanced guards the host: it lets an index use what the machine can spare and stops a runaway worker before the host is exhausted. Strict holds indexing to the daemon's own budget, so a repository that needs more fails fast and attributed instead of finishing at the host's expense. Any individual key, including an explicit off, replaces that dimension of the selected profile, so a profile is a baseline rather than a lock. The balanced worker ceiling is derived from detected host memory instead of being tabled. Large-repository indexing peaks in the tens of gigabytes, so any round number low enough to feel safe would reject repositories that index successfully today. When host memory cannot be read, balanced falls back to a fixed floor. Profiles also bound directory count, entry count, traversal depth and discovery time. These have no individual keys for the same reason as the storage internals: they are only meaningful inside a composed decision. The resolved profile travels through the same trusted worker contract as the individual limits, so every entry point enforces the same decision. That contract now also carries the resolved profile and the override mask, so the shell fixture standing in for the supervisor sends both. The coverage that proves a forged policy cannot fork a second job asserts that an explicit request joins the auto-index job already running for the same root, and that assertion found the join broken on Windows. A project root reaches the daemon through the platform canonicalizer, which answers in backslash form there, while every tool handler normalizes the separators of the path it canonicalizes. One directory therefore had two spellings, and they met in comparisons that are exact: whether an index request may join the running job, and whether a watch is still live for that root. Both were wrong on Windows, in opposite directions, so aligning either one alone moved the failure rather than removing it. The session context now stores the normalized spelling, and the daemon canonicalizes project roots through a single function that normalizes on the way out. Only the storage side is directly assertable off Windows, since separators are folded on every platform; the watch-liveness side is held by the existing watcher-ownership tests, which is where the second half of this surfaced. Signed-off-by: 刘冲 <mail@liuchong.dev>
1 parent cf862a0 commit 8a3b8d4

15 files changed

Lines changed: 938 additions & 44 deletions

‎README.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -667,6 +667,7 @@ codebase-memory-mcp config list # show all settings
667667
codebase-memory-mcp config set auto_index true # auto-index on session start
668668
codebase-memory-mcp config set auto_index_limit 50000 # max files for auto-index
669669
codebase-memory-mcp config set auto_watch false # don't register background git watcher (default: true)
670+
codebase-memory-mcp config set index_resource_profile balanced # opt-in composed resource limits
670671
codebase-memory-mcp config set index_max_files 250000 # optional per-index source-file limit
671672
codebase-memory-mcp config set index_max_source_mb 16384 # optional per-index source-size limit
672673
codebase-memory-mcp config set index_max_rss_mb 8192 # optional worker-tree current RSS limit
@@ -676,10 +677,13 @@ codebase-memory-mcp config set index_min_free_disk_mb 4096 # optional free-space
676677
codebase-memory-mcp config reset auto_index # reset to default
677678
```
678679

679-
The six index resource settings default to `off`. Exceeding one fails the
680-
complete index attempt rather than publishing a partial graph; an existing
681-
serving index is preserved. Storage probes also fail closed when an enabled
682-
measurement cannot be completed. See
680+
`index_resource_profile` defaults to `off` and accepts `off`, `balanced`, or
681+
`strict`. The six individual resource settings also default to `off`; an
682+
explicit value, including `off`, replaces that dimension of a selected
683+
profile. Exceeding an effective limit fails the complete index attempt rather
684+
than publishing a partial graph, and an existing serving index is preserved.
685+
Storage probes also fail closed when an enabled measurement cannot be
686+
completed. See
683687
[Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
684688

685689
### Environment Variables

‎docs/CONFIGURATION.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,15 +86,20 @@ Current keys:
8686
|---|---|---|
8787
| `auto_index` | `false` | Automatically index new projects when an MCP session starts. |
8888
| `auto_index_limit` | `50000` | Maximum file count allowed for automatic indexing of a new project. |
89+
| `index_resource_profile` | `off` | Composed index resource baseline: `off`, `balanced`, or `strict`. |
8990
| `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |
9091
| `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
9192
| `index_max_rss_mb` | `off` | Optional maximum current RSS in MiB for the complete contained index-worker process tree (`64..1048576`). |
9293
| `index_max_duration_seconds` | `off` | Optional maximum total worker duration in seconds (`1..86400`). |
9394
| `index_cache_max_mb` | `off` | Optional maximum projected cache size in MiB after publication. |
9495
| `index_min_free_disk_mb` | `off` | Optional minimum free MiB reserved on the cache filesystem while indexing. |
9596

96-
The six index resource settings are independent and disabled by default.
97-
They apply to explicit indexing, automatic indexing, and watcher re-indexing,
97+
The profile and all six individual resource settings are disabled by default.
98+
An individual stored value replaces its profile dimension; storing `off`
99+
disables only that dimension. Profile-only limits also bound traversed
100+
directories, directory entries, depth, discovery time, final database size,
101+
staging size, and task temporary size. Effective limits apply to explicit
102+
indexing, automatic indexing, and watcher re-indexing,
98103
but not to `cross-repo-intelligence`, which does not scan or publish repository
99104
source indexes. Equality is allowed; exceeding a setting fails the complete
100105
index request and preserves any previously serving database. Worker RSS covers

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,51 @@ Index resource limits are optional operator controls for repositories whose
44
discovery breadth or worker runtime is not known in advance. They are disabled
55
by default so existing large-repository workloads retain their current behavior.
66

7+
## Resource profiles
8+
9+
`index_resource_profile` accepts `off`, `balanced`, or `strict` and defaults to
10+
`off`. A profile supplies a baseline; each explicitly stored individual
11+
resource key then replaces its dimension. An explicit `off` disables only that
12+
dimension.
13+
14+
| Dimension | `balanced` | `strict` |
15+
|---|---:|---:|
16+
| Accepted files | 500,000 | 100,000 |
17+
| Traversed directories | 250,000 | 20,000 |
18+
| Directory entries | 2,000,000 | 500,000 |
19+
| Directory depth | 128 | 64 |
20+
| Accepted source | 65,536 MiB | 4,096 MiB |
21+
| Discovery duration | 1,800 seconds | 30 seconds |
22+
| Worker duration | 7,200 seconds | 3,600 seconds |
23+
| Final database | 65,536 MiB | 16,384 MiB |
24+
| Staging artifacts | 81,920 MiB | 20,480 MiB |
25+
| Task temporary artifacts | 98,304 MiB | 24,576 MiB |
26+
| Projected cache | 131,072 MiB | 32,768 MiB |
27+
| Free-disk reserve | 4,096 MiB | 4,096 MiB |
28+
29+
The two profiles answer different questions about worker memory, so their RSS
30+
ceilings are derived rather than tabled.
31+
32+
Balanced protects the host. Its ceiling is 80% of detected host memory, which
33+
leaves an index free to use everything this machine can spare and still stops a
34+
runaway worker before the host is exhausted. A fixed number is deliberately not
35+
used here: large-repository indexing peaks in the tens of gigabytes, so any
36+
round ceiling low enough to feel safe would reject repositories that index
37+
successfully with the profile off. When host memory cannot be read, balanced
38+
falls back to the higher of twice the worker soft budget or 8,192 MiB.
39+
40+
Strict protects the budget. Its ceiling is the lower of the worker soft budget
41+
and 8,192 MiB, which is the point of the profile: a repository that needs more
42+
memory than the daemon budgets for itself fails fast with an attributed
43+
`rss_bytes` violation instead of completing at the host's expense. On a large
44+
repository this is expected to fail, and the failure names the limit and the
45+
observed value so the operator can choose `balanced`, raise
46+
`index_max_rss_mb`, or leave the profile `off`.
47+
48+
Both derived values are held inside the same 64 to 1,048,576 MiB range that
49+
`index_max_rss_mb` accepts, and detected host memory can lower but never raise
50+
an explicitly configured `index_max_rss_mb`.
51+
752
## Discovery settings
853

954
| Key | Default | Accepted value | Protects |
@@ -63,6 +108,12 @@ rules, filename and suffix filters, language detection, and the existing
63108
per-file size rule. `index_max_source_mb` sums the filesystem sizes of that same
64109
accepted set.
65110

111+
Profile directory counting includes the request root and each non-skipped
112+
directory admitted for traversal. Entry counting happens before ignore,
113+
language, and file-size filtering. Root depth is zero. The monotonic discovery
114+
deadline is checked before opening a directory and before processing each
115+
entry. Equality is allowed for every dimension.
116+
66117
Equality is allowed. The first file or byte that makes an observed value greater
67118
than its limit stops discovery. CBM discards the partial file list and does not
68119
publish a partial graph as a complete index.

‎scripts/test-runtime.sh‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -109,10 +109,13 @@ _cbm_test_runtime_daemon() {
109109
# worker in argv; a worker that finds no complete policy refuses to start rather
110110
# than index unbounded. A shell test that spawns `cli --index-worker` itself
111111
# stands in for the supervisor and owes the worker the same object. Mirrors
112-
# cbm_mcp_index_policy_add_to_args: every key in cbm_index_policy_key_at, and
113-
# nothing else.
112+
# cbm_mcp_index_policy_add_to_args: the resolved profile, the override mask
113+
# that says which keys the operator set by hand, every key in
114+
# cbm_index_policy_key_at, and nothing else.
114115
cbm_test_index_worker_policy_json() {
115-
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"'
116+
printf '%s' '"_cbm_index_policy":{"index_resource_profile":"off"'
117+
printf '%s' ',"_cbm_index_override_mask":0'
118+
printf '%s' ',"index_max_files":"off","index_max_source_mb":"off"'
116119
printf '%s' ',"index_max_rss_mb":"off","index_max_duration_seconds":"off"'
117120
printf '%s' ',"index_cache_max_mb":"off","index_min_free_disk_mb":"off"}'
118121
}

‎src/cli/cli.c‎

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
#include "foundation/platform.h"
2222
#include "foundation/constants.h"
2323
#include "foundation/log.h"
24+
#include "foundation/mem.h"
2425
#include "foundation/sha256.h"
2526
#include "cli/client_adapter.h"
2627
#include "mcp/mcp.h" // cbm_mcp_tool_input_schema — CLI flag parser + per-tool --help
@@ -6743,13 +6744,19 @@ bool cbm_config_load_index_policy(cbm_config_t *cfg, cbm_index_resource_policy_t
67436744
return false;
67446745
}
67456746
cbm_index_policy_init(policy);
6747+
const char *profile = cbm_config_get(cfg, CBM_INDEX_CONFIG_RESOURCE_PROFILE, "off");
6748+
if (!cbm_index_policy_set_profile(policy, profile, error, error_size)) {
6749+
return false;
6750+
}
67466751
for (size_t index = 0; index < cbm_index_policy_key_count(); index++) {
67476752
const char *key = cbm_index_policy_key_at(index);
6748-
const char *value = cbm_config_get(cfg, key, cbm_index_policy_default_value(key));
6749-
if (!cbm_index_policy_set(policy, key, value, error, error_size)) {
6753+
const char *value = cbm_config_get(cfg, key, NULL);
6754+
if (value && !cbm_index_policy_set(policy, key, value, error, error_size)) {
67506755
return false;
67516756
}
67526757
}
6758+
cbm_system_info_t system = cbm_system_info();
6759+
cbm_index_policy_finalize(policy, (uint64_t)system.total_ram, (uint64_t)cbm_mem_budget());
67536760
return true;
67546761
}
67556762

@@ -6775,6 +6782,7 @@ static const config_key_def_t CONFIG_KEYS[] = {
67756782
{CBM_CONFIG_UI_LANG, "auto", "Pin graph UI language: en, zh, or auto"},
67766783
{CBM_CONFIG_UI_ENABLED, "false", "Serve the graph UI on a loopback HTTP port"},
67776784
{CBM_CONFIG_UI_PORT, "9749", "Port for the graph UI listener when enabled"},
6785+
{CBM_INDEX_CONFIG_RESOURCE_PROFILE, "off", "Index resource profile: off, balanced, or strict"},
67786786
{CBM_INDEX_CONFIG_MAX_FILES, "off", "Max accepted source files per index, or off"},
67796787
{CBM_INDEX_CONFIG_MAX_SOURCE_MB, "off", "Max accepted source MiB per index, or off"},
67806788
{CBM_INDEX_CONFIG_MAX_RSS_MB, "off", "Max worker process-tree RSS MiB, or off"},
@@ -6808,19 +6816,17 @@ static bool config_key_is_ui(const char *key) {
68086816
}
68096817

68106818
static bool config_key_is_index_policy(const char *key) {
6811-
for (size_t index = 0; key && index < cbm_index_policy_key_count(); index++) {
6812-
if (strcmp(key, cbm_index_policy_key_at(index)) == 0) {
6813-
return true;
6814-
}
6815-
}
6816-
return false;
6819+
return cbm_index_policy_is_config_key(key);
68176820
}
68186821

68196822
static int config_index_policy_write(cbm_config_t *config, const char *key, const char *value) {
68206823
cbm_index_resource_policy_t candidate;
68216824
cbm_index_policy_init(&candidate);
68226825
char error[CLI_BUF_256];
6823-
if (!cbm_index_policy_set(&candidate, key, value, error, sizeof(error))) {
6826+
bool valid = strcmp(key, CBM_INDEX_CONFIG_RESOURCE_PROFILE) == 0
6827+
? cbm_index_policy_set_profile(&candidate, value, error, sizeof(error))
6828+
: cbm_index_policy_set(&candidate, key, value, error, sizeof(error));
6829+
if (!valid) {
68246830
(void)fprintf(stderr, "error: %s\n", error);
68256831
return CLI_ERR;
68266832
}

‎src/daemon/application.c‎

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -595,6 +595,20 @@ static void application_tmp_unlock(void) {
595595
atomic_flag_clear_explicit(&g_application_tmp_lock, memory_order_release);
596596
}
597597

598+
/* The daemon's only spelling of a project root. Tool handlers normalize the
599+
* separators of every path they canonicalize, so a root left in the platform's
600+
* native form is a second name for one directory — and the two names then meet
601+
* in comparisons that are exact: whether a watch is still live for this root,
602+
* and whether an index request may join the job already running for it. The
603+
* forms differ on Windows, which is where both comparisons were wrong. */
604+
static bool application_canonical_root(const char *path, char *out, size_t out_size) {
605+
if (!path || !out || out_size == 0 || !cbm_canonical_path(path, out, out_size)) {
606+
return false;
607+
}
608+
cbm_normalize_path_sep(out);
609+
return true;
610+
}
611+
598612
static bool application_cache_dir(char out[APPLICATION_PATH_CAP]) {
599613
char configured[APPLICATION_PATH_CAP] = {0};
600614
if (cbm_safe_getenv("CBM_CACHE_DIR", configured, sizeof(configured), NULL) && configured[0]) {
@@ -1935,9 +1949,32 @@ static bool application_index_args_add_policy(cbm_daemon_application_t *applicat
19351949
cbm_log_error("daemon.index.policy", "error", error);
19361950
return false;
19371951
}
1952+
cbm_system_info_t system = cbm_system_info();
1953+
cbm_index_policy_finalize(&policy, (uint64_t)system.total_ram,
1954+
application ? (uint64_t)application->worker_memory_budget_bytes : 0);
19381955
return cbm_mcp_index_policy_add_to_args(document, root, &policy);
19391956
}
19401957

1958+
static char *application_index_args_replace_policy(cbm_daemon_application_t *application,
1959+
const char *args_json) {
1960+
yyjson_doc *source = args_json ? yyjson_read(args_json, strlen(args_json), 0) : NULL;
1961+
yyjson_mut_doc *document = source ? yyjson_doc_mut_copy(source, NULL) : NULL;
1962+
yyjson_doc_free(source);
1963+
yyjson_mut_val *root = document ? yyjson_mut_doc_get_root(document) : NULL;
1964+
if (!root || !yyjson_mut_is_obj(root)) {
1965+
yyjson_mut_doc_free(document);
1966+
return NULL;
1967+
}
1968+
while (yyjson_mut_obj_get(root, "_cbm_index_policy")) {
1969+
(void)yyjson_mut_obj_remove_key(root, "_cbm_index_policy");
1970+
}
1971+
char *rewritten = application_index_args_add_policy(application, document, root)
1972+
? yyjson_mut_write(document, 0, NULL)
1973+
: NULL;
1974+
yyjson_mut_doc_free(document);
1975+
return rewritten;
1976+
}
1977+
19411978
static char *application_auto_index_args(cbm_daemon_application_t *application,
19421979
const char *root_path) {
19431980
yyjson_mut_doc *document = yyjson_mut_doc_new(NULL);
@@ -2218,14 +2255,19 @@ static char *application_index_execute(void *context, const char *root_path,
22182255
if (!session || !root_path || !args_json) {
22192256
return NULL;
22202257
}
2221-
char *project_key = application_index_project_key(root_path, args_json);
2258+
char *trusted_args = application_index_args_replace_policy(session->application, args_json);
2259+
if (!trusted_args) {
2260+
return cbm_mcp_text_result("failed to resolve daemon index resource policy", true);
2261+
}
2262+
char *project_key = application_index_project_key(root_path, trusted_args);
22222263
if (!project_key) {
2264+
free(trusted_args);
22232265
return cbm_mcp_text_result("failed to derive index project identity", true);
22242266
}
22252267
application_job_subscribe_status_t subscribe_status = APPLICATION_JOB_SUBSCRIBE_UNAVAILABLE;
22262268
cbm_daemon_application_job_t *job = NULL;
22272269
for (;;) {
2228-
job = application_job_subscribe(session->application, project_key, root_path, args_json,
2270+
job = application_job_subscribe(session->application, project_key, root_path, trusted_args,
22292271
&subscribe_status);
22302272
if (job || (subscribe_status != APPLICATION_JOB_SUBSCRIBE_BUSY &&
22312273
subscribe_status != APPLICATION_JOB_SUBSCRIBE_CANCELLING)) {
@@ -2244,11 +2286,13 @@ static char *application_index_execute(void *context, const char *root_path,
22442286
cbm_mutex_unlock(&session->application->mutex);
22452287
if (queued_cancelled) {
22462288
free(project_key);
2289+
free(trusted_args);
22472290
return cbm_mcp_text_result("index operation cancelled for this session", true);
22482291
}
22492292
cbm_usleep(APPLICATION_JOB_POLL_US);
22502293
}
22512294
free(project_key);
2295+
free(trusted_args);
22522296
if (!job) {
22532297
const char *message = "daemon index coordinator is stopping or unavailable";
22542298
if (subscribe_status == APPLICATION_JOB_SUBSCRIBE_OPTIONS_CONFLICT) {
@@ -2360,9 +2404,10 @@ static cbm_daemon_runtime_application_status_t application_set_context(
23602404
}
23612405
char canonical_root[APPLICATION_PATH_CAP] = {0};
23622406
char canonical_allowed[APPLICATION_PATH_CAP] = {0};
2363-
bool canonical = cbm_canonical_path(root, canonical_root, sizeof(canonical_root));
2407+
bool canonical = application_canonical_root(root, canonical_root, sizeof(canonical_root));
23642408
if (canonical && allowed_present) {
2365-
canonical = cbm_canonical_path(allowed, canonical_allowed, sizeof(canonical_allowed));
2409+
canonical =
2410+
application_canonical_root(allowed, canonical_allowed, sizeof(canonical_allowed));
23662411
}
23672412
struct stat root_status;
23682413
canonical =
@@ -3380,7 +3425,7 @@ static int application_background_index(cbm_daemon_application_t *application,
33803425
}
33813426
char canonical_root[APPLICATION_PATH_CAP];
33823427
struct stat root_status;
3383-
if (!cbm_canonical_path(root_path, canonical_root, sizeof(canonical_root)) ||
3428+
if (!application_canonical_root(root_path, canonical_root, sizeof(canonical_root)) ||
33843429
stat(canonical_root, &root_status) != 0 || !S_ISDIR(root_status.st_mode)) {
33853430
return -1;
33863431
}

0 commit comments

Comments
 (0)