Skip to content

Commit 87da442

Browse files
committed
feat(index): add worker resource watchdogs
Discovery limits bound what indexing accepts, not what it then costs. A repository well inside those bounds can still exhaust the host through parser memory, or simply never finish, and a supervised worker that hangs leaves the parent waiting with nothing to report. Add index_max_rss_mb and index_max_duration_seconds, enforced by the parent against the worker process tree rather than the worker process alone, so a runaway child cannot hide behind a small parent. Resident memory is sampled through the platform interface on macOS, Linux and Windows. Crossing a limit terminates the tree and yields one trusted, structured terminal result that attributes the failure to the resource that caused it. Both limits default to off. A measurement that cannot be taken fails the attempt instead of passing it: a watchdog that quietly stops watching is worse than no watchdog at all. The shell fixture that stands in for the supervisor names the two new keys. The worker accepts only a policy that spells out every key it knows, which is what keeps a stale supervisor from starting a worker it cannot bound. Signed-off-by: 刘冲 <mail@liuchong.dev>
1 parent 21ae021 commit 87da442

21 files changed

Lines changed: 1452 additions & 98 deletions

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -735,10 +735,12 @@ codebase-memory-mcp config set auto_watch false # don't register backgr
735735
codebase-memory-mcp config set watcher_enabled false # stop the watcher thread entirely (default: true)
736736
codebase-memory-mcp config set index_max_files 250000 # optional per-index source-file limit
737737
codebase-memory-mcp config set index_max_source_mb 16384 # optional per-index source-size limit
738+
codebase-memory-mcp config set index_max_rss_mb 8192 # optional worker-tree current RSS limit
739+
codebase-memory-mcp config set index_max_duration_seconds 3600 # optional total worker duration
738740
codebase-memory-mcp config reset auto_index # reset to default
739741
```
740742

741-
The two `index_max_*` settings default to `off`. Exceeding one fails the complete
743+
The four `index_max_*` settings default to `off`. Exceeding one fails the complete
742744
index attempt rather than publishing a partial graph; an existing serving index
743745
is preserved. See [Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
744746

‎docs/CONFIGURATION.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,8 @@ Current keys:
9191
| `watcher_enabled` | `true` | Master switch for the background watcher subsystem. Set `false` to stop the watcher from starting at all — no poll thread and no project registration. Reindex manually with `index_repository` when disabled. |
9292
| `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |
9393
| `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
94+
| `index_max_rss_mb` | `off` | Optional maximum current RSS in MiB for the complete contained index-worker process tree (`64..1048576`). |
95+
| `index_max_duration_seconds` | `off` | Optional maximum total worker duration in seconds (`1..86400`). |
9496

9597
> **`watcher_enabled` vs `auto_watch`.** `watcher_enabled` controls whether the
9698
> watcher *subsystem* starts at all (the background poll thread). `auto_watch` is
@@ -118,11 +120,13 @@ Current keys:
118120
> `auto_index` still runs, and `index_repository` stays available for manual
119121
> reindexing.
120122
121-
The two `index_max_*` settings are independent and disabled by default. They
123+
The four `index_max_*` settings are independent and disabled by default. They
122124
apply to explicit indexing, automatic indexing, and watcher re-indexing, but not
123125
to `cross-repo-intelligence`, which does not scan repository source files.
124-
Equality is allowed; exceeding either setting fails the complete index request
125-
and preserves any previously serving database. See
126+
Equality is allowed; exceeding any setting fails the complete index request and
127+
preserves any previously serving database. Worker RSS covers descendants and is
128+
not the same as the internal `CBM_MEM_BUDGET_MB` allocation budget. Total
129+
duration is independent of the existing 15-minute no-log-progress timeout. See
126130
[Index resource limits](INDEX_RESOURCE_LIMITS.md) for counting, validation, and
127131
error-response details.
128132

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Index resource limits
22

33
Index resource limits are optional operator controls for repositories whose
4-
discovery breadth is not known in advance. They are disabled by default so
5-
existing large-repository workloads retain their current behavior.
4+
discovery breadth or worker runtime is not known in advance. They are disabled
5+
by default so existing large-repository workloads retain their current behavior.
66

77
## Discovery settings
88

@@ -23,6 +23,39 @@ Values use base-10 integers. MiB means 1,048,576 bytes. Empty values, zero,
2323
negative values, suffixes, trailing characters, and values outside the stated
2424
ranges are rejected without changing the stored value.
2525

26+
## Worker settings
27+
28+
| Key | Default | Accepted value | Protects |
29+
|---|---:|---:|---|
30+
| `index_max_rss_mb` | `off` | `off` or `64..1048576` | Current RSS of the complete worker process tree |
31+
| `index_max_duration_seconds` | `off` | `off` or `1..86400` | Total worker wall-clock duration |
32+
33+
```bash
34+
codebase-memory-mcp config set index_max_rss_mb 8192
35+
codebase-memory-mcp config set index_max_duration_seconds 3600
36+
```
37+
38+
RSS is the current resident memory of the contained worker and every descendant,
39+
not the worker's allocation budget and not peak memory. This hard watchdog is
40+
separate from the internal `CBM_MEM_BUDGET_MB` soft budget. The supervisor
41+
samples RSS at most once every 250 milliseconds so the watchdog does not turn
42+
full process-table enumeration into a busy loop.
43+
44+
Duration uses a monotonic clock from successful spawn. It is independent of the
45+
existing 15-minute quiet timeout: continuous log progress does not reset total
46+
duration, while the quiet timeout continues to identify a worker that stops
47+
making progress.
48+
49+
Equality is allowed. The first RSS or elapsed-duration observation above its
50+
limit starts the existing graceful-to-force process-tree shutdown. CBM reports
51+
terminal only after the tree is quiescent or a bounded containment failure is
52+
explicitly surfaced. Resource termination is not retried and does not
53+
quarantine a source file.
54+
55+
If RSS is enabled and three consecutive probes cannot obtain any trustworthy
56+
tree measurement while the root worker is still running, CBM fails closed with
57+
`code=resource_probe_failed`.
58+
2659
## Counting and failure semantics
2760

2861
`index_max_files` counts a file only after it passes directory pruning, ignore
@@ -55,6 +88,12 @@ The previous database remains available because publication occurs only after a
5588
complete discovery and successful staged build. If no previous database exists,
5689
`serving_index_preserved` is false.
5790

91+
Worker limit failures use the same shape with `stage=worker`,
92+
`resource=rss_bytes` and `unit=bytes`, or `resource=duration_ms` and
93+
`unit=milliseconds`. RSS measurement failures use `code=resource_probe_failed`
94+
and omit `observed`, `limit`, and `unit` because no trustworthy observation was
95+
available.
96+
5897
## Trust and compatibility
5998

6099
Limits are read from the CLI-managed `_config.db`; they are not MCP request
@@ -64,5 +103,6 @@ parent policy.
64103

65104
These settings do not replace or increase `auto_index_limit`, change the 512 MiB
66105
single-file cap, alter workspace-root authorization, or affect
67-
`cross-repo-intelligence`. With both settings `off`, discovery follows the
68-
existing unbounded path.
106+
`cross-repo-intelligence`. With all settings `off`, discovery follows the
107+
existing path and the supervisor performs no periodic RSS probe or total-duration
108+
termination.

‎scripts/test-runtime.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,8 @@ _cbm_test_runtime_daemon() {
112112
# cbm_mcp_index_policy_add_to_args: every key in cbm_index_policy_key_at, and
113113
# nothing else.
114114
cbm_test_index_worker_policy_json() {
115-
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"}'
115+
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"'
116+
printf '%s' ',"index_max_rss_mb":"off","index_max_duration_seconds":"off"}'
116117
}
117118

118119
cbm_test_runtime_cleanup() {

‎src/cli/cli.c‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7162,6 +7162,8 @@ static const config_key_def_t CONFIG_KEYS[] = {
71627162
{CBM_CONFIG_UI_PORT, "9749", "Port for the graph UI listener when enabled"},
71637163
{CBM_INDEX_CONFIG_MAX_FILES, "off", "Max accepted source files per index, or off"},
71647164
{CBM_INDEX_CONFIG_MAX_SOURCE_MB, "off", "Max accepted source MiB per index, or off"},
7165+
{CBM_INDEX_CONFIG_MAX_RSS_MB, "off", "Max worker process-tree RSS MiB, or off"},
7166+
{CBM_INDEX_CONFIG_MAX_DURATION_SECONDS, "off", "Max worker duration in seconds, or off"},
71657167
};
71667168

71677169
/* #1558: ui_enabled and ui_port were reachable ONLY by hand-editing
@@ -7188,8 +7190,12 @@ static bool config_key_is_ui(const char *key) {
71887190
}
71897191

71907192
static bool config_key_is_index_policy(const char *key) {
7191-
return key && (strcmp(key, CBM_INDEX_CONFIG_MAX_FILES) == 0 ||
7192-
strcmp(key, CBM_INDEX_CONFIG_MAX_SOURCE_MB) == 0);
7193+
for (size_t index = 0; key && index < cbm_index_policy_key_count(); index++) {
7194+
if (strcmp(key, cbm_index_policy_key_at(index)) == 0) {
7195+
return true;
7196+
}
7197+
}
7198+
return false;
71937199
}
71947200

71957201
static int config_index_policy_write(cbm_config_t *config, const char *key, const char *value) {

‎src/daemon/application.c‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -305,9 +305,18 @@ static int application_worker_start_default(void *context, const char *args_json
305305
const char *quarantine_file,
306306
cbm_daemon_application_worker_t *worker_out) {
307307
(void)context;
308+
cbm_index_resource_policy_t resource_policy;
309+
char error[CBM_SZ_256] = {0};
310+
if (!cbm_mcp_index_policy_from_internal_args(args_json, &resource_policy, error,
311+
sizeof(error))) {
312+
cbm_log_error("daemon.index.policy", "error", error);
313+
*worker_out = NULL;
314+
return -1;
315+
}
308316
cbm_index_worker_handle_t *worker = NULL;
309-
int result = cbm_index_worker_start(args_json, memory_budget_bytes, false, marker_file,
310-
quarantine_file, &worker);
317+
int result =
318+
cbm_index_worker_start_with_policy(args_json, memory_budget_bytes, &resource_policy, false,
319+
marker_file, quarantine_file, &worker);
311320
*worker_out = worker;
312321
return result;
313322
}
@@ -1283,6 +1292,12 @@ static application_attempt_decision_t application_consume_attempt(
12831292
application_attempt_free(attempt);
12841293
return APPLICATION_ATTEMPT_DECISION_SUCCESS;
12851294
}
1295+
if (disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_FAILURE) {
1296+
execution->response =
1297+
cbm_mcp_index_worker_resource_response(job->args_json, &attempt->result);
1298+
application_attempt_free(attempt);
1299+
return APPLICATION_ATTEMPT_DECISION_STOP;
1300+
}
12861301
if (disposition == CBM_MCP_SUPERVISED_RESULT_UNSAFE_TERMINAL) {
12871302
execution->unsafe_terminal = true;
12881303
execution->supervision_failed =

‎src/foundation/index_policy.c‎

Lines changed: 89 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,38 @@
11
#include "foundation/index_policy.h"
22

3+
#include <stddef.h>
34
#include <stdio.h>
45
#include <string.h>
56

6-
static const char *const INDEX_POLICY_KEYS[] = {
7-
CBM_INDEX_CONFIG_MAX_FILES,
8-
CBM_INDEX_CONFIG_MAX_SOURCE_MB,
7+
typedef struct {
8+
const char *key;
9+
size_t field_offset;
10+
uint64_t minimum;
11+
uint64_t maximum;
12+
uint64_t multiplier;
13+
} index_policy_metadata_t;
14+
15+
static const index_policy_metadata_t INDEX_POLICY_METADATA[] = {
16+
{CBM_INDEX_CONFIG_MAX_FILES, offsetof(cbm_index_resource_policy_t, max_files), 1,
17+
CBM_INDEX_MAX_FILES_VALUE, 1},
18+
{CBM_INDEX_CONFIG_MAX_SOURCE_MB, offsetof(cbm_index_resource_policy_t, max_source_bytes), 1,
19+
CBM_INDEX_MAX_SOURCE_MB_VALUE, CBM_INDEX_MIB_BYTES},
20+
{CBM_INDEX_CONFIG_MAX_RSS_MB, offsetof(cbm_index_resource_policy_t, max_rss_bytes),
21+
CBM_INDEX_MIN_RSS_MB_VALUE, CBM_INDEX_MAX_RSS_MB_VALUE, CBM_INDEX_MIB_BYTES},
22+
{CBM_INDEX_CONFIG_MAX_DURATION_SECONDS, offsetof(cbm_index_resource_policy_t, max_duration_ms),
23+
1, CBM_INDEX_MAX_DURATION_SECONDS_VALUE, UINT64_C(1000)},
924
};
1025

11-
static void set_error(char *error, size_t error_size, const char *key, uint64_t maximum) {
26+
static void set_error(char *error, size_t error_size, const char *key, uint64_t minimum,
27+
uint64_t maximum) {
1228
if (error && error_size > 0) {
13-
(void)snprintf(error, error_size, "%s must be off or an integer from 1 to %llu", key,
14-
(unsigned long long)maximum);
29+
(void)snprintf(error, error_size, "%s must be off or an integer from %llu to %llu", key,
30+
(unsigned long long)minimum, (unsigned long long)maximum);
1531
}
1632
}
1733

18-
static bool parse_bounded_uint64(const char *value, uint64_t maximum, uint64_t *parsed) {
34+
static bool parse_bounded_uint64(const char *value, uint64_t minimum, uint64_t maximum,
35+
uint64_t *parsed) {
1936
if (!value || !value[0] || !parsed) {
2037
return false;
2138
}
@@ -30,7 +47,7 @@ static bool parse_bounded_uint64(const char *value, uint64_t maximum, uint64_t *
3047
}
3148
result = result * 10U + digit;
3249
}
33-
if (result == 0) {
50+
if (result < minimum) {
3451
return false;
3552
}
3653
*parsed = result;
@@ -44,20 +61,39 @@ void cbm_index_policy_init(cbm_index_resource_policy_t *policy) {
4461
}
4562

4663
bool cbm_index_policy_enabled(const cbm_index_resource_policy_t *policy) {
64+
if (!policy) {
65+
return false;
66+
}
67+
for (size_t index = 0; index < cbm_index_policy_key_count(); index++) {
68+
const cbm_index_limit_u64_t *limit =
69+
(const cbm_index_limit_u64_t *)((const unsigned char *)policy +
70+
INDEX_POLICY_METADATA[index].field_offset);
71+
if (limit->enabled) {
72+
return true;
73+
}
74+
}
75+
return false;
76+
}
77+
78+
bool cbm_index_policy_discovery_enabled(const cbm_index_resource_policy_t *policy) {
4779
return policy && (policy->max_files.enabled || policy->max_source_bytes.enabled);
4880
}
4981

82+
bool cbm_index_policy_worker_enabled(const cbm_index_resource_policy_t *policy) {
83+
return policy && (policy->max_rss_bytes.enabled || policy->max_duration_ms.enabled);
84+
}
85+
5086
size_t cbm_index_policy_key_count(void) {
51-
return sizeof(INDEX_POLICY_KEYS) / sizeof(INDEX_POLICY_KEYS[0]);
87+
return sizeof(INDEX_POLICY_METADATA) / sizeof(INDEX_POLICY_METADATA[0]);
5288
}
5389

5490
const char *cbm_index_policy_key_at(size_t index) {
55-
return index < cbm_index_policy_key_count() ? INDEX_POLICY_KEYS[index] : NULL;
91+
return index < cbm_index_policy_key_count() ? INDEX_POLICY_METADATA[index].key : NULL;
5692
}
5793

5894
const char *cbm_index_policy_default_value(const char *key) {
5995
for (size_t index = 0; index < cbm_index_policy_key_count(); index++) {
60-
if (key && strcmp(key, INDEX_POLICY_KEYS[index]) == 0) {
96+
if (key && strcmp(key, INDEX_POLICY_METADATA[index].key) == 0) {
6197
return "off";
6298
}
6399
}
@@ -70,34 +106,33 @@ bool cbm_index_policy_set(cbm_index_resource_policy_t *policy, const char *key,
70106
error[0] = '\0';
71107
}
72108
if (!policy || !key || !value) {
73-
set_error(error, error_size, key ? key : "index resource limit", 0);
109+
set_error(error, error_size, key ? key : "index resource limit", 0, 0);
74110
return false;
75111
}
76112

77-
cbm_index_limit_u64_t *target = NULL;
78-
uint64_t maximum = 0;
79-
uint64_t multiplier = 1;
80-
if (strcmp(key, CBM_INDEX_CONFIG_MAX_FILES) == 0) {
81-
target = &policy->max_files;
82-
maximum = CBM_INDEX_MAX_FILES_VALUE;
83-
} else if (strcmp(key, CBM_INDEX_CONFIG_MAX_SOURCE_MB) == 0) {
84-
target = &policy->max_source_bytes;
85-
maximum = CBM_INDEX_MAX_SOURCE_MB_VALUE;
86-
multiplier = CBM_INDEX_MIB_BYTES;
87-
} else {
88-
set_error(error, error_size, key, 0);
113+
const index_policy_metadata_t *metadata = NULL;
114+
for (size_t index = 0; index < cbm_index_policy_key_count(); index++) {
115+
if (strcmp(key, INDEX_POLICY_METADATA[index].key) == 0) {
116+
metadata = &INDEX_POLICY_METADATA[index];
117+
break;
118+
}
119+
}
120+
if (!metadata) {
121+
set_error(error, error_size, key, 0, 0);
89122
return false;
90123
}
124+
cbm_index_limit_u64_t *target =
125+
(cbm_index_limit_u64_t *)((unsigned char *)policy + metadata->field_offset);
91126

92127
cbm_index_limit_u64_t candidate = {0};
93128
if (strcmp(value, "off") != 0) {
94129
uint64_t parsed = 0;
95-
if (!parse_bounded_uint64(value, maximum, &parsed)) {
96-
set_error(error, error_size, key, maximum);
130+
if (!parse_bounded_uint64(value, metadata->minimum, metadata->maximum, &parsed)) {
131+
set_error(error, error_size, key, metadata->minimum, metadata->maximum);
97132
return false;
98133
}
99134
candidate.enabled = true;
100-
candidate.value = parsed * multiplier;
135+
candidate.value = parsed * metadata->multiplier;
101136
}
102137
*target = candidate;
103138
return true;
@@ -108,18 +143,21 @@ bool cbm_index_policy_format(const cbm_index_resource_policy_t *policy, const ch
108143
if (!policy || !key || !out || out_size == 0) {
109144
return false;
110145
}
111-
const cbm_index_limit_u64_t *limit = NULL;
112-
uint64_t divisor = 1;
113-
if (strcmp(key, CBM_INDEX_CONFIG_MAX_FILES) == 0) {
114-
limit = &policy->max_files;
115-
} else if (strcmp(key, CBM_INDEX_CONFIG_MAX_SOURCE_MB) == 0) {
116-
limit = &policy->max_source_bytes;
117-
divisor = CBM_INDEX_MIB_BYTES;
118-
} else {
146+
const index_policy_metadata_t *metadata = NULL;
147+
for (size_t index = 0; index < cbm_index_policy_key_count(); index++) {
148+
if (strcmp(key, INDEX_POLICY_METADATA[index].key) == 0) {
149+
metadata = &INDEX_POLICY_METADATA[index];
150+
break;
151+
}
152+
}
153+
if (!metadata) {
119154
return false;
120155
}
156+
const cbm_index_limit_u64_t *limit =
157+
(const cbm_index_limit_u64_t *)((const unsigned char *)policy + metadata->field_offset);
121158
int length = limit->enabled
122-
? snprintf(out, out_size, "%llu", (unsigned long long)(limit->value / divisor))
159+
? snprintf(out, out_size, "%llu",
160+
(unsigned long long)(limit->value / metadata->multiplier))
123161
: snprintf(out, out_size, "off");
124162
return length >= 0 && (size_t)length < out_size;
125163
}
@@ -130,14 +168,24 @@ const char *cbm_index_resource_name(cbm_index_resource_t resource) {
130168
return "files";
131169
case CBM_INDEX_RESOURCE_SOURCE_BYTES:
132170
return "source_bytes";
171+
case CBM_INDEX_RESOURCE_RSS_BYTES:
172+
return "rss_bytes";
173+
case CBM_INDEX_RESOURCE_DURATION_MS:
174+
return "duration_ms";
133175
case CBM_INDEX_RESOURCE_NONE:
134176
default:
135177
return "unknown";
136178
}
137179
}
138180

139181
const char *cbm_index_resource_unit(cbm_index_resource_t resource) {
140-
return resource == CBM_INDEX_RESOURCE_FILES ? "files" : "bytes";
182+
if (resource == CBM_INDEX_RESOURCE_FILES) {
183+
return "files";
184+
}
185+
if (resource == CBM_INDEX_RESOURCE_DURATION_MS) {
186+
return "milliseconds";
187+
}
188+
return "bytes";
141189
}
142190

143191
const char *cbm_index_resource_config_key(cbm_index_resource_t resource) {
@@ -146,6 +194,10 @@ const char *cbm_index_resource_config_key(cbm_index_resource_t resource) {
146194
return CBM_INDEX_CONFIG_MAX_FILES;
147195
case CBM_INDEX_RESOURCE_SOURCE_BYTES:
148196
return CBM_INDEX_CONFIG_MAX_SOURCE_MB;
197+
case CBM_INDEX_RESOURCE_RSS_BYTES:
198+
return CBM_INDEX_CONFIG_MAX_RSS_MB;
199+
case CBM_INDEX_RESOURCE_DURATION_MS:
200+
return CBM_INDEX_CONFIG_MAX_DURATION_SECONDS;
149201
case CBM_INDEX_RESOURCE_NONE:
150202
default:
151203
return "index_resource_limit";

0 commit comments

Comments
 (0)