Skip to content

Commit d3b8d25

Browse files
committed
feat(index): doc_candidates setting and index_repository parameter
The doc<->code candidate scoring (DeusData#2574, doc_section_worker in semantic_edges) scores every documentation section against the functions that share a key term with it, key terms being those held by at most funcs/20 functions. Its cost therefore grows with sections x functions. Measured on 11 corpora (10-09 main vs 4d9592b, semantic_edges): under a second up to ~100k functions, +12.6 s on java (325k), +14.6 s on csharp (451k), +174 s on the Linux kernel (784k functions, 28k sections), almost all of the kernel's regression against v0.11.0. A new boolean setting, doc_candidates (default true), and an optional index_repository argument of the same name switch the scoring off: no candidate rows are stored and nodes and edges are unchanged. The argument wins over the setting. The index handler resolves the value once and passes it to a daemon job or supervised worker as an argument, the same way as the resource policy, so every route sees one decision; the session auto-index reads the setting. With the switch off, a delta-route reindex replaces the candidate rows it would otherwise keep, so turning it off leaves no stale candidates. The default output is unchanged. Tests (each fails on its assertion when the switch has no effect): - pipeline_doc_candidates_off_stores_none_and_keeps_graph: no rows, same node and edge counts as the default run, which has rows; - pipeline_doc_candidates_off_clears_rows_on_delta: a closure-repair reindex with the switch off removes the earlier rows; - tool_index_repository_doc_candidates_switch: argument false -> none; setting default -> rows; setting false -> none; argument true over setting false -> rows. Proof, release-built binaries on macOS arm64 (M5 Pro 18c/48 GB): - django, main 4d9592b vs this change with defaults: nodes (63,441), edges (307,584) and doc_link_candidates rows (11,304) hash-identical; - django with doc_candidates false: nodes and edges hash-identical, no candidate rows; - Linux kernel with doc_candidates false: semantic_edges 17.9 s (191.9 s on main, 17.6 s before DeusData#2574), whole index 210.9 s (v0.11.0: ~234 s), no candidate rows. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 4d9592b commit d3b8d25

11 files changed

Lines changed: 367 additions & 16 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -793,6 +793,7 @@ codebase-memory-mcp config list # show all settings
793793
codebase-memory-mcp config set auto_index true # auto-index on session start
794794
codebase-memory-mcp config set auto_index_limit 50000 # max files for auto-index
795795
codebase-memory-mcp config set auto_watch false # don't register background git watcher (default: true)
796+
codebase-memory-mcp config set doc_candidates false # skip doc<->code candidate scoring on huge repos (default: true)
796797
codebase-memory-mcp config set watcher_enabled false # stop the watcher thread entirely (default: true)
797798
codebase-memory-mcp config set watch_non_git true # also poll non-git project roots (default: false)
798799
codebase-memory-mcp config set index_max_files 250000 # optional per-index source-file limit

‎docs/CONFIGURATION.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ Current keys:
8888
| `auto_index` | `false` | Automatically index new projects when an MCP session starts. |
8989
| `auto_index_limit` | `50000` | Maximum file count allowed for automatic indexing of a new project. |
9090
| `auto_watch` | `true` | Register the session's project with the background git watcher on connect. Set `false` to keep a session from registering its project (the watcher still runs for other projects). |
91+
| `doc_candidates` | `true` | Score documentation sections against code to store doc↔code candidates (the possible targets of a section, and the sections that may describe a function or file, each with a judged probability). The scoring grows with sections × functions: under a second up to ~100k functions, ~15 s at ~400k, ~3 minutes on the Linux kernel. Set `false` to skip it; no candidates are stored and the rest of the graph is unchanged. `index_repository` also takes `doc_candidates` for a single index, which wins over this setting. Applies from the next index. |
9192
| `watcher_enabled` | `true` | Master switch for the background watcher subsystem. Set `false` to stop the watcher from starting at all — no poll thread and no project registration. Reindex manually with `index_repository` when disabled. Only an open MCP session's own project is ever watched; `index_status` reports the current state in its `watch` object. |
9293
| `watch_non_git` | `false` | Also poll project roots that are **not git repositories**. By default the watcher only follows git projects, so a project indexed from a plain directory is never refreshed after its first index — reindex it manually with `index_repository`. Set `true` to poll such roots on the same adaptive cadence with a file-tree scan: the indexer's own discovery walk (same skip lists, `.gitignore` and `.cbmignore` rules) hashed over each file's path, size and mtime. Any change reindexes once; paths the indexer skips (including cbm's own `.codebase-memory/` output) never trigger. The first poll after the daemon starts reindexes each such project once, since nothing records which tree state the index holds. The scan walks the whole tree every poll, so it costs more than git polling on very large trees. Read once when the daemon starts, like `watcher_enabled`. |
9394
| `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |

‎src/cli/cli.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -452,6 +452,7 @@ bool cbm_config_load_index_policy(cbm_config_t *cfg, cbm_index_resource_policy_t
452452
#define CBM_CONFIG_AUTO_INDEX "auto_index"
453453
#define CBM_CONFIG_AUTO_INDEX_LIMIT "auto_index_limit"
454454
#define CBM_CONFIG_AUTO_WATCH "auto_watch"
455+
#define CBM_CONFIG_DOC_CANDIDATES "doc_candidates"
455456
#define CBM_CONFIG_UI_LANG "ui-lang"
456457
#define CBM_CONFIG_WATCHER_ENABLED "watcher_enabled"
457458
#define CBM_CONFIG_WATCH_NON_GIT "watch_non_git"

‎src/cli/runtime_settings.c‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,10 @@ static const setting_t settings[] = {
4646
CFG("auto_watch", "Automatic watching",
4747
"Register a background Git watcher when an MCP session connects.", "boolean", "true", 0, 0,
4848
NULL, "src/mcp/mcp.c", "next-session"),
49+
CFG("doc_candidates", "Doc-code candidates",
50+
"Score documentation sections against code for doc-code candidates. Costly on very large "
51+
"repositories; off stores none.",
52+
"boolean", "true", 0, 0, NULL, "src/mcp/mcp.c", "next-session"),
4953
CFG("watcher_enabled", "Background watcher", "Enable the daemon's background watcher thread.",
5054
"boolean", "true", 0, 0, NULL, "src/daemon/application.c", "restart"),
5155
{"ui-lang", "Interface language", "Language preference used by subsequent HTTP requests.",

‎src/mcp/mcp.c‎

Lines changed: 34 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -548,6 +548,8 @@ static const tool_def_t TOOLS[] = {
548548
"\"Name override; Non-ASCII bytes are encoded; unsafe characters normalized.\"},"
549549
"\"persistence\":{\"type\":\"boolean\",\"default\":false,\"description\":"
550550
"\"Write .codebase-memory/graph.db.zst.\"},"
551+
"\"doc_candidates\":{\"type\":\"boolean\",\"description\":"
552+
"\"Doc-code candidates; default: the doc_candidates setting (on).\"},"
551553
"\"async\":{\"type\":\"boolean\",\"default\":false,\"description\":"
552554
"\"Start the index in the background and return immediately; it keeps running if this "
553555
"call is cancelled or times out. Not with status or cross-repo-intelligence. Refused for a "
@@ -11738,14 +11740,32 @@ static bool resolve_session_repo_path(cbm_mcp_server_t *srv, char **repo_path) {
1173811740
return true;
1173911741
}
1174011742

11743+
/* doc_candidates for one index: an explicit boolean argument wins, else the
11744+
* doc_candidates setting (default on). The parent resolves it once and passes
11745+
* the decided value on, so a daemon job or supervised worker gets it as an
11746+
* argument instead of reading a config of its own. */
11747+
static bool index_doc_candidates_enabled(cbm_mcp_server_t *srv, const char *args) {
11748+
yyjson_doc *doc = args ? yyjson_read(args, strlen(args), 0) : NULL;
11749+
yyjson_val *val = doc ? yyjson_obj_get(yyjson_doc_get_root(doc), "doc_candidates") : NULL;
11750+
bool explicit_value = val && yyjson_is_bool(val);
11751+
bool enabled = explicit_value ? yyjson_get_bool(val) : true;
11752+
yyjson_doc_free(doc);
11753+
if (!explicit_value && srv && srv->config) {
11754+
enabled = cbm_config_get_bool(srv->config, CBM_CONFIG_DOC_CANDIDATES, true);
11755+
}
11756+
return enabled;
11757+
}
11758+
1174111759
/* Preserve every index option while replacing all caller-supplied repo_path
1174211760
* keys with the one canonical path that was actually authorized. A non-empty
1174311761
* project_name likewise replaces any caller "name" (#2134: the project key the
1174411762
* parent resolved for this root travels to the daemon job / worker verbatim,
11745-
* so lease key and written index cannot diverge). */
11763+
* so lease key and written index cannot diverge). doc_candidates is replaced
11764+
* by the value the parent resolved. */
1174611765
static char *index_args_with_repo_path(const char *args, const char *canonical_repo_path,
1174711766
const char *project_name,
11748-
const cbm_index_resource_policy_t *policy) {
11767+
const cbm_index_resource_policy_t *policy,
11768+
bool doc_candidates) {
1174911769
if (!args || !canonical_repo_path || !policy) {
1175011770
return NULL;
1175111771
}
@@ -11783,9 +11803,13 @@ static char *index_args_with_repo_path(const char *args, const char *canonical_r
1178311803
while (has_name && yyjson_mut_obj_get(copy_root, "name")) {
1178411804
(void)yyjson_mut_obj_remove_key(copy_root, "name");
1178511805
}
11806+
while (yyjson_mut_obj_get(copy_root, "doc_candidates")) {
11807+
(void)yyjson_mut_obj_remove_key(copy_root, "doc_candidates");
11808+
}
1178611809
if ((has_name && !yyjson_mut_obj_add_strcpy(copy, copy_root, "name", project_name)) ||
1178711810
!yyjson_mut_obj_add_strcpy(copy, copy_root, "repo_path", canonical_repo_path) ||
11788-
!cbm_mcp_index_policy_add_to_args(copy, copy_root, policy)) {
11811+
!cbm_mcp_index_policy_add_to_args(copy, copy_root, policy) ||
11812+
!yyjson_mut_obj_add_bool(copy, copy_root, "doc_candidates", doc_candidates)) {
1178911813
yyjson_mut_doc_free(copy);
1179011814
return NULL;
1179111815
}
@@ -12151,12 +12175,13 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) {
1215112175
index_args_free(repo_path, mode_str, name_override);
1215212176
return cbm_mcp_text_result(policy_error, true);
1215312177
}
12178+
bool doc_candidates = index_doc_candidates_enabled(srv, args);
1215412179

1215512180
/* A daemon session delegates the one physical write to its shared job
1215612181
* registry only after path canonicalization and workspace authorization. */
1215712182
if (srv->index_executor) {
12158-
char *worker_args =
12159-
index_args_with_repo_path(args, repo_path, name_override, &resource_policy);
12183+
char *worker_args = index_args_with_repo_path(args, repo_path, name_override,
12184+
&resource_policy, doc_candidates);
1216012185
char *coordinated = worker_args ? srv->index_executor(srv->index_executor_context,
1216112186
repo_path, worker_args, async_mode)
1216212187
: NULL;
@@ -12184,8 +12209,8 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) {
1218412209
* installs the same guard before running the in-process pipeline. A marked
1218512210
* host fails closed if preparation or worker startup cannot complete. */
1218612211
if (cbm_index_supervisor_should_wrap()) {
12187-
char *worker_args =
12188-
index_args_with_repo_path(args, repo_path, name_override, &resource_policy);
12212+
char *worker_args = index_args_with_repo_path(args, repo_path, name_override,
12213+
&resource_policy, doc_candidates);
1218912214
if (!worker_args) {
1219012215
free(mutation_project);
1219112216
index_args_free(repo_path, mode_str, name_override);
@@ -12249,6 +12274,7 @@ static char *handle_index_repository(cbm_mcp_server_t *srv, const char *args) {
1224912274
free(name_override);
1225012275
cbm_pipeline_set_persistence(p, persistence);
1225112276
cbm_pipeline_set_resource_policy(p, &resource_policy);
12277+
cbm_pipeline_set_doc_candidates_enabled(p, doc_candidates);
1225212278

1225312279
char *project_name = heap_strdup(cbm_pipeline_project_name(p));
1225412280

@@ -19957,6 +19983,7 @@ static void *autoindex_thread(void *arg) {
1995719983
cbm_log_warn("autoindex.err", "msg", "pipeline_create_failed");
1995819984
return NULL;
1995919985
}
19986+
cbm_pipeline_set_doc_candidates_enabled(p, index_doc_candidates_enabled(srv, NULL));
1996019987

1996119988
/* Block until any concurrent pipeline finishes */
1996219989
cbm_pipeline_lock();

‎src/pipeline/pass_semantic_edges.c‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3029,6 +3029,12 @@ static void phase6c_doc_sections(cbm_pipeline_ctx_t *ctx, const cbm_sem_func_t *
30293029
if (!corpus || func_count <= 0) {
30303030
return;
30313031
}
3032+
/* doc_candidates off: no scoring and no rows (the scoring grows with
3033+
* sections x functions; on the Linux kernel it is ~174 s of the index). */
3034+
if (!cbm_pipeline_doc_candidates_enabled(ctx->pipeline)) {
3035+
cbm_log_info("pass.semantic.doc_candidates", "skipped", "doc_candidates_off");
3036+
return;
3037+
}
30323038
const cbm_gbuf_node_t **found = NULL;
30333039
int nsec = 0;
30343040
if (cbm_gbuf_find_by_label(gbuf, "Section", &found, &nsec) != 0 || nsec == 0) {

‎src/pipeline/pipeline.c‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,8 @@ struct cbm_pipeline {
278278
* cbm_pipeline_free). NULL when the pass found none or did not run. */
279279
cbm_doc_candidate_t *doc_candidates;
280280
int doc_candidate_count;
281+
/* doc_candidates setting off (calloc'd false = on, the default). */
282+
bool doc_candidates_off;
281283

282284
/* Deterministic test-only seam at the final publication boundary. Kept
283285
* per pipeline so concurrent test/process activity cannot cross-trigger. */
@@ -871,6 +873,16 @@ void cbm_pipeline_set_persistence(cbm_pipeline_t *p, bool enabled) {
871873
}
872874
}
873875

876+
void cbm_pipeline_set_doc_candidates_enabled(cbm_pipeline_t *p, bool enabled) {
877+
if (p) {
878+
p->doc_candidates_off = !enabled;
879+
}
880+
}
881+
882+
bool cbm_pipeline_doc_candidates_enabled(const cbm_pipeline_t *p) {
883+
return !p || !p->doc_candidates_off;
884+
}
885+
874886
void cbm_pipeline_set_resource_policy(cbm_pipeline_t *p,
875887
const cbm_index_resource_policy_t *policy) {
876888
if (p && policy) {

‎src/pipeline/pipeline.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,13 @@ void cbm_pipeline_set_persistence(cbm_pipeline_t *p, bool enabled);
5656
/* Apply a validated discovery resource policy. The value is copied. */
5757
void cbm_pipeline_set_resource_policy(cbm_pipeline_t *p, const cbm_index_resource_policy_t *policy);
5858

59+
/* Doc -> code candidates (the doc_candidates setting; default on). Off skips
60+
* scoring documentation sections against code and leaves the project with no
61+
* doc_link_candidates rows; nodes and edges are unchanged. The scoring grows
62+
* with sections x functions, which is the cost on very large repositories. */
63+
void cbm_pipeline_set_doc_candidates_enabled(cbm_pipeline_t *p, bool enabled);
64+
bool cbm_pipeline_doc_candidates_enabled(const cbm_pipeline_t *p);
65+
5966
/* Copy the exact discovery violation from the most recent run. */
6067
void cbm_pipeline_get_resource_violation(const cbm_pipeline_t *p,
6168
cbm_index_resource_violation_t *violation);

‎src/pipeline/pipeline_incremental.c‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3172,7 +3172,9 @@ static int run_closure_delta(cbm_pipeline_t *p, const char *db_path, const char
31723172
.doc_link_rows = doc_rows,
31733173
.doc_link_row_count = doc_row_count,
31743174
.doc_links_failed = doc_failed,
3175-
.doc_candidates_in_place = true, /* the clone keeps the previous rows */
3175+
/* The clone keeps the previous rows; with doc_candidates off they
3176+
* are replaced by none, so turning it off leaves no stale rows. */
3177+
.doc_candidates_in_place = cbm_pipeline_doc_candidates_enabled(p),
31763178
};
31773179
cbm_store_close(staging);
31783180
staging = NULL;

0 commit comments

Comments
 (0)