Repository navigation
feat(index): doc_candidates switch for the doc-code candidate scoring - #2610
Merged
Merged
Conversation
The doc<->code candidate scoring (#2574, doc_section_worker in semantic_edges) scores every documentation section against the functions that share a key term with it, key terms being those held by at most funcs/20 functions. Its cost therefore grows with sections x functions. Measured on 11 corpora (10-09 main vs 4d9592b, semantic_edges): under a second up to ~100k functions, +12.6 s on java (325k), +14.6 s on csharp (451k), +174 s on the Linux kernel (784k functions, 28k sections), almost all of the kernel's regression against v0.11.0. A new boolean setting, doc_candidates (default true), and an optional index_repository argument of the same name switch the scoring off: no candidate rows are stored and nodes and edges are unchanged. The argument wins over the setting. The index handler resolves the value once and passes it to a daemon job or supervised worker as an argument, the same way as the resource policy, so every route sees one decision; the session auto-index reads the setting. With the switch off, a delta-route reindex replaces the candidate rows it would otherwise keep, so turning it off leaves no stale candidates. The default output is unchanged. Tests (each fails on its assertion when the switch has no effect): - pipeline_doc_candidates_off_stores_none_and_keeps_graph: no rows, same node and edge counts as the default run, which has rows; - pipeline_doc_candidates_off_clears_rows_on_delta: a closure-repair reindex with the switch off removes the earlier rows; - tool_index_repository_doc_candidates_switch: argument false -> none; setting default -> rows; setting false -> none; argument true over setting false -> rows. Proof, release-built binaries on macOS arm64 (M5 Pro 18c/48 GB): - django, main 4d9592b vs this change with defaults: nodes (63,441), edges (307,584) and doc_link_candidates rows (11,304) hash-identical; - django with doc_candidates false: nodes and edges hash-identical, no candidate rows; - Linux kernel with doc_candidates false: semantic_edges 17.9 s (191.9 s on main, 17.6 s before #2574), whole index 210.9 s (v0.11.0: ~234 s), no candidate rows. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
yermakoffivan
pushed a commit
to yermakoffivan/codebase-memory-mcp
that referenced
this pull request
Oct 13, 2026
Doc-section -> code candidates (doc_link_candidates, DeusData#2574/DeusData#2610) are the documentation-in-the-graph feature of this release. They now run by default for full indexes and not for moderate or fast ones; an explicit `doc_candidates` argument, or a `doc_candidates` setting the user set, overrides the default either way in every mode. - pipeline: the switch is unset/on/off; unset means on only when the effective mode is full. - MCP: forwards only an explicit choice (argument, then a user-set config value) to daemon jobs and workers; daemon auto-index and watcher/UI jobs now carry a user-set value too (their workers have no config, so `doc_candidates false` was ignored there). - fast + doc_candidates on runs the semantic pass's corpus phases and the doc phase only: candidates, and still no SIMILAR_TO or SEMANTICALLY_RELATED edges. - config: `doc_candidates` joins the config key table (default `auto`); `config set doc_candidates ...` was refused as an unknown key before. README, docs/CONFIGURATION.md and the MCP schema describe the default. Proof (macOS arm64, django fb11376): full 11,304 rows · moderate 0 · fast 0 · full+off 0 · moderate+on 1 · fast+on 1 (moderate/fast discovery skips docs/) -- nodes and edges are identical between on and off in every mode; fast+on adds 0 semantic edges. - New tests (RED on the old code): pipeline pipeline_doc_candidates_default_follows_mode, pipeline_doc_candidates_explicit_wins_over_mode, pipeline_doc_candidates_fast_on_legacy_incremental; mcp tool_index_repository_doc_candidates_follow_mode, tool_index_repository_doc_candidates_forwarded_only_when_explicit; daemon_application daemon_application_programmatic_index_forwards_set_doc_candidates. - mcp 374 passed (4 Windows-only skips), daemon_application 73 passed, pipeline/store_arch/cli/runtime_settings/index_policy/pipeline_frozen green on the release tree. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The doc↔code candidate scoring added in #2574 (
doc_section_worker, insidesemantic_edges) is the largest indexing cost on big repositories since v0.11.0. This adds a switch for it, on by default, so very large repositories can skip it.Why
For each documentation section, the pass gathers every function that shares a key term with it (key terms: held by at most
funcs/20functions), computes a full dot product for each, and keeps the top ten. Its cost grows with sections × functions. Measured on 11 corpora,semantic_edgeson the 10-09 main vs4d9592b5, macOS arm64:During that phase the kernel's samples are all in
doc_section_worker. The candidates feed a real MCP feature: sections show the functions and files they may describe, and functions and files show the sections that may describe them, each with a judged probability. So the default stays on.What
doc_candidates(boolean, defaulttrue):codebase-memory-mcp config set doc_candidates false.index_repositoryargumentdoc_candidates(optional boolean): overrides the setting for one index. The handler resolves the value once and passes it to a daemon job or supervised worker as an argument, like the resource policy. The session auto-index reads the setting.pass.semantic.doc_candidates skipped=doc_candidates_off), and nodoc_link_candidatesrows are stored. Nodes and edges are unchanged. A delta-route reindex, which otherwise keeps the clone's candidate rows, replaces them with none, so turning it off leaves no stale candidates.docs/CONFIGURATION.md.Checks
4d9592b5and by this branch. Nodes (63,441), edges (307,584) and candidate rows (11,304) are hash-identical.semantic_edgesdrops from 191.9 s to 17.9 s and the whole index takes 210.9 s (v0.11.0: ~234 s), with 0 candidate rows.pipeline_doc_candidates_off_stores_none_and_keeps_graph;pipeline_doc_candidates_off_clears_rows_on_delta;tool_index_repository_doc_candidates_switch: argument false → none; setting default → rows; setting false → none; argument true over setting false → rows.pipeline,mcpandruntime_settingssuites: 725 passed, including the tool-metadata byte budget.