Skip to content

Commit 65acd38

Browse files
Merge branch 'main' into perf/py-cross-registry-index
Signed-off-by: Nelson Spence <nelson@projectnavi.ai>
2 parents a46dc5a + 0f52d30 commit 65acd38

99 files changed

Lines changed: 7618 additions & 867 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile.cbm‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,7 @@ FOUNDATION_SRCS = \
327327
src/foundation/limits.c \
328328
src/foundation/index_policy.c \
329329
src/foundation/subprocess.c \
330+
src/foundation/git_env.c \
330331
src/foundation/sha256.c \
331332
src/foundation/secure_random.c \
332333
src/foundation/macos_acl.c \

‎README.md‎

Lines changed: 33 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ The install script placed beside the binary is **reported, not deleted** — uni
199199
- **Cypher-like queries**: `MATCH (f:Function)-[:CALLS]->(g) WHERE f.name = 'main' RETURN g.name`
200200

201201
### Search
202-
- **Semantic search** (`semantic_query`): vector search across the entire graph, powered by bundled Nomic `nomic-embed-code` embeddings (40K tokens, 768d int8) compiled into the binary — no API key, no Ollama, no Docker. 11-signal combined scoring (TF-IDF, RRI, API/Type/Decorator signatures, AST profiles, data flow, Halstead-lite, MinHash, module proximity, graph diffusion).
202+
- **Semantic search** (the `semantic_query` parameter of `search_graph`, not a separate tool): vector search across the entire graph, powered by bundled Nomic `nomic-embed-code` embeddings (40K tokens, 768d int8) compiled into the binary — no API key, no Ollama, no Docker. 11-signal combined scoring (TF-IDF, RRI, API/Type/Decorator signatures, AST profiles, data flow, Halstead-lite, MinHash, module proximity, graph diffusion).
203203
- **BM25 full-text search** via SQLite FTS5 with `cbm_camel_split` tokenizer (camelCase / snake_case aware)
204204
- **Structural search** (`search_graph`): regex name patterns, label filters, min/max degree, file scoping
205205
- **Code search** (`search_code`): graph-augmented grep over indexed files only
@@ -243,12 +243,12 @@ The install script placed beside the binary is **reported, not deleted** — uni
243243

244244
Commit a single compressed file to your repo and your teammates skip the reindex.
245245

246-
`.codebase-memory/graph.db.zst` is a zstd-compressed snapshot of the knowledge graph that lives next to your source. When you index, the artifact is written or refreshed; when a teammate clones the repo and runs `codebase-memory-mcp` for the first time, the artifact is decompressed and incremental indexing fills in their local diff.
246+
`.codebase-memory/graph.db.zst` is a zstd-compressed snapshot of the knowledge graph that lives next to your source. The artifact is opt-in: `index_repository` writes it only with `persistence: true` (the default is `false`), and once it exists later indexes refresh it; when a teammate clones the repo and runs `codebase-memory-mcp` for the first time, the artifact is decompressed and incremental indexing fills in their local diff.
247247

248248
- **Format**: SQLite database, indexes stripped, `VACUUM INTO` compacted, then zstd 1.5.7 compressed (8–13:1 ratio typical)
249249
- **Two tiers**:
250-
- **Best** (`zstd -9` + index strip + `VACUUM INTO`) — written on explicit `index_repository`
251-
- **Fast** (`zstd -3`) — written by the watcher for low-latency incremental updates
250+
- **Best** (`zstd -9` + index strip + `VACUUM INTO`) — written by `index_repository` with `persistence: true`
251+
- **Fast** (`zstd -3`) — refreshes an existing artifact on every other index (including the watcher's low-latency incremental updates)
252252
- **Bootstrap**: when no local DB exists but the artifact is present, `index_repository` imports the artifact first, then runs incremental indexing — avoiding the full reindex cost
253253
- **No merge pain**: a `.codebase-memory/.gitattributes` line with `merge=ours` is auto-created on first export, so concurrent edits don't produce conflicts on the binary artifact
254254
- **Commit it deliberately**: the artifact is rewritten on every index, including the watcher's Fast tier, and git stores each rewrite as a full new blob. Committing every refresh is what turns a 20 MB file into gigabytes of history — one team reached ~6 GB across ~350 commits of this single path. Pick a cadence (a release, a milestone, a nightly job) rather than committing every save.
@@ -669,12 +669,39 @@ JSON arguments can also be piped on stdin, for tools that take arguments. A tool
669669

670670
| Tool | Description |
671671
|------|-------------|
672-
| `index_repository` | Index a repository into the graph. Auto-sync keeps it fresh after that. |
672+
| `index_repository` | Index a repository into the graph. Auto-sync keeps it fresh after that. Waits for the whole index by default; pass `async: true` to start it in the daemon and return at once, then poll with `status: true` (see below). |
673673
| `list_projects` | List all indexed projects with node/edge counts. |
674674
| `delete_project` | Remove a project and all its graph data. |
675675
| `index_status` | Check indexing status of a project. |
676676
| `check_index_coverage` | Check whether exact paths or a scope are indexed and fresh. A clean result means no recorded gap, not proof of completeness. |
677677

678+
**Long indexes and client call deadlines.** A synchronous `index_repository` on a large
679+
repository can take longer than an MCP client allows one tool call (some IDE clients give up
680+
after a fixed deadline). When a client cancels or disconnects, the daemon cancels an index that
681+
nobody else is waiting for, so retrying the same blocking call never finishes. Use the async
682+
mode instead:
683+
684+
1. `index_repository(repo_path="/abs/path", async: true)` starts the index in the daemon (or
685+
joins the one already running for that project) and returns immediately with
686+
`state` (`queued`/`running`). The job keeps running even if the client cancels, times out or
687+
disconnects; only stopping the daemon ends it.
688+
2. `index_repository(repo_path="/abs/path", status: true)` reports `state`
689+
(`queued`, `running`, `cancelling`, `succeeded`, `failed`, `cancelled`), `started_at`,
690+
`finished_at` and an `error` summary. Poll it until the state is `succeeded`, `failed` or
691+
`cancelled`. Pass the same `repo_path` (and `name`, if the index call used one).
692+
693+
`async` and `status` are exclusive; `async` does not apply to `cross-repo-intelligence`. Both
694+
need the daemon-backed server (the default `codebase-memory-mcp` entry point). A temporary
695+
daemon (started on demand rather than by `codebase-memory-mcp daemon start`) stops, and
696+
cancels its jobs, when its last client disconnects. A connected MCP session keeps it alive, so
697+
async from your editor works. A one-shot `codebase-memory-mcp cli index_repository --async`
698+
that is the daemon's only client is refused with a clear error, because the job would die the
699+
moment the command exits: run `codebase-memory-mcp daemon start` first, keep an MCP session
700+
open, or call without `--async`. `status` works everywhere. When a synchronous call was cut
701+
short, the next
702+
`index_repository` or `status` call for that project carries a `notice` suggesting the async
703+
mode. `index_status` keeps describing the published graph and its freshness.
704+
678705
### Querying
679706

680707
| Tool | Description |
@@ -799,6 +826,7 @@ SQLite databases stored at `~/.cache/codebase-memory-mcp/`. Persists across rest
799826
|---------|-----|
800827
| `/mcp` doesn't show the server | Check `.mcp.json` path is absolute. Restart agent. Test: `echo '{}' \| /path/to/binary` should output JSON. |
801828
| `index_repository` fails | Pass absolute path: `index_repository(repo_path="/absolute/path")` |
829+
| `index_repository` times out in the client | Start it with `async: true`, then poll with `status: true` (see [Indexing](#indexing)). |
802830
| `trace_path` returns 0 results | Use `search_graph(name_pattern=".*PartialName.*")` first to find the exact name. |
803831
| Queries return wrong project results | Add `project="name"` parameter. Use `list_projects` to see names. |
804832
| Binary not found after install | Add to PATH: `export PATH="$HOME/.local/bin:$PATH"` |

‎docs/CONFIGURATION.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,20 @@ process that should share one daemon must see the same value — set it in the
200200
environment of your MCP client and your shell alike, or a CLI invocation without
201201
it will coordinate through the default location instead.
202202

203+
**WSL2 and Windows drives (`/mnt/c`, `/mnt/e`, ...).** `CBM_CACHE_DIR` goes through
204+
the same private-directory check. With WSL's default automount options, DrvFs
205+
reports every directory as `0777` and ignores `chmod`, so a cache under
206+
`/mnt/<drive>` is refused, and the refusal names this remedy. Either turn on
207+
permission metadata in `/etc/wsl.conf`, then run `wsl --shutdown` and reopen WSL:
208+
209+
```ini
210+
[automount]
211+
options = "metadata,umask=22,fmask=11"
212+
```
213+
214+
or keep the cache on the Linux filesystem (the default `~/.cache/codebase-memory-mcp`),
215+
which is also much faster than a 9p-mounted Windows drive.
216+
203217
Environment used by daemon-owned components—such as diagnostics, daemon logging, and process-wide indexing resource limits—is captured from the first daemon-backed session that starts the daemon. Later sessions join the existing process and cannot replace those values. To change them, close every daemon-backed session, update the relevant agent configurations consistently, and restart a session. `CBM_ALLOWED_ROOT` remains session-specific, a conflicting `CBM_CACHE_DIR` is rejected, and one-shot CLI commands use their own current environment without starting the daemon.
204218

205219

‎docs/cbmignore.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,12 @@ For directories:
7070

7171
1. **Built-in skip list** — `.git`, `node_modules`, `dist`, `target`,
7272
`vendor`, tool caches, etc. (60+ names; the fast/moderate index modes add
73-
more, e.g. `docs`, `examples`, `testdata`). A `.cbmignore` negation
73+
more, e.g. `docs`, `examples`, `testdata`). Generated output whose
74+
folder name alone is too common to skip everywhere is matched by path
75+
suffix instead: Laravel's compiled Blade view cache
76+
`storage/framework/views/` (at any depth) is skipped even without
77+
Laravel's stock `.gitignore` there, while other `views/` folders stay
78+
indexed. A `.cbmignore` negation
7479
(for example `!target/`) can un-skip these directories, except the
7580
non-negatable safety core: `.git`, `node_modules`, `.worktrees`, and
7681
`.claude-worktrees`.

‎internal/cbm/cbm.h‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,14 @@ typedef struct {
238238
* that declared this method. Kept at the tail so zero-initialised
239239
* callers in every other language remain ABI/source compatible. */
240240
const char *impl_trait;
241+
/* JS/TS only (#1916): a module-level binding initialised by
242+
* `axios.create(...)` records the client library here ("axios"), and the
243+
* literal `baseURL` string (or NULL when absent / not a string literal).
244+
* The Module def carries the pair when its default export is such a
245+
* client. The call resolver composes `<binding>.get('/p')` into an
246+
* HTTP_CALLS edge to base + path. Tail fields: zero-init stays valid. */
247+
const char *http_client;
248+
const char *http_base_url;
241249
} CBMDefinition;
242250

243251
/* Argument captured from a call expression */
@@ -299,6 +307,7 @@ typedef struct {
299307
typedef struct {
300308
const char *local_name; // local alias or name
301309
const char *module_path; // resolved module path / QN
310+
bool is_default; // ES default import (`import X from "Y"`), JS/TS only (#1916)
302311
} CBMImport;
303312

304313
typedef enum {
@@ -729,9 +738,6 @@ const char *cbm_index_quarantine_phase(const char *rel_path);
729738
void cbm_index_mark_start(const char *rel_path);
730739
void cbm_index_mark_done(const char *rel_path);
731740

732-
// Extract all data from one file. Caller must call cbm_free_result().
733-
// source must remain valid for the duration of the call.
734-
// timeout_micros: per-file parse timeout in microseconds (0 = no timeout).
735741
/* Compact a finished result: copy everything reachable from it -- every
736742
* record array at exact count, every string once (interned by content within
737743
* the file), the retained source -- into one exact-size arena, and destroy the
@@ -767,6 +773,11 @@ void cbm_work_arena_keep_begin(void);
767773
/* Free the compaction scratch this thread kept (cbm_work_arena_release calls it). */
768774
void cbm_result_compact_release_thread(void);
769775

776+
// Extract all data from one file. Caller must call cbm_free_result().
777+
// source must remain valid for the duration of the call.
778+
// timeout_micros: per-file tree-sitter parse budget in microseconds of the
779+
// calling thread's CPU time, with a wall-clock backstop of
780+
// CBM_PARSE_WALL_CEILING_FACTOR x the budget (0 = no budget).
770781
CBMFileResult *cbm_extract_file(const char *source, int source_len, CBMLanguage language,
771782
const char *project, const char *rel_path, int64_t timeout_micros,
772783
const char **extra_defines, // NULL-terminated, or NULL

‎internal/cbm/extract_calls.c‎

Lines changed: 66 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2105,6 +2105,56 @@ static void process_keyword_arg(CBMExtractCtx *ctx, TSNode arg_node, CBMCallArg
21052105
}
21062106
}
21072107

2108+
static const char *extract_string_value(CBMExtractCtx *ctx, TSNode val_node);
2109+
2110+
/* A JS/TS request-config object -- `http({url: `/users/${id}`, method: 'GET'})`,
2111+
* `axios({url: '/orders'})`, the shape Orval and similar generators emit -- names
2112+
* its request URL in the `url` property. Return that URL (template literals
2113+
* flattened to the canonical "{}" form, like a bare template argument), or NULL.
2114+
* Only the `url` key counts: `path`/`endpoint` in an options object name other
2115+
* things too often. Route registrations are excluded: an object there is the
2116+
* route definition (Fastify `route({url, handler})`), not a client request, and
2117+
* its handler lives inside the object where the route pass cannot read it. So is
2118+
* axios `getUri(config)`: it takes the same config but only formats the URL and
2119+
* sends nothing (Orval's `get<Op>Url` helpers). (#2235) */
2120+
static bool config_object_callee_eligible(const CBMExtractCtx *ctx, const char *callee_name) {
2121+
if (ctx->language != CBM_LANG_JAVASCRIPT && ctx->language != CBM_LANG_TYPESCRIPT &&
2122+
ctx->language != CBM_LANG_TSX && ctx->language != CBM_LANG_ARKTS) {
2123+
return false;
2124+
}
2125+
if (!callee_name || cbm_service_pattern_route_method(callee_name) != NULL) {
2126+
return false;
2127+
}
2128+
static const char get_uri[] = "getUri";
2129+
size_t clen = strlen(callee_name);
2130+
size_t glen = sizeof(get_uri) - 1;
2131+
return !(clen >= glen && strcmp(callee_name + clen - glen, get_uri) == 0);
2132+
}
2133+
2134+
static const char *js_config_object_url(CBMExtractCtx *ctx, TSNode obj, const char *callee_name) {
2135+
if (strcmp(ts_node_type(obj), "object") != 0 ||
2136+
!config_object_callee_eligible(ctx, callee_name)) {
2137+
return NULL;
2138+
}
2139+
uint32_t n = ts_node_named_child_count(obj);
2140+
for (uint32_t i = 0; i < n; i++) {
2141+
TSNode pair = ts_node_named_child(obj, i);
2142+
if (strcmp(ts_node_type(pair), "pair") != 0) {
2143+
continue;
2144+
}
2145+
TSNode key_n = ts_node_child_by_field_name(pair, TS_FIELD("key"));
2146+
TSNode val_n = ts_node_child_by_field_name(pair, TS_FIELD("value"));
2147+
if (ts_node_is_null(key_n) || ts_node_is_null(val_n)) {
2148+
continue;
2149+
}
2150+
const char *key = strip_quotes(ctx->arena, cbm_node_text(ctx->arena, key_n, ctx->source));
2151+
if (key && strcmp(key, "url") == 0) {
2152+
return extract_string_value(ctx, val_n);
2153+
}
2154+
}
2155+
return NULL;
2156+
}
2157+
21082158
/* Extract all arguments from a call expression into call->args[]. */
21092159
static void extract_call_args(CBMExtractCtx *ctx, TSNode args, CBMCall *call) {
21102160
uint32_t argc = ts_node_named_child_count(args);
@@ -2147,6 +2197,10 @@ static void extract_call_args(CBMExtractCtx *ctx, TSNode args, CBMCall *call) {
21472197
ca->value = cbm_template_string_text(ctx->arena, arg_node, ctx->source);
21482198
} else if (strcmp(ak, "identifier") == 0 && ca->expr) {
21492199
ca->value = lookup_string_constant(ctx, ca->expr);
2200+
} else if (strcmp(ak, "object") == 0) {
2201+
/* Request-config object: its `url` is what the arg-url
2202+
* heuristic reads for a local fetch wrapper (#2235). */
2203+
ca->value = js_config_object_url(ctx, arg_node, call->callee_name);
21502204
} else if (strcmp(ak, "call_expression") == 0) {
21512205
/* URL-builder helper call (issue #1009): resolve
21522206
* client(buildPath(id)) through the per-file builder map. */
@@ -2411,7 +2465,8 @@ static const char *extract_positional_url(CBMExtractCtx *ctx, TSNode arg, const
24112465
}
24122466

24132467
// Extract URL/topic from keyword or positional args.
2414-
static const char *extract_url_or_topic_arg(CBMExtractCtx *ctx, TSNode args) {
2468+
static const char *extract_url_or_topic_arg(CBMExtractCtx *ctx, TSNode args,
2469+
const char *callee_name) {
24152470
uint32_t nc = ts_node_named_child_count(args);
24162471
for (uint32_t ai = 0; ai < nc; ai++) {
24172472
TSNode arg = ts_node_named_child(args, ai);
@@ -2439,6 +2494,15 @@ static const char *extract_url_or_topic_arg(CBMExtractCtx *ctx, TSNode args) {
24392494
continue;
24402495
}
24412496

2497+
/* JS/TS request-config object: `axios({url: '/orders'})` (#2235). */
2498+
if (strcmp(ak, "object") == 0) {
2499+
const char *val = js_config_object_url(ctx, arg, callee_name);
2500+
if (val) {
2501+
return val;
2502+
}
2503+
continue;
2504+
}
2505+
24422506
/* Cloud SDK dispatch via input struct: the queue/topic target is a field
24432507
* of a composite literal (Go `&sqs.SendMessageInput{QueueUrl: ...}`), not
24442508
* a bare string arg. Recover it so the async edge forms. */
@@ -3876,7 +3940,7 @@ CBMInvocationDescriptor handle_calls(CBMExtractCtx *ctx, TSNode node, const CBML
38763940
args = swift_call_args(node);
38773941
}
38783942
if (!ts_node_is_null(args)) {
3879-
call.first_string_arg = extract_url_or_topic_arg(ctx, args);
3943+
call.first_string_arg = extract_url_or_topic_arg(ctx, args, call.callee_name);
38803944
/* #952: routes registered inside Laravel `prefix()->group()`
38813945
* closures must carry the composed path — the resolve passes
38823946
* only see the flat CBMCall, so the enclosing chain can only

0 commit comments

Comments
 (0)