Skip to content

Commit 28b41c3

Browse files
committed
feat(mcp): opt-in async index_repository with status polling (DeusData#2144)
index_repository blocks until the whole index completes. MCP clients with a per-call deadline (Copilot for IntelliJ) give up on a large repository; their cancel drops the daemon job's last subscriber, the daemon cancels the worker, and every retry starts over, so the index never finishes. Add an opt-in async mode and a status query to the same tool; the default synchronous behaviour is unchanged: - async: true starts the project's index job in the daemon, or joins the one already running for it, and returns at once with its state. The application holds one subscriber reference for an async job until its terminal publish, so the request returning, the client cancelling or the starting session closing no longer cancels it. Only daemon shutdown (or the final session of a non-permanent generation) ends it. Async requests never queue behind the physical job limit; a full daemon answers busy. async/status are stripped from the worker args, so an async request coalesces with an identical synchronous one. - status: true reports the running or last job for the project (queued/running/cancelling/succeeded/failed/cancelled, started_at, finished_at, error summary) from a small per-project record kept in the daemon's job registry after the job itself is reaped. An unknown project is an error; a project indexed before this daemon generation reports idle. No freshness key is added: index_status keeps the DeusData#1561 freshness object. - async + status together, non-boolean values, and async with cross-repo-intelligence are refused. In-process servers (index worker, embedders) refuse both modes: nothing there outlives the call. - On a temporary (non-permanent) daemon, async is refused when the request arrives on the one-shot `cli` tool channel and that session is the daemon's only live session: the daemon would stop and cancel the job the moment the command exits. The error points at `daemon start` (or an open MCP session). A long-lived MCP session stays a valid host even when it is the only one, which is the IDE case this exists for; status stays allowed everywhere. The daemon lifecycle is unchanged. - New allocations go through the memory core (cbm_alloc/cbm_calloc/ cbm_mem_strdup/cbm_free); blocks returned by non-core APIs are released with safe_free, so no file's raw allocator count grows. - A cut-short synchronous call now offers the async alternative: in the cancellation text the daemon or frontend delivers (including the -32800 reply to a cancelled index_repository), and, because a timed-out client never reads that reply, as a notice on the next index_repository or status call for the project. - The tool description and schema, the CLI help and the README explain the flags, the deadline problem and the polling pattern. No server-side timeout or budget is introduced. Tests (deterministic, held fake worker, bounded observation waits): async returns before completion and status observes running then succeeded; an async job survives its session closing while a sync job in the same shape is still cancelled; validation errors; the cancellation reply and the next-call notice carry the advice; a lone one-shot client of a temporary daemon is refused async while status and an MCP session on the same daemon are allowed; in-process refusal; schema; notice helper. Refs DeusData#2031 Fixes DeusData#2144 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 64c23fa commit 28b41c3

10 files changed

Lines changed: 1295 additions & 28 deletions

File tree

‎README.md‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -669,12 +669,39 @@ JSON arguments can also be piped on stdin, for tools that take arguments. A tool
669669

670670
| Tool | Description |
671671
|------|-------------|
672-
| `index_repository` | Index a repository into the graph. Auto-sync keeps it fresh after that. |
672+
| `index_repository` | Index a repository into the graph. Auto-sync keeps it fresh after that. Waits for the whole index by default; pass `async: true` to start it in the daemon and return at once, then poll with `status: true` (see below). |
673673
| `list_projects` | List all indexed projects with node/edge counts. |
674674
| `delete_project` | Remove a project and all its graph data. |
675675
| `index_status` | Check indexing status of a project. |
676676
| `check_index_coverage` | Check whether exact paths or a scope are indexed and fresh. A clean result means no recorded gap, not proof of completeness. |
677677

678+
**Long indexes and client call deadlines.** A synchronous `index_repository` on a large
679+
repository can take longer than an MCP client allows one tool call (some IDE clients give up
680+
after a fixed deadline). When a client cancels or disconnects, the daemon cancels an index that
681+
nobody else is waiting for, so retrying the same blocking call never finishes. Use the async
682+
mode instead:
683+
684+
1. `index_repository(repo_path="/abs/path", async: true)` starts the index in the daemon (or
685+
joins the one already running for that project) and returns immediately with
686+
`state` (`queued`/`running`). The job keeps running even if the client cancels, times out or
687+
disconnects; only stopping the daemon ends it.
688+
2. `index_repository(repo_path="/abs/path", status: true)` reports `state`
689+
(`queued`, `running`, `cancelling`, `succeeded`, `failed`, `cancelled`), `started_at`,
690+
`finished_at` and an `error` summary. Poll it until the state is `succeeded`, `failed` or
691+
`cancelled`. Pass the same `repo_path` (and `name`, if the index call used one).
692+
693+
`async` and `status` are exclusive; `async` does not apply to `cross-repo-intelligence`. Both
694+
need the daemon-backed server (the default `codebase-memory-mcp` entry point). A temporary
695+
daemon (started on demand rather than by `codebase-memory-mcp daemon start`) stops, and
696+
cancels its jobs, when its last client disconnects. A connected MCP session keeps it alive, so
697+
async from your editor works. A one-shot `codebase-memory-mcp cli index_repository --async`
698+
that is the daemon's only client is refused with a clear error, because the job would die the
699+
moment the command exits: run `codebase-memory-mcp daemon start` first, keep an MCP session
700+
open, or call without `--async`. `status` works everywhere. When a synchronous call was cut
701+
short, the next
702+
`index_repository` or `status` call for that project carries a `notice` suggesting the async
703+
mode. `index_status` keeps describing the published graph and its freshness.
704+
678705
### Querying
679706

680707
| Tool | Description |
@@ -799,6 +826,7 @@ SQLite databases stored at `~/.cache/codebase-memory-mcp/`. Persists across rest
799826
|---------|-----|
800827
| `/mcp` doesn't show the server | Check `.mcp.json` path is absolute. Restart agent. Test: `echo '{}' \| /path/to/binary` should output JSON. |
801828
| `index_repository` fails | Pass absolute path: `index_repository(repo_path="/absolute/path")` |
829+
| `index_repository` times out in the client | Start it with `async: true`, then poll with `status: true` (see [Indexing](#indexing)). |
802830
| `trace_path` returns 0 results | Use `search_graph(name_pattern=".*PartialName.*")` first to find the exact name. |
803831
| Queries return wrong project results | Add `project="name"` parameter. Use `list_projects` to see names. |
804832
| Binary not found after install | Add to PATH: `export PATH="$HOME/.local/bin:$PATH"` |

‎src/daemon/application.c‎

Lines changed: 429 additions & 16 deletions
Large diffs are not rendered by default.

‎src/daemon/frontend.c‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -382,8 +382,26 @@ static bool frontend_write_response(FILE *out, const uint8_t *response, uint32_t
382382
return fflush(out) == 0 && written;
383383
}
384384

385+
const char *cbm_daemon_frontend_cancelled_error_message(const char *request_message) {
386+
static const char plain[] = "Request cancelled";
387+
static const char index_call[] = "Request cancelled. " CBM_MCP_INDEX_ASYNC_HINT;
388+
cbm_jsonrpc_request_t request = {0};
389+
if (!request_message || cbm_jsonrpc_parse(request_message, &request) != 0) {
390+
return plain;
391+
}
392+
char *tool = request.method && strcmp(request.method, "tools/call") == 0 && request.params_raw
393+
? cbm_mcp_get_string_arg(request.params_raw, "name")
394+
: NULL;
395+
bool index_repository = tool && strcmp(tool, "index_repository") == 0;
396+
safe_free(tool);
397+
cbm_jsonrpc_request_free(&request);
398+
return index_repository ? index_call : plain;
399+
}
400+
385401
static bool frontend_write_cancelled_response(FILE *out, const frontend_item_t *item) {
386-
static const char cancelled_error[] = "{\"code\":-32800,\"message\":\"Request cancelled\"}";
402+
char cancelled_error[512];
403+
(void)snprintf(cancelled_error, sizeof(cancelled_error), "{\"code\":-32800,\"message\":\"%s\"}",
404+
cbm_daemon_frontend_cancelled_error_message(item->message));
387405
cbm_jsonrpc_response_t response = {
388406
.id = item->id,
389407
.id_str = item->id_str,

‎src/daemon/frontend.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,12 @@ bool cbm_daemon_frontend_is_cancellation_notification(const char *message);
2929
bool cbm_daemon_frontend_cancellation_matches_request(const char *message, int64_t active_id,
3030
const char *active_id_str);
3131

32+
/* The JSON-RPC error message answered for a cancelled request (#2144). A
33+
* cancelled index_repository call is usually a client deadline on a long
34+
* index, so its reply names the async alternative; every other request keeps
35+
* the plain message. Returns a static string free of JSON metacharacters. */
36+
const char *cbm_daemon_frontend_cancelled_error_message(const char *request_message);
37+
3238
/* Start a temporary observer for a one-shot local CLI command or physical
3339
* supervised worker. manager and cancel_context are borrowed until stop. On
3440
* maintenance intent the observer invokes cancel once, permits a fixed bounded

0 commit comments

Comments
 (0)