Skip to content

Commit e5eb0b7

Browse files
committed
feat(index): record and report the latest index attempt
A failed index is currently a log line. A watcher-triggered rebuild that dies on a resource limit leaves the previously published graph in service and nothing durable behind it, so index_status reports a healthy generation and the operator has no way to learn that it is no longer the tree. Record one attempt per project under the cache directory, updated under a file lock across queued, running and terminal states, carrying the origin, timestamps, the effective profile source and the resource failure when there is one. index_status returns it as last_index_attempt, and delete_project removes a matching record when no database was published. Freshness is reported alongside it and is deliberately conservative. A generation is comparable only when matching clean Git snapshots were observed before and after indexing, and it is fresh only while the worktree is still clean at that HEAD. A different clean HEAD is stale. Non-Git roots, dirty trees, a tree that changed during indexing, a failed probe and a corrupt record are all unknown, because a dirty tree is evidence of neither freshness nor staleness. Daemon startup fails attempts whose worker no longer exists. Liveness is judged by process identity rather than the pid alone, so a recycled pid cannot keep a dead attempt alive, and an unavailable start time falls back to plain liveness rather than declaring the worker gone. Projects with no record keep the previous response shape. Signed-off-by: 刘冲 <mail@liuchong.dev>
1 parent 8a3b8d4 commit e5eb0b7

12 files changed

Lines changed: 1706 additions & 11 deletions

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -682,8 +682,8 @@ codebase-memory-mcp config reset auto_index # reset to default
682682
explicit value, including `off`, replaces that dimension of a selected
683683
profile. Exceeding an effective limit fails the complete index attempt rather
684684
than publishing a partial graph, and an existing serving index is preserved.
685-
Storage probes also fail closed when an enabled measurement cannot be
686-
completed. See
685+
`index_status` reports the latest physical indexing attempt and Git-based
686+
freshness after the first recorded attempt. See
687687
[Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
688688

689689
### Environment Variables

‎docs/CONFIGURATION.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ This page documents the configuration files that `codebase-memory-mcp` reads or
1010
| Per-project custom extension mapping | `{repo_root}/.codebase-memory.json` | JSON | Overrides conflicting global `extra_extensions` entries. |
1111
| CLI-managed runtime settings | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/_config.db` | SQLite | Written by `codebase-memory-mcp config set/reset`. |
1212
| UI settings | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/config.json` | JSON | Stores `ui_enabled` and `ui_port`. |
13+
| Latest indexing attempt | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/status/<project>.json` | JSON | Owner-private, atomically replaced status used by `index_status`. |
1314
| Daemon operation log | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/logs/cbm-daemon.log` | Structured log | Durable daemon lifecycle, watcher/indexing, UI, resource, and error events. |
1415
| Admission conflict log | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/logs/daemon-conflicts.ndjson` | NDJSON | Exact-build, ABI, and canonical-cache conflicts. |
1516
| Activation log | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/logs/activation-events.ndjson` | NDJSON | Install/update/uninstall activation progress and outcomes. |
@@ -109,6 +110,14 @@ timeout. See
109110
[Index resource limits](INDEX_RESOURCE_LIMITS.md) for counting, validation, and
110111
error-response details.
111112

113+
After the first physical indexing attempt for a project, `index_status` adds
114+
`last_index_attempt` and `freshness`. Freshness is `fresh` only when a clean
115+
Git snapshot observed both before and after indexing still matches the current
116+
clean `HEAD`. A different clean `HEAD` is `stale`, while non-Git, dirty,
117+
changed-during-index, missing, or unreadable snapshots are `unknown`.
118+
`delete_project` also removes a matching attempt record when no database was
119+
published.
120+
112121
## 3. UI Settings
113122

114123
The optional built-in graph UI stores its settings in:

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,25 @@ Both cases preserve the old serving database. Staging files created by a
170170
terminated supervised worker are tagged with a private task token and removed
171171
after its process tree is quiescent; cleanup cannot match another attempt.
172172

173+
## Attempt visibility and freshness
174+
175+
The latest physical attempt for each project is atomically stored in an
176+
owner-private file below `${CBM_CACHE_DIR}/status/`. It records the
177+
`explicit`, `auto`, or `watcher` origin; `queued`, `running`, `completed`,
178+
`failed`, or `cancelled` state; effective profile source; timestamps; and
179+
stable resource failure details when applicable. Daemon startup changes
180+
abandoned `queued` or `running` records to `failed` with
181+
`failure_code=worker_lost`.
182+
183+
After a record exists, `index_status` returns `last_index_attempt` and
184+
`freshness`. A generation is comparable only when matching clean Git snapshots
185+
were observed before and after indexing, and it is `fresh` only while the
186+
current worktree remains clean at that `HEAD`. A different clean `HEAD`, or a
187+
failed watcher rebuild after an observed change, is `stale`. Non-Git roots,
188+
dirty or changed-during-index snapshots, failed Git probes, and corrupt records
189+
are `unknown`. Dirty state is never proof of freshness or staleness. Projects
190+
with no attempt record retain the previous response shape.
191+
173192
## Trust and compatibility
174193

175194
Limits are read from the CLI-managed `_config.db`; they are not MCP request

‎src/daemon/application.c‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,8 @@ struct cbm_daemon_application_job {
146146
bool cancelled;
147147
bool cancel_requested;
148148
bool supervision_failed;
149+
bool attempt_recorded;
150+
char attempt_id[33];
149151
cbm_daemon_application_job_t *next;
150152
};
151153

@@ -1212,6 +1214,63 @@ static void application_record_cancelled(cbm_index_worker_result_t *last_result,
12121214
last_log[0] = '\0';
12131215
}
12141216

1217+
static cbm_index_resource_t application_resource_from_name(const char *name) {
1218+
for (int resource = CBM_INDEX_RESOURCE_FILES;
1219+
resource <= CBM_INDEX_RESOURCE_DISCOVERY_DURATION_MS; resource++) {
1220+
if (name && strcmp(name, cbm_index_resource_name((cbm_index_resource_t)resource)) == 0) {
1221+
return (cbm_index_resource_t)resource;
1222+
}
1223+
}
1224+
return CBM_INDEX_RESOURCE_NONE;
1225+
}
1226+
1227+
static bool application_response_error_metadata(const char *response, char *code, size_t code_size,
1228+
cbm_index_resource_violation_t *violation) {
1229+
if (code && code_size > 0) {
1230+
code[0] = '\0';
1231+
}
1232+
if (violation) {
1233+
*violation = (cbm_index_resource_violation_t){0};
1234+
}
1235+
yyjson_doc *outer = response ? yyjson_read(response, strlen(response), 0) : NULL;
1236+
yyjson_val *outer_root = outer ? yyjson_doc_get_root(outer) : NULL;
1237+
yyjson_val *is_error =
1238+
outer_root && yyjson_is_obj(outer_root) ? yyjson_obj_get(outer_root, "isError") : NULL;
1239+
yyjson_val *content =
1240+
outer_root && yyjson_is_obj(outer_root) ? yyjson_obj_get(outer_root, "content") : NULL;
1241+
yyjson_val *first = content && yyjson_is_arr(content) ? yyjson_arr_get_first(content) : NULL;
1242+
yyjson_val *text = first && yyjson_is_obj(first) ? yyjson_obj_get(first, "text") : NULL;
1243+
yyjson_doc *inner = text && yyjson_is_str(text)
1244+
? yyjson_read(yyjson_get_str(text), yyjson_get_len(text), 0)
1245+
: NULL;
1246+
yyjson_val *inner_root = inner ? yyjson_doc_get_root(inner) : NULL;
1247+
yyjson_val *inner_status =
1248+
inner_root && yyjson_is_obj(inner_root) ? yyjson_obj_get(inner_root, "status") : NULL;
1249+
bool failed = (is_error && yyjson_is_bool(is_error) && yyjson_get_bool(is_error)) ||
1250+
(inner_status && yyjson_is_str(inner_status) &&
1251+
strcmp(yyjson_get_str(inner_status), "error") == 0);
1252+
if (failed && inner_root && yyjson_is_obj(inner_root)) {
1253+
yyjson_val *failure_code = yyjson_obj_get(inner_root, "code");
1254+
if (code && code_size > 0 && failure_code && yyjson_is_str(failure_code)) {
1255+
(void)snprintf(code, code_size, "%s", yyjson_get_str(failure_code));
1256+
}
1257+
yyjson_val *resource = yyjson_obj_get(inner_root, "resource");
1258+
yyjson_val *observed = yyjson_obj_get(inner_root, "observed");
1259+
yyjson_val *limit = yyjson_obj_get(inner_root, "limit");
1260+
if (violation && resource && yyjson_is_str(resource)) {
1261+
violation->resource = application_resource_from_name(yyjson_get_str(resource));
1262+
violation->probe_failed = code && strcmp(code, "resource_probe_failed") == 0;
1263+
if (observed && yyjson_is_uint(observed) && limit && yyjson_is_uint(limit)) {
1264+
violation->observed = yyjson_get_uint(observed);
1265+
violation->limit = yyjson_get_uint(limit);
1266+
}
1267+
}
1268+
}
1269+
yyjson_doc_free(inner);
1270+
yyjson_doc_free(outer);
1271+
return failed;
1272+
}
1273+
12151274
static bool application_result_is_attributable_failure(
12161275
const application_attempt_t *attempt, cbm_mcp_supervised_result_disposition_t disposition) {
12171276
return disposition == CBM_MCP_SUPERVISED_RESULT_CONTAINED_FAILURE && attempt->has_result &&
@@ -1227,6 +1286,7 @@ typedef enum {
12271286
typedef struct {
12281287
char *response;
12291288
bool successful;
1289+
bool response_is_error;
12301290
bool unsafe_terminal;
12311291
bool supervision_failed;
12321292
cbm_index_worker_result_t last_result;
@@ -1256,12 +1316,17 @@ static application_attempt_decision_t application_consume_attempt(
12561316
execution->response = attempt->result.response;
12571317
attempt->result.response = NULL;
12581318
execution->successful = execution->response != NULL;
1319+
execution->response_is_error =
1320+
execution->response &&
1321+
application_response_error_metadata(execution->response, NULL, 0, NULL);
12591322
application_attempt_free(attempt);
12601323
return APPLICATION_ATTEMPT_DECISION_SUCCESS;
12611324
}
12621325
if (disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_FAILURE) {
12631326
execution->response =
12641327
cbm_mcp_index_worker_resource_response(job->args_json, &attempt->result);
1328+
execution->successful = execution->response != NULL;
1329+
execution->response_is_error = execution->response != NULL;
12651330
application_attempt_free(attempt);
12661331
return APPLICATION_ATTEMPT_DECISION_STOP;
12671332
}
@@ -1461,6 +1526,46 @@ static void application_job_publish(cbm_daemon_application_job_t *job,
14611526
execution->have_last_result ? &execution->last_result : NULL,
14621527
execution->last_log[0] ? execution->last_log : NULL);
14631528
}
1529+
if (job->attempt_recorded) {
1530+
bool cancelled =
1531+
execution->have_last_result && execution->last_result.cancellation_requested;
1532+
bool attempt_completed = execution->successful && !execution->response_is_error;
1533+
const char *state = attempt_completed ? "completed" : cancelled ? "cancelled" : "failed";
1534+
const char *failure_code = NULL;
1535+
const cbm_index_resource_violation_t *violation = NULL;
1536+
char response_failure_code[CBM_SZ_128] = {0};
1537+
cbm_index_resource_violation_t response_violation = {0};
1538+
if (!attempt_completed && !cancelled) {
1539+
if (application_response_error_metadata(execution->response, response_failure_code,
1540+
sizeof(response_failure_code),
1541+
&response_violation)) {
1542+
failure_code = response_failure_code[0] ? response_failure_code : "index_failed";
1543+
if (response_violation.resource != CBM_INDEX_RESOURCE_NONE) {
1544+
violation = &response_violation;
1545+
}
1546+
} else if (execution->have_last_result &&
1547+
execution->last_result.resource_violation.resource !=
1548+
CBM_INDEX_RESOURCE_NONE) {
1549+
violation = &execution->last_result.resource_violation;
1550+
failure_code =
1551+
violation->probe_failed ? "resource_probe_failed" : "resource_limit_exceeded";
1552+
} else if (execution->supervision_failed || execution->unsafe_terminal) {
1553+
failure_code = "supervision_failed";
1554+
} else if (!execution->have_last_result ||
1555+
execution->last_result.outcome == CBM_PROC_SPAWN_FAILED) {
1556+
failure_code = "worker_start_failed";
1557+
} else {
1558+
failure_code = "index_failed";
1559+
}
1560+
} else if (cancelled) {
1561+
failure_code = "cancelled";
1562+
}
1563+
if (!cbm_mcp_index_attempt_transition(job->project_key, job->root_path, job->attempt_id,
1564+
state, failure_code, violation, attempt_completed)) {
1565+
cbm_log_warn("daemon.index.attempt_record", "project", job->project_key, "state",
1566+
"terminal_write_failed");
1567+
}
1568+
}
14641569
cbm_daemon_application_t *application = job->application;
14651570
cbm_mutex_lock(&application->mutex);
14661571
job->response = execution->response;
@@ -1508,6 +1613,12 @@ static void *application_job_thread(void *opaque) {
15081613
if (!application_job_wait_for_mutations(job)) {
15091614
application_job_execution_cancel(&execution);
15101615
} else {
1616+
if (job->attempt_recorded &&
1617+
!cbm_mcp_index_attempt_transition(job->project_key, job->root_path, job->attempt_id,
1618+
"running", NULL, NULL, false)) {
1619+
cbm_log_warn("daemon.index.attempt_record", "project", job->project_key, "state",
1620+
"running_write_failed");
1621+
}
15111622
application_attempt_t attempt;
15121623
application_attempt_status_t status =
15131624
application_job_run_attempt(job, NULL, NULL, &attempt);
@@ -1556,6 +1667,18 @@ static char *application_index_project_key(const char *root_path, const char *ar
15561667
return key;
15571668
}
15581669

1670+
static const char *application_index_origin(const char *args_json, char **owned_origin) {
1671+
*owned_origin = cbm_mcp_get_string_arg(args_json, "_cbm_index_origin");
1672+
if (*owned_origin &&
1673+
(strcmp(*owned_origin, "auto") == 0 || strcmp(*owned_origin, "watcher") == 0 ||
1674+
strcmp(*owned_origin, "explicit") == 0)) {
1675+
return *owned_origin;
1676+
}
1677+
free(*owned_origin);
1678+
*owned_origin = NULL;
1679+
return "explicit";
1680+
}
1681+
15591682
static size_t application_active_job_count_locked(cbm_daemon_application_t *application) {
15601683
size_t count = 0;
15611684
for (cbm_daemon_application_job_t *job = application->jobs; job; job = job->next) {
@@ -1587,6 +1710,7 @@ static bool application_index_args_normalize_defaults(yyjson_mut_val *root) {
15871710
if (name && (!yyjson_mut_is_str(name) || yyjson_mut_get_len(name) == 0)) {
15881711
(void)yyjson_mut_obj_remove_key(root, "name");
15891712
}
1713+
(void)yyjson_mut_obj_remove_key(root, "_cbm_index_origin");
15901714
return true;
15911715
}
15921716

@@ -1619,6 +1743,9 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
16191743
if (application->stopping) {
16201744
return NULL;
16211745
}
1746+
char *requested_origin = cbm_mcp_get_string_arg(args_json, "_cbm_index_origin");
1747+
bool watcher_request = requested_origin && strcmp(requested_origin, "watcher") == 0;
1748+
free(requested_origin);
16221749
cbm_daemon_application_job_t *job =
16231750
application_find_active_job_locked(application, project_key);
16241751
if (job) {
@@ -1631,6 +1758,9 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
16311758
return NULL;
16321759
}
16331760
job->subscribers++;
1761+
if (watcher_request && job->attempt_recorded) {
1762+
(void)cbm_mcp_index_attempt_mark_watcher_change(job->project_key, job->attempt_id);
1763+
}
16341764
*status_out = APPLICATION_JOB_SUBSCRIBE_OK;
16351765
return job;
16361766
}
@@ -1656,6 +1786,20 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
16561786
}
16571787
job->application = application;
16581788
job->subscribers = 1;
1789+
cbm_index_resource_policy_t policy;
1790+
char policy_error[CBM_SZ_256] = {0};
1791+
char *owned_origin = NULL;
1792+
const char *origin = application_index_origin(args_json, &owned_origin);
1793+
if (cbm_mcp_index_policy_from_internal_args(args_json, &policy, policy_error,
1794+
sizeof(policy_error))) {
1795+
job->attempt_recorded =
1796+
cbm_mcp_index_attempt_begin(project_key, root_path, origin, &policy,
1797+
strcmp(origin, "watcher") == 0, job->attempt_id);
1798+
}
1799+
free(owned_origin);
1800+
if (!job->attempt_recorded) {
1801+
cbm_log_warn("daemon.index.attempt_record", "project", project_key, "state", "unavailable");
1802+
}
16591803
job->next = application->jobs;
16601804
application->jobs = job;
16611805
if (application_job_thread_create(&job->thread, job) == 0) {
@@ -1666,6 +1810,11 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
16661810
* reservation back synchronously and let background callers retry. */
16671811
application->jobs = job->next;
16681812
job->next = NULL;
1813+
if (job->attempt_recorded) {
1814+
(void)cbm_mcp_index_attempt_transition(job->project_key, job->root_path,
1815+
job->attempt_id, "failed", "worker_start_failed",
1816+
NULL, false);
1817+
}
16691818
application_job_free(job);
16701819
*status_out = APPLICATION_JOB_SUBSCRIBE_UNAVAILABLE;
16711820
cbm_log_warn("daemon.index.thread_start_failed", "action", "retry");
@@ -1985,6 +2134,7 @@ static char *application_auto_index_args(cbm_daemon_application_t *application,
19852134
}
19862135
yyjson_mut_doc_set_root(document, root);
19872136
char *args = yyjson_mut_obj_add_strcpy(document, root, "repo_path", root_path) &&
2137+
yyjson_mut_obj_add_str(document, root, "_cbm_index_origin", "auto") &&
19882138
application_index_args_add_policy(application, document, root)
19892139
? yyjson_mut_write(document, 0, NULL)
19902140
: NULL;
@@ -3013,6 +3163,7 @@ cbm_daemon_application_t *cbm_daemon_application_new(
30133163
application->watcher, application_watcher_mutation_begin,
30143164
application_watcher_mutation_end, application_watcher_project_pruned, application);
30153165
}
3166+
cbm_mcp_index_attempt_recover_abandoned();
30163167
return application;
30173168
}
30183169

@@ -3437,6 +3588,8 @@ static int application_background_index(cbm_daemon_application_t *application,
34373588
}
34383589
yyjson_mut_doc_set_root(document, root);
34393590
bool encoded = yyjson_mut_obj_add_strcpy(document, root, "repo_path", canonical_root) &&
3591+
yyjson_mut_obj_add_str(document, root, "_cbm_index_origin",
3592+
require_live_watch ? "watcher" : "explicit") &&
34403593
application_index_args_add_policy(application, document, root);
34413594
char *default_project = cbm_project_name_from_path(canonical_root);
34423595
bool custom_project =

‎src/git/git_context.c‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,35 @@ static int git_capture(const char *repo_path, const char *git_args, char **out)
8686
return *out ? 0 : CBM_NOT_FOUND;
8787
}
8888

89+
static int git_dirty_status(const char *repo_path, bool *dirty_out) {
90+
if (!dirty_out || !repo_path || !git_validate_repo_path(repo_path)) {
91+
return CBM_NOT_FOUND;
92+
}
93+
*dirty_out = false;
94+
#ifdef _WIN32
95+
const char *null_dev = "NUL";
96+
#else
97+
const char *null_dev = "/dev/null";
98+
#endif
99+
char cmd[GIT_CMD_MAX];
100+
int n = snprintf(
101+
cmd, sizeof(cmd),
102+
"git --no-optional-locks -C \"%s\" status --porcelain --untracked-files=normal 2>%s",
103+
repo_path, null_dev);
104+
if (n < 0 || n >= (int)sizeof(cmd)) {
105+
return CBM_NOT_FOUND;
106+
}
107+
FILE *fp = cbm_popen(cmd, "r");
108+
if (!fp) {
109+
return CBM_NOT_FOUND;
110+
}
111+
char buffer[CBM_SZ_4K];
112+
while (fread(buffer, 1, sizeof(buffer), fp) > 0) {
113+
*dirty_out = true;
114+
}
115+
return cbm_pclose(fp) == 0 ? 0 : CBM_NOT_FOUND;
116+
}
117+
89118
static bool path_is_absolute(const char *path) {
90119
if (!path || !path[0]) {
91120
return false;
@@ -239,6 +268,36 @@ void cbm_git_context_free(cbm_git_context_t *ctx) {
239268
memset(ctx, 0, sizeof(*ctx));
240269
}
241270

271+
int cbm_git_snapshot_read(const char *path, cbm_git_snapshot_t *out) {
272+
if (!out) {
273+
return CBM_NOT_FOUND;
274+
}
275+
memset(out, 0, sizeof(*out));
276+
char *head_sha = NULL;
277+
int status = git_capture(path, "rev-parse --verify HEAD", &head_sha);
278+
if (status != 0 || !head_sha || !head_sha[0]) {
279+
free(head_sha);
280+
return status == 0 ? CBM_NOT_FOUND : status;
281+
}
282+
bool dirty = false;
283+
if (git_dirty_status(path, &dirty) != 0) {
284+
free(head_sha);
285+
return CBM_NOT_FOUND;
286+
}
287+
out->head_sha = head_sha;
288+
out->available = true;
289+
out->dirty = dirty;
290+
return 0;
291+
}
292+
293+
void cbm_git_snapshot_free(cbm_git_snapshot_t *snapshot) {
294+
if (!snapshot) {
295+
return;
296+
}
297+
free(snapshot->head_sha);
298+
memset(snapshot, 0, sizeof(*snapshot));
299+
}
300+
242301
int cbm_git_context_resolve(const char *path, cbm_git_context_t *out) {
243302
if (!out) {
244303
return CBM_NOT_FOUND;

0 commit comments

Comments
 (0)