Skip to content

Commit 579336c

Browse files
committed
feat(index): record and report the latest index attempt
A failed index is currently a log line. A watcher-triggered rebuild that dies on a resource limit leaves the previously published graph in service and nothing durable behind it, so index_status reports a healthy generation and the operator has no way to learn that it is no longer the tree. Record one attempt per project under the cache directory, updated under a file lock across queued, running and terminal states, carrying the origin, timestamps, the effective profile source and the resource failure when there is one. index_status returns it as last_index_attempt, and delete_project removes a matching record when no database was published. Freshness is reported alongside it and is deliberately conservative. A generation is comparable only when matching clean Git snapshots were observed before and after indexing, and it is fresh only while the worktree is still clean at that HEAD. A different clean HEAD is stale. Non-Git roots, dirty trees, a tree that changed during indexing, a failed probe and a corrupt record are all unknown, because a dirty tree is evidence of neither freshness nor staleness. Daemon startup fails attempts whose worker no longer exists. Liveness is judged by process identity rather than the pid alone, so a recycled pid cannot keep a dead attempt alive, and an unavailable start time falls back to plain liveness rather than declaring the worker gone. Projects with no record keep the previous response shape. Signed-off-by: 刘冲 <mail@liuchong.dev>
1 parent 9380b22 commit 579336c

12 files changed

Lines changed: 1704 additions & 11 deletions

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -748,8 +748,8 @@ codebase-memory-mcp config reset auto_index # reset to default
748748
explicit value, including `off`, replaces that dimension of a selected
749749
profile. Exceeding an effective limit fails the complete index attempt rather
750750
than publishing a partial graph, and an existing serving index is preserved.
751-
Storage probes also fail closed when an enabled measurement cannot be
752-
completed. See
751+
`index_status` reports the latest physical indexing attempt and Git-based
752+
freshness after the first recorded attempt. See
753753
[Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
754754

755755
### Environment Variables

‎docs/CONFIGURATION.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ This page documents the configuration files that `codebase-memory-mcp` reads or
1010
| Per-project custom extension mapping | `{repo_root}/.codebase-memory.json` | JSON | Overrides conflicting global `extra_extensions` entries. |
1111
| CLI-managed runtime settings | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/_config.db` | SQLite | Written by `codebase-memory-mcp config set/reset`. |
1212
| UI settings | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/config.json` | JSON | Stores `ui_enabled` and `ui_port`. |
13+
| Latest indexing attempt | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/status/<project>.json` | JSON | Owner-private, atomically replaced status used by `index_status`. |
1314
| Daemon operation log | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/logs/cbm-daemon.log` | Structured log | Durable daemon lifecycle, watcher/indexing, UI, resource, and error events. |
1415
| Admission conflict log | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/logs/daemon-conflicts.ndjson` | NDJSON | Exact-build, ABI, and canonical-cache conflicts. |
1516
| Activation log | `${CBM_CACHE_DIR:-~/.cache/codebase-memory-mcp}/logs/activation-events.ndjson` | NDJSON | Install/update/uninstall activation progress and outcomes. |
@@ -138,6 +139,14 @@ timeout. See
138139
[Index resource limits](INDEX_RESOURCE_LIMITS.md) for counting, validation, and
139140
error-response details.
140141
142+
After the first physical indexing attempt for a project, `index_status` adds
143+
`last_index_attempt` and `freshness`. Freshness is `fresh` only when a clean
144+
Git snapshot observed both before and after indexing still matches the current
145+
clean `HEAD`. A different clean `HEAD` is `stale`, while non-Git, dirty,
146+
changed-during-index, missing, or unreadable snapshots are `unknown`.
147+
`delete_project` also removes a matching attempt record when no database was
148+
published.
149+
141150
## 3. UI Settings
142151
143152
The optional built-in graph UI stores its settings in:

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,25 @@ Both cases preserve the old serving database. Staging files created by a
170170
terminated supervised worker are tagged with a private task token and removed
171171
after its process tree is quiescent; cleanup cannot match another attempt.
172172

173+
## Attempt visibility and freshness
174+
175+
The latest physical attempt for each project is atomically stored in an
176+
owner-private file below `${CBM_CACHE_DIR}/status/`. It records the
177+
`explicit`, `auto`, or `watcher` origin; `queued`, `running`, `completed`,
178+
`failed`, or `cancelled` state; effective profile source; timestamps; and
179+
stable resource failure details when applicable. Daemon startup changes
180+
abandoned `queued` or `running` records to `failed` with
181+
`failure_code=worker_lost`.
182+
183+
After a record exists, `index_status` returns `last_index_attempt` and
184+
`freshness`. A generation is comparable only when matching clean Git snapshots
185+
were observed before and after indexing, and it is `fresh` only while the
186+
current worktree remains clean at that `HEAD`. A different clean `HEAD`, or a
187+
failed watcher rebuild after an observed change, is `stale`. Non-Git roots,
188+
dirty or changed-during-index snapshots, failed Git probes, and corrupt records
189+
are `unknown`. Dirty state is never proof of freshness or staleness. Projects
190+
with no attempt record retain the previous response shape.
191+
173192
## Trust and compatibility
174193

175194
Limits are read from the CLI-managed `_config.db`; they are not MCP request

‎src/daemon/application.c‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,8 @@ struct cbm_daemon_application_job {
147147
bool cancelled;
148148
bool cancel_requested;
149149
bool supervision_failed;
150+
bool attempt_recorded;
151+
char attempt_id[33];
150152
cbm_daemon_application_job_t *next;
151153
};
152154

@@ -1248,6 +1250,63 @@ static void application_record_cancelled(cbm_index_worker_result_t *last_result,
12481250
last_log[0] = '\0';
12491251
}
12501252

1253+
static cbm_index_resource_t application_resource_from_name(const char *name) {
1254+
for (int resource = CBM_INDEX_RESOURCE_FILES;
1255+
resource <= CBM_INDEX_RESOURCE_DISCOVERY_DURATION_MS; resource++) {
1256+
if (name && strcmp(name, cbm_index_resource_name((cbm_index_resource_t)resource)) == 0) {
1257+
return (cbm_index_resource_t)resource;
1258+
}
1259+
}
1260+
return CBM_INDEX_RESOURCE_NONE;
1261+
}
1262+
1263+
static bool application_response_error_metadata(const char *response, char *code, size_t code_size,
1264+
cbm_index_resource_violation_t *violation) {
1265+
if (code && code_size > 0) {
1266+
code[0] = '\0';
1267+
}
1268+
if (violation) {
1269+
*violation = (cbm_index_resource_violation_t){0};
1270+
}
1271+
yyjson_doc *outer = response ? yyjson_read(response, strlen(response), 0) : NULL;
1272+
yyjson_val *outer_root = outer ? yyjson_doc_get_root(outer) : NULL;
1273+
yyjson_val *is_error =
1274+
outer_root && yyjson_is_obj(outer_root) ? yyjson_obj_get(outer_root, "isError") : NULL;
1275+
yyjson_val *content =
1276+
outer_root && yyjson_is_obj(outer_root) ? yyjson_obj_get(outer_root, "content") : NULL;
1277+
yyjson_val *first = content && yyjson_is_arr(content) ? yyjson_arr_get_first(content) : NULL;
1278+
yyjson_val *text = first && yyjson_is_obj(first) ? yyjson_obj_get(first, "text") : NULL;
1279+
yyjson_doc *inner = text && yyjson_is_str(text)
1280+
? yyjson_read(yyjson_get_str(text), yyjson_get_len(text), 0)
1281+
: NULL;
1282+
yyjson_val *inner_root = inner ? yyjson_doc_get_root(inner) : NULL;
1283+
yyjson_val *inner_status =
1284+
inner_root && yyjson_is_obj(inner_root) ? yyjson_obj_get(inner_root, "status") : NULL;
1285+
bool failed = (is_error && yyjson_is_bool(is_error) && yyjson_get_bool(is_error)) ||
1286+
(inner_status && yyjson_is_str(inner_status) &&
1287+
strcmp(yyjson_get_str(inner_status), "error") == 0);
1288+
if (failed && inner_root && yyjson_is_obj(inner_root)) {
1289+
yyjson_val *failure_code = yyjson_obj_get(inner_root, "code");
1290+
if (code && code_size > 0 && failure_code && yyjson_is_str(failure_code)) {
1291+
(void)snprintf(code, code_size, "%s", yyjson_get_str(failure_code));
1292+
}
1293+
yyjson_val *resource = yyjson_obj_get(inner_root, "resource");
1294+
yyjson_val *observed = yyjson_obj_get(inner_root, "observed");
1295+
yyjson_val *limit = yyjson_obj_get(inner_root, "limit");
1296+
if (violation && resource && yyjson_is_str(resource)) {
1297+
violation->resource = application_resource_from_name(yyjson_get_str(resource));
1298+
violation->probe_failed = code && strcmp(code, "resource_probe_failed") == 0;
1299+
if (observed && yyjson_is_uint(observed) && limit && yyjson_is_uint(limit)) {
1300+
violation->observed = yyjson_get_uint(observed);
1301+
violation->limit = yyjson_get_uint(limit);
1302+
}
1303+
}
1304+
}
1305+
yyjson_doc_free(inner);
1306+
yyjson_doc_free(outer);
1307+
return failed;
1308+
}
1309+
12511310
static bool application_result_is_attributable_failure(
12521311
const application_attempt_t *attempt, cbm_mcp_supervised_result_disposition_t disposition) {
12531312
return disposition == CBM_MCP_SUPERVISED_RESULT_CONTAINED_FAILURE && attempt->has_result &&
@@ -1263,6 +1322,7 @@ typedef enum {
12631322
typedef struct {
12641323
char *response;
12651324
bool successful;
1325+
bool response_is_error;
12661326
bool unsafe_terminal;
12671327
bool supervision_failed;
12681328
cbm_index_worker_result_t last_result;
@@ -1292,12 +1352,17 @@ static application_attempt_decision_t application_consume_attempt(
12921352
execution->response = attempt->result.response;
12931353
attempt->result.response = NULL;
12941354
execution->successful = execution->response != NULL;
1355+
execution->response_is_error =
1356+
execution->response &&
1357+
application_response_error_metadata(execution->response, NULL, 0, NULL);
12951358
application_attempt_free(attempt);
12961359
return APPLICATION_ATTEMPT_DECISION_SUCCESS;
12971360
}
12981361
if (disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_FAILURE) {
12991362
execution->response =
13001363
cbm_mcp_index_worker_resource_response(job->args_json, &attempt->result);
1364+
execution->successful = execution->response != NULL;
1365+
execution->response_is_error = execution->response != NULL;
13011366
application_attempt_free(attempt);
13021367
return APPLICATION_ATTEMPT_DECISION_STOP;
13031368
}
@@ -1502,6 +1567,46 @@ static void application_job_publish(cbm_daemon_application_job_t *job,
15021567
execution->have_last_result ? &execution->last_result : NULL,
15031568
execution->last_log[0] ? execution->last_log : NULL);
15041569
}
1570+
if (job->attempt_recorded) {
1571+
bool cancelled =
1572+
execution->have_last_result && execution->last_result.cancellation_requested;
1573+
bool attempt_completed = execution->successful && !execution->response_is_error;
1574+
const char *state = attempt_completed ? "completed" : cancelled ? "cancelled" : "failed";
1575+
const char *failure_code = NULL;
1576+
const cbm_index_resource_violation_t *violation = NULL;
1577+
char response_failure_code[CBM_SZ_128] = {0};
1578+
cbm_index_resource_violation_t response_violation = {0};
1579+
if (!attempt_completed && !cancelled) {
1580+
if (application_response_error_metadata(execution->response, response_failure_code,
1581+
sizeof(response_failure_code),
1582+
&response_violation)) {
1583+
failure_code = response_failure_code[0] ? response_failure_code : "index_failed";
1584+
if (response_violation.resource != CBM_INDEX_RESOURCE_NONE) {
1585+
violation = &response_violation;
1586+
}
1587+
} else if (execution->have_last_result &&
1588+
execution->last_result.resource_violation.resource !=
1589+
CBM_INDEX_RESOURCE_NONE) {
1590+
violation = &execution->last_result.resource_violation;
1591+
failure_code =
1592+
violation->probe_failed ? "resource_probe_failed" : "resource_limit_exceeded";
1593+
} else if (execution->supervision_failed || execution->unsafe_terminal) {
1594+
failure_code = "supervision_failed";
1595+
} else if (!execution->have_last_result ||
1596+
execution->last_result.outcome == CBM_PROC_SPAWN_FAILED) {
1597+
failure_code = "worker_start_failed";
1598+
} else {
1599+
failure_code = "index_failed";
1600+
}
1601+
} else if (cancelled) {
1602+
failure_code = "cancelled";
1603+
}
1604+
if (!cbm_mcp_index_attempt_transition(job->project_key, job->root_path, job->attempt_id,
1605+
state, failure_code, violation, attempt_completed)) {
1606+
cbm_log_warn("daemon.index.attempt_record", "project", job->project_key, "state",
1607+
"terminal_write_failed");
1608+
}
1609+
}
15051610
cbm_daemon_application_t *application = job->application;
15061611
cbm_mutex_lock(&application->mutex);
15071612
job->response = execution->response;
@@ -1549,6 +1654,12 @@ static void *application_job_thread(void *opaque) {
15491654
if (!application_job_wait_for_mutations(job)) {
15501655
application_job_execution_cancel(&execution);
15511656
} else {
1657+
if (job->attempt_recorded &&
1658+
!cbm_mcp_index_attempt_transition(job->project_key, job->root_path, job->attempt_id,
1659+
"running", NULL, NULL, false)) {
1660+
cbm_log_warn("daemon.index.attempt_record", "project", job->project_key, "state",
1661+
"running_write_failed");
1662+
}
15521663
application_attempt_t attempt;
15531664
application_attempt_status_t status =
15541665
application_job_run_attempt(job, NULL, NULL, &attempt);
@@ -1597,6 +1708,18 @@ static char *application_index_project_key(const char *root_path, const char *ar
15971708
return key;
15981709
}
15991710

1711+
static const char *application_index_origin(const char *args_json, char **owned_origin) {
1712+
*owned_origin = cbm_mcp_get_string_arg(args_json, "_cbm_index_origin");
1713+
if (*owned_origin &&
1714+
(strcmp(*owned_origin, "auto") == 0 || strcmp(*owned_origin, "watcher") == 0 ||
1715+
strcmp(*owned_origin, "explicit") == 0)) {
1716+
return *owned_origin;
1717+
}
1718+
free(*owned_origin);
1719+
*owned_origin = NULL;
1720+
return "explicit";
1721+
}
1722+
16001723
static size_t application_active_job_count_locked(cbm_daemon_application_t *application) {
16011724
size_t count = 0;
16021725
for (cbm_daemon_application_job_t *job = application->jobs; job; job = job->next) {
@@ -1644,6 +1767,7 @@ static bool application_index_args_normalize_defaults(yyjson_mut_val *root) {
16441767
if (name && (!yyjson_mut_is_str(name) || yyjson_mut_get_len(name) == 0)) {
16451768
(void)yyjson_mut_obj_remove_key(root, "name");
16461769
}
1770+
(void)yyjson_mut_obj_remove_key(root, "_cbm_index_origin");
16471771
return true;
16481772
}
16491773

@@ -1709,6 +1833,9 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
17091833
if (application->stopping) {
17101834
return NULL;
17111835
}
1836+
char *requested_origin = cbm_mcp_get_string_arg(args_json, "_cbm_index_origin");
1837+
bool watcher_request = requested_origin && strcmp(requested_origin, "watcher") == 0;
1838+
free(requested_origin);
17121839
cbm_daemon_application_job_t *job =
17131840
application_find_active_job_locked(application, project_key);
17141841
if (job) {
@@ -1721,6 +1848,9 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
17211848
return NULL;
17221849
}
17231850
job->subscribers++;
1851+
if (watcher_request && job->attempt_recorded) {
1852+
(void)cbm_mcp_index_attempt_mark_watcher_change(job->project_key, job->attempt_id);
1853+
}
17241854
*status_out = APPLICATION_JOB_SUBSCRIBE_OK;
17251855
return job;
17261856
}
@@ -1746,6 +1876,20 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
17461876
}
17471877
job->application = application;
17481878
job->subscribers = 1;
1879+
cbm_index_resource_policy_t policy;
1880+
char policy_error[CBM_SZ_256] = {0};
1881+
char *owned_origin = NULL;
1882+
const char *origin = application_index_origin(args_json, &owned_origin);
1883+
if (cbm_mcp_index_policy_from_internal_args(args_json, &policy, policy_error,
1884+
sizeof(policy_error))) {
1885+
job->attempt_recorded =
1886+
cbm_mcp_index_attempt_begin(project_key, root_path, origin, &policy,
1887+
strcmp(origin, "watcher") == 0, job->attempt_id);
1888+
}
1889+
free(owned_origin);
1890+
if (!job->attempt_recorded) {
1891+
cbm_log_warn("daemon.index.attempt_record", "project", project_key, "state", "unavailable");
1892+
}
17491893
job->next = application->jobs;
17501894
application->jobs = job;
17511895
if (application_job_thread_create(&job->thread, job) == 0) {
@@ -1756,6 +1900,11 @@ static cbm_daemon_application_job_t *application_job_subscribe_locked(
17561900
* reservation back synchronously and let background callers retry. */
17571901
application->jobs = job->next;
17581902
job->next = NULL;
1903+
if (job->attempt_recorded) {
1904+
(void)cbm_mcp_index_attempt_transition(job->project_key, job->root_path,
1905+
job->attempt_id, "failed", "worker_start_failed",
1906+
NULL, false);
1907+
}
17591908
application_job_free(job);
17601909
*status_out = APPLICATION_JOB_SUBSCRIBE_UNAVAILABLE;
17611910
cbm_log_warn("daemon.index.thread_start_failed", "action", "retry");
@@ -2075,6 +2224,7 @@ static char *application_auto_index_args(cbm_daemon_application_t *application,
20752224
}
20762225
yyjson_mut_doc_set_root(document, root);
20772226
char *args = yyjson_mut_obj_add_strcpy(document, root, "repo_path", root_path) &&
2227+
yyjson_mut_obj_add_str(document, root, "_cbm_index_origin", "auto") &&
20782228
application_index_args_add_policy(application, document, root)
20792229
? yyjson_mut_write(document, 0, NULL)
20802230
: NULL;
@@ -3106,6 +3256,7 @@ cbm_daemon_application_t *cbm_daemon_application_new(
31063256
application->watcher, application_watcher_mutation_begin,
31073257
application_watcher_mutation_end, application_watcher_project_pruned, application);
31083258
}
3259+
cbm_mcp_index_attempt_recover_abandoned();
31093260
return application;
31103261
}
31113262

@@ -3529,6 +3680,8 @@ static int application_background_index(cbm_daemon_application_t *application,
35293680
}
35303681
yyjson_mut_doc_set_root(document, root);
35313682
bool encoded = yyjson_mut_obj_add_strcpy(document, root, "repo_path", canonical_root) &&
3683+
yyjson_mut_obj_add_str(document, root, "_cbm_index_origin",
3684+
require_live_watch ? "watcher" : "explicit") &&
35323685
application_index_args_add_policy(application, document, root);
35333686
char *default_project = cbm_project_name_from_path(canonical_root);
35343687
bool custom_project =

‎src/git/git_context.c‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,35 @@ static int git_capture(const char *repo_path, const char *git_args, char **out)
8686
return *out ? 0 : CBM_NOT_FOUND;
8787
}
8888

89+
static int git_dirty_status(const char *repo_path, bool *dirty_out) {
90+
if (!dirty_out || !repo_path || !git_validate_repo_path(repo_path)) {
91+
return CBM_NOT_FOUND;
92+
}
93+
*dirty_out = false;
94+
#ifdef _WIN32
95+
const char *null_dev = "NUL";
96+
#else
97+
const char *null_dev = "/dev/null";
98+
#endif
99+
char cmd[GIT_CMD_MAX];
100+
int n = snprintf(
101+
cmd, sizeof(cmd),
102+
"git --no-optional-locks -C \"%s\" status --porcelain --untracked-files=normal 2>%s",
103+
repo_path, null_dev);
104+
if (n < 0 || n >= (int)sizeof(cmd)) {
105+
return CBM_NOT_FOUND;
106+
}
107+
FILE *fp = cbm_popen(cmd, "r");
108+
if (!fp) {
109+
return CBM_NOT_FOUND;
110+
}
111+
char buffer[CBM_SZ_4K];
112+
while (fread(buffer, 1, sizeof(buffer), fp) > 0) {
113+
*dirty_out = true;
114+
}
115+
return cbm_pclose(fp) == 0 ? 0 : CBM_NOT_FOUND;
116+
}
117+
89118
static bool path_is_absolute(const char *path) {
90119
if (!path || !path[0]) {
91120
return false;
@@ -239,6 +268,36 @@ void cbm_git_context_free(cbm_git_context_t *ctx) {
239268
memset(ctx, 0, sizeof(*ctx));
240269
}
241270

271+
int cbm_git_snapshot_read(const char *path, cbm_git_snapshot_t *out) {
272+
if (!out) {
273+
return CBM_NOT_FOUND;
274+
}
275+
memset(out, 0, sizeof(*out));
276+
char *head_sha = NULL;
277+
int status = git_capture(path, "rev-parse --verify HEAD", &head_sha);
278+
if (status != 0 || !head_sha || !head_sha[0]) {
279+
free(head_sha);
280+
return status == 0 ? CBM_NOT_FOUND : status;
281+
}
282+
bool dirty = false;
283+
if (git_dirty_status(path, &dirty) != 0) {
284+
free(head_sha);
285+
return CBM_NOT_FOUND;
286+
}
287+
out->head_sha = head_sha;
288+
out->available = true;
289+
out->dirty = dirty;
290+
return 0;
291+
}
292+
293+
void cbm_git_snapshot_free(cbm_git_snapshot_t *snapshot) {
294+
if (!snapshot) {
295+
return;
296+
}
297+
free(snapshot->head_sha);
298+
memset(snapshot, 0, sizeof(*snapshot));
299+
}
300+
242301
int cbm_git_context_resolve(const char *path, cbm_git_context_t *out) {
243302
if (!out) {
244303
return CBM_NOT_FOUND;

0 commit comments

Comments
 (0)