Skip to content

Commit 6140094

Browse files
DeusDatamoofone
andcommitted
feat(artifact): reconcile imported hashes against git on bootstrap
An imported artifact carries the EXPORTER's mtime/size in every file_hashes row, so the first incremental run after a bootstrap sees every file as changed and re-parses ~the whole repository — most of the time the artifact was supposed to save. Reconcile the rows instead. After a successful import, ask the local git which files are unchanged between the artifact's commit and the working tree, and re-stamp exactly those rows with local stat() values. Everything git cannot vouch for stays foreign and is re-parsed. Export writes an optional "reconcile_basis":"git-clean-head" marker into the existing artifact.json, and only when the tree is provably clean outside .codebase-memory/ AND every hash row still matches disk. Unknown fields are ignored by older binaries, so no schema_version bump. Eligibility is deliberately narrow, and the polarity is skip-on-doubt: a row is restamped only if it is tracked at the artifact commit (git diff/ls-files are blind to ignored files — a .cbmignore negation can index a gitignored file, DeusData#500) and absent from the changed set. Any uncertainty — no git, missing/non-hex commit, shallow clone, truncated -z output, allocation failure — skips reconciliation entirely and falls back to today's behavior. Reconciliation can never fail the import. Windows correctness: every git argument uses DOUBLE quotes, per the rule already stated in artifact.c. cbm_popen runs cmd.exe /c on Windows, which does not honor single quotes, so a single-quoted ":(exclude)" pathspec made the clean-tree probe report dirty and the marker was never written there. The commit-existence probe uses `cat-file -t <oid>` rather than `cat-file -e <oid>^{commit}` because ^ is cmd.exe's escape character. Fail closed on allocation failure: both cbm_ht_create() results are checked, and every insertion is verified to have landed (cbm_ht_set discards the underlying insert's status, so OOM drops entries silently). A dropped entry in the changed set would make a genuinely modified file read as unchanged and be restamped — a silently stale graph. This is the one place where the membership-set primitive has unsafe polarity; classify_files uses it where a dropped entry only means "re-parse". Symlinks are refused rather than followed when stamping: git tracks a symlink's link text, not its target's content, so "unchanged" from git says nothing about the bytes the indexer parses. Uses the established lstat + S_ISLNK / reparse-point idiom (discover.c, pass_pkgmap.c, pass_envscan.c), which on Windows also keeps non-ASCII paths off the ANSI CRT. The trust trade-off is documented at both the import site and in artifact.h: foreign mtimes currently force a re-parse that auto-scrubs a poisoned artifact at an unpredictable clone time, so today's exposure is transient and self-healing. Restamping makes it durable and gates it on a producer-written string — acceptable only because the marker alone never suffices, as every restamped row must additionally be proven unchanged by the local git against a commit present in this clone. Also skip synthetic file_hashes rows. The pipeline stores semantic-input digests under .codebase-memory/.semantic-input/ that have no file on disk; stat()ing them made the export-side check fail unconditionally, so the marker was never written on ANY platform and the whole feature was inert. find_deleted_files skips the same class. Both halves of the trust check now ignore the .codebase-memory/ subtree, matching the clean-tree pathspec. Distilled from DeusData#868 by moofone; DeusData#869 and DeusData#870 are not included. Co-authored-by: moofone <35233167+moofone@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 8f112f1 commit 6140094

4 files changed

Lines changed: 1065 additions & 11 deletions

File tree

‎src/mcp/mcp.c‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7698,7 +7698,28 @@ static void try_artifact_bootstrap(const char *project_name, const char *repo_pa
76987698
project_db_path(project_name, db_buf, sizeof(db_buf));
76997699
if (cbm_file_size(db_buf) < 0 && cbm_artifact_exists(repo_path)) {
77007700
cbm_log_info("index.artifact_bootstrap", "project", project_name);
7701-
cbm_artifact_import(repo_path, db_buf);
7701+
/* An imported artifact is trusted for graph CONTENT as-is — nothing
7702+
* verifies that its nodes/edges describe the code they claim to. What
7703+
* has been limiting the blast radius is mechanical, not a check: every
7704+
* imported row carries the EXPORTER's mtime, so the first incremental
7705+
* run re-parses ~everything and auto-scrubs a poisoned artifact at a
7706+
* clone time the producer cannot predict. That exposure is transient
7707+
* and self-healing.
7708+
*
7709+
* cbm_artifact_reconcile_hashes deliberately trades part of that away
7710+
* for the speed the artifact is supposed to deliver (#885): rows it
7711+
* restamps are no longer re-parsed, so poisoned nodes for those files
7712+
* persist until the file changes — a DURABLE exposure gated on a
7713+
* producer-written marker. It stays acceptable only because the marker
7714+
* alone never suffices: each restamped row must additionally be proven
7715+
* unchanged by the LOCAL git against a commit present in this clone.
7716+
* Read the trade-off note in artifact.h before widening it.
7717+
*
7718+
* Best-effort: a -1 return leaves every row foreign and falls back to
7719+
* today's behavior, so a failure here can never fail the import. */
7720+
if (cbm_artifact_import(repo_path, db_buf) == 0) {
7721+
(void)cbm_artifact_reconcile_hashes(repo_path, db_buf, project_name);
7722+
}
77027723
}
77037724
}
77047725

0 commit comments

Comments
 (0)