Repository navigation
Commit 6140094
feat(artifact): reconcile imported hashes against git on bootstrap
An imported artifact carries the EXPORTER's mtime/size in every file_hashes
row, so the first incremental run after a bootstrap sees every file as
changed and re-parses ~the whole repository — most of the time the artifact
was supposed to save.
Reconcile the rows instead. After a successful import, ask the local git
which files are unchanged between the artifact's commit and the working
tree, and re-stamp exactly those rows with local stat() values. Everything
git cannot vouch for stays foreign and is re-parsed.
Export writes an optional "reconcile_basis":"git-clean-head" marker into the
existing artifact.json, and only when the tree is provably clean outside
.codebase-memory/ AND every hash row still matches disk. Unknown fields are
ignored by older binaries, so no schema_version bump.
Eligibility is deliberately narrow, and the polarity is skip-on-doubt:
a row is restamped only if it is tracked at the artifact commit (git
diff/ls-files are blind to ignored files — a .cbmignore negation can index
a gitignored file, DeusData#500) and absent from the changed set. Any uncertainty —
no git, missing/non-hex commit, shallow clone, truncated -z output,
allocation failure — skips reconciliation entirely and falls back to
today's behavior. Reconciliation can never fail the import.
Windows correctness: every git argument uses DOUBLE quotes, per the rule
already stated in artifact.c. cbm_popen runs cmd.exe /c on Windows, which
does not honor single quotes, so a single-quoted ":(exclude)" pathspec made
the clean-tree probe report dirty and the marker was never written there.
The commit-existence probe uses `cat-file -t <oid>` rather than
`cat-file -e <oid>^{commit}` because ^ is cmd.exe's escape character.
Fail closed on allocation failure: both cbm_ht_create() results are checked,
and every insertion is verified to have landed (cbm_ht_set discards the
underlying insert's status, so OOM drops entries silently). A dropped entry
in the changed set would make a genuinely modified file read as unchanged
and be restamped — a silently stale graph. This is the one place where the
membership-set primitive has unsafe polarity; classify_files uses it where a
dropped entry only means "re-parse".
Symlinks are refused rather than followed when stamping: git tracks a
symlink's link text, not its target's content, so "unchanged" from git says
nothing about the bytes the indexer parses. Uses the established
lstat + S_ISLNK / reparse-point idiom (discover.c, pass_pkgmap.c,
pass_envscan.c), which on Windows also keeps non-ASCII paths off the ANSI
CRT.
The trust trade-off is documented at both the import site and in
artifact.h: foreign mtimes currently force a re-parse that auto-scrubs a
poisoned artifact at an unpredictable clone time, so today's exposure is
transient and self-healing. Restamping makes it durable and gates it on a
producer-written string — acceptable only because the marker alone never
suffices, as every restamped row must additionally be proven unchanged by
the local git against a commit present in this clone.
Also skip synthetic file_hashes rows. The pipeline stores semantic-input
digests under .codebase-memory/.semantic-input/ that have no file on disk;
stat()ing them made the export-side check fail unconditionally, so the
marker was never written on ANY platform and the whole feature was inert.
find_deleted_files skips the same class. Both halves of the trust check now
ignore the .codebase-memory/ subtree, matching the clean-tree pathspec.
Distilled from DeusData#868 by moofone; DeusData#869 and DeusData#870 are not included.
Co-authored-by: moofone <35233167+moofone@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>1 parent 8f112f1 commit 6140094
4 files changed
Lines changed: 1065 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7698 | 7698 | | |
7699 | 7699 | | |
7700 | 7700 | | |
7701 | | - | |
| 7701 | + | |
| 7702 | + | |
| 7703 | + | |
| 7704 | + | |
| 7705 | + | |
| 7706 | + | |
| 7707 | + | |
| 7708 | + | |
| 7709 | + | |
| 7710 | + | |
| 7711 | + | |
| 7712 | + | |
| 7713 | + | |
| 7714 | + | |
| 7715 | + | |
| 7716 | + | |
| 7717 | + | |
| 7718 | + | |
| 7719 | + | |
| 7720 | + | |
| 7721 | + | |
| 7722 | + | |
7702 | 7723 | | |
7703 | 7724 | | |
7704 | 7725 | | |
| |||
0 commit comments