Skip to content

Commit 98d7dba

Browse files
committed
feat(ci): add a third release candidate (debug-stripped) as an extra VT draw
Release evidence from run 31744302624 shows the tolerated Microsoft `!ml` verdict is close to a coin flip per byte image rather than a property of the code. Across the eight targets the stripped and unstripped candidates of the SAME linker output disagreed on four, and in both directions: linux-amd64 stripped microsoft-ml unstripped clean darwin-arm64 stripped clean unstripped microsoft-ml linux-arm64 stripped clean unstripped microsoft-ml If the classifier were keying on something intrinsic to our code the siblings would agree; they do not. So each variant is close to an independent draw, and 5 of 16 candidates drew the flag. Two draws is not always enough. On that run linux-amd64-portable came back microsoft-ml on BOTH candidates, leaving no clean binary to ship for that target. A third independent draw at a ~31% observed per-candidate hit rate takes the both/all-flagged case from roughly 1-in-10 per target to roughly 1-in-30. The third candidate is `--strip-debug` (Apple: `-S`): debug information removed, symbol table kept. Behaviourally identical to the other two — same linker output, only metadata differs — but a distinct byte image, which is all VirusTotal needs to scan it as its own file. Verified on the real v0.10.4 candidates: linux-amd64 gives three distinct hashes (294,634,656 / 294,623,208 / 293,746,096 bytes) and darwin-arm64 likewise, with the ad-hoc signature verifying after strip. Selection is unchanged in spirit and now ordered: smallest artifact first (stripped, debug-stripped, unstripped), take the first CLEAN one, and only if every candidate drew the tolerated verdict ship the smallest flagged one. A hard verdict on any candidate still blocks the release before selection. Cost is 24 objects per release instead of 16. Derivation enforces that all three hashes differ — identical candidates would be one draw wearing three hats, and the selector would believe it had alternatives it does not have. Public claims in README, SECURITY.md and docs/index.html updated from "both stripped and unstripped" to the three candidates. All three release contract tests pass, including the native derivation test which exercises the real strip and codesign path on this host. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent db26e96 commit 98d7dba

9 files changed

Lines changed: 114 additions & 48 deletions

‎.github/workflows/_build.yml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -295,10 +295,10 @@ jobs:
295295
if-no-files-found: error
296296

297297
# One blocking join owns the security-sensitive transition from candidates
298-
# to public artifacts. Each target tuple contributes exactly two behavior-
299-
# equivalent candidates (stripped/unstripped); the selector may choose only
300-
# within that tuple. A hard or incomplete result for either sibling blocks
301-
# the all-target release rather than selecting around missing evidence.
298+
# to public artifacts. Each target tuple contributes exactly three behavior-
299+
# equivalent candidates (unstripped/debug-stripped/stripped); the selector may
300+
# only choose within that tuple. A hard result on ANY sibling blocks
301+
# the all-target release rather than selecting around a real detection.
302302
select-package:
303303
needs: [build-unix, build-windows, build-windows-arm64, build-linux-portable]
304304
runs-on: ubuntu-latest
@@ -324,14 +324,14 @@ jobs:
324324
"$RUNNER_TEMP/release-candidate-artifacts" \
325325
release-candidate-scan \
326326
--expect-targets 8 \
327-
--expect-candidates 16
327+
--expect-candidates 24
328328
# Staging copied and re-hashed every byte into the immutable object
329329
# set. Drop the redundant artifact-download tree before the VT wait.
330330
test -n "$RUNNER_TEMP"
331331
rm -rf -- "$RUNNER_TEMP/release-candidate-artifacts"
332332
test ! -e "$RUNNER_TEMP/release-candidate-artifacts"
333333
334-
- name: Submit all 16 candidates to VirusTotal
334+
- name: Submit all 24 candidates to VirusTotal
335335
if: ${{ inputs.scan_candidates }}
336336
uses: crazy-max/ghaction-virustotal@936d8c5c00afe97d3d9a1af26d017cfdf26800a2 # v5.0.0
337337
id: virustotal-candidates
@@ -379,7 +379,7 @@ jobs:
379379
- name: Reclaim rejected and redundant candidate bytes
380380
run: |
381381
# release-selection owns fresh, hash-verified copies of the eight
382-
# chosen executables. No later release step may consume the 16-object
382+
# chosen executables. No later release step may consume the 24-object
383383
# scan pool, so discard it before packaging to bound runner disk use.
384384
rm -rf -- release-candidate-scan/objects
385385
test ! -e release-candidate-scan/objects

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-si
2020

2121
> **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration.
2222
23-
> **Security & Trust** — This tool reads your codebase and writes to your agent configuration files. That is what it is designed to do. If you prefer to audit before running, the [full source is here](https://github.com/DeusData/codebase-memory-mcp). For each release product, both stripped and unstripped executable candidates are submitted to VirusTotal before testing; the selected candidate is then packaged with its SHA-256 unchanged. Release notes link every measured candidate result. Publication permits only the narrowly documented single-Microsoft `!ml` tolerance in [SECURITY.md](SECURITY.md#our-release-policy). All processing happens 100% locally; your code never leaves your machine. Found a security issue? We want to know — see [SECURITY.md](SECURITY.md). Security is Priority #1 for us.
23+
> **Security & Trust** — This tool reads your codebase and writes to your agent configuration files. That is what it is designed to do. If you prefer to audit before running, the [full source is here](https://github.com/DeusData/codebase-memory-mcp). For each release product, three behaviourally identical executable candidates (unstripped, debug-stripped, stripped) are submitted to VirusTotal before testing; the selected candidate is then packaged with its SHA-256 unchanged. Release notes link every measured candidate result. Publication permits only the narrowly documented single-Microsoft `!ml` tolerance in [SECURITY.md](SECURITY.md#our-release-policy). All processing happens 100% locally; your code never leaves your machine. Found a security issue? We want to know — see [SECURITY.md](SECURITY.md). Security is Priority #1 for us.
2424
2525
<p align="center">
2626
<img src="docs/graph-ui-screenshot.png" alt="Graph visualization UI showing the codebase-memory-mcp knowledge graph" width="800">
@@ -794,7 +794,7 @@ internal/cbm/ Vendored tree-sitter grammars (158 languages) + AST extrac
794794

795795
Every release is verified through a multi-layer pipeline before publication:
796796

797-
- **VirusTotal** — all 16 stripped/unstripped executable candidates across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Archive containers are checksummed, not redundantly resubmitted.
797+
- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Archive containers are checksummed, not redundantly resubmitted.
798798
- **SLSA Level 3** — cryptographic build provenance generated by the trusted GitHub Actions build workflow; verify with `gh attestation verify <file> --repo DeusData/codebase-memory-mcp --signer-workflow DeusData/codebase-memory-mcp/.github/workflows/_build.yml`
799799
- **Sigstore cosign** — keyless signatures on all artifacts; bundles included in every release
800800
- **SHA-256 checksums** — `checksums.txt` published with every release; verified by both install scripts before extraction

‎SECURITY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ Releases are created as **drafts** (invisible to users) and only published after
126126
2. **Sigstore cosign signing** — keyless digital signatures verifiable by anyone
127127
3. **SBOM** — Software Bill of Materials (SPDX) listing all vendored dependencies
128128
4. **SHA-256 checksums** — published with every release
129-
5. **VirusTotal scanning** — both stripped and unstripped executable candidates for every release product are scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256; archive containers are checksummed rather than redundantly rescanned. Release notes link every candidate result and the selection evidence.
129+
5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256; archive containers are checksummed rather than redundantly rescanned. Release notes link every candidate result and the selection evidence.
130130
6. **OpenSSF Scorecard** — repository security health score
131131

132132
Scope of the SLSA claim: this is a build provenance claim for release

‎docs/index.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@
165165
"name": "Does codebase-memory-mcp send my code anywhere?",
166166
"acceptedAnswer": {
167167
"@type": "Answer",
168-
"text": "No. All indexing and querying happen 100% locally on your machine. There is no embedded LLM and no API key — your MCP client acts as the intelligence layer. Before release testing, both stripped and unstripped executable candidates for every platform and link mode are submitted to VirusTotal. The selected candidate is packaged with its SHA-256 unchanged, and release notes link every measured result. Publication permits only the documented single-Microsoft !ml tolerance."
168+
"text": "No. All indexing and querying happen 100% locally on your machine. There is no embedded LLM and no API key — your MCP client acts as the intelligence layer. Before release testing, three behaviourally identical executable candidates (unstripped, debug-stripped, stripped) for every platform and link mode are submitted to VirusTotal. The selected candidate is packaged with its SHA-256 unchanged, and release notes link every measured result. Publication permits only the documented single-Microsoft !ml tolerance."
169169
}
170170
},
171171
{
@@ -791,8 +791,8 @@ <h2>Frequently asked questions</h2>
791791

792792
<h3>Does codebase-memory-mcp send my code anywhere?</h3>
793793
<p>No. All indexing and querying happen 100% locally. There is no embedded LLM and no API key. Release
794-
archives are signed and checksummed. Before testing, both stripped and unstripped executable candidates
795-
for every platform and link mode are submitted to VirusTotal; the selected candidate is packaged with its
794+
archives are signed and checksummed. Before testing, three behaviourally identical executable candidates
795+
(unstripped, debug-stripped, stripped) for every platform and link mode are submitted to VirusTotal; the selected candidate is packaged with its
796796
SHA-256 unchanged. Release notes link every measured result. Publication permits only the
797797
documented single-Microsoft <code>!ml</code> tolerance.</p>
798798

‎scripts/ci/prepare-release-candidates.sh‎

Lines changed: 49 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env bash
2-
# Derive the two final, immutable release candidates for one existing product
2+
# Derive the three final, immutable release candidates for one existing product
33
# tuple from a single linker output. Strip/signing happens here and nowhere
44
# after this boundary.
55
set -euo pipefail
@@ -11,8 +11,9 @@ usage() {
1111
Usage: scripts/ci/prepare-release-candidates.sh <goos> <goarch> \
1212
--binary FILE --out-dir DIR
1313
14-
Derive one unstripped and one stripped final executable for exactly one of the
15-
eight release product tuples. Both candidates come from the same linker output
14+
Derive three final executables - unstripped, debug-stripped and stripped - for
15+
exactly one of the eight release product tuples. All three come from the same
16+
linker output
1617
and are finalized before hashing and atomic publication. Runtime testing starts
1718
only after VirusTotal selection, using the selected candidate.
1819
@@ -139,12 +140,14 @@ trap cleanup EXIT
139140
STAGE="$WORK/$TARGET"
140141
BINARY_NAME=codebase-memory-mcp
141142
[ "$GOOS" = windows ] && BINARY_NAME=codebase-memory-mcp.exe
142-
mkdir -p "$STAGE/unstripped" "$STAGE/stripped"
143+
mkdir -p "$STAGE/unstripped" "$STAGE/debug-stripped" "$STAGE/stripped"
143144
UNSTRIPPED="$STAGE/unstripped/$BINARY_NAME"
145+
DEBUG_STRIPPED="$STAGE/debug-stripped/$BINARY_NAME"
144146
STRIPPED="$STAGE/stripped/$BINARY_NAME"
145147
cp "$BINARY" "$UNSTRIPPED"
148+
cp "$BINARY" "$DEBUG_STRIPPED"
146149
cp "$BINARY" "$STRIPPED"
147-
chmod 0755 "$UNSTRIPPED" "$STRIPPED"
150+
chmod 0755 "$UNSTRIPPED" "$DEBUG_STRIPPED" "$STRIPPED"
148151

149152
STRIP_INVOCATION="$(basename "$STRIP_TOOL") --strip-all"
150153
if ! "$STRIP_TOOL" --strip-all "$STRIPPED" 2>/dev/null; then
@@ -161,7 +164,31 @@ if ! "$STRIP_TOOL" --strip-all "$STRIPPED" 2>/dev/null; then
161164
STRIP_INVOCATION="$(basename "$STRIP_TOOL")"
162165
fi
163166

167+
# Third candidate: debug information removed, symbol table kept. Behaviourally
168+
# identical to the other two — same linker output, nothing but metadata differs
169+
# — but a distinct byte image, so VirusTotal scans it as its own file.
170+
#
171+
# That is the entire point. Release-run evidence shows the single tolerated
172+
# Microsoft `!ml` verdict landing on stripped and unstripped candidates of the
173+
# SAME build essentially at random (they disagreed on 4 of 8 targets, in both
174+
# directions), so each variant is an independent draw. With two candidates one
175+
# target came back flagged on both and had no clean binary to ship; a third
176+
# independent draw makes that outcome substantially rarer.
177+
DEBUG_STRIP_INVOCATION="$(basename "$STRIP_TOOL") --strip-debug"
178+
if ! "$STRIP_TOOL" --strip-debug "$DEBUG_STRIPPED" 2>/dev/null; then
179+
# Apple's strip rejects --strip-debug; -S is its measured equivalent.
180+
if [ "$GOOS" != darwin ]; then
181+
echo "prepare-release-candidates: $STRIP_TOOL --strip-debug failed" >&2
182+
exit 2
183+
fi
184+
cp "$BINARY" "$DEBUG_STRIPPED"
185+
chmod 0755 "$DEBUG_STRIPPED"
186+
"$STRIP_TOOL" -S "$DEBUG_STRIPPED"
187+
DEBUG_STRIP_INVOCATION="$(basename "$STRIP_TOOL") -S"
188+
fi
189+
164190
UNSTRIPPED_PRE_SIGN_SHA="$(sha256_file "$UNSTRIPPED")"
191+
DEBUG_STRIPPED_PRE_SIGN_SHA="$(sha256_file "$DEBUG_STRIPPED")"
165192
STRIPPED_PRE_SIGN_SHA="$(sha256_file "$STRIPPED")"
166193
SIGNATURE=not-applicable
167194
if [ "$GOOS" = darwin ]; then
@@ -171,21 +198,26 @@ if [ "$GOOS" = darwin ]; then
171198
exit 2
172199
}
173200
CODESIGN_TOOL="$(command -v "$CODESIGN_TOOL")"
174-
for candidate in "$UNSTRIPPED" "$STRIPPED"; do
201+
for candidate in "$UNSTRIPPED" "$DEBUG_STRIPPED" "$STRIPPED"; do
175202
"$CODESIGN_TOOL" --sign - --force --timestamp=none "$candidate"
176203
"$CODESIGN_TOOL" --verify --strict "$candidate"
177204
done
178205
SIGNATURE=adhoc-verified
179206
fi
180207

181208
UNSTRIPPED_SHA="$(sha256_file "$UNSTRIPPED")"
209+
DEBUG_STRIPPED_SHA="$(sha256_file "$DEBUG_STRIPPED")"
182210
STRIPPED_SHA="$(sha256_file "$STRIPPED")"
183-
[ "$UNSTRIPPED_SHA" != "$STRIPPED_SHA" ] || {
211+
# All three must differ: identical candidates would be one draw wearing three
212+
# hats, and the selector would believe it had alternatives it does not have.
213+
if [ "$UNSTRIPPED_SHA" = "$STRIPPED_SHA" ] || \
214+
[ "$UNSTRIPPED_SHA" = "$DEBUG_STRIPPED_SHA" ] || \
215+
[ "$DEBUG_STRIPPED_SHA" = "$STRIPPED_SHA" ]; then
184216
echo "prepare-release-candidates: strip did not produce byte-distinct candidates" >&2
185217
exit 2
186-
}
218+
fi
187219

188-
for candidate in "$UNSTRIPPED" "$STRIPPED"; do
220+
for candidate in "$UNSTRIPPED" "$DEBUG_STRIPPED" "$STRIPPED"; do
189221
bash "$ROOT/scripts/ci/check-binary-composition.sh" "$candidate" >/dev/null
190222
done
191223

@@ -194,6 +226,7 @@ done
194226
exit 2
195227
}
196228
[ "$(sha256_file "$UNSTRIPPED")" = "$UNSTRIPPED_SHA" ] && \
229+
[ "$(sha256_file "$DEBUG_STRIPPED")" = "$DEBUG_STRIPPED_SHA" ] && \
197230
[ "$(sha256_file "$STRIPPED")" = "$STRIPPED_SHA" ] || {
198231
echo "prepare-release-candidates: a final candidate changed during validation" >&2
199232
exit 2
@@ -212,6 +245,7 @@ linux-*) LINKAGE=dynamic ;;
212245
esac
213246

214247
UNSTRIPPED_SIZE="$(wc -c < "$UNSTRIPPED" | tr -d '[:space:]')"
248+
DEBUG_STRIPPED_SIZE="$(wc -c < "$DEBUG_STRIPPED" | tr -d '[:space:]')"
215249
STRIPPED_SIZE="$(wc -c < "$STRIPPED" | tr -d '[:space:]')"
216250
MANIFEST="$STAGE/candidate-provenance.tsv"
217251
{
@@ -221,13 +255,17 @@ MANIFEST="$STAGE/candidate-provenance.tsv"
221255
"$TARGET" "$BINARY_NAME" "$SOURCE_SHA" "$UNSTRIPPED_PRE_SIGN_SHA" \
222256
"$UNSTRIPPED_SHA" "$UNSTRIPPED_SIZE" "$FORMAT" "$ARCHITECTURE" \
223257
"$LINKAGE" "$SIGNATURE" "$STRIP_INVOCATION" "$STRIP_VERSION"
258+
printf '%s\tdebug-stripped\tdebug-stripped/%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\tstrip-debug\t%s\t%s\t%s\tsame-linker-output-v1\n' \
259+
"$TARGET" "$BINARY_NAME" "$SOURCE_SHA" "$DEBUG_STRIPPED_PRE_SIGN_SHA" \
260+
"$DEBUG_STRIPPED_SHA" "$DEBUG_STRIPPED_SIZE" "$FORMAT" "$ARCHITECTURE" \
261+
"$LINKAGE" "$SIGNATURE" "$DEBUG_STRIP_INVOCATION" "$STRIP_VERSION"
224262
printf '%s\tstripped\tstripped/%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\tstrip\t%s\t%s\t%s\tsame-linker-output-v1\n' \
225263
"$TARGET" "$BINARY_NAME" "$SOURCE_SHA" "$STRIPPED_PRE_SIGN_SHA" \
226264
"$STRIPPED_SHA" "$STRIPPED_SIZE" "$FORMAT" "$ARCHITECTURE" \
227265
"$LINKAGE" "$SIGNATURE" "$STRIP_INVOCATION" "$STRIP_VERSION"
228266
} > "$MANIFEST"
229267

230-
chmod 0555 "$UNSTRIPPED" "$STRIPPED"
268+
chmod 0555 "$UNSTRIPPED" "$DEBUG_STRIPPED" "$STRIPPED"
231269
chmod 0444 "$MANIFEST"
232270
mv "$STAGE" "$PUBLISHED"
233-
echo "prepare-release-candidates: published $TARGET stripped/unstripped pair"
271+
echo "prepare-release-candidates: published $TARGET unstripped/debug-stripped/stripped candidates"

0 commit comments

Comments
 (0)