Skip to content

Commit db26e96

Browse files
committed
fix(ci): stop gating the release on how many VirusTotal engines answered
The build+VT dry-run (run 31744302624) blocked all eight targets on: BLOCKED: objects/3eb486a4... completed with only 48/75 decisive engines (< 50) That object is the windows-arm64 STRIPPED candidate with ZERO detections. It was classified "hard" — the same bucket as a real malware verdict — purely because 48 engines answered instead of 50. Its own sibling scanned clean with 66. How many engines return a decisive result is VirusTotal fleet availability on the day, not a property of our binary. These artifacts are ~300 MB and many engines skip or time out at that size, so the count sits near the floor and moves run to run: on this run one object landed at 48 and the other fifteen spread 59-68. A floor there makes shipping a lottery decided by someone else's infrastructure, which is exactly what our CI-determinism rule forbids, and it is not something we can influence by fixing anything on our side. Classification now depends only on what engines FOUND: hard - two or more engines, any non-Microsoft engine, any label that is not `!ml`, or anything suspicious. Still blocks the release. microsoft-ml- exactly one Microsoft `!ml`. Tolerated and disclosed. clean - nothing flagged. The decisive-engine count is still recorded per candidate in the results manifest and printed as a NOTE when it is below the reference value, so the evidence is unchanged; only its power to block is gone. The selector keeps rejecting an incoherent response (total < completed). Public claims updated to match in README.md, SECURITY.md and docs/index.html — all three promised "at least 50 decisive engines" as a publication requirement, which would no longer have been true. Contract test updated: a clean verdict now stands regardless of engine count, and the below-reference NOTE must still be reported. All three release contract tests pass (vt-gate-policy, release-gate-chain, candidate-derivation). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 4cf9579 commit db26e96

6 files changed

Lines changed: 46 additions & 25 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-si
2020

2121
> **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration.
2222
23-
> **Security & Trust** — This tool reads your codebase and writes to your agent configuration files. That is what it is designed to do. If you prefer to audit before running, the [full source is here](https://github.com/DeusData/codebase-memory-mcp). For each release product, both stripped and unstripped executable candidates are submitted to VirusTotal before testing; the selected candidate is then packaged with its SHA-256 unchanged. Release notes link every measured candidate result. Publication requires at least 50 decisive engines and permits only the narrowly documented single-Microsoft `!ml` tolerance in [SECURITY.md](SECURITY.md#our-release-policy). All processing happens 100% locally; your code never leaves your machine. Found a security issue? We want to know — see [SECURITY.md](SECURITY.md). Security is Priority #1 for us.
23+
> **Security & Trust** — This tool reads your codebase and writes to your agent configuration files. That is what it is designed to do. If you prefer to audit before running, the [full source is here](https://github.com/DeusData/codebase-memory-mcp). For each release product, both stripped and unstripped executable candidates are submitted to VirusTotal before testing; the selected candidate is then packaged with its SHA-256 unchanged. Release notes link every measured candidate result. Publication permits only the narrowly documented single-Microsoft `!ml` tolerance in [SECURITY.md](SECURITY.md#our-release-policy). All processing happens 100% locally; your code never leaves your machine. Found a security issue? We want to know — see [SECURITY.md](SECURITY.md). Security is Priority #1 for us.
2424
2525
<p align="center">
2626
<img src="docs/graph-ui-screenshot.png" alt="Graph visualization UI showing the codebase-memory-mcp knowledge graph" width="800">
@@ -794,7 +794,7 @@ internal/cbm/ Vendored tree-sitter grammars (158 languages) + AST extrac
794794

795795
Every release is verified through a multi-layer pipeline before publication:
796796

797-
- **VirusTotal** — all 16 stripped/unstripped executable candidates across the eight release products are scanned before smoke/soak (minimum 50 decisive engines; clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Archive containers are checksummed, not redundantly resubmitted.
797+
- **VirusTotal** — all 16 stripped/unstripped executable candidates across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Archive containers are checksummed, not redundantly resubmitted.
798798
- **SLSA Level 3** — cryptographic build provenance generated by the trusted GitHub Actions build workflow; verify with `gh attestation verify <file> --repo DeusData/codebase-memory-mcp --signer-workflow DeusData/codebase-memory-mcp/.github/workflows/_build.yml`
799799
- **Sigstore cosign** — keyless signatures on all artifacts; bundles included in every release
800800
- **SHA-256 checksums** — `checksums.txt` published with every release; verified by both install scripts before extraction

‎SECURITY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ Releases are created as **drafts** (invisible to users) and only published after
126126
2. **Sigstore cosign signing** — keyless digital signatures verifiable by anyone
127127
3. **SBOM** — Software Bill of Materials (SPDX) listing all vendored dependencies
128128
4. **SHA-256 checksums** — published with every release
129-
5. **VirusTotal scanning** — both stripped and unstripped executable candidates for every release product are scanned before smoke/soak. Each completed analysis must contain at least 50 decisive engine results. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256; archive containers are checksummed rather than redundantly rescanned. Release notes link every candidate result and the selection evidence.
129+
5. **VirusTotal scanning** — both stripped and unstripped executable candidates for every release product are scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256; archive containers are checksummed rather than redundantly rescanned. Release notes link every candidate result and the selection evidence.
130130
6. **OpenSSF Scorecard** — repository security health score
131131

132132
Scope of the SLSA claim: this is a build provenance claim for release

‎docs/index.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@
165165
"name": "Does codebase-memory-mcp send my code anywhere?",
166166
"acceptedAnswer": {
167167
"@type": "Answer",
168-
"text": "No. All indexing and querying happen 100% locally on your machine. There is no embedded LLM and no API key — your MCP client acts as the intelligence layer. Before release testing, both stripped and unstripped executable candidates for every platform and link mode are submitted to VirusTotal. The selected candidate is packaged with its SHA-256 unchanged, and release notes link every measured result. Publication requires at least 50 decisive engines and permits only the documented single-Microsoft !ml tolerance."
168+
"text": "No. All indexing and querying happen 100% locally on your machine. There is no embedded LLM and no API key — your MCP client acts as the intelligence layer. Before release testing, both stripped and unstripped executable candidates for every platform and link mode are submitted to VirusTotal. The selected candidate is packaged with its SHA-256 unchanged, and release notes link every measured result. Publication permits only the documented single-Microsoft !ml tolerance."
169169
}
170170
},
171171
{
@@ -793,8 +793,8 @@ <h3>Does codebase-memory-mcp send my code anywhere?</h3>
793793
<p>No. All indexing and querying happen 100% locally. There is no embedded LLM and no API key. Release
794794
archives are signed and checksummed. Before testing, both stripped and unstripped executable candidates
795795
for every platform and link mode are submitted to VirusTotal; the selected candidate is packaged with its
796-
SHA-256 unchanged. Release notes link every measured result. Publication requires at least 50 decisive
797-
engines and permits only the documented single-Microsoft <code>!ml</code> tolerance.</p>
796+
SHA-256 unchanged. Release notes link every measured result. Publication permits only the
797+
documented single-Microsoft <code>!ml</code> tolerance.</p>
798798

799799
<h3>Does it support semantic or natural-language code search?</h3>
800800
<p>Yes. Alongside structural and full-text search, <code>search_graph</code>'s <code>semantic_query</code>

‎scripts/ci/check-virustotal.sh‎

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -658,14 +658,26 @@ def is_tolerated_detection(result: CompletedResult) -> bool:
658658
return engine == "Microsoft" and category == "malicious" and label.endswith("!ml")
659659
660660
661+
# Classification depends ONLY on what engines found, never on how many answered.
662+
#
663+
# "hard" means an engine actually found something we will not ship: two or more
664+
# engines, any non-Microsoft engine, any label that is not `!ml`, or anything
665+
# suspicious.
666+
#
667+
# How many engines returned a decisive result is NOT a policy input. It is a
668+
# property of VirusTotal's fleet on the day, which we cannot influence: these
669+
# binaries are ~300 MB and many engines skip or time out at that size, so the
670+
# count varies run to run (observed on one run: one object at 48, the other
671+
# fifteen spread 59-68). A 50-engine floor turned that variance into a release
672+
# blocker — it failed an 8-target release on a windows-arm64 candidate with
673+
# ZERO detections whose own sibling scanned clean at 66. Gating on it makes
674+
# shipping a lottery decided by someone else's infrastructure, so the count is
675+
# recorded as evidence and nothing more.
661676
def classify_result(result: CompletedResult, min_engines: int) -> str:
662-
if result.completed_engines < min_engines:
663-
return "hard"
664-
if result.malicious == 0 and result.suspicious == 0:
665-
return "clean"
666-
if is_tolerated_detection(result):
667-
return "microsoft-ml"
668-
return "hard"
677+
del min_engines # retained for signature stability; not a policy input
678+
if result.malicious or result.suspicious:
679+
return "microsoft-ml" if is_tolerated_detection(result) else "hard"
680+
return "clean"
669681
670682
671683
def main() -> None:
@@ -733,14 +745,14 @@ def main() -> None:
733745
completed.append(result)
734746
tolerated = is_tolerated_detection(result)
735747
if result.completed_engines < min_engines:
736-
message = (
737-
f"{submission.expected.scan_path} completed with only "
748+
# Reported, never fatal. See classify_result: engine count is
749+
# VirusTotal's fleet availability, not a property of our binary.
750+
print(
751+
f"NOTE: {submission.expected.scan_path} was judged by "
738752
f"{result.completed_engines}/{result.total_engines} decisive engines "
739-
f"(< {min_engines})"
753+
f"(below the {min_engines} reference); verdict still applies"
740754
)
741-
failures.append(message)
742-
print(f"BLOCKED: {message}")
743-
elif result.malicious or result.suspicious:
755+
if result.malicious or result.suspicious:
744756
message = (
745757
f"{submission.expected.scan_path} flagged "
746758
f"({result.malicious} malicious, {result.suspicious} suspicious / "

‎scripts/ci/select-release-candidates.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -320,8 +320,12 @@ def validate_result_row(row: dict[str, str], candidate: dict[str, str]) -> None:
320320
total = parse_nonnegative(row, "total_engines", scan_path=scan_path)
321321
malicious = parse_nonnegative(row, "malicious", scan_path=scan_path)
322322
suspicious = parse_nonnegative(row, "suspicious", scan_path=scan_path)
323-
if completed < MIN_ENGINES or total < completed:
324-
raise ContractError(f"VT result has incomplete engine coverage: {scan_path}")
323+
# `total < completed` stays: that is an incoherent response. A low decisive
324+
# count does NOT — how many engines answered is VirusTotal fleet
325+
# availability on the day, not a property of this binary, and gating on it
326+
# made an 8-target release fail on a candidate with zero detections.
327+
if total < completed:
328+
raise ContractError(f"VT result has incoherent engine coverage: {scan_path}")
325329
if ANALYSIS_ID_RE.fullmatch(row["analysis_id"]) is None:
326330
raise ContractError(f"VT result has invalid analysis id: {scan_path}")
327331
for field in ("microsoft_engine_version", "microsoft_engine_update"):

‎tests/test_vt_gate_policy_contract.sh‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -302,9 +302,14 @@ for id in one-malicious one-suspicious; do
302302
[ "$(run_gate "binaries/objects/probe=$(url_for "$id")")" != "0" ] || \
303303
fail "$id must block"
304304
done
305-
[ "$(run_gate "binaries/objects/probe=$(url_for low-engines)")" != "0" ] || \
306-
fail "completed analysis below the 50-engine policy must block"
307-
grep -q 'only 49/49 decisive engines' "$FIX/last.log" || fail "low-engine failure is not explicit"
305+
# A low decisive-engine count must NOT block. It is VirusTotal fleet
306+
# availability on the day, not a property of our binary: a ~300 MB release
307+
# artifact is skipped or timed out by many engines, so the count moves run to
308+
# run and a floor turns shipping into a lottery. It is reported, not enforced.
309+
[ "$(run_gate "binaries/objects/probe=$(url_for low-engines)")" = "0" ] || \
310+
fail "a clean verdict must stand regardless of how many engines answered: $(cat "$FIX/last.log")"
311+
grep -q 'NOTE: .*49/49 decisive engines' "$FIX/last.log" || \
312+
fail "a below-reference engine count must still be reported explicitly"
308313

309314
# A single Microsoft `!ml` verdict is the tolerated case: it passes, but it is
310315
# reported and recorded rather than silently downgraded to "clean".
@@ -394,4 +399,4 @@ printf 'tampered\n' > "$FIX/work/binaries/objects/probe"
394399
[ "$(run_gate "$clean_output")" != "0" ] || fail "tampered staged object must block before polling"
395400
grep -q 'changed after extraction' "$FIX/last.log" || fail "tamper failure is not diagnosable"
396401

397-
echo 'PASS: VT gate is exact-set, content-bound, >=50-engine; tolerates exactly one Microsoft !ml verdict and nothing else'
402+
echo 'PASS: VT gate is exact-set and content-bound; tolerates exactly one Microsoft !ml verdict and nothing else; engine count is evidence, not a gate'

0 commit comments

Comments
 (0)