Skip to content

Commit 5f82732

Browse files
committed
fix(ci): release notes report the shipped binary, not every candidate
Release notes now carry one row per product — the bytes actually published, with their VirusTotal verdict — instead of a column per candidate. Several candidates are scanned so the selector has an alternative when an opaque classifier flags one, but a reader installing cbm cares about what they receive, not about what we discarded. The rejected candidates' verdicts remain in the published evidence TSVs for anyone auditing the selection, and they stay useful to US as a development signal; in a changelog they are noise, and listing a "microsoft-ml" verdict next to a binary nobody can download invites exactly the wrong conclusion. Also removes a claim that had become FALSE: the section asserted that every scan had a minimum of 50 decisive engines. That floor was removed deliberately (it is VirusTotal fleet availability, not a property of our binary) and the last release observed 31. Published notes would have stated something untrue. The observed range is still reported, now over the shipped binaries. The disclosure of a tolerated Microsoft `!ml` is preserved and narrowed to the binaries that ship. The contract test now pins both directions: a flagged binary that SHIPS must be disclosed, and flagged candidates that were rejected must not appear at all. Its fixture gained a target that draws the tolerated verdict on all three candidates, so the disclosure branch is actually exercised — previously every flagged candidate was rejected in favour of a clean sibling, leaving that path untested. README and SECURITY.md updated: they promised notes "link every candidate result". All five release/VT contract tests pass. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent c0bd4bb commit 5f82732

4 files changed

Lines changed: 46 additions & 22 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -794,7 +794,7 @@ internal/cbm/ Vendored tree-sitter grammars (158 languages) + AST extrac
794794

795795
Every release is verified through a multi-layer pipeline before publication:
796796

797-
- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Every distinct object extracted from the shipped containers — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — is then scanned as well, so the full published surface is covered, not just the executables.
797+
- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, release notes link the verdict for the exact bytes shipped, and the full per-candidate evidence is published alongside the release as TSVs for anyone auditing the selection. Every distinct object extracted from the shipped containers — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — is then scanned as well, so the full published surface is covered, not just the executables.
798798
- **SLSA Level 3** — cryptographic build provenance generated by the trusted GitHub Actions build workflow; verify with `gh attestation verify <file> --repo DeusData/codebase-memory-mcp --signer-workflow DeusData/codebase-memory-mcp/.github/workflows/_build.yml`
799799
- **Sigstore cosign** — keyless signatures on all artifacts; bundles included in every release
800800
- **SHA-256 checksums** — `checksums.txt` published with every release; verified by both install scripts before extraction

‎SECURITY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ Releases are created as **drafts** (invisible to users) and only published after
126126
2. **Sigstore cosign signing** — keyless digital signatures verifiable by anyone
127127
3. **SBOM** — Software Bill of Materials (SPDX) listing all vendored dependencies
128128
4. **SHA-256 checksums** — published with every release
129-
5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256. After packaging, every distinct object extracted from the 14 shipped containers is scanned under the same policy — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — so the covered surface is everything we publish, not only the executables. Release notes link every candidate result and the selection evidence.
129+
5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256. After packaging, every distinct object extracted from the 14 shipped containers is scanned under the same policy — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — so the covered surface is everything we publish, not only the executables. Release notes link the verdict for the exact bytes shipped; the per-candidate results and the selection table are published as release assets for independent audit.
130130
6. **OpenSSF Scorecard** — repository security health score
131131

132132
Scope of the SLSA claim: this is a build provenance claim for release

‎scripts/ci/append-vt-notes.sh‎

Lines changed: 27 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -211,40 +211,52 @@ asset_base = (
211211
)
212212
213213
214-
def verdict_cell(target: str, variant: str) -> str:
215-
result = results_by_path[candidate_by_pair[target, variant]["scan_path"]]
214+
# Release notes report the SHIPPED binary and nothing else.
215+
#
216+
# Several candidates per product are scanned so the selector has an alternative
217+
# when an opaque classifier flags one of them, but a reader installing cbm cares
218+
# about the bytes they receive, not about the ones we discarded. The rejected
219+
# candidates' verdicts stay in the published evidence TSVs for anyone auditing
220+
# the selection, and they matter to US in development as a signal; they are
221+
# noise in a changelog.
222+
def shipped_result(target: str) -> dict:
223+
selection = selection_by_target[target]
224+
variant = selection["selected_variant"]
225+
return results_by_path[candidate_by_pair[target, variant]["scan_path"]]
226+
227+
228+
def verdict_cell(target: str) -> str:
229+
result = shipped_result(target)
216230
label = "clean" if result["policy_classification"] == "clean" else "Microsoft `!ml`"
217231
return f"[{label}]({result['virustotal_url']})"
218232
219233
220-
ml_count = sum(
221-
result["policy_classification"] == "microsoft-ml" for result in results
222-
)
223-
engine_counts = [int(result["completed_engines"]) for result in results]
234+
shipped = [shipped_result(target) for target in TARGETS]
235+
ml_count = sum(result["policy_classification"] == "microsoft-ml" for result in shipped)
236+
engine_counts = [int(result["completed_engines"]) for result in shipped]
224237
engine_range = str(min(engine_counts)) if min(engine_counts) == max(engine_counts) else f"{min(engine_counts)}–{max(engine_counts)}"
225238
section = [
226239
START,
227240
"## Security Verification",
228241
"",
229242
(
230-
f"Before smoke and soak testing, VirusTotal completed **{len(TARGETS) * len(VARIANTS)} executable scans**: "
231-
"stripped and unstripped candidates for each of the **8 release products**. "
232-
f"Every scan had at least 50 decisive engines (observed range: {engine_range})."
243+
f"Every binary published below was scanned by VirusTotal before smoke and soak "
244+
f"testing, and the verdict for the exact shipped bytes is linked per product "
245+
f"(decisive engines: {engine_range})."
233246
),
234247
(
235-
"All candidates were clean."
248+
"Every shipped binary was clean."
236249
if ml_count == 0
237-
else f"**{ml_count} candidate(s)** had only the documented single Microsoft machine-learning `!ml` result; no other decisive engine reported malicious or suspicious."
250+
else f"**{ml_count} shipped binary/binaries** carried only the documented single Microsoft machine-learning `!ml` result; no other decisive engine reported malicious or suspicious."
238251
),
239252
"",
240-
"| Product | Stripped candidate | Unstripped candidate | Shipped |",
241-
"|---|---|---|---|",
253+
"| Product | Shipped binary | VirusTotal verdict |",
254+
"|---|---|---|",
242255
]
243256
for target in TARGETS:
244257
selection = selection_by_target[target]
245258
section.append(
246-
f"| `{target}` | {verdict_cell(target, 'stripped')} | "
247-
f"{verdict_cell(target, 'unstripped')} | `{selection['selected_variant']}` (`{selection['selected_sha256']}`) |"
259+
f"| `{target}` | `{selection['selected_sha256']}` | {verdict_cell(target)} |"
248260
)
249261
section.extend(
250262
[

‎tests/test_vt_release_notes_contract.sh‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,12 @@ classes = {(target, variant): "clean" for target in targets for variant in varia
6565
classes["linux-amd64", "stripped"] = "microsoft-ml"
6666
classes["darwin-arm64", "stripped"] = "microsoft-ml"
6767
classes["darwin-arm64", "unstripped"] = "microsoft-ml"
68+
# windows-amd64 draws the tolerated verdict on ALL THREE candidates, so the
69+
# SHIPPED binary carries it and the notes must disclose exactly that one. The
70+
# two targets above keep a clean sibling, so their flagged candidates are
71+
# rejected and must NOT appear in the notes at all.
72+
for _variant in variants:
73+
classes["windows-amd64", _variant] = "microsoft-ml"
6874
candidates = []
6975
results = []
7076
by_pair = {}
@@ -190,12 +196,18 @@ run_notes "$FIX/current.md" "$FIX/first.md"
190196
grep -q 'Intro text.' "$FIX/first.md" || fail "content before section was lost"
191197
grep -q 'Outro text.' "$FIX/first.md" || fail "content after section was lost"
192198
! grep -q 'stale data' "$FIX/first.md" || fail "stale section was appended"
193-
grep -q '24 executable scans' "$FIX/first.md" || fail "candidate scan scope missing"
194-
grep -q '8 release products' "$FIX/first.md" || fail "product scope missing"
195-
grep -q '3 candidate(s).*Microsoft' "$FIX/first.md" || fail "tolerated findings not disclosed"
199+
# Release notes report ONLY the shipped bytes. Rejected candidates stay in the
200+
# published evidence TSVs; they are development signal, not changelog content.
201+
grep -q 'verdict for the exact shipped bytes is linked per product' "$FIX/first.md" || \
202+
fail "shipped-binary scan scope missing"
203+
[ "$(grep -c '^| `' "$FIX/first.md")" = "8" ] || fail "product scope missing: expected one row per release product"
204+
grep -q '1 shipped binary.*Microsoft' "$FIX/first.md" || fail "tolerated finding on a shipped binary not disclosed"
205+
grep -q 'candidate(s)' "$FIX/first.md" && fail "notes must not count rejected candidates"
196206
grep -q 'no other decisive engine reported malicious or suspicious' "$FIX/first.md" || fail "non-decisive engines are overstated as clean"
197207
! grep -q 'every other engine was clean' "$FIX/first.md" || fail "non-decisive engines are incorrectly described as clean"
198-
grep -q '| Product | Stripped candidate | Unstripped candidate | Shipped |' "$FIX/first.md" || fail "candidate table missing"
208+
grep -q '| Product | Shipped binary | VirusTotal verdict |' "$FIX/first.md" || fail "shipped-binary table missing"
209+
grep -q 'Stripped candidate' "$FIX/first.md" && \
210+
fail "release notes must not enumerate rejected candidates"
199211
grep -q '| `linux-amd64` |' "$FIX/first.md" || fail "Linux product row missing"
200212
grep -q '| `windows-arm64` |' "$FIX/first.md" || fail "Windows product row missing"
201213
grep -q 'archive containers were not redundantly submitted' "$FIX/first.md" || fail "no-rescan boundary missing"
@@ -228,4 +240,4 @@ if run_notes "$FIX/reversed.md" "$FIX/reversed-capture.md"; then
228240
fail "reversed verification markers must fail closed"
229241
fi
230242

231-
echo 'PASS: release notes reuse all 24 candidate verdicts, disclose selection, and avoid a duplicate scan'
243+
echo 'PASS: release notes report the shipped binary only, disclose selection, and avoid a duplicate scan'

0 commit comments

Comments
 (0)