Repository navigation
harden: ship integration templates as a verified JSON asset, not embedded bytes - #1493
Merged
Merged
Conversation
…dded bytes
The product binary embedded nine complete shebang'd shell scripts, their
PowerShell/.cmd twins, and two node:child_process client-adapter modules as C
string literals, and wrote them into agent config dirs at 0755 on install. A
block of script-shaped bytes inside a large unsigned executable is exactly the
surface Microsoft Defender's ML scored Trojan:Script/Wacatac.B!ml — the
detection that has recurred across the last several release attempts, on the
Linux ELF and darwin-arm64 standard binaries while every UI and Windows variant
stayed clean.
This moves every integration template out of the binary into a single shipped
data file, assets/cbm-integrations.json (compact: one asset, because each
shipped file is itself scanned). The binary now carries only the templates'
identity — one embedded SHA-256, generated at build time from the JSON so it
can never drift — plus the code to load, verify, and render them.
Why a hash and not just a loose file: content compiled into the binary was
tamper-evident with the binary. Moving it to disk must not turn it into
unauthenticated code the installer blindly executes. install verifies the asset
against the embedded SHA-256 and fails closed on mismatch ("integration assets
missing or modified - reinstall from the release archive") before writing
anything. Verified end to end: a one-byte edit to the asset makes install
refuse.
Ownership, for safe upgrade/removal: the verified copy is published to
~/.cbm/assets/<version>/ — a sibling of the DB cache, never inside it, so
clearing the cache cannot strip a user's hooks. A deployed integration file is
ours to remove only if it matches a template rendered from that stored copy;
foreign files and user-edited files at the reserved paths are preserved. The
historical template bodies travel in the JSON's released[] arrays so files
written by older versions are still recognised and cleaned up on uninstall.
Removed from src/: cmm_gate/session/subagent script-prefix and suffix
constants, cbm_build_released_gate_script, the shell/PowerShell/.cmd bodies in
cli.c, and the JS/TS generators in client_adapter.c. Confirmed on the built
standard binary: 0 occurrences of '#!/usr/bin/env bash', '#!/bin/sh',
'node:child_process', and 'ExecutionPolicy' (was non-zero for each).
test_no_embedded_scripts_contract.sh makes the removal a build property, not a
reachability assumption: no production C source may contain a shebang or
node:child_process literal, and the built binary is scanned directly. It strips
comments first (a contract a comment can satisfy is a false guard), and is
revert-checked — reintroducing one shebang literal makes it fail naming the
file:line. check-binary-composition.sh (A6) enforces the same needles on every
release artifact post-strip.
This does not, by itself, prove the detection is cured: the clean UI binary is a
strict superset of the flagged standard one, so no single passage is a
deterministic trigger. It removes the most coherent malware-shaped composite
from every shipped binary, aligns with how gh/starship/zoxide externalise
integration material, and makes the templates maintainable as data rather than
escaped C literals. Causal confirmation still needs a VirusTotal ablation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…t a JSON-license false positive Two CI failures on the integration-asset change, both build/gate plumbing, not the architecture: 1. scripts/gen-integrations-hash.sh was committed 100644 while Makefile.cbm invoked it directly, so a fresh CI checkout died with "Permission denied" building the hash header — which failed EVERY test leg at step 0, since the contract step builds first. It passed locally only because the generated header was already cached, so make never re-ran the generator. Fixed both ways: the recipe now runs it via `sh` (mode-independent, cannot regress from a checkout mode), and the file is committed 100755 to match its siblings. Note: test_script_exec_bit_contract.sh scans shell call sites, not Makefile recipes, so it did not catch this — the `sh` prefix is the durable guard. 2. The license gate (ScanCode) flagged scripts/package-release.sh with the SPDX 'JSON' license. The archive member lists place 'cbm-integrations.json' and 'LICENSE' adjacently, and ScanCode reads the '.json LICENSE' token adjacency as a JSON-license reference. The order is not free to change — the Windows single-binary contract locks the exact member sequence — so this is a genuine false positive on a first-party MIT build script. Added it to the policy's ignored_paths with the same justification the six existing first-party entries carry (the gate scripts, provenance auditor, and discover.c all name licenses for legitimate reasons). No allow-listed SPDX id was added; the JSON license is NOT now permitted anywhere else. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
enabled auto-merge
August 8, 2026 02:18
pr-smoke failed on all three platforms: the binary's install/uninstall render templates from the integration asset, and every place that stages the binary without the asset made those operations fail closed with "integration assets missing". The asset must travel with the binary in EVERY layout, not just the release archive. Five staging paths were missing it: - scripts/build.sh — stage cbm-integrations.json next to the freshly built binary, so `build/c/codebase-memory-mcp install` works straight out of a build tree (dev, and the base for the smoke fixture). - scripts/smoke-local.sh — include the asset in the fixture tarball and its required-sidecar check. Member set and ORDER now mirror package-release.sh exactly (binary, cbm-integrations.json, LICENSE, install.sh, notices); the fixture was smoking an archive layout we never actually ship. - install.sh / install.ps1 — after installing the binary, copy the asset beside it in the install dir. `install` already publishes a verified copy to ~/.cbm/assets/<version>/, but a later install/uninstall run from the install dir resolves the asset NEXT TO THE BINARY first, so without the adjacent copy that lookup misses and a re-install or uninstall fails on a machine that just installed successfully. Best-effort atomic rename, same shape as the existing install.sh/ps1 sidecar copy. install.ps1 stays pure ASCII. - scripts/smoke-test.sh Phase 14 — this phase hand-stages the binary into a fresh HOME without going through install, so nothing populates ~/.cbm/assets; stage the asset next to each staged copy so the uninstall it drives resolves. Verified: scripts/smoke-local.sh on the standard binary now runs clean through all 16 phases — zero "integration assets missing" / hook_script_uninstall / agent-cleanup-failed lines (was 5). windows-bundle, smoke-fixture, exec-bit and no-embedded-scripts contracts pass; install.ps1 has zero non-ASCII bytes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Round 2's smoke fix added the archive member list 'cbm-integrations.json LICENSE install.sh ...' to scripts/smoke-local.sh (mirroring package-release.sh), which reintroduced the exact ScanCode false positive that commit 83df500 quieted for package-release.sh: the '.json LICENSE' token adjacency reads as the SPDX 'JSON' license. Same class, same fix — smoke-local.sh is a first-party MIT build script listing its own archive members, added to the license policy's ignored_paths with the shared justification. Swept every ScanCode-scanned file (src/pkg/scripts, the gate's own file set) for the adjacency: the only three carrying it — package-release.sh, smoke-local.sh, and license-policy.json itself — are all now in ignored_paths, so this closes it rather than trading one flagged file for another. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows pr-smoke uses test-infrastructure/vm/vm-smoke.sh, the Windows analog of smoke-local.sh, which had the same three staging gaps the Unix fix closed: its required-sidecar check, fixture copy, and zip member list all omitted cbm-integrations.json, so the served archive lacked the asset and install/ uninstall failed closed with "integration assets missing". Plus one Windows-specific spot: vm-smoke.sh copies the smoke binary into $SMOKE_DIR and runs smoke-test.sh against it there, but the asset only reached $FIXTURE_DIR — so Phase 14 (which hand-stages the binary and drives an uninstall, resolving the asset next to the binary) missed it. Now staged next to the binary in $SMOKE_DIR as well. build.sh already places it next to build/c/*.exe, so the non-artifact source is $(dirname BINARY_SRC); artifact mode takes the release copy. Member set + order in the zip mirror scripts/package-release.sh (Windows). vm-smoke.sh lives under test-infrastructure/, outside the license gate's scan set, so the cbm-integrations.json/LICENSE adjacency raises no ScanCode finding there. Syntax-checked; vm-worktree-manifest, smoke-fixture and windows-bundle contracts pass. The Unix smoke already runs clean through all 16 phases; this mirrors that staging onto the one path I cannot exercise locally. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…namespace The prior fix put cbm-integrations.json into the Windows archive, and install.ps1 promptly rejected it: its pre-extraction namespace guard permits only the exact release entries in $WindowsArchiveNames and throws "archive contains an unexpected root entry" on anything else. The asset is now a real release member, so it belongs in that allow-list. Added in archive order (binary, cbm-integrations.json, LICENSE, install.ps1, THIRD_PARTY_NOTICES.md), matching package-release.sh and the windows-bundle contract's windows_archive_names. The guard also REQUIRES each listed entry exactly once, so this both permits and mandates the asset — an archive missing it now fails install.ps1 up front rather than at first hook render. install.sh has no equivalent namespace allow-list (it extracts the tarball directly), which is why the Unix smoke passed; no change needed there. install.ps1 stays pure ASCII (0 non-ASCII bytes); windows-bundle contract passes. This was the Windows smoke advancing PAST the archive-staging fix (asset now present) to the installer's own entry validation. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…ash header The -ui build failed on Windows with "cbm_integrations_hash.h file not found". integration_assets.c includes that generated header, and the subagent added it to PROJECT_HDRS so every single-invocation binary target picks it up — but cbm-with-ui is a phony target with its own direct recipe that does NOT list PROJECT_HDRS, so nothing generated the header before its one-shot compile. Only the UI variant hit this; the standard binary, test-runner, test-repro-runner and tsan targets all carry PROJECT_HDRS and built fine. Added $(INTEGRATIONS_HASH_HDR) as an explicit prerequisite of cbm-with-ui. Confirmed by dry-run: `make cbm-with-ui` now schedules the generator before the compile, and the header regenerates from clean. Audited every target that compiles integration_assets.c (via PROD_SRCS) — all now resolve the header, so this closes the build gap rather than moving it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
thientrile
pushed a commit
to thientrile/codebase-memory-mcp
that referenced
this pull request
Aug 10, 2026
Externalizing the integration templates (DeusData#1492/DeusData#1493) and the UI bundle (DeusData#1501/DeusData#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did not work: across dry runs the flagged artifact count stayed at ~3 and the detections merely moved between artifacts. Dissection of run 31286803592 shows there is no structural cause to fix. The verdicts split across every axis at once — linux-amd64 (dynamic) flagged while linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while linux-arm64-portable (static) is flagged. The two macOS binaries have identical segment structure and split clean/flagged. Siblings from one build landed in different variant buckets (.B vs .C). Entropy is low everywhere (code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for packed payloads), so the packed-payload hypothesis is excluded too. So the complexity bought nothing, and installation goes back to being self-contained: one binary that carries its own UI and agent integration templates, with no adjacent data file that has to resolve before `install` works. Only the UI-capable composition ships from now on, under the historical unsuffixed archive name. Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c, scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h}, assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the --verify-runtime-assets probe (nothing adjacent left to verify), and the composition gates A6/A7 whose property is now deliberately inverted. Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path. Kept from the reverted commits, re-applied by hand where a wholesale file restore would have dropped them: - cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders through the ANSI code page and mangles non-ASCII install paths. - the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so `daemon start --open` still waits for a genuine CBM listener. - X-Content-Type-Options: nosniff on served assets. - the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes. Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the names every static package manifest already points at — aur, chocolatey, homebrew, scoop, winget and glama were all broken by the -ui- rename. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1492 (which removed the
-ExecutionPolicy Bypassliteral). This is the structural change behind theTrojan:Script/Wacatac.B!mlwork.What & why
The binary embedded nine complete shebang'd shell scripts, their PowerShell/
.cmdtwins, and twonode:child_processclient-adapter modules as C string literals, and wrote them to agent config dirs at0755on install. Script-shaped bytes inside a large unsigned executable are exactly the surface Defender's ML scoredTrojan:Script/Wacatac.B!ml— the recurring detection on the Linux ELF and darwin-arm64 standard binaries, while every UI and Windows variant stayed clean.Every template now lives in one compact shipped data file,
assets/cbm-integrations.json(~8.5k). The binary carries only its identity — one build-time-generated SHA-256 — plus the code to load, verify and render it.Integrity (the security property we'd otherwise lose)
Content compiled into the binary was tamper-evident with the binary. Moving it to disk must not turn it into unauthenticated code the installer blindly runs.
installverifies the asset against the embedded SHA-256 and fails closed on mismatch before writing anything. Verified end-to-end: a one-byte edit →installrefuses with "integration assets missing or modified - reinstall from the release archive".Placement & ownership
~/.cbm/assets/<version>/— a sibling of the DB cache, never inside it, so clearing the cache can't strip a user's hooks.released[]arrays so files written by older versions are still recognised on uninstall.Verification
#!/usr/bin/env bash,#!/bin/sh,node:child_process,ExecutionPolicy~/.cbm/assets/dev/test_no_embedded_scripts_contract.shTEST_SEAMS=1, confirmed pass with it)check-binary-composition.sh(A6) enforces the same needles on every release artifact post-strip, andscripts/test.shStep 0l runs the contract on every leg.Honest scope
This does not by itself prove the detection is cured — the clean UI binary is a strict superset of the flagged standard one, so no single passage is a deterministic trigger. It removes the most coherent malware-shaped composite from every shipped binary, aligns with how gh/starship/zoxide externalise integration material, and makes the templates maintainable as data. Causal confirmation needs a VirusTotal ablation (blocked on
VT_KEY).Provenance note
Built with a Fable subagent that was interrupted mid-verification and left two guards disabled (
return true || <hash check>anddry_run || g_assets.length). Both were found by sweeping the diff for revert-check markers and restored; the integrity and install-copy paths are verified working above.