From ad701ac43a6a46b5ecf9f00a6a909f0ced22473f Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Sun, 23 Aug 2026 21:08:40 +0300 Subject: [PATCH 01/20] =?UTF-8?q?docs(specs):=20add=20101-tpa-db=20?= =?UTF-8?q?=E2=80=94=20versioned=20signed=20offline=20TPA=20signature=20da?= =?UTF-8?q?tabase?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec-only (speckit stage 1) for the roadmap P1 tpa-db epic: signed, sequence-versioned bundle format with anti-downgrade + rollback; loader verify-before-parse extending the 086 fail-closed pipeline; seed corpus strategy (>=25 signatures, provenance/license, eval sample pairs); the publication channel riding the spec-087 refresh lifecycle; post-activation informational re-scan; freshness telemetry. Eval-gated on the existing scan-eval recall/FP CI bar. --- specs/101-tpa-db/spec.md | 229 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 specs/101-tpa-db/spec.md diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md new file mode 100644 index 000000000..66efb97c1 --- /dev/null +++ b/specs/101-tpa-db/spec.md @@ -0,0 +1,229 @@ +# Feature Specification: Versioned, Signed, Offline-First TPA Signature Database (tpa-db) + +**Feature Branch**: `101-tpa-db` +**Created**: 2026-08-23 +**Status**: Draft +**Input**: Roadmap P1 epic `tpa-db` — "Build a versioned, offline-first signature/pattern database (known TPA campaigns, malicious phrase corpora, IoC hashes) that the engine consumes — bundled with the binary, refreshable out-of-band, community-contributable, and guarded by the existing scan-eval recall/FP CI gate." Tasks: `tpa-db-format` (signature DB format + loader: versioned, signed, bundled default), `tpa-db-corpus` (seed corpus of known public TPA campaigns/patterns), `tpa-db-refresh` (out-of-band refresh, offline-friendly, eval-gated). + +## Positioning *(context)* + + + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Signed database format + trust-anchored, downgrade-proof loader (Priority: P1) + +An operator receives a TPA signature database out of band — a `scanner-bundle.json` plus its detached signature file — and drops both into the configured bundle location (or lets the opt-in Spec 087 fetch retrieve them). Before anything else touches the bytes, mcpproxy verifies the signature against its built-in publisher trust anchors (or an operator-configured key), checks that the candidate's publish sequence is not older than what is already active, and only then runs the existing validate-before-activate pipeline (version compatibility, pattern compilation, activation self-check). A tampered bundle, a bundle signed by an unknown key, or a replayed older bundle never activates: the last-known-good database keeps serving and the rejection reason is recorded and surfaced. When the operator genuinely needs to roll back (a bad signature pushed by the publisher), an explicit, loudly-logged rollback action reverts to the embedded default or a named prior bundle. + +**Why this priority**: The database is a code-adjacent security input: whoever controls its bytes controls what the scanner ignores. Without signing and anti-rollback, the "refreshable signature DB" pillar creates a new supply-chain surface (swap the file → blind the scanner; replay an old bundle → resurrect a fixed blind spot). Signing + sequence + fail-closed loading is what makes every other story safe to ship, so it goes first. + +**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check. Finally, invoke the explicit rollback action and assert the embedded default is restored and the override is logged. + +**Acceptance Scenarios**: + +1. **Given** a candidate bundle whose detached signature verifies against a configured trust anchor and whose sequence is greater than the active bundle's, **When** refresh runs, **Then** the candidate proceeds through the existing validation pipeline and, on success, becomes active with its sequence persisted and `signature_verified=true` surfaced in bundle status. +2. **Given** a candidate whose bytes were altered after signing (signature mismatch), **When** refresh runs, **Then** the candidate is rejected BEFORE its JSON is parsed or any pattern compiled, the active bundle is unchanged, and a "signature verification failed" reason is recorded. +3. **Given** a validly-signed candidate whose sequence is lower than the highest sequence ever activated on this install, **When** refresh runs, **Then** it is refused as a downgrade, the active bundle is unchanged, and a "downgrade refused" reason names both sequences. +4. **Given** `require_signed_bundle` is enabled and a file-drop candidate has no signature sidecar, **When** refresh runs, **Then** the candidate is refused (fail-closed) with a "signature required" reason; with the setting disabled (default), an unsigned drop is accepted through the existing Spec 086 validation path but surfaced as `signature_verified=false`. +5. **Given** a network-fetched candidate (Spec 087 opt-in path), **When** it arrives, **Then** signature verification is ALWAYS mandatory regardless of `require_signed_bundle` — an unsigned or wrongly-signed fetched bundle is never activated (parity with Spec 087 FR-013/FR-014). +6. **Given** an operator invokes the explicit rollback action naming the embedded default (or a prior bundle file), **When** it runs, **Then** the target is re-validated, activated even though its sequence is lower, the anti-downgrade floor is reset to the rolled-back sequence, and the override is loudly logged and visible in bundle status. +7. **Given** an air-gapped host with no network at any point, **When** any of the above scenarios run via file drop, **Then** behavior is identical — signature verification, sequence checks, and rollback require no network. + +--- + +### User Story 2 - Seed corpus: catalog known public TPA campaigns, eval-gated (Priority: P2) + +A security-conscious developer installs mcpproxy and gets, out of the box, a signature database that actually catalogs the publicly known Tool Poisoning Attack landscape — not 6 demo signatures. Hidden-instruction blocks (the `TPA-2026-0001` `…read ~/.ssh/id_rsa…` class), hidden HTML/comment directives, tool shadowing and cross-server override instructions, exfiltration redirects ("also send the result to…"), sensitive-file coaxing, and rug-pull phrasings each have cataloged signatures with a stable `TPA-YYYY-NNNN` identity, a provenance record pointing at the public disclosure that motivated them, and a redistributable license. Every signature ships with labeled eval samples, and no corpus change — addition, tightening, or removal — can land unless the existing scan-eval CI gate stays green. + +**Why this priority**: The format (US1) is worthless empty, and the refresh channel (US3) has nothing to carry without a corpus. This is the "database" in tpa-db. It is P2 only because the P1 trust machinery must exist before a larger corpus becomes an attractive distribution target. + +**Independent Test**: Run the corpus build and assert every signature carries id, category, detector(s), provenance source, and license; assert every gated signature has ≥1 labeled malicious eval sample and ≥1 category-matched hard-negative in the eval dataset; run `go run ./cmd/scan-eval --corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json --gate --min-recall 0.90 --max-fp 0.05` with the new embedded default and assert PASS; feed each campaign class's canonical fixture through the offline scanner and assert a hard-tier finding naming the TPA id. + +**Acceptance Scenarios**: + +1. **Given** the seed corpus is built into the embedded default bundle, **When** the offline scanner inspects a tool description carrying any cataloged campaign class's canonical payload (e.g. the TPA-2026-0001 hidden-instruction string), **Then** a hard-tier finding fires naming the matched `TPA-YYYY-NNNN` id, and a `scan`-mode gate holds the tool/server. +2. **Given** any signature in the corpus, **When** its metadata is inspected, **Then** it carries provenance (public source reference) and a license permitting redistribution; a contribution without a redistributable license is refused at corpus build time (Spec 087 FR-021 parity). +3. **Given** a proposed corpus change that would drop gated-category recall below 0.90 or push hard-negative false positives above 0.05 on the frozen eval dataset, **When** the CI gate runs, **Then** the change fails CI and cannot merge; the same regressing bundle, if force-published, is additionally rejected at activation by the Spec 087 self-check (two independent layers). +4. **Given** a new signature intended to gate approvals, **When** it is authored, **Then** it emits hard-tier signals (the eval gate scores hard tier only) and its eval samples follow the dataset validator's conventions, including `hn__*` naming for hard-negatives. +5. **Given** benign-but-spicy tool descriptions (security tooling that legitimately mentions credentials, docs that quote attack examples), **When** scanned with the full seed corpus, **Then** they do not fire hard-tier findings — each campaign class is paired with hard-negatives that pin this down. + +--- + +### User Story 3 - Publication channel + adoption loop (freshness you can see, refresh that re-protects) (Priority: P3) + +The publisher cuts a new database release (new signature for a fresh campaign), signs it, and publishes it to a stable public location. A connected install with the Spec 087 opt-in fetch enabled picks it up within one daily cycle; an air-gapped operator downloads the same two files from any machine and drops them in. In both cases the operator can see — in `mcpproxy security` CLI output, the REST status, and the Web UI — which database version/sequence is active, whether its signature verified, and how stale it is. After a new database activates, mcpproxy re-evaluates the already-approved toolsets it has cached against the new signatures off the hot path and surfaces any new hits as review findings — without auto-revoking approvals — so a signature published today protects servers approved last month. Anonymous telemetry gains the active bundle version/sequence and source so the funnel "published → fetched → active → detected" is finally measurable. + +**Why this priority**: This closes the loop that makes the DB worth updating. Verified adoption analysis (2026-08-22) showed TPA adoption is structurally gated: refresh as spec'd in 087 only swaps the bundle and never re-scans, so a refreshed DB would protect only future admissions. P3 because it strictly builds on US1's artifacts and 087's lifecycle, and the product is already safer with US1+US2 alone. + +**Independent Test**: Serve a signed release from a local test endpoint; enable the 087 fetch against it; assert activation within one refresh cycle and that bundle status (CLI, REST, Web UI) reports the new version, sequence, source=fetch, signature_verified=true, and generated_at. Separately: with a server already approved whose cached tool description matches ONLY a signature added in the new bundle, activate the new bundle and assert a review finding naming the TPA id appears for that tool without the approval being revoked or the server quarantined. Assert heartbeat telemetry (when enabled) carries the active bundle version/sequence/source, and carries nothing when telemetry is opted out. + +**Acceptance Scenarios**: + +1. **Given** a signed database release published at the channel's stable location, **When** an install with the opt-in fetch enabled completes its next daily cycle, **Then** the new database is verified, gated, and active, and the same artifact pair downloaded manually and file-dropped on an air-gapped install activates identically. +2. **Given** an active database, **When** the operator inspects bundle status on any surface (CLI, REST, Web UI), **Then** they see bundle version, sequence, fingerprint, source (embedded/file/fetch), signature-verified state, generated_at, and last refresh outcome incl. the last rejection reason. +3. **Given** a newly-activated database containing a signature absent from the previous one, **When** the post-activation re-scan runs over cached, already-approved tool metadata, **Then** tools matching the new signature surface as review findings naming the TPA id — approvals are NOT auto-revoked and servers are NOT auto-quarantined by refresh alone. +4. **Given** telemetry is enabled, **When** the heartbeat fires, **Then** it includes active bundle version, sequence, and source alongside the existing anonymous TPA-scanner stats; **Given** telemetry is disabled, **Then** nothing bundle-related is sent (existing opt-out respected). + +--- + +### Edge Cases + +- **Air-gapped host, forever**: every P1 behavior (verification, sequence check, rollback, activation self-check) is network-free; the trust anchors ship inside the binary; staleness is surfaced but never blocks scanning — an old database is degraded coverage, not an outage. +- **Signature key rotation**: the trust anchor is a SET of publisher keys embedded per binary release. Rotation is a binary-release event: a release ships old+new keys, bundles are signed with the new key, the old key is removed in a later release. Keys are NEVER delivered over the network, and rotation is not done casually — every shipped binary pins its key set, so a hasty rotation strands older installs on unsigned-refresh only (same philosophy as the app-update signing keys). Key compromise → remove the key in the next release AND publish a higher-sequence bundle signed by the surviving key. +- **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade floor still blocks sequence replay. +- **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. +- **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted floor does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the floor retained and the situation surfaced. +- **Anti-downgrade state lost** (data directory wiped/reset): the floor resets; this is accepted — the floor is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. +- **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). +- **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. +- **Huge or adversarial database** (thousands of rules, pathological regex): size and rule-count ceilings are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. +- **Post-activation re-scan storms**: the re-scan is off the hot path, rate-limited, runs once per activation over cached metadata only (no upstream reconnects), and produces at most one review finding per tool per bundle activation — no notification storm. +- **Signature removed from a newer bundle** (false-positive retired): tools previously held by it are NOT auto-approved by refresh; the held state persists for human review (state changes only flow toward review, never silently toward approval). + +## Requirements *(mandatory)* + +### Functional Requirements + +**Database format & signing** + +- **FR-001**: The publishable database artifact MUST be the existing compiled bundle (`scanner-bundle.json`, Scanner Bundle Contract) plus a detached signature sidecar over the exact bundle bytes. Bundle bytes MUST remain deterministic for a given corpus (byte-identical rebuilds), so the signature and the fingerprint are stable. +- **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), and a publisher key identifier. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). +- **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. +- **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. +- **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure (drop key next release + publish higher-sequence bundle under surviving key) and the "do not rotate casually" operational stance. + +**Loader & activation** + +- **FR-006**: For any candidate carrying a signature, verification MUST run BEFORE JSON parsing or pattern compilation; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). +- **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature (Spec 087 FR-013/FR-014 unchanged); file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. +- **FR-008**: The full validate-before-activate order MUST be: signature (per policy) → sequence/anti-downgrade → version compatibility → parse → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. +- **FR-009**: The system MUST persist the highest sequence ever activated (per data directory) and refuse any candidate with a lower sequence — and any candidate with an equal sequence but different content fingerprint — as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. The embedded default competes under the same rule; when no higher-sequence valid candidate exists, the embedded default still serves (scanning never stops) with the retained floor surfaced. +- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, the persisted floor is reset to the rolled-back sequence, and the override is loudly logged and visible in bundle status. Rollback MUST be the ONLY path that lowers the floor. +- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). + +**Seed corpus** + +- **FR-012**: The seed corpus MUST catalog known public TPA campaign/technique classes as signatures with stable `TPA-YYYY-NNNN` identities, covering at minimum: hidden-instruction blocks (the TPA-2026-0001 class), hidden HTML/comment directives, tool shadowing and cross-server override, exfiltration redirects, sensitive-file coaxing (SSH keys, cloud credentials), and rug-pull phrasing patterns. Each signature MUST carry category, detector(s), level, confidence, provenance (public source reference), and a redistributable license; corpus build MUST refuse contributions lacking either (Spec 087 FR-021 parity). +- **FR-013**: Every signature intended to gate approvals MUST ship with eval evidence in the frozen eval dataset: at least one labeled gated-malicious sample and at least one category-matched hard-negative following the dataset validator's conventions (including `hn__*` naming), so the gate can measure both its recall contribution and its false-positive risk. +- **FR-014**: Gating signatures MUST emit hard-tier signals (the eval gate and the `scan`-mode approval gates score hard tier only); any soft-tier-only signature MUST be explicitly marked non-gating in its metadata and MUST NOT be counted toward gate coverage. +- **FR-015**: Signatures MUST be authored in the corpus source-of-truth pipeline (signature sources compiled to the bundle); mcpproxy MUST continue to consume only the compiled bundle and never parse signature sources (Spec 087 FR-002 parity). The contribution pipeline MUST run the same validation the loader runs (compile, schema, license, eval gate) before a bundle can be signed. +- **FR-016**: The CI eval gate (`cmd/scan-eval --gate`, recall ≥ 0.90 over gated categories, hard-negative FP rate ≤ 0.05, existing vacuity guard) MUST pass for: every change to the embedded default corpus (merge-blocking, existing `eval.yml` D2 job) and every published bundle BEFORE it is signed — a bundle that fails the gate is never signed or published. Thresholds are reused verbatim, not redefined. + +**Publication & refresh channel** + +- **FR-017**: Published databases MUST be versioned, signed artifact pairs (bundle + sidecar) at a stable public location with a discoverable "latest" reference; the Spec 087 opt-in fetch consumes this channel unchanged, and manual download + file drop MUST remain a first-class, documented, equally-capable path. [NEEDS CLARIFICATION: hosting location for the published artifacts — GitHub Releases on the tpa-db corpus repo vs. a mcpproxy.app static path (affects URL stability, availability SLO, and bandwidth, not behavior)] +- **FR-018**: This feature MUST NOT introduce a second refresh lifecycle: cadence, single-flight, on-demand trigger, activation self-check, and fail-safe behavior are Spec 087's; this spec only inserts the signature and sequence stages (FR-008) and defines the artifacts flowing through it. +- **FR-019**: After a new database activates, the system MUST run a post-activation informational re-scan over cached tool metadata of already-approved toolsets, off the hot path, without connecting to upstreams: new hard-tier hits surface as review findings naming the TPA id. Refresh alone MUST NOT auto-revoke approvals, auto-quarantine servers, or auto-approve previously-held tools; all state changes from re-scan flow toward human review only. + +**Observability, telemetry & docs** + +- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". +- **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. +- **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. + +### Key Entities *(include if feature involves data)* + +- **Signature Database (bundle + sidecar)**: the publishable unit — the deterministic compiled `scanner-bundle.json` plus a detached signature over its exact bytes. Identified by (bundle_version, sequence, fingerprint). +- **Bundle Manifest metadata**: `bundle_version`/`schema_version` (existing), plus additive `generated_at`, `sequence`, publisher key id. +- **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. +- **Signature (TPA record)**: one cataloged campaign/technique — `TPA-YYYY-NNNN` id, detectors, category, level, confidence, tier intent (gating vs non-gating), provenance reference, license. +- **Eval Sample Pair**: the labeled gated-malicious sample(s) plus category-matched hard-negative(s) a gating signature must contribute to the frozen eval dataset. +- **Sequence Floor**: the persisted highest-activated sequence per install; the anti-downgrade/anti-replay state, lowered only by explicit rollback. +- **Publication Channel**: the stable public location of versioned signed artifact pairs with a "latest" reference; consumed by the Spec 087 fetch or by manual download. +- **Post-Activation Re-scan**: the one-shot, off-hot-path evaluation of cached approved tool metadata against a newly-activated database, emitting review findings only. +- **Active Database / Last-Known-Good**: unchanged from Spec 086/087 — the serving bundle and the fail-closed fallback target. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: 100% of tampered candidates (any byte of bundle or sidecar altered, or mismatched pair) are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. +- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart, embedded fallback race), and an explicit rollback both succeeds and leaves an audit trail — demonstrated by fixture in CI. +- **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. +- **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. +- **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded, and a deliberately-regressing corpus change is blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. +- **SC-006**: Fleet freshness is answerable from telemetry: for opted-in installs, the distribution of active database version/sequence/source is reportable, enabling the metric "% of active installs on a database ≤ 30 days old" — a metric that is impossible to compute today. +- **SC-007**: A newly published signature reaches detection within one daily refresh cycle on a fetch-enabled install (and immediately on manual drop), measured end to end: publish → active → the new signature's fixture fires. +- **SC-008**: With a database at the design load ceilings (≥ 500 regex rules), p95 scan latency over a 100-tool server changes by less than 10% versus the 6-signature baseline, and bundle load-plus-verify completes under one second on commodity hardware. +- **SC-009**: After activating a database containing a new signature, an already-approved tool whose cached description matches it is surfaced for review within the same refresh cycle, with zero approvals auto-revoked and zero servers auto-quarantined by the refresh itself. + +## Assumptions + +- Spec 087's refresh lifecycle (daily tick, single-flight, activation self-check, opt-in fetch, fail-safe) is implemented before or together with this feature's refresh story; this spec inserts stages into that pipeline rather than duplicating it. US1's signature/sequence verification is also exercised by the existing hot-reload/file-drop path (Spec 086 `ConfigureBundle`), so US1 does not hard-depend on 087 landing first. +- The Scanner Bundle Contract v0.1 line remains the wire format; signing metadata is additive and the sidecar is a separate file, so existing loaders and the byte-determinism guarantee are unaffected. +- The corpus source-of-truth pipeline (signature sources → compiled bundle) lives outside this repo's runtime (the tpa-db authoring pipeline); this repo consumes compiled bundles only and embeds one at build time (a release build with no embeddable bundle fails, per Spec 087 FR-001). +- The eval gate's thresholds, dataset conventions (including hard-negative naming), vacuity guard, and hard-tier-only scoring are reused verbatim; growing the corpus means growing the dataset alongside it. +- The offline tier's runnable surface remains `engine: regex` × `target: tool_description` for v1; `structural_diff`, `resource_content`, and `server_manifest` rules stay declared-not-runnable (skipped, never clean coverage) exactly as today. +- Cached tool metadata already held by mcpproxy (approval baselines, index) is sufficient for the post-activation re-scan; no upstream connection is initiated by refresh. +- [NEEDS CLARIFICATION: back-compat window for unsigned file drops — should `require_signed_bundle` flip to default-on after one or two release cycles once signed publishing is live, or remain opt-in indefinitely for air-gapped/self-built-corpus operators?] + +## Out of Scope + +- Any change to the detect engine's check semantics, tiers, thresholds, or the shared position classifier (a corpus change must never require touching `ClassifyPosition`; recall fixes belong in signatures + eval samples, not classifier cues). +- Running `structural_diff`/stateful rules, or adding `resource_content`/`server_manifest` scan surfaces. +- LLM-assisted or networked detection tiers; this database feeds the deterministic offline tier only. +- Remote/networked trust-anchor distribution, certificate hierarchies, or transparency logs (key set ships with the binary; revisit only if the publisher set ever grows beyond the project). +- Auto-revoking approvals, auto-quarantining, or auto-approving anything as a side effect of a database refresh (re-scan is informational; state moves toward review only). +- IoC hash feeds and package/registry reputation data (the roadmap note's "IoC hashes" are deferred: v1 targets are description-borne patterns; the format's additive versioning leaves room for later rule engines). +- mcpproxy binary self-update or release-awareness changes (Spec 087 US3 owns that surface). +- Changing the scan-eval gate's thresholds or scoring. + +## Constitution Check *(note)* + +Principle IV (Security by Default) governs this spec: the database is treated as a hostile input until proven otherwise (verify-before-parse, fail-closed to last-known-good, anti-downgrade floor, mandatory signatures on the network path), and refresh can never silently widen approvals. Principle III (Configuration-Driven Architecture): `require_signed_bundle`, operator keys, and the existing bundle path/fetch settings live in `mcp_config.json` with env override and hot-reload; no hardcoded URLs or paths. Principle V (TDD): every rejection class (tamper, downgrade, unsigned, gate regression, ceiling breach) is built against failing fixtures first, and the corpus itself is test-gated by scan-eval. Principle I (Performance at Scale): verification, activation, and the post-activation re-scan stay off the scan hot path with atomic swaps (SC-008). + +## Commit Message Conventions *(mandatory)* + +When committing changes for this feature, follow these guidelines: + +### Issue References +- ✅ **Use**: `Related #[issue-number]` - Links the commit to the issue without auto-closing +- ❌ **Do NOT use**: `Fixes #[issue-number]`, `Closes #[issue-number]`, `Resolves #[issue-number]` - These auto-close issues on merge + +**Rationale**: Issues should only be closed manually after verification and testing in production, not automatically on merge. + +### Co-Authorship +- ❌ **Do NOT include**: `Co-Authored-By: Claude ` +- ❌ **Do NOT include**: "🤖 Generated with [Claude Code](https://claude.com/claude-code)" + +**Rationale**: Commit authorship should reflect the human contributors, not the AI tools used. + +### Example Commit Message +``` +feat(security): [brief description of change] + +Related #[issue-number] + +[Detailed description of what was changed and why] + +## Changes +- [Bulleted list of key changes] +- [Each change on a new line] + +## Testing +- [Test results summary] +- [Key test scenarios covered] +``` From 91d3becfe37a2269fe3ce7dc337e688c7ba53ca1 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Sun, 23 Aug 2026 21:18:16 +0300 Subject: [PATCH 02/20] docs(specs): close trust-model and eval-gate gaps in 101-tpa-db MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode/gpt-5.6-sol) plus local verification against the shipped loader and eval harness found ten genuine gaps in the stage-1 spec. All were confirmed against the code before being addressed. ## Changes Trust model / anti-downgrade - FR-009: the sequence floor is now a property of SIGNED artifacts only. `sequence` lives in the manifest, so an unsigned drop's sequence is attacker-chosen; an unsigned candidate could previously claim a huge sequence and permanently lock out every legitimate signed release. Unsigned candidates no longer advance the floor, and a signature ratchet stops signing from being stripped off an install that already ran signed. - FR-010: rollback no longer lowers the floor. Resetting the floor to the rolled-back sequence meant the next refresh could immediately re-activate the exact release the operator just rejected. Rollback is now a pinned exception over a monotonic high-watermark, with the rejected (sequence, fingerprint) deny-listed. - FR-009a: the embedded-default availability fallback is stated as the single explicit exemption to SC-002 (it previously contradicted it outright) and must serve as DEGRADED coverage, so deleting the external bundle cannot be used to obtain scan-mode auto-approvals under a weaker corpus. - FR-007: an active unsigned/unverified bundle likewise marks coverage degraded. `inprocess.go` derives `coverageOK` from bundle *presence*, so without this an attacker-supplied unsigned corpus keeps auto-approving. - FR-003: key ids must be canonical public-key fingerprints, duplicates rejected, and manifest id / sidecar id / verifying-key id must all match. - FR-006/FR-008: corrected an impossible pipeline order — the spec required checking the manifest-resident `sequence` before parsing the manifest. Verification is what precedes parsing; it runs over the raw bytes that are then parsed from memory, closing a verify/parse swap race. - FR-011a: activation must be crash-consistent. Spec 087 FR-010's atomicity is in-memory only; a crash between activation and floor persistence reopens the replay window FR-009 exists to close. Eval gate (the gate was measuring nothing) - FR-016a/SC-005/US2: `cmd/scan-eval/gate.go`'s `gateChecks()` registers only the four built-in detect checks — `scanner.BundleCheck` is absent and scan-eval never loads a bundle, so FR-016's "merge-blocking on corpus change" and SC-005 would both have passed vacuously. The harness must load a candidate through the production loader, register the bundle check, and map each new campaign category (`gatedCategory()` enforces a category only when its mapped check id is registered). SC-005 gains a negative control. Telemetry - FR-021: the schema-v8 anonymity backstop hard-whitelists `tpa_scanner` to four integer counters, so string/bool bundle fields would be rejected before transmit. Requires typed whitelist entries, an enum-bounded `source`, and publisher-issued `bundle_version` values only (a self-built corpus version string would itself be a fleet identifier). Consistency - FR-007: states explicitly that this spec supersedes Spec 087 FR-014's precondition, since embedding publisher keys means a fetch is no longer refused merely because the operator configured no key of their own. Related #1028 --- specs/101-tpa-db/spec.md | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 66efb97c1..9ae31e27e 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -40,16 +40,17 @@ An operator receives a TPA signature database out of band — a `scanner-bundle. **Why this priority**: The database is a code-adjacent security input: whoever controls its bytes controls what the scanner ignores. Without signing and anti-rollback, the "refreshable signature DB" pillar creates a new supply-chain surface (swap the file → blind the scanner; replay an old bundle → resurrect a fixed blind spot). Signing + sequence + fail-closed loading is what makes every other story safe to ship, so it goes first. -**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check. Finally, invoke the explicit rollback action and assert the embedded default is restored and the override is logged. +**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), and (g) an unsigned bundle claiming a huge sequence — assert the floor is unchanged so a later legitimate signed release still activates. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. **Acceptance Scenarios**: 1. **Given** a candidate bundle whose detached signature verifies against a configured trust anchor and whose sequence is greater than the active bundle's, **When** refresh runs, **Then** the candidate proceeds through the existing validation pipeline and, on success, becomes active with its sequence persisted and `signature_verified=true` surfaced in bundle status. 2. **Given** a candidate whose bytes were altered after signing (signature mismatch), **When** refresh runs, **Then** the candidate is rejected BEFORE its JSON is parsed or any pattern compiled, the active bundle is unchanged, and a "signature verification failed" reason is recorded. 3. **Given** a validly-signed candidate whose sequence is lower than the highest sequence ever activated on this install, **When** refresh runs, **Then** it is refused as a downgrade, the active bundle is unchanged, and a "downgrade refused" reason names both sequences. -4. **Given** `require_signed_bundle` is enabled and a file-drop candidate has no signature sidecar, **When** refresh runs, **Then** the candidate is refused (fail-closed) with a "signature required" reason; with the setting disabled (default), an unsigned drop is accepted through the existing Spec 086 validation path but surfaced as `signature_verified=false`. +4. **Given** `require_signed_bundle` is enabled and a file-drop candidate has no signature sidecar, **When** refresh runs, **Then** the candidate is refused (fail-closed) with a "signature required" reason; with the setting disabled (default) and no signature-verified bundle ever having been active on this install, an unsigned drop is accepted through the existing Spec 086 validation path but surfaced as `signature_verified=false`, does NOT advance the sequence floor, and marks coverage degraded so `scan`-mode auto-approval is suspended while it is active. +4a. **Given** an install that has already activated a signature-verified bundle, **When** an unsigned candidate is dropped (with `require_signed_bundle` still off), **Then** it is refused by the signature ratchet — signing cannot be stripped back off an install by whoever can write the bundle path — and an unsigned candidate claiming an arbitrarily high sequence never becomes the floor, so it cannot lock out subsequent legitimate signed releases. 5. **Given** a network-fetched candidate (Spec 087 opt-in path), **When** it arrives, **Then** signature verification is ALWAYS mandatory regardless of `require_signed_bundle` — an unsigned or wrongly-signed fetched bundle is never activated (parity with Spec 087 FR-013/FR-014). -6. **Given** an operator invokes the explicit rollback action naming the embedded default (or a prior bundle file), **When** it runs, **Then** the target is re-validated, activated even though its sequence is lower, the anti-downgrade floor is reset to the rolled-back sequence, and the override is loudly logged and visible in bundle status. +6. **Given** an operator invokes the explicit rollback action naming the embedded default (or a prior bundle file), **When** it runs, **Then** the target is re-validated and activated even though its sequence is lower, the historical high-watermark is NOT lowered, the artifact being rolled away from is deny-listed so the next refresh cycle cannot silently re-activate the very release the operator just rejected, and the override is loudly logged and visible in bundle status. 7. **Given** an air-gapped host with no network at any point, **When** any of the above scenarios run via file drop, **Then** behavior is identical — signature verification, sequence checks, and rollback require no network. --- @@ -60,7 +61,7 @@ A security-conscious developer installs mcpproxy and gets, out of the box, a sig **Why this priority**: The format (US1) is worthless empty, and the refresh channel (US3) has nothing to carry without a corpus. This is the "database" in tpa-db. It is P2 only because the P1 trust machinery must exist before a larger corpus becomes an attractive distribution target. -**Independent Test**: Run the corpus build and assert every signature carries id, category, detector(s), provenance source, and license; assert every gated signature has ≥1 labeled malicious eval sample and ≥1 category-matched hard-negative in the eval dataset; run `go run ./cmd/scan-eval --corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json --gate --min-recall 0.90 --max-fp 0.05` with the new embedded default and assert PASS; feed each campaign class's canonical fixture through the offline scanner and assert a hard-tier finding naming the TPA id. +**Independent Test**: Run the corpus build and assert every signature carries id, category, detector(s), provenance source, and license; assert every gated signature has ≥1 labeled malicious eval sample and ≥1 category-matched hard-negative in the eval dataset; run `go run ./cmd/scan-eval --corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json --gate --min-recall 0.90 --max-fp 0.05` with the new embedded default and assert PASS — and, because that command today scores built-in checks only, assert the accompanying NEGATIVE CONTROL that proves the bundle is being scored at all: with a gating signature removed from the bundle the same command must FAIL. Feed each campaign class's canonical fixture through the offline scanner and assert a hard-tier finding naming the TPA id. **Acceptance Scenarios**: @@ -95,7 +96,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Signature key rotation**: the trust anchor is a SET of publisher keys embedded per binary release. Rotation is a binary-release event: a release ships old+new keys, bundles are signed with the new key, the old key is removed in a later release. Keys are NEVER delivered over the network, and rotation is not done casually — every shipped binary pins its key set, so a hasty rotation strands older installs on unsigned-refresh only (same philosophy as the app-update signing keys). Key compromise → remove the key in the next release AND publish a higher-sequence bundle signed by the surviving key. - **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade floor still blocks sequence replay. - **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. -- **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted floor does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the floor retained and the situation surfaced. +- **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted floor does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the floor retained and the situation surfaced, but per FR-009a as DEGRADED coverage, so an attacker who merely deletes the external bundle cannot use the weaker embedded corpus to obtain `scan`-mode auto-approvals. This is the one explicit exemption to SC-002. - **Anti-downgrade state lost** (data directory wiped/reset): the floor resets; this is accepted — the floor is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. - **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). - **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. @@ -110,19 +111,21 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Database format & signing** - **FR-001**: The publishable database artifact MUST be the existing compiled bundle (`scanner-bundle.json`, Scanner Bundle Contract) plus a detached signature sidecar over the exact bundle bytes. Bundle bytes MUST remain deterministic for a given corpus (byte-identical rebuilds), so the signature and the fingerprint are stable. -- **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), and a publisher key identifier. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). -- **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. +- **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), and a publisher key identifier. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). `generated_at` MUST be derived from deterministic release metadata (the release commit's timestamp or `SOURCE_DATE_EPOCH`), NEVER from build wall-clock time — a wall-clock stamp would change the bundle bytes, fingerprint, and signature on every rebuild of an identical corpus and so would directly violate FR-001's byte-identical-rebuild guarantee. +- **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. A key id MUST be a canonical full-length fingerprint of the public key itself (not an operator-chosen label), the trust set MUST reject duplicate ids at build/config time, and verification MUST succeed only when the manifest key id, the sidecar key id, and the id of the trust-set key whose signature actually verified are all identical — a manifest naming one publisher while the bytes are signed by another MUST be a verification failure, not a cosmetic mismatch. - **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. - **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure (drop key next release + publish higher-sequence bundle under surviving key) and the "do not rotate casually" operational stance. **Loader & activation** -- **FR-006**: For any candidate carrying a signature, verification MUST run BEFORE JSON parsing or pattern compilation; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). -- **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature (Spec 087 FR-013/FR-014 unchanged); file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. -- **FR-008**: The full validate-before-activate order MUST be: signature (per policy) → sequence/anti-downgrade → version compatibility → parse → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. -- **FR-009**: The system MUST persist the highest sequence ever activated (per data directory) and refuse any candidate with a lower sequence — and any candidate with an equal sequence but different content fingerprint — as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. The embedded default competes under the same rule; when no higher-sequence valid candidate exists, the embedded default still serves (scanning never stops) with the retained floor surfaced. -- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, the persisted floor is reset to the rolled-back sequence, and the override is loudly logged and visible in bundle status. Rollback MUST be the ONLY path that lowers the floor. +- **FR-006**: For any candidate carrying a signature, verification MUST run over the exact raw candidate bytes BEFORE those bytes are parsed as JSON or any pattern is compiled; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). The bytes that are verified MUST be the same bytes that are subsequently parsed (read once into memory, verify, then parse that buffer) — the file MUST NOT be re-read from disk after verification, or a writer racing the loader could swap content between verify and parse. +- **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an UNSIGNED (or signature-unverified) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. +- **FR-008**: The full validate-before-activate order MUST be: size/rule-count ceiling (FR-011) → signature over the raw bytes (per policy, FR-006) → parse manifest → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. Sequence is a manifest field, so it is necessarily read after parsing; what MUST precede parsing is signature verification, not the sequence check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. +- **FR-009**: The system MUST persist the highest sequence ever activated (per data directory) and refuse any candidate with a lower sequence — and any candidate with an equal sequence but different content fingerprint — as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The floor is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the floor (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from the floor as an availability fallback (see FR-009a). +- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the floor. Because it may be an older, weaker corpus than the floor implies, this state MUST be treated as degraded coverage (auto-approval suspended, same as FR-007's unsigned case), the retained floor and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This is the single explicit exemption to SC-002. +- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). +- **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "floor written" reopens exactly the replay window FR-009 exists to close (while writing the floor first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, watermark, signature-verified state, any FR-010 pin — MUST be persisted as one transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. **Seed corpus** @@ -131,6 +134,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-014**: Gating signatures MUST emit hard-tier signals (the eval gate and the `scan`-mode approval gates score hard tier only); any soft-tier-only signature MUST be explicitly marked non-gating in its metadata and MUST NOT be counted toward gate coverage. - **FR-015**: Signatures MUST be authored in the corpus source-of-truth pipeline (signature sources compiled to the bundle); mcpproxy MUST continue to consume only the compiled bundle and never parse signature sources (Spec 087 FR-002 parity). The contribution pipeline MUST run the same validation the loader runs (compile, schema, license, eval gate) before a bundle can be signed. - **FR-016**: The CI eval gate (`cmd/scan-eval --gate`, recall ≥ 0.90 over gated categories, hard-negative FP rate ≤ 0.05, existing vacuity guard) MUST pass for: every change to the embedded default corpus (merge-blocking, existing `eval.yml` D2 job) and every published bundle BEFORE it is signed — a bundle that fails the gate is never signed or published. Thresholds are reused verbatim, not redefined. +- **FR-016a**: The gate harness MUST first be taught to see the database at all — today it does not, so FR-016 and SC-005 would otherwise pass vacuously. `cmd/scan-eval/gate.go`'s `gateChecks()` returns only the four built-in `detect.Check`s (`unicode.hidden`, `shadowing.cross_server`, `payload.decoded`, `phrase.injection`); `scanner.BundleCheck` — which production DOES append in `internal/security/scanner/inprocess.go` — is absent, and `scan-eval` never loads a bundle. Therefore: (a) `scan-eval` MUST accept a candidate bundle path, load it through the production loader (never a parallel parser), and register the resulting bundle check, defaulting to the embedded default so the existing `eval.yml` D2 invocation starts scoring the shipped corpus; (b) `gate.go`'s `categoryCheck` map MUST gain an entry for every new TPA campaign category, since `gatedCategory()` enforces a category ONLY when its mapped check id is registered — an unmapped category is measured but can never fail the build, which is precisely the vacuous-pass failure mode; (c) the same candidate-loading path MUST be what the Spec 087 FR-008 activation self-check runs, so the CI layer and the activation layer cannot drift apart while SC-005 claims they are independent. **Publication & refresh channel** @@ -141,7 +145,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Observability, telemetry & docs** - **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". -- **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. +- **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. - **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. ### Key Entities *(include if feature involves data)* @@ -161,10 +165,10 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes - **SC-001**: 100% of tampered candidates (any byte of bundle or sidecar altered, or mismatched pair) are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. -- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart, embedded fallback race), and an explicit rollback both succeeds and leaves an audit trail — demonstrated by fixture in CI. +- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart), and an unsigned candidate never advances the floor nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. -- **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded, and a deliberately-regressing corpus change is blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. +- **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a negative control: removing a gating signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. - **SC-006**: Fleet freshness is answerable from telemetry: for opted-in installs, the distribution of active database version/sequence/source is reportable, enabling the metric "% of active installs on a database ≤ 30 days old" — a metric that is impossible to compute today. - **SC-007**: A newly published signature reaches detection within one daily refresh cycle on a fetch-enabled install (and immediately on manual drop), measured end to end: publish → active → the new signature's fixture fires. - **SC-008**: With a database at the design load ceilings (≥ 500 regex rules), p95 scan latency over a 100-tool server changes by less than 10% versus the 6-signature baseline, and bundle load-plus-verify completes under one second on commodity hardware. From bbc1b795ef12d6c876fd6af10a640920990e824c Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Sun, 23 Aug 2026 21:23:21 +0300 Subject: [PATCH 03/20] docs(specs): fix watermark lockout + embedded-default regression in 101-tpa-db MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-2 cross-model review (opencode/gpt-5.6-sol) on the round-1 revisions. Six findings, all verified genuine; two of them were regressions the round-1 fixes introduced. ## Changes Watermark could be weaponized into a permanent update lockout - FR-005a (new): the trust set carries a monotonic trust EPOCH that only a binary release can advance, and anti-downgrade is scoped (authority, epoch, sequence). Without it, key-compromise recovery was impossible rather than merely painful: an attacker holding a compromised publisher key signs at the maximum representable sequence, and since round-1 made the watermark monotonic, no surviving key could ever outbid it — a one-shot permanent denial of updates. FR-005 and the key-rotation edge case now route compromise recovery through the epoch bump. - FR-004a (new): anti-downgrade state is tracked per SIGNING AUTHORITY, not in one global namespace. An operator with a self-built signed corpus (an explicitly supported persona) would otherwise raise the single global watermark past the publisher's release line and permanently block official updates. All publisher keys group into one authority so per-authority watermarks stay compatible with key rotation. - SC-002a (new): both lockout scenarios must be proven by fixture. Regressions introduced by the round-1 fixes - FR-007/FR-009a: round-1 made "any unsigned active bundle" degrade coverage, but the embedded default has no sidecar — so every fresh install would have shipped with scan-mode auto-approval silently disabled. Degradation now applies to unsigned EXTERNAL candidates only; the embedded default is binary-trusted and degrades solely when serving BELOW a retained signed watermark. - FR-008: round-1 put the rule-count ceiling before signature verification and parsing, which is impossible — rule count is not knowable from raw bytes. The two ceilings are now split: byte size before verification, rule count after parse. Consistency - FR-011a: the crash-consistent transaction now explicitly includes the watermark fingerprint and the FR-010 deny-list. Omitting them let a crash preserve a rollback while losing the state that rejects the rolled-away release, silently undoing the rollback on the next refresh. - Key Entities: "Sequence Floor" still said the floor is "lowered only by explicit rollback", directly contradicting revised FR-010. Renamed to Sequence Watermark, defined as monotonic, with rollback as a pinned exception. Terminology normalized from floor to watermark throughout so the two do not read as separate mechanisms. - Added a threat-model boundary note: an attacker with config write is out of scope (they can already disable quarantine), which is why the ratchet keys on install state and require_signed_bundle is config-resident. Related #1028 --- specs/101-tpa-db/spec.md | 45 ++++++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 9ae31e27e..c652e5909 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -40,15 +40,15 @@ An operator receives a TPA signature database out of band — a `scanner-bundle. **Why this priority**: The database is a code-adjacent security input: whoever controls its bytes controls what the scanner ignores. Without signing and anti-rollback, the "refreshable signature DB" pillar creates a new supply-chain surface (swap the file → blind the scanner; replay an old bundle → resurrect a fixed blind spot). Signing + sequence + fail-closed loading is what makes every other story safe to ship, so it goes first. -**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), and (g) an unsigned bundle claiming a huge sequence — assert the floor is unchanged so a later legitimate signed release still activates. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. +**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), and (g) an unsigned bundle claiming a huge sequence — assert the watermark is unchanged so a later legitimate signed release still activates. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. **Acceptance Scenarios**: 1. **Given** a candidate bundle whose detached signature verifies against a configured trust anchor and whose sequence is greater than the active bundle's, **When** refresh runs, **Then** the candidate proceeds through the existing validation pipeline and, on success, becomes active with its sequence persisted and `signature_verified=true` surfaced in bundle status. 2. **Given** a candidate whose bytes were altered after signing (signature mismatch), **When** refresh runs, **Then** the candidate is rejected BEFORE its JSON is parsed or any pattern compiled, the active bundle is unchanged, and a "signature verification failed" reason is recorded. 3. **Given** a validly-signed candidate whose sequence is lower than the highest sequence ever activated on this install, **When** refresh runs, **Then** it is refused as a downgrade, the active bundle is unchanged, and a "downgrade refused" reason names both sequences. -4. **Given** `require_signed_bundle` is enabled and a file-drop candidate has no signature sidecar, **When** refresh runs, **Then** the candidate is refused (fail-closed) with a "signature required" reason; with the setting disabled (default) and no signature-verified bundle ever having been active on this install, an unsigned drop is accepted through the existing Spec 086 validation path but surfaced as `signature_verified=false`, does NOT advance the sequence floor, and marks coverage degraded so `scan`-mode auto-approval is suspended while it is active. -4a. **Given** an install that has already activated a signature-verified bundle, **When** an unsigned candidate is dropped (with `require_signed_bundle` still off), **Then** it is refused by the signature ratchet — signing cannot be stripped back off an install by whoever can write the bundle path — and an unsigned candidate claiming an arbitrarily high sequence never becomes the floor, so it cannot lock out subsequent legitimate signed releases. +4. **Given** `require_signed_bundle` is enabled and a file-drop candidate has no signature sidecar, **When** refresh runs, **Then** the candidate is refused (fail-closed) with a "signature required" reason; with the setting disabled (default) and no signature-verified bundle ever having been active on this install, an unsigned drop is accepted through the existing Spec 086 validation path but surfaced as `signature_verified=false`, does NOT advance the sequence watermark, and marks coverage degraded so `scan`-mode auto-approval is suspended while it is active. +4a. **Given** an install that has already activated a signature-verified bundle, **When** an unsigned candidate is dropped (with `require_signed_bundle` still off), **Then** it is refused by the signature ratchet — signing cannot be stripped back off an install by whoever can write the bundle path — and an unsigned candidate claiming an arbitrarily high sequence never becomes the watermark, so it cannot lock out subsequent legitimate signed releases. 5. **Given** a network-fetched candidate (Spec 087 opt-in path), **When** it arrives, **Then** signature verification is ALWAYS mandatory regardless of `require_signed_bundle` — an unsigned or wrongly-signed fetched bundle is never activated (parity with Spec 087 FR-013/FR-014). 6. **Given** an operator invokes the explicit rollback action naming the embedded default (or a prior bundle file), **When** it runs, **Then** the target is re-validated and activated even though its sequence is lower, the historical high-watermark is NOT lowered, the artifact being rolled away from is deny-listed so the next refresh cycle cannot silently re-activate the very release the operator just rejected, and the override is loudly logged and visible in bundle status. 7. **Given** an air-gapped host with no network at any point, **When** any of the above scenarios run via file drop, **Then** behavior is identical — signature verification, sequence checks, and rollback require no network. @@ -93,11 +93,12 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Edge Cases - **Air-gapped host, forever**: every P1 behavior (verification, sequence check, rollback, activation self-check) is network-free; the trust anchors ship inside the binary; staleness is surfaced but never blocks scanning — an old database is degraded coverage, not an outage. -- **Signature key rotation**: the trust anchor is a SET of publisher keys embedded per binary release. Rotation is a binary-release event: a release ships old+new keys, bundles are signed with the new key, the old key is removed in a later release. Keys are NEVER delivered over the network, and rotation is not done casually — every shipped binary pins its key set, so a hasty rotation strands older installs on unsigned-refresh only (same philosophy as the app-update signing keys). Key compromise → remove the key in the next release AND publish a higher-sequence bundle signed by the surviving key. -- **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade floor still blocks sequence replay. +- **Threat-model boundary — attacker with config write**: FR-004 lets operator-configured keys join the trust set, so anyone who can write `mcp_config.json` can trust their own key and sign anything. This is explicitly OUT of the threat model and is not a regression: an attacker with config write can already disable quarantine outright or set a server's trust mode to `auto`. The attacker this design defends against is the one who can write the BUNDLE PATH or intercept the FETCH but not the config — which is precisely why the signature ratchet (FR-009) is keyed to install state rather than to the bundle path, and why `require_signed_bundle` is config-resident (it must not be defeatable from the data dir alone). +- **Signature key rotation**: the trust anchor is a SET of publisher keys embedded per binary release. Rotation is a binary-release event: a release ships old+new keys, bundles are signed with the new key, the old key is removed in a later release. Keys are NEVER delivered over the network, and rotation is not done casually — every shipped binary pins its key set, so a hasty rotation strands older installs on unsigned-refresh only (same philosophy as the app-update signing keys). Key compromise → remove the key in the next release, **bump the trust epoch (FR-005a)**, and publish a bundle in the new epoch signed by the surviving key. The epoch bump is what makes recovery possible at all: a compromised key can sign at the maximum representable sequence, and against a monotonic watermark no surviving key could ever outbid that within the same epoch, so revocation alone would leave installs permanently unable to accept any future release. +- **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade watermark still blocks sequence replay. - **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. -- **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted floor does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the floor retained and the situation surfaced, but per FR-009a as DEGRADED coverage, so an attacker who merely deletes the external bundle cannot use the weaker embedded corpus to obtain `scan`-mode auto-approvals. This is the one explicit exemption to SC-002. -- **Anti-downgrade state lost** (data directory wiped/reset): the floor resets; this is accepted — the floor is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. +- **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted watermark does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the watermark retained and the situation surfaced, but per FR-009a as DEGRADED coverage, so an attacker who merely deletes the external bundle cannot use the weaker embedded corpus to obtain `scan`-mode auto-approvals. This is the one explicit exemption to SC-002. +- **Anti-downgrade state lost** (data directory wiped/reset): the watermark resets, and so does the FR-009 signature-ratchet state (both live in the data dir), so a wiped install accepts an unsigned drop again. This is accepted — the watermark and ratchet are best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. - **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). - **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. - **Huge or adversarial database** (thousands of rules, pathological regex): size and rule-count ceilings are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. @@ -113,19 +114,21 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-001**: The publishable database artifact MUST be the existing compiled bundle (`scanner-bundle.json`, Scanner Bundle Contract) plus a detached signature sidecar over the exact bundle bytes. Bundle bytes MUST remain deterministic for a given corpus (byte-identical rebuilds), so the signature and the fingerprint are stable. - **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), and a publisher key identifier. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). `generated_at` MUST be derived from deterministic release metadata (the release commit's timestamp or `SOURCE_DATE_EPOCH`), NEVER from build wall-clock time — a wall-clock stamp would change the bundle bytes, fingerprint, and signature on every rebuild of an identical corpus and so would directly violate FR-001's byte-identical-rebuild guarantee. - **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. A key id MUST be a canonical full-length fingerprint of the public key itself (not an operator-chosen label), the trust set MUST reject duplicate ids at build/config time, and verification MUST succeed only when the manifest key id, the sidecar key id, and the id of the trust-set key whose signature actually verified are all identical — a manifest naming one publisher while the bytes are signed by another MUST be a verification failure, not a cosmetic mismatch. -- **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. -- **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure (drop key next release + publish higher-sequence bundle under surviving key) and the "do not rotate casually" operational stance. +- **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. These are two distinct **signing authorities**, and the design MUST keep them so: all publisher keys (including keys being rotated in or out, FR-005) belong to ONE publisher authority, while operator-configured keys form a separate operator authority. +- **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. +- **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure and the "do not rotate casually" operational stance. +- **FR-005a**: The trust set MUST carry a **trust epoch** — a monotonic integer that advances ONLY via a binary release — and every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle from a higher epoch MUST supersede the retained watermark and re-baseline it. Epoch advancement is safe to treat as authoritative because it arrives only inside a release binary, whose integrity is already the root of trust for the embedded key set (FR-004) — it is not attacker-reachable over the bundle or fetch path. **Loader & activation** - **FR-006**: For any candidate carrying a signature, verification MUST run over the exact raw candidate bytes BEFORE those bytes are parsed as JSON or any pattern is compiled; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). The bytes that are verified MUST be the same bytes that are subsequently parsed (read once into memory, verify, then parse that buffer) — the file MUST NOT be re-read from disk after verification, or a writer racing the loader could swap content between verify and parse. -- **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an UNSIGNED (or signature-unverified) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. -- **FR-008**: The full validate-before-activate order MUST be: size/rule-count ceiling (FR-011) → signature over the raw bytes (per policy, FR-006) → parse manifest → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. Sequence is a manifest field, so it is necessarily read after parsing; what MUST precede parsing is signature verification, not the sequence check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. -- **FR-009**: The system MUST persist the highest sequence ever activated (per data directory) and refuse any candidate with a lower sequence — and any candidate with an equal sequence but different content fingerprint — as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The floor is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the floor (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from the floor as an availability fallback (see FR-009a). -- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the floor. Because it may be an older, weaker corpus than the floor implies, this state MUST be treated as degraded coverage (auto-approval suspended, same as FR-007's unsigned case), the retained floor and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This is the single explicit exemption to SC-002. +- **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a case — when it is serving as a fallback BELOW a retained signed watermark, i.e. when the install has evidence a newer signed corpus should be active. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. +- **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. +- **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. +- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark (the ordinary fresh install), this is simply the normal full-coverage state. Where a signed watermark IS retained and the embedded default is below it, the embedded corpus may be older and weaker than the watermark implies, so THAT state MUST be treated as degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case), the retained watermark and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). -- **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "floor written" reopens exactly the replay window FR-009 exists to close (while writing the floor first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, watermark, signature-verified state, any FR-010 pin — MUST be persisted as one transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. +- **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. **Seed corpus** @@ -144,7 +147,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Observability, telemetry & docs** -- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". +- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, unsigned-refused-by-ratchet, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. - **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. - **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. @@ -152,10 +155,11 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Signature Database (bundle + sidecar)**: the publishable unit — the deterministic compiled `scanner-bundle.json` plus a detached signature over its exact bytes. Identified by (bundle_version, sequence, fingerprint). - **Bundle Manifest metadata**: `bundle_version`/`schema_version` (existing), plus additive `generated_at`, `sequence`, publisher key id. -- **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. +- **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. Carries a **trust epoch** (FR-005a), a monotonic integer that only a binary release can advance. +- **Signing Authority**: the grouping anti-downgrade state is keyed by (FR-004a) — one publisher authority spanning all publisher keys including those mid-rotation, and a separate operator authority for operator-configured keys. Each holds its own watermark so neither can lock the other out. - **Signature (TPA record)**: one cataloged campaign/technique — `TPA-YYYY-NNNN` id, detectors, category, level, confidence, tier intent (gating vs non-gating), provenance reference, license. - **Eval Sample Pair**: the labeled gated-malicious sample(s) plus category-matched hard-negative(s) a gating signature must contribute to the frozen eval dataset. -- **Sequence Floor**: the persisted highest-activated sequence per install; the anti-downgrade/anti-replay state, lowered only by explicit rollback. +- **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving in a binary release. - **Publication Channel**: the stable public location of versioned signed artifact pairs with a "latest" reference; consumed by the Spec 087 fetch or by manual download. - **Post-Activation Re-scan**: the one-shot, off-hot-path evaluation of cached approved tool metadata against a newly-activated database, emitting review findings only. - **Active Database / Last-Known-Good**: unchanged from Spec 086/087 — the serving bundle and the fail-closed fallback target. @@ -165,7 +169,8 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes - **SC-001**: 100% of tampered candidates (any byte of bundle or sidecar altered, or mismatched pair) are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. -- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart), and an unsigned candidate never advances the floor nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. +- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart), and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. +- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; and (b) an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so the next official publisher release still activates — proving the two signing authorities cannot lock each other out. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. - **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a negative control: removing a gating signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. @@ -182,7 +187,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - The eval gate's thresholds, dataset conventions (including hard-negative naming), vacuity guard, and hard-tier-only scoring are reused verbatim; growing the corpus means growing the dataset alongside it. - The offline tier's runnable surface remains `engine: regex` × `target: tool_description` for v1; `structural_diff`, `resource_content`, and `server_manifest` rules stay declared-not-runnable (skipped, never clean coverage) exactly as today. - Cached tool metadata already held by mcpproxy (approval baselines, index) is sufficient for the post-activation re-scan; no upstream connection is initiated by refresh. -- [NEEDS CLARIFICATION: back-compat window for unsigned file drops — should `require_signed_bundle` flip to default-on after one or two release cycles once signed publishing is live, or remain opt-in indefinitely for air-gapped/self-built-corpus operators?] +- [NEEDS CLARIFICATION: back-compat window for unsigned file drops — should `require_signed_bundle` flip to default-on after one or two release cycles once signed publishing is live, or remain opt-in indefinitely for air-gapped/self-built-corpus operators? Note this is now a smaller question than it looks: FR-009's signature ratchet already means any install that has ever run a signed bundle refuses unsigned drops, and FR-007 suspends auto-approval while an unsigned bundle is active, so the default-on flip would only change behavior for installs that have never seen a signed bundle at all.] ## Out of Scope @@ -197,7 +202,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ## Constitution Check *(note)* -Principle IV (Security by Default) governs this spec: the database is treated as a hostile input until proven otherwise (verify-before-parse, fail-closed to last-known-good, anti-downgrade floor, mandatory signatures on the network path), and refresh can never silently widen approvals. Principle III (Configuration-Driven Architecture): `require_signed_bundle`, operator keys, and the existing bundle path/fetch settings live in `mcp_config.json` with env override and hot-reload; no hardcoded URLs or paths. Principle V (TDD): every rejection class (tamper, downgrade, unsigned, gate regression, ceiling breach) is built against failing fixtures first, and the corpus itself is test-gated by scan-eval. Principle I (Performance at Scale): verification, activation, and the post-activation re-scan stay off the scan hot path with atomic swaps (SC-008). +Principle IV (Security by Default) governs this spec: the database is treated as a hostile input until proven otherwise (verify-before-parse, fail-closed to last-known-good, anti-downgrade watermark, mandatory signatures on the network path), and refresh can never silently widen approvals. Principle III (Configuration-Driven Architecture): `require_signed_bundle`, operator keys, and the existing bundle path/fetch settings live in `mcp_config.json` with env override and hot-reload; no hardcoded URLs or paths. Principle V (TDD): every rejection class (tamper, downgrade, unsigned, gate regression, ceiling breach) is built against failing fixtures first, and the corpus itself is test-gated by scan-eval. Principle I (Performance at Scale): verification, activation, and the post-activation re-scan stay off the scan hot path with atomic swaps (SC-008). ## Commit Message Conventions *(mandatory)* From f2ce0fec764ce2bdf43c7d432cd481f9f0345cf0 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Sun, 23 Aug 2026 21:26:20 +0300 Subject: [PATCH 04/20] docs(specs): bind the trust epoch into signed bundle bytes in 101-tpa-db MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-3 cross-model review (opencode/gpt-5.6-sol) found one CRITICAL defect, verified genuine: the trust epoch introduced in round 2 was itself a replay window. ## The defect FR-005a made the epoch a property of the binary's trust set only; FR-002's manifest carried sequence and key id but no epoch. After a compromise recovery bumped the epoch and re-baselined the watermark, an OLD, weaker bundle legitimately signed by a still-trusted surviving key would be reinterpreted as belonging to the NEW epoch, clear the fresh (near-zero) watermark, and activate with signature_verified=true and full coverage — reinstating precisely the blind spots the recovery release was published to close. SC-002a tested recovery but not this replay. ## Changes - FR-002: the trust epoch is now carried inside the SIGNED manifest, so it is cryptographically bound to the bundle and cannot be reinterpreted. - FR-005a: epochs are per-authority, and a candidate is accepted only when its manifest epoch EQUALS the current binary-trusted epoch of the verifying authority — never merely less-than-or-equal. A below-epoch bundle is refused as stale-epoch replay even when its signature verifies against a still trusted key. Documents the accepted consequence: a binary on the old epoch stops receiving database updates until updated, which is correct because it still trusts the compromised key, and must surface as a stale-epoch degraded state rather than failing silently. - FR-008: epoch match added to the pipeline, before the anti-downgrade stage. - SC-002a: new fixture (a2) — a pre-bump bundle signed by the surviving key is refused after the bump rather than clearing the re-baselined watermark. - FR-020: stale-epoch replay added to the surfaced rejection reasons; bundle status also reports signing authority and trust epoch. Also, from a self-audit of the round-2 per-authority model: - FR-009a: the embedded default degrades when below a watermark retained by ANY authority (previously ambiguous which authority was meant). - FR-004a: choosing an operator-signed corpus over a higher-sequence publisher one is a deliberate operator decision, not a downgrade and not degraded coverage — but the active authority must be visible in bundle status. Related #1028 --- specs/101-tpa-db/spec.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index c652e5909..6f5c8f218 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -112,20 +112,20 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Database format & signing** - **FR-001**: The publishable database artifact MUST be the existing compiled bundle (`scanner-bundle.json`, Scanner Bundle Contract) plus a detached signature sidecar over the exact bundle bytes. Bundle bytes MUST remain deterministic for a given corpus (byte-identical rebuilds), so the signature and the fingerprint are stable. -- **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), and a publisher key identifier. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). `generated_at` MUST be derived from deterministic release metadata (the release commit's timestamp or `SOURCE_DATE_EPOCH`), NEVER from build wall-clock time — a wall-clock stamp would change the bundle bytes, fingerprint, and signature on every rebuild of an identical corpus and so would directly violate FR-001's byte-identical-rebuild guarantee. +- **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), a publisher key identifier, and the **trust epoch** the artifact was published under (FR-005a). Because these keys live inside the signed bytes, the epoch is cryptographically bound to the bundle and cannot be reinterpreted after the fact. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). `generated_at` MUST be derived from deterministic release metadata (the release commit's timestamp or `SOURCE_DATE_EPOCH`), NEVER from build wall-clock time — a wall-clock stamp would change the bundle bytes, fingerprint, and signature on every rebuild of an identical corpus and so would directly violate FR-001's byte-identical-rebuild guarantee. - **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. A key id MUST be a canonical full-length fingerprint of the public key itself (not an operator-chosen label), the trust set MUST reject duplicate ids at build/config time, and verification MUST succeed only when the manifest key id, the sidecar key id, and the id of the trust-set key whose signature actually verified are all identical — a manifest naming one publisher while the bytes are signed by another MUST be a verification failure, not a cosmetic mismatch. - **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. These are two distinct **signing authorities**, and the design MUST keep them so: all publisher keys (including keys being rotated in or out, FR-005) belong to ONE publisher authority, while operator-configured keys form a separate operator authority. -- **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. +- **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Choosing an operator-signed corpus over a higher-sequence publisher one is a deliberate operator decision and MUST NOT by itself be treated as a downgrade or as degraded coverage — the operator configured that key precisely to assert their own corpus — but the active authority MUST be visible in bundle status (FR-020) so the choice is never invisible. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. - **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure and the "do not rotate casually" operational stance. -- **FR-005a**: The trust set MUST carry a **trust epoch** — a monotonic integer that advances ONLY via a binary release — and every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle from a higher epoch MUST supersede the retained watermark and re-baseline it. Epoch advancement is safe to treat as authoritative because it arrives only inside a release binary, whose integrity is already the root of trust for the embedded key set (FR-004) — it is not attacker-reachable over the bundle or fetch path. +- **FR-005a**: Each signing authority's trust set MUST carry a **trust epoch** — a monotonic per-authority integer that advances ONLY via a binary release — and every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. The epoch MUST be carried inside the SIGNED manifest (FR-002) and a candidate MUST be accepted only when its manifest epoch EQUALS the current binary-trusted epoch of the authority whose key verified it — never merely less-than-or-equal. Binding the epoch to the signed bytes and requiring exact equality is what stops the epoch bump from becoming a replay window of its own: if the epoch lived only in the binary, then after a bump-and-revoke the re-baselined watermark would start from zero, and an OLD, weaker bundle legitimately signed by a still-trusted surviving key would be silently reinterpreted as belonging to the new epoch, clear the fresh watermark, and activate with `signature_verified=true` and full coverage — reinstating exactly the blind spots the recovery release was published to close. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle whose manifest epoch matches a newly-bumped binary epoch MUST supersede the retained watermark and re-baseline it. A bundle whose manifest epoch is BELOW the current binary-trusted epoch MUST be refused as stale-epoch replay even when its signature verifies against a still-trusted key. Epoch advancement is safe to treat as authoritative because it arrives only inside a release binary, whose integrity is already the root of trust for the embedded key set (FR-004) — it is not attacker-reachable over the bundle or fetch path. The accepted consequence of exact-equality matching is that a binary still on the OLD epoch will refuse the new epoch's bundles and stop receiving database updates until it is itself updated. That is deliberate, not incidental: an epoch bump means a key was compromised, and an un-updated binary still trusts that compromised key, so freezing its database is safer than feeding it artifacts whose trust basis it cannot evaluate. It also matches the existing "a hasty rotation strands older installs" stance, and it MUST be surfaced as a stale-epoch degraded state (FR-020) rather than failing silently. **Loader & activation** - **FR-006**: For any candidate carrying a signature, verification MUST run over the exact raw candidate bytes BEFORE those bytes are parsed as JSON or any pattern is compiled; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). The bytes that are verified MUST be the same bytes that are subsequently parsed (read once into memory, verify, then parse that buffer) — the file MUST NOT be re-read from disk after verification, or a writer racing the loader could swap content between verify and parse. - **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a case — when it is serving as a fallback BELOW a retained signed watermark, i.e. when the install has evidence a newer signed corpus should be active. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. -- **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. +- **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. -- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark (the ordinary fresh install), this is simply the normal full-coverage state. Where a signed watermark IS retained and the embedded default is below it, the embedded corpus may be older and weaker than the watermark implies, so THAT state MUST be treated as degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case), the retained watermark and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. +- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark (the ordinary fresh install), this is simply the normal full-coverage state. Where a signed watermark IS retained by ANY authority (FR-004a) and the embedded default is below it, the embedded corpus may be older and weaker than that watermark implies, so THAT state MUST be treated as degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case), the retained watermark and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. @@ -147,7 +147,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Observability, telemetry & docs** -- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, unsigned-refused-by-ratchet, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. +- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signing authority and trust epoch, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, stale-epoch replay, unsigned-refused-by-ratchet, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. - **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. - **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. @@ -155,7 +155,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Signature Database (bundle + sidecar)**: the publishable unit — the deterministic compiled `scanner-bundle.json` plus a detached signature over its exact bytes. Identified by (bundle_version, sequence, fingerprint). - **Bundle Manifest metadata**: `bundle_version`/`schema_version` (existing), plus additive `generated_at`, `sequence`, publisher key id. -- **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. Carries a **trust epoch** (FR-005a), a monotonic integer that only a binary release can advance. +- **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. Each authority carries its own **trust epoch** (FR-005a), a monotonic integer that only a binary release can advance and that a candidate's signed manifest must match exactly. - **Signing Authority**: the grouping anti-downgrade state is keyed by (FR-004a) — one publisher authority spanning all publisher keys including those mid-rotation, and a separate operator authority for operator-configured keys. Each holds its own watermark so neither can lock the other out. - **Signature (TPA record)**: one cataloged campaign/technique — `TPA-YYYY-NNNN` id, detectors, category, level, confidence, tier intent (gating vs non-gating), provenance reference, license. - **Eval Sample Pair**: the labeled gated-malicious sample(s) plus category-matched hard-negative(s) a gating signature must contribute to the frozen eval dataset. @@ -170,7 +170,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **SC-001**: 100% of tampered candidates (any byte of bundle or sidecar altered, or mismatched pair) are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. - **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart), and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. -- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; and (b) an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so the next official publisher release still activates — proving the two signing authorities cannot lock each other out. +- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so the next official publisher release still activates — proving the two signing authorities cannot lock each other out. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. - **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a negative control: removing a gating signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. From 0527bfe0df0ea76dfe0ca1adc70c4e03c12ac564 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Sun, 23 Aug 2026 21:29:45 +0300 Subject: [PATCH 05/20] docs(specs): seed the anti-downgrade watermark from a trusted baseline (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-4 cross-model review (opencode/gpt-5.6-sol) found one HIGH defect, verified genuine: the anti-downgrade guarantee did not exist at first use. ## The defect Every anti-downgrade mechanism in the spec binds only once persisted state exists. On a fresh install — or after any data-directory reset — there is no watermark, so an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle (sequence 5 while the shipping release is at 100). Signature verifies, epoch matches, nothing to compare the sequence against: it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. The FR-009 ratchet and the FR-010 pin/deny-list do not help, since both also require pre-existing state. Confirmed no new contradiction was introduced by the round-3 epoch fix. ## Changes - FR-009b (new): watermarks are SEEDED from a trusted baseline rather than starting empty. The publisher authority seeds from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and is refused — and the baseline is re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The operator authority, which has no binary-trusted baseline, gets a minimum sequence configurable alongside the operator key. This also makes the FR-005a epoch re-baseline safe: a bumped epoch starts at the embedded sequence, not zero. - SC-002 / US1 Independent Test: fresh-install and post-wipe downgrade are now explicitly in scope, with fixture (h) covering the old-but-validly-signed bundle on an install with no persisted state. - Edge case "Anti-downgrade state lost": corrected — a wipe resets the watermark to the seeded baseline, not to zero. What a wipe genuinely loses is the ratchet state and any pin/deny-list. - Key Entities: Sequence Watermark is documented as never empty. Related #1028 --- specs/101-tpa-db/spec.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 6f5c8f218..ea1deadd8 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -40,7 +40,7 @@ An operator receives a TPA signature database out of band — a `scanner-bundle. **Why this priority**: The database is a code-adjacent security input: whoever controls its bytes controls what the scanner ignores. Without signing and anti-rollback, the "refreshable signature DB" pillar creates a new supply-chain surface (swap the file → blind the scanner; replay an old bundle → resurrect a fixed blind spot). Signing + sequence + fail-closed loading is what makes every other story safe to ship, so it goes first. -**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), and (g) an unsigned bundle claiming a huge sequence — assert the watermark is unchanged so a later legitimate signed release still activates. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. +**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), (g) an unsigned bundle claiming a huge sequence — assert the watermark is unchanged so a later legitimate signed release still activates, and (h) on a FRESH install with no persisted state, a genuinely publisher-signed current-epoch bundle whose sequence is older than the binary's embedded corpus — assert it is refused by the FR-009b seeded baseline rather than activating at full coverage. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. **Acceptance Scenarios**: @@ -98,7 +98,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade watermark still blocks sequence replay. - **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. - **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted watermark does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the watermark retained and the situation surfaced, but per FR-009a as DEGRADED coverage, so an attacker who merely deletes the external bundle cannot use the weaker embedded corpus to obtain `scan`-mode auto-approvals. This is the one explicit exemption to SC-002. -- **Anti-downgrade state lost** (data directory wiped/reset): the watermark resets, and so does the FR-009 signature-ratchet state (both live in the data dir), so a wiped install accepts an unsigned drop again. This is accepted — the watermark and ratchet are best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. +- **Anti-downgrade state lost** (data directory wiped/reset): the persisted watermark resets to the FR-009b seeded baseline — NOT to zero, so a wiped install still refuses any publisher-signed bundle older than the corpus its own binary ships. What a wipe does lose is the FR-009 signature-ratchet state and any FR-010 pin/deny-list (both data-dir resident), so a wiped install accepts an unsigned drop again. This is accepted — the ratchet is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. - **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). - **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. - **Huge or adversarial database** (thousands of rules, pathological regex): size and rule-count ceilings are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. @@ -125,6 +125,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a case — when it is serving as a fallback BELOW a retained signed watermark, i.e. when the install has evidence a newer signed corpus should be active. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. - **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. +- **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its minimum sequence MUST be configurable alongside the operator key (defaulting to zero, with the residual exposure being the already-out-of-scope config-write attacker). The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the embedded bundle's sequence, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark (the ordinary fresh install), this is simply the normal full-coverage state. Where a signed watermark IS retained by ANY authority (FR-004a) and the embedded default is below it, the embedded corpus may be older and weaker than that watermark implies, so THAT state MUST be treated as degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case), the retained watermark and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). @@ -159,7 +160,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Signing Authority**: the grouping anti-downgrade state is keyed by (FR-004a) — one publisher authority spanning all publisher keys including those mid-rotation, and a separate operator authority for operator-configured keys. Each holds its own watermark so neither can lock the other out. - **Signature (TPA record)**: one cataloged campaign/technique — `TPA-YYYY-NNNN` id, detectors, category, level, confidence, tier intent (gating vs non-gating), provenance reference, license. - **Eval Sample Pair**: the labeled gated-malicious sample(s) plus category-matched hard-negative(s) a gating signature must contribute to the frozen eval dataset. -- **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving in a binary release. +- **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. Never empty — it is seeded from a trusted baseline (the embedded bundle's sequence, for the publisher authority) per FR-009b, so "no state yet" is not an accept-anything state. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving in a binary release. - **Publication Channel**: the stable public location of versioned signed artifact pairs with a "latest" reference; consumed by the Spec 087 fetch or by manual download. - **Post-Activation Re-scan**: the one-shot, off-hot-path evaluation of cached approved tool metadata against a newly-activated database, emitting review findings only. - **Active Database / Last-Known-Good**: unchanged from Spec 086/087 — the serving bundle and the fail-closed fallback target. @@ -169,7 +170,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes - **SC-001**: 100% of tampered candidates (any byte of bundle or sidecar altered, or mismatched pair) are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. -- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart), and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. +- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. - **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so the next official publisher release still activates — proving the two signing authorities cannot lock each other out. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. From c178144722a5022e7b8c6760c5855f49cd71daf1 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Sun, 23 Aug 2026 21:32:16 +0300 Subject: [PATCH 06/20] docs(specs): make the operator-authority baseline explicit and durable (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-5 cross-model review (opencode/gpt-5.6-sol) found one HIGH defect, verified genuine. It confirmed FR-009b closes the publisher-side replay without contradicting FR-005a, FR-009, FR-009a, or FR-010, but caught that the operator authority was left with the same hole. ## The defect FR-009b gave the operator authority a minimum sequence "defaulting to zero, with the residual exposure being the already-out-of-scope config-write attacker". That parenthetical was wrong. With a zero default, no config write is needed: after a data-directory wipe (or on a fresh install), a bundle-path attacker — squarely inside the threat model — can replay any older genuinely operator-signed bundle at full coverage. The operator authority therefore had no post-wipe anti-downgrade at all by default, while the spec read as though it did. ## Changes - FR-009b: the operator baseline must be declared explicitly in mcp_config.json alongside the operator key, not defaulted to zero and not stored in the data directory — a baseline held in resettable state gives no protection exactly when it is needed. If an operator key is configured without a baseline, operator-authority anti-downgrade is treated as unsupported and coverage is marked degraded while an operator-signed bundle is active, instead of presenting it as fully trusted. The data-directory watermark may only raise the effective minimum above the configured baseline, never lower it. Epoch re-baseline now starts at the authority's baseline rather than zero. - Edge case "Anti-downgrade state lost": states plainly that the operator-side guarantee survives a wipe only with a config-resident baseline. - SC-002a: fixture (c) covers the operator authority after a wipe, both with and without a declared baseline. Note: per the repo's per-PR cross-model review cap, this is the fifth round. This fix has NOT itself been re-reviewed. Related #1028 --- specs/101-tpa-db/spec.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index ea1deadd8..7233ebbed 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -98,7 +98,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade watermark still blocks sequence replay. - **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. - **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted watermark does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the watermark retained and the situation surfaced, but per FR-009a as DEGRADED coverage, so an attacker who merely deletes the external bundle cannot use the weaker embedded corpus to obtain `scan`-mode auto-approvals. This is the one explicit exemption to SC-002. -- **Anti-downgrade state lost** (data directory wiped/reset): the persisted watermark resets to the FR-009b seeded baseline — NOT to zero, so a wiped install still refuses any publisher-signed bundle older than the corpus its own binary ships. What a wipe does lose is the FR-009 signature-ratchet state and any FR-010 pin/deny-list (both data-dir resident), so a wiped install accepts an unsigned drop again. This is accepted — the ratchet is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. +- **Anti-downgrade state lost** (data directory wiped/reset): the persisted watermark resets to the FR-009b seeded baseline — NOT to zero, so a wiped install still refuses any publisher-signed bundle older than the corpus its own binary ships. For the operator authority the same protection holds only if the operator declared a config-resident baseline; without one, operator-authority anti-downgrade does not survive the wipe and the install runs degraded while an operator-signed bundle is active, by FR-009b, rather than pretending to a guarantee it does not have. What a wipe does lose is the FR-009 signature-ratchet state and any FR-010 pin/deny-list (both data-dir resident), so a wiped install accepts an unsigned drop again. This is accepted — the ratchet is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. - **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). - **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. - **Huge or adversarial database** (thousands of rules, pathological regex): size and rule-count ceilings are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. @@ -125,7 +125,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a case — when it is serving as a fallback BELOW a retained signed watermark, i.e. when the install has evidence a newer signed corpus should be active. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. - **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. -- **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its minimum sequence MUST be configurable alongside the operator key (defaulting to zero, with the residual exposure being the already-out-of-scope config-write attacker). The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the embedded bundle's sequence, not at zero. +- **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark (the ordinary fresh install), this is simply the normal full-coverage state. Where a signed watermark IS retained by ANY authority (FR-004a) and the embedded default is below it, the embedded corpus may be older and weaker than that watermark implies, so THAT state MUST be treated as degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case), the retained watermark and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). @@ -171,7 +171,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **SC-001**: 100% of tampered candidates (any byte of bundle or sidecar altered, or mismatched pair) are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. - **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. -- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so the next official publisher release still activates — proving the two signing authorities cannot lock each other out. +- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. - **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a negative control: removing a gating signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. From 49c4e7edf70697deddd288927e72adbda5c40d18 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 06:40:02 +0300 Subject: [PATCH 07/20] docs(specs): close the authority-swap hole and de-vacuate the gate control (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode / gpt-5.6-sol) round 1 on specs/101-tpa-db/spec.md. All five findings verified against the current code before changing anything. ## Changes - FR-004b (new): the active signing authority is a config-resident declaration, default publisher; a candidate signed by a non-active authority is refused ("authority not active"). Per-authority watermarks alone let a bundle-path attacker — in the threat model — swap a publisher bundle at sequence 100 for any operator-signed artifact clearing the operator baseline, silently narrowing coverage while still reporting signature_verified=true at full coverage, because the comparison never crosses namespaces. - FR-009a: the embedded default's sequence is compared ONLY against the publisher watermark (same namespace); the "any authority" rule made an operator corpus at sequence 900 permanently degrade a current publisher fallback at 100. The intent behind it is restated directly as condition (b): a previously-active external bundle of the active authority having gone missing. - FR-016a: corrected the claim about unmapped eval categories — they are excluded from OverallRecall so their MISSES cannot fail the gate, but their false positives still count, since FPRate is computed over the whole hard-negative set regardless of category mapping (gate.go:193-197, 287-289). - FR-016b (new) + SC-005 + US2 independent test: the anti-vacuity negative control is made deterministic. Aggregate recall over 25+ signatures absorbs a single removal within the 0.10 threshold headroom, and built-in checks (phrase.injection especially) often flag the dropped signature's sample anyway, so "remove a gating signature and the gate must go red" could pass vacuously. A designated control signature with no built-in or sibling overlap, an asserted gate-breach exit code, and a CI assertion of the no-overlap property replace it. - SC-001: tampering is defined over the fields the detached signature actually covers (bundle bytes, signature value, sidecar identity fields, mismatched pair) instead of "any byte of ... sidecar", which no detached scheme can guarantee against whitespace/key-order edits. - SC-002a gains fixture (b2) and US1's independent test gains case (i) for the authority-not-active refusal; FR-020, the threat-model edge case, Key Entities, and the Constitution Check note the new config-resident field. ## Testing - Docs-only change; no Go sources touched. - Every codebase claim re-verified before editing: cmd/scan-eval/gate.go (gateChecks/categoryCheck/OverallRecall/FPRate, no --bundle flag today), internal/security/scanner/inprocess.go (coverageOK), and internal/telemetry/anonymity.go (schema-v8 tpa_scanner whitelist). --- specs/101-tpa-db/spec.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 7233ebbed..d42ea97ae 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -40,7 +40,7 @@ An operator receives a TPA signature database out of band — a `scanner-bundle. **Why this priority**: The database is a code-adjacent security input: whoever controls its bytes controls what the scanner ignores. Without signing and anti-rollback, the "refreshable signature DB" pillar creates a new supply-chain surface (swap the file → blind the scanner; replay an old bundle → resurrect a fixed blind spot). Signing + sequence + fail-closed loading is what makes every other story safe to ship, so it goes first. -**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), (g) an unsigned bundle claiming a huge sequence — assert the watermark is unchanged so a later legitimate signed release still activates, and (h) on a FRESH install with no persisted state, a genuinely publisher-signed current-epoch bundle whose sequence is older than the binary's embedded corpus — assert it is refused by the FR-009b seeded baseline rather than activating at full coverage. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. +**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), (g) an unsigned bundle claiming a huge sequence — assert the watermark is unchanged so a later legitimate signed release still activates, and (h) on a FRESH install with no persisted state, a genuinely publisher-signed current-epoch bundle whose sequence is older than the binary's embedded corpus — assert it is refused by the FR-009b seeded baseline rather than activating at full coverage, and (i) with an operator key configured but the publisher authority declared active, a valid operator-signed bundle clearing the operator baseline — assert it is refused as "authority not active" (FR-004b) and the publisher bundle keeps serving. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. **Acceptance Scenarios**: @@ -61,7 +61,7 @@ A security-conscious developer installs mcpproxy and gets, out of the box, a sig **Why this priority**: The format (US1) is worthless empty, and the refresh channel (US3) has nothing to carry without a corpus. This is the "database" in tpa-db. It is P2 only because the P1 trust machinery must exist before a larger corpus becomes an attractive distribution target. -**Independent Test**: Run the corpus build and assert every signature carries id, category, detector(s), provenance source, and license; assert every gated signature has ≥1 labeled malicious eval sample and ≥1 category-matched hard-negative in the eval dataset; run `go run ./cmd/scan-eval --corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json --gate --min-recall 0.90 --max-fp 0.05` with the new embedded default and assert PASS — and, because that command today scores built-in checks only, assert the accompanying NEGATIVE CONTROL that proves the bundle is being scored at all: with a gating signature removed from the bundle the same command must FAIL. Feed each campaign class's canonical fixture through the offline scanner and assert a hard-tier finding naming the TPA id. +**Independent Test**: Run the corpus build and assert every signature carries id, category, detector(s), provenance source, and license; assert every gated signature has ≥1 labeled malicious eval sample and ≥1 category-matched hard-negative in the eval dataset; run `go run ./cmd/scan-eval --corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json --gate --min-recall 0.90 --max-fp 0.05` with the new embedded default and assert PASS — and, because that command today scores built-in checks only, assert the accompanying NEGATIVE CONTROL that proves the bundle is being scored at all: with the designated FR-016b control signature removed from the bundle, the same command must FAIL with the gate-breach exit code (the control signature is the one whose sample no built-in check and no other signature detects, so its removal necessarily crosses the threshold rather than being absorbed by aggregate recall headroom). Feed each campaign class's canonical fixture through the offline scanner and assert a hard-tier finding naming the TPA id. **Acceptance Scenarios**: @@ -93,7 +93,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Edge Cases - **Air-gapped host, forever**: every P1 behavior (verification, sequence check, rollback, activation self-check) is network-free; the trust anchors ship inside the binary; staleness is surfaced but never blocks scanning — an old database is degraded coverage, not an outage. -- **Threat-model boundary — attacker with config write**: FR-004 lets operator-configured keys join the trust set, so anyone who can write `mcp_config.json` can trust their own key and sign anything. This is explicitly OUT of the threat model and is not a regression: an attacker with config write can already disable quarantine outright or set a server's trust mode to `auto`. The attacker this design defends against is the one who can write the BUNDLE PATH or intercept the FETCH but not the config — which is precisely why the signature ratchet (FR-009) is keyed to install state rather than to the bundle path, and why `require_signed_bundle` is config-resident (it must not be defeatable from the data dir alone). +- **Threat-model boundary — attacker with config write**: FR-004 lets operator-configured keys join the trust set, so anyone who can write `mcp_config.json` can trust their own key and sign anything. This is explicitly OUT of the threat model and is not a regression: an attacker with config write can already disable quarantine outright or set a server's trust mode to `auto`. The attacker this design defends against is the one who can write the BUNDLE PATH or intercept the FETCH but not the config — which is precisely why the signature ratchet (FR-009) is keyed to install state rather than to the bundle path, why the active-authority declaration (FR-004b) and the operator baseline (FR-009b) are config-resident, and why `require_signed_bundle` is config-resident (none of them must be defeatable from the data dir alone). - **Signature key rotation**: the trust anchor is a SET of publisher keys embedded per binary release. Rotation is a binary-release event: a release ships old+new keys, bundles are signed with the new key, the old key is removed in a later release. Keys are NEVER delivered over the network, and rotation is not done casually — every shipped binary pins its key set, so a hasty rotation strands older installs on unsigned-refresh only (same philosophy as the app-update signing keys). Key compromise → remove the key in the next release, **bump the trust epoch (FR-005a)**, and publish a bundle in the new epoch signed by the surviving key. The epoch bump is what makes recovery possible at all: a compromised key can sign at the maximum representable sequence, and against a monotonic watermark no surviving key could ever outbid that within the same epoch, so revocation alone would leave installs permanently unable to accept any future release. - **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade watermark still blocks sequence replay. - **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. @@ -115,7 +115,8 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), a publisher key identifier, and the **trust epoch** the artifact was published under (FR-005a). Because these keys live inside the signed bytes, the epoch is cryptographically bound to the bundle and cannot be reinterpreted after the fact. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). `generated_at` MUST be derived from deterministic release metadata (the release commit's timestamp or `SOURCE_DATE_EPOCH`), NEVER from build wall-clock time — a wall-clock stamp would change the bundle bytes, fingerprint, and signature on every rebuild of an identical corpus and so would directly violate FR-001's byte-identical-rebuild guarantee. - **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. A key id MUST be a canonical full-length fingerprint of the public key itself (not an operator-chosen label), the trust set MUST reject duplicate ids at build/config time, and verification MUST succeed only when the manifest key id, the sidecar key id, and the id of the trust-set key whose signature actually verified are all identical — a manifest naming one publisher while the bytes are signed by another MUST be a verification failure, not a cosmetic mismatch. - **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. These are two distinct **signing authorities**, and the design MUST keep them so: all publisher keys (including keys being rotated in or out, FR-005) belong to ONE publisher authority, while operator-configured keys form a separate operator authority. -- **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Choosing an operator-signed corpus over a higher-sequence publisher one is a deliberate operator decision and MUST NOT by itself be treated as a downgrade or as degraded coverage — the operator configured that key precisely to assert their own corpus — but the active authority MUST be visible in bundle status (FR-020) so the choice is never invisible. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. +- **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Choosing an operator-signed corpus over a higher-sequence publisher one is a deliberate operator decision (declared per FR-004b) and MUST NOT by itself be treated as a downgrade or as degraded coverage — the operator configured that key precisely to assert their own corpus — but the active authority MUST be visible in bundle status (FR-020) so the choice is never invisible. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. +- **FR-004b**: The **active signing authority MUST be a config-resident declaration**, defaulting to the publisher authority, and MUST NOT be inferred solely from whichever key happens to verify a candidate. A candidate whose verifying key belongs to a non-active authority MUST be refused with an "authority not active" reason, exactly as any other failed candidate (last-known-good keeps serving). Without this, per-authority watermarks (FR-004a) open the very hole they were introduced to avoid: because each authority is compared only against its own counter, an attacker who can write the bundle path — squarely inside the threat model, unlike the config-write attacker — can replace a current publisher bundle at sequence 100 with ANY operator-signed artifact that merely clears the operator authority's FR-009b baseline, silently swapping the install onto an older or deliberately narrower corpus while it still reports `signature_verified=true` at FULL coverage. The publisher watermark cannot detect this, since the comparison never crosses namespaces, and no operator decision is involved — whoever writes the path picks the authority. Config-residency is what puts that choice out of the attacker's reach, matching `require_signed_bundle` (FR-007) and the operator baseline (FR-009b), which are config-resident for the same reason. Both key sets MAY remain configured simultaneously; only artifacts of the declared active authority activate, and switching authorities is a config edit, not a file drop. - **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure and the "do not rotate casually" operational stance. - **FR-005a**: Each signing authority's trust set MUST carry a **trust epoch** — a monotonic per-authority integer that advances ONLY via a binary release — and every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. The epoch MUST be carried inside the SIGNED manifest (FR-002) and a candidate MUST be accepted only when its manifest epoch EQUALS the current binary-trusted epoch of the authority whose key verified it — never merely less-than-or-equal. Binding the epoch to the signed bytes and requiring exact equality is what stops the epoch bump from becoming a replay window of its own: if the epoch lived only in the binary, then after a bump-and-revoke the re-baselined watermark would start from zero, and an OLD, weaker bundle legitimately signed by a still-trusted surviving key would be silently reinterpreted as belonging to the new epoch, clear the fresh watermark, and activate with `signature_verified=true` and full coverage — reinstating exactly the blind spots the recovery release was published to close. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle whose manifest epoch matches a newly-bumped binary epoch MUST supersede the retained watermark and re-baseline it. A bundle whose manifest epoch is BELOW the current binary-trusted epoch MUST be refused as stale-epoch replay even when its signature verifies against a still-trusted key. Epoch advancement is safe to treat as authoritative because it arrives only inside a release binary, whose integrity is already the root of trust for the embedded key set (FR-004) — it is not attacker-reachable over the bundle or fetch path. The accepted consequence of exact-equality matching is that a binary still on the OLD epoch will refuse the new epoch's bundles and stop receiving database updates until it is itself updated. That is deliberate, not incidental: an epoch bump means a key was compromised, and an un-updated binary still trusts that compromised key, so freezing its database is safer than feeding it artifacts whose trust basis it cannot evaluate. It also matches the existing "a hasty rotation strands older installs" stance, and it MUST be surfaced as a stale-epoch degraded state (FR-020) rather than failing silently. @@ -126,7 +127,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. -- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark (the ordinary fresh install), this is simply the normal full-coverage state. Where a signed watermark IS retained by ANY authority (FR-004a) and the embedded default is below it, the embedded corpus may be older and weaker than that watermark implies, so THAT state MUST be treated as degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case), the retained watermark and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. +- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) an external bundle of the currently active authority (FR-004b) was previously activated and is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for and states it directly: the trigger is evidence that a corpus the install was actually running has gone missing, not an arithmetic accident. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. @@ -138,7 +139,8 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-014**: Gating signatures MUST emit hard-tier signals (the eval gate and the `scan`-mode approval gates score hard tier only); any soft-tier-only signature MUST be explicitly marked non-gating in its metadata and MUST NOT be counted toward gate coverage. - **FR-015**: Signatures MUST be authored in the corpus source-of-truth pipeline (signature sources compiled to the bundle); mcpproxy MUST continue to consume only the compiled bundle and never parse signature sources (Spec 087 FR-002 parity). The contribution pipeline MUST run the same validation the loader runs (compile, schema, license, eval gate) before a bundle can be signed. - **FR-016**: The CI eval gate (`cmd/scan-eval --gate`, recall ≥ 0.90 over gated categories, hard-negative FP rate ≤ 0.05, existing vacuity guard) MUST pass for: every change to the embedded default corpus (merge-blocking, existing `eval.yml` D2 job) and every published bundle BEFORE it is signed — a bundle that fails the gate is never signed or published. Thresholds are reused verbatim, not redefined. -- **FR-016a**: The gate harness MUST first be taught to see the database at all — today it does not, so FR-016 and SC-005 would otherwise pass vacuously. `cmd/scan-eval/gate.go`'s `gateChecks()` returns only the four built-in `detect.Check`s (`unicode.hidden`, `shadowing.cross_server`, `payload.decoded`, `phrase.injection`); `scanner.BundleCheck` — which production DOES append in `internal/security/scanner/inprocess.go` — is absent, and `scan-eval` never loads a bundle. Therefore: (a) `scan-eval` MUST accept a candidate bundle path, load it through the production loader (never a parallel parser), and register the resulting bundle check, defaulting to the embedded default so the existing `eval.yml` D2 invocation starts scoring the shipped corpus; (b) `gate.go`'s `categoryCheck` map MUST gain an entry for every new TPA campaign category, since `gatedCategory()` enforces a category ONLY when its mapped check id is registered — an unmapped category is measured but can never fail the build, which is precisely the vacuous-pass failure mode; (c) the same candidate-loading path MUST be what the Spec 087 FR-008 activation self-check runs, so the CI layer and the activation layer cannot drift apart while SC-005 claims they are independent. +- **FR-016a**: The gate harness MUST first be taught to see the database at all — today it does not, so FR-016 and SC-005 would otherwise pass vacuously. `cmd/scan-eval/gate.go`'s `gateChecks()` returns only the four built-in `detect.Check`s (`unicode.hidden`, `shadowing.cross_server`, `payload.decoded`, `phrase.injection`); `scanner.BundleCheck` — which production DOES append in `internal/security/scanner/inprocess.go` — is absent, and `scan-eval` never loads a bundle. Therefore: (a) `scan-eval` MUST accept a candidate bundle path, load it through the production loader (never a parallel parser), and register the resulting bundle check, defaulting to the embedded default so the existing `eval.yml` D2 invocation starts scoring the shipped corpus; (b) `gate.go`'s `categoryCheck` map MUST gain an entry for every new TPA campaign category, since `gatedCategory()` enforces a category ONLY when its mapped check id is registered — an unmapped category's malicious samples are tallied and reported but excluded from `gatedMalicious`/`OverallRecall`, so its MISSES can never fail the gate (its false positives still count, because `FPRate` is computed over the whole hard-negative set regardless of category mapping). A category whose recall is unenforceable is precisely the vacuous-pass failure mode; (c) the same candidate-loading path MUST be what the Spec 087 FR-008 activation self-check runs, so the CI layer and the activation layer cannot drift apart while SC-005 claims they are independent. +- **FR-016b**: The FR-016a/SC-005 negative control MUST be constructed to fail DETERMINISTICALLY, not left as "remove any signature and hope the gate turns red". As written it can pass vacuously twice over: `OverallRecall` is an aggregate `gatedDetected/gatedMalicious` over ALL gated categories, so with a 25-signature corpus dropping one signature typically moves recall by well under the 0.10 headroom the 0.90 threshold allows; and the dropped signature's malicious sample may still be flagged at hard tier by a built-in check (`phrase.injection` in particular overlaps most instruction-injection payloads), in which case recall does not move at all. Therefore the corpus MUST designate at least one **control signature** whose canonical malicious sample is verified to be hard-flagged by NO built-in `detect.Check` and by no other bundle signature, and the control MUST assert the gate's exit code is `exitGateBreach` when that signature alone is removed. The control MUST be evaluated over a scope in which the removal necessarily crosses the threshold (a control corpus restricted to the control signature's category, or an equivalent per-category assertion), and the no-overlap property of the control sample MUST itself be asserted in CI so that a later built-in check gaining coverage of it turns the control red rather than silently re-vacuating it. **Publication & refresh channel** @@ -148,7 +150,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Observability, telemetry & docs** -- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signing authority and trust epoch, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, stale-epoch replay, unsigned-refused-by-ratchet, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. +- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signing authority and trust epoch, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, stale-epoch replay, unsigned-refused-by-ratchet, authority-not-active, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. - **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. - **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. @@ -157,7 +159,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Signature Database (bundle + sidecar)**: the publishable unit — the deterministic compiled `scanner-bundle.json` plus a detached signature over its exact bytes. Identified by (bundle_version, sequence, fingerprint). - **Bundle Manifest metadata**: `bundle_version`/`schema_version` (existing), plus additive `generated_at`, `sequence`, publisher key id. - **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. Each authority carries its own **trust epoch** (FR-005a), a monotonic integer that only a binary release can advance and that a candidate's signed manifest must match exactly. -- **Signing Authority**: the grouping anti-downgrade state is keyed by (FR-004a) — one publisher authority spanning all publisher keys including those mid-rotation, and a separate operator authority for operator-configured keys. Each holds its own watermark so neither can lock the other out. +- **Signing Authority**: the grouping anti-downgrade state is keyed by (FR-004a) — one publisher authority spanning all publisher keys including those mid-rotation, and a separate operator authority for operator-configured keys. Each holds its own watermark so neither can lock the other out. Exactly one authority is **active** at a time, declared in config (FR-004b, default publisher); artifacts of a non-active authority are refused, so per-authority isolation cannot be turned into an authority-swap by whoever can write the bundle path. - **Signature (TPA record)**: one cataloged campaign/technique — `TPA-YYYY-NNNN` id, detectors, category, level, confidence, tier intent (gating vs non-gating), provenance reference, license. - **Eval Sample Pair**: the labeled gated-malicious sample(s) plus category-matched hard-negative(s) a gating signature must contribute to the frozen eval dataset. - **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. Never empty — it is seeded from a trusted baseline (the embedded bundle's sequence, for the publisher authority) per FR-009b, so "no state yet" is not an accept-anything state. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving in a binary release. @@ -169,12 +171,12 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes -- **SC-001**: 100% of tampered candidates (any byte of bundle or sidecar altered, or mismatched pair) are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. +- **SC-001**: 100% of tampered candidates are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. "Tampered" is defined precisely, because the signature covers the bundle bytes and NOT the sidecar's own encoding: any altered bundle byte, any altered signature value, any altered sidecar identity field (algorithm or key id, whose agreement with the manifest FR-003 already requires), and any mismatched bundle/sidecar pair. Byte changes to the sidecar that alter neither the signature value nor an identity field (whitespace, key order) are explicitly NOT required to be rejected — a detached signature cannot detect them — so the criterion is stated in terms of the fields that carry meaning rather than an unachievable "every byte of the sidecar". - **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. -- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. +- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) with the operator authority declared active (FR-004b), an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so after the declaration is switched back the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A fixture (b2) covers the declaration itself: the same operator-signed bundle dropped while the PUBLISHER authority is declared active is refused as "authority not active" with the publisher bundle still serving, so the choice of authority cannot be made by whoever writes the bundle path. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. -- **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a negative control: removing a gating signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. +- **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a **deterministic** negative control (FR-016b): removing a designated control signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. - **SC-006**: Fleet freshness is answerable from telemetry: for opted-in installs, the distribution of active database version/sequence/source is reportable, enabling the metric "% of active installs on a database ≤ 30 days old" — a metric that is impossible to compute today. - **SC-007**: A newly published signature reaches detection within one daily refresh cycle on a fetch-enabled install (and immediately on manual drop), measured end to end: publish → active → the new signature's fixture fires. - **SC-008**: With a database at the design load ceilings (≥ 500 regex rules), p95 scan latency over a 100-tool server changes by less than 10% versus the 6-signature baseline, and bundle load-plus-verify completes under one second on commodity hardware. @@ -203,7 +205,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ## Constitution Check *(note)* -Principle IV (Security by Default) governs this spec: the database is treated as a hostile input until proven otherwise (verify-before-parse, fail-closed to last-known-good, anti-downgrade watermark, mandatory signatures on the network path), and refresh can never silently widen approvals. Principle III (Configuration-Driven Architecture): `require_signed_bundle`, operator keys, and the existing bundle path/fetch settings live in `mcp_config.json` with env override and hot-reload; no hardcoded URLs or paths. Principle V (TDD): every rejection class (tamper, downgrade, unsigned, gate regression, ceiling breach) is built against failing fixtures first, and the corpus itself is test-gated by scan-eval. Principle I (Performance at Scale): verification, activation, and the post-activation re-scan stay off the scan hot path with atomic swaps (SC-008). +Principle IV (Security by Default) governs this spec: the database is treated as a hostile input until proven otherwise (verify-before-parse, fail-closed to last-known-good, anti-downgrade watermark, mandatory signatures on the network path), and refresh can never silently widen approvals. Principle III (Configuration-Driven Architecture): `require_signed_bundle`, operator keys, the active-authority declaration and its baseline, and the existing bundle path/fetch settings live in `mcp_config.json` with env override and hot-reload; no hardcoded URLs or paths. Principle V (TDD): every rejection class (tamper, downgrade, unsigned, gate regression, ceiling breach) is built against failing fixtures first, and the corpus itself is test-gated by scan-eval. Principle I (Performance at Scale): verification, activation, and the post-activation re-scan stay off the scan hot path with atomic swaps (SC-008). ## Commit Message Conventions *(mandatory)* From eea76d30f751972c4e0b6f528dcc56a1817946b3 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 06:44:41 +0300 Subject: [PATCH 08/20] docs(specs): bind rollback to a fingerprint and narrow the degraded fallback (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode / gpt-5.6-sol) round 2 on specs/101-tpa-db/spec.md. All three findings verified against the document and gate.go before changing anything. ## Changes - FR-010: the rollback target must be identified by content fingerprint, not by pathname alone. Rollback is the one path that deliberately bypasses the watermark, so naming its target by an attacker-writable path lets a bundle-path attacker choose what the bypass lands on — substituting a genuinely publisher-signed, current-epoch bundle at sequence 5 for the sequence-90 artifact the operator intended, which then gets PINNED. The operator now supplies the expected fingerprint, the path is only a hint, a mismatch is refused, and a rollback target must sit at or above the FR-009b seeded baseline (the embedded default excepted, since it is that baseline). - FR-009a condition (b): narrowed to a missing OPERATOR-authority bundle. As written it degraded coverage whenever any previously-activated external bundle went missing, including the ordinary case of a publisher external file removed after a binary upgrade shipped a strictly newer embedded corpus — where condition (a) already answers the question on the same axis and says full coverage. - US2 independent test: the negative control now runs over the FR-016b control scope instead of re-running the full-corpus command. One miss out of a 25-signature corpus leaves aggregate recall near 0.96, above the 0.90 threshold, so the full-corpus form contradicted FR-016b and would have proven nothing. - US1 independent test, SC-002, FR-020: assertions and rejection reason for the rollback-target mismatch. ## Testing - Docs-only change; no Go sources touched. - Recall/FP semantics re-checked in cmd/scan-eval/gate.go (OverallRecall is gatedDetected/gatedMalicious aggregated across gated categories). --- specs/101-tpa-db/spec.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index d42ea97ae..1910b1338 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -40,7 +40,7 @@ An operator receives a TPA signature database out of band — a `scanner-bundle. **Why this priority**: The database is a code-adjacent security input: whoever controls its bytes controls what the scanner ignores. Without signing and anti-rollback, the "refreshable signature DB" pillar creates a new supply-chain surface (swap the file → blind the scanner; replay an old bundle → resurrect a fixed blind spot). Signing + sequence + fail-closed loading is what makes every other story safe to ship, so it goes first. -**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), (g) an unsigned bundle claiming a huge sequence — assert the watermark is unchanged so a later legitimate signed release still activates, and (h) on a FRESH install with no persisted state, a genuinely publisher-signed current-epoch bundle whose sequence is older than the binary's embedded corpus — assert it is refused by the FR-009b seeded baseline rather than activating at full coverage, and (i) with an operator key configured but the publisher authority declared active, a valid operator-signed bundle clearing the operator baseline — assert it is refused as "authority not active" (FR-004b) and the publisher bundle keeps serving. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release. +**Independent Test**: Fully offline, no live upstream. Sign a fixture bundle with a test key configured as a trust anchor; drop bundle+signature and assert activation with `signature_verified=true` and the new sequence recorded. Then assert each of the following is rejected with the active bundle unchanged and a machine-readable reason: (a) one flipped byte in the bundle, (b) a signature from a non-trusted key, (c) a missing sidecar with `require_signed_bundle=true`, (d) a validly-signed bundle whose sequence is lower than the active one, (e) a validly-signed bundle failing the activation self-check, (f) an unsigned bundle dropped after a signed one was active (ratchet), (g) an unsigned bundle claiming a huge sequence — assert the watermark is unchanged so a later legitimate signed release still activates, and (h) on a FRESH install with no persisted state, a genuinely publisher-signed current-epoch bundle whose sequence is older than the binary's embedded corpus — assert it is refused by the FR-009b seeded baseline rather than activating at full coverage, and (i) with an operator key configured but the publisher authority declared active, a valid operator-signed bundle clearing the operator baseline — assert it is refused as "authority not active" (FR-004b) and the publisher bundle keeps serving. Finally, invoke the explicit rollback action and assert the embedded default is restored, the override is logged, the high-watermark is unchanged, and a subsequent refresh does not re-activate the deny-listed release; and assert that a rollback whose named file has been substituted (valid signature, current epoch, but a fingerprint other than the one the operator supplied) is REFUSED as a target mismatch rather than activated through the watermark bypass (FR-010). **Acceptance Scenarios**: @@ -61,7 +61,7 @@ A security-conscious developer installs mcpproxy and gets, out of the box, a sig **Why this priority**: The format (US1) is worthless empty, and the refresh channel (US3) has nothing to carry without a corpus. This is the "database" in tpa-db. It is P2 only because the P1 trust machinery must exist before a larger corpus becomes an attractive distribution target. -**Independent Test**: Run the corpus build and assert every signature carries id, category, detector(s), provenance source, and license; assert every gated signature has ≥1 labeled malicious eval sample and ≥1 category-matched hard-negative in the eval dataset; run `go run ./cmd/scan-eval --corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json --gate --min-recall 0.90 --max-fp 0.05` with the new embedded default and assert PASS — and, because that command today scores built-in checks only, assert the accompanying NEGATIVE CONTROL that proves the bundle is being scored at all: with the designated FR-016b control signature removed from the bundle, the same command must FAIL with the gate-breach exit code (the control signature is the one whose sample no built-in check and no other signature detects, so its removal necessarily crosses the threshold rather than being absorbed by aggregate recall headroom). Feed each campaign class's canonical fixture through the offline scanner and assert a hard-tier finding naming the TPA id. +**Independent Test**: Run the corpus build and assert every signature carries id, category, detector(s), provenance source, and license; assert every gated signature has ≥1 labeled malicious eval sample and ≥1 category-matched hard-negative in the eval dataset; run `go run ./cmd/scan-eval --corpus specs/065-evaluation-foundation/datasets/detect_corpus_v1.json --gate --min-recall 0.90 --max-fp 0.05` with the new embedded default and assert PASS — and, because that command today scores built-in checks only, assert the accompanying NEGATIVE CONTROL that proves the bundle is being scored at all: with the designated FR-016b control signature removed from the bundle, the gate must exit with the gate-breach code when run over the FR-016b control scope (the corpus restricted to that signature's category, or the equivalent per-category recall assertion). The control MUST NOT be expressed as "remove the signature and re-run the full-corpus command" — one miss out of a 25-signature corpus leaves aggregate recall at roughly 0.96, comfortably above the 0.90 threshold, so the full-corpus run would stay green and the control would prove nothing. Feed each campaign class's canonical fixture through the offline scanner and assert a hard-tier finding naming the TPA id. **Acceptance Scenarios**: @@ -127,8 +127,8 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. -- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) an external bundle of the currently active authority (FR-004b) was previously activated and is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for and states it directly: the trigger is evidence that a corpus the install was actually running has gone missing, not an arithmetic accident. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. -- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. +- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. +- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above the FR-009b seeded baseline (the running binary's own embedded corpus) — the embedded default itself being the one target always permitted, since it IS that baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. @@ -150,7 +150,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Observability, telemetry & docs** -- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signing authority and trust epoch, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, stale-epoch replay, unsigned-refused-by-ratchet, authority-not-active, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. +- **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signing authority and trust epoch, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, stale-epoch replay, unsigned-refused-by-ratchet, authority-not-active, rollback-target mismatch, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. - **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. - **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. @@ -172,7 +172,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes - **SC-001**: 100% of tampered candidates are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. "Tampered" is defined precisely, because the signature covers the bundle bytes and NOT the sidecar's own encoding: any altered bundle byte, any altered signature value, any altered sidecar identity field (algorithm or key id, whose agreement with the manifest FR-003 already requires), and any mismatched bundle/sidecar pair. Byte changes to the sidecar that alter neither the signature value nor an identity field (whitespace, key order) are explicitly NOT required to be rejected — a detached signature cannot detect them — so the criterion is stated in terms of the fields that carry meaning rather than an unachievable "every byte of the sidecar". -- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. +- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. - **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) with the operator authority declared active (FR-004b), an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so after the declaration is switched back the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A fixture (b2) covers the declaration itself: the same operator-signed bundle dropped while the PUBLISHER authority is declared active is refused as "authority not active" with the publisher bundle still serving, so the choice of authority cannot be made by whoever writes the bundle path. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. From 9feda6ea4a8b68802bae2071fb66e1df1a10923b Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 06:49:28 +0300 Subject: [PATCH 09/20] docs(specs): cap the sidecar, scope the rollback floor, pin the gate control (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode / gpt-5.6-sol) round 3 on specs/101-tpa-db/spec.md. All three findings verified against the document and gate.go before changing anything. ## Changes - FR-011 + FR-008 + edge cases: the sidecar gets its own small byte cap and a bounded read, applied before it is parsed, on both the file-drop and fetch paths. The sidecar is the one artifact that must be read before any signature is verified — its algorithm id, key id, and signature value are verification's own inputs — so FR-006's verify-before-parse protection does not cover it, and a bundle-path attacker could drop a multi-gigabyte sidecar next to an otherwise valid bundle and stall or exhaust the refresh path. The pipeline order and the "two ceilings" note are updated accordingly. - FR-010: the rollback floor is now its OWN authority's FR-009b baseline, not the publisher one in both cases. As written it compared an operator-signed rollback target at sequence 9 against the publisher embedded corpus at 100 and would have refused every legitimate operator rollback, contradicting the per-authority isolation of FR-004a. - FR-016b: "restrict the control to its category" is not by itself provably failing — decide() breaches only on OverallRecall < minRecall, so a control category with ten gated malicious samples lands on exactly 0.90 after one miss and stays green on the boundary. The canonical construction is now a control corpus whose gated malicious set is the control sample alone; any other scope must compute and assert the recall drop rather than assume it. ## Testing - Docs-only change; no Go sources touched. - Boundary semantics re-checked in cmd/scan-eval/gate.go decide() (strict `<` against min-recall) and the per-authority baseline rules in FR-009b. --- specs/101-tpa-db/spec.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 1910b1338..6228caaba 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -101,7 +101,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Anti-downgrade state lost** (data directory wiped/reset): the persisted watermark resets to the FR-009b seeded baseline — NOT to zero, so a wiped install still refuses any publisher-signed bundle older than the corpus its own binary ships. For the operator authority the same protection holds only if the operator declared a config-resident baseline; without one, operator-authority anti-downgrade does not survive the wipe and the install runs degraded while an operator-signed bundle is active, by FR-009b, rather than pretending to a guarantee it does not have. What a wipe does lose is the FR-009 signature-ratchet state and any FR-010 pin/deny-list (both data-dir resident), so a wiped install accepts an unsigned drop again. This is accepted — the ratchet is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. - **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). - **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. -- **Huge or adversarial database** (thousands of rules, pathological regex): size and rule-count ceilings are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. +- **Huge or adversarial database** (thousands of rules, pathological regex) **or an adversarial sidecar**: the sidecar byte cap (read before it is parsed, since verification's own inputs come from it), plus the bundle size and rule-count ceilings, are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. - **Post-activation re-scan storms**: the re-scan is off the hot path, rate-limited, runs once per activation over cached metadata only (no upstream reconnects), and produces at most one review finding per tool per bundle activation — no notification storm. - **Signature removed from a newer bundle** (false-positive retired): tools previously held by it are NOT auto-approved by refresh; the held state persists for human review (state changes only flow toward review, never silently toward approval). @@ -124,12 +124,12 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-006**: For any candidate carrying a signature, verification MUST run over the exact raw candidate bytes BEFORE those bytes are parsed as JSON or any pattern is compiled; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). The bytes that are verified MUST be the same bytes that are subsequently parsed (read once into memory, verify, then parse that buffer) — the file MUST NOT be re-read from disk after verification, or a writer racing the loader could swap content between verify and parse. - **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a case — when it is serving as a fallback BELOW a retained signed watermark, i.e. when the install has evidence a newer signed corpus should be active. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. -- **FR-008**: The full validate-before-activate order MUST be: byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's two ceilings MUST therefore be split across the pipeline rather than applied as one stage: byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. +- **FR-008**: The full validate-before-activate order MUST be: sidecar byte cap → parse sidecar → bundle byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's ceilings MUST therefore be split across the pipeline rather than applied as one stage: the sidecar cap comes first because the sidecar is read before any signature exists to check, bundle byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. -- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above the FR-009b seeded baseline (the running binary's own embedded corpus) — the embedded default itself being the one target always permitted, since it IS that baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. -- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). +- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. +- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. **Seed corpus** @@ -140,7 +140,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-015**: Signatures MUST be authored in the corpus source-of-truth pipeline (signature sources compiled to the bundle); mcpproxy MUST continue to consume only the compiled bundle and never parse signature sources (Spec 087 FR-002 parity). The contribution pipeline MUST run the same validation the loader runs (compile, schema, license, eval gate) before a bundle can be signed. - **FR-016**: The CI eval gate (`cmd/scan-eval --gate`, recall ≥ 0.90 over gated categories, hard-negative FP rate ≤ 0.05, existing vacuity guard) MUST pass for: every change to the embedded default corpus (merge-blocking, existing `eval.yml` D2 job) and every published bundle BEFORE it is signed — a bundle that fails the gate is never signed or published. Thresholds are reused verbatim, not redefined. - **FR-016a**: The gate harness MUST first be taught to see the database at all — today it does not, so FR-016 and SC-005 would otherwise pass vacuously. `cmd/scan-eval/gate.go`'s `gateChecks()` returns only the four built-in `detect.Check`s (`unicode.hidden`, `shadowing.cross_server`, `payload.decoded`, `phrase.injection`); `scanner.BundleCheck` — which production DOES append in `internal/security/scanner/inprocess.go` — is absent, and `scan-eval` never loads a bundle. Therefore: (a) `scan-eval` MUST accept a candidate bundle path, load it through the production loader (never a parallel parser), and register the resulting bundle check, defaulting to the embedded default so the existing `eval.yml` D2 invocation starts scoring the shipped corpus; (b) `gate.go`'s `categoryCheck` map MUST gain an entry for every new TPA campaign category, since `gatedCategory()` enforces a category ONLY when its mapped check id is registered — an unmapped category's malicious samples are tallied and reported but excluded from `gatedMalicious`/`OverallRecall`, so its MISSES can never fail the gate (its false positives still count, because `FPRate` is computed over the whole hard-negative set regardless of category mapping). A category whose recall is unenforceable is precisely the vacuous-pass failure mode; (c) the same candidate-loading path MUST be what the Spec 087 FR-008 activation self-check runs, so the CI layer and the activation layer cannot drift apart while SC-005 claims they are independent. -- **FR-016b**: The FR-016a/SC-005 negative control MUST be constructed to fail DETERMINISTICALLY, not left as "remove any signature and hope the gate turns red". As written it can pass vacuously twice over: `OverallRecall` is an aggregate `gatedDetected/gatedMalicious` over ALL gated categories, so with a 25-signature corpus dropping one signature typically moves recall by well under the 0.10 headroom the 0.90 threshold allows; and the dropped signature's malicious sample may still be flagged at hard tier by a built-in check (`phrase.injection` in particular overlaps most instruction-injection payloads), in which case recall does not move at all. Therefore the corpus MUST designate at least one **control signature** whose canonical malicious sample is verified to be hard-flagged by NO built-in `detect.Check` and by no other bundle signature, and the control MUST assert the gate's exit code is `exitGateBreach` when that signature alone is removed. The control MUST be evaluated over a scope in which the removal necessarily crosses the threshold (a control corpus restricted to the control signature's category, or an equivalent per-category assertion), and the no-overlap property of the control sample MUST itself be asserted in CI so that a later built-in check gaining coverage of it turns the control red rather than silently re-vacuating it. +- **FR-016b**: The FR-016a/SC-005 negative control MUST be constructed to fail DETERMINISTICALLY, not left as "remove any signature and hope the gate turns red". As written it can pass vacuously twice over: `OverallRecall` is an aggregate `gatedDetected/gatedMalicious` over ALL gated categories, so with a 25-signature corpus dropping one signature typically moves recall by well under the 0.10 headroom the 0.90 threshold allows; and the dropped signature's malicious sample may still be flagged at hard tier by a built-in check (`phrase.injection` in particular overlaps most instruction-injection payloads), in which case recall does not move at all. Therefore the corpus MUST designate at least one **control signature** whose canonical malicious sample is verified to be hard-flagged by NO built-in `detect.Check` and by no other bundle signature, and the control MUST assert the gate's exit code is `exitGateBreach` when that signature alone is removed. The control MUST be evaluated over a scope in which the removal PROVABLY crosses the threshold, and merely "restricting to the control signature's category" does not establish that: `decide()` fails only on `OverallRecall < minRecall`, so a control category holding ten gated malicious samples lands on exactly 0.90 after one miss and the gate stays green on the boundary. The canonical construction is therefore a control corpus whose gated malicious set is the control signature's sample ALONE, so removal drives recall to 0 — any other scope MUST be accompanied by an explicit assertion that the control category's recall is 1.0 with the signature present and strictly below the threshold without it, computed rather than assumed. The no-overlap property of the control sample MUST itself be asserted in CI so that a later built-in check gaining coverage of it turns the control red rather than silently re-vacuating it. **Publication & refresh channel** From c2fbc7fe6a000848baad01114b93bc44b451a148 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 06:53:06 +0300 Subject: [PATCH 10/20] docs(specs): refuse non-regular candidates and scope the freshness metric (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode / gpt-5.6-sol) round 4 on specs/101-tpa-db/spec.md. Findings verified against the document before changing anything; the telemetry finding was accepted in substance but not in its proposed remedy (see below). ## Changes - FR-011 + Edge Cases: a byte cap bounds bytes, not time. Local candidate artifacts (bundle and sidecar) must be REGULAR files — a FIFO, device, socket, or symlink resolving to one is refused outright rather than read — and every candidate read, local or fetched, runs under an enforceable I/O deadline whose expiry releases Spec 087's single-flight slot. Otherwise the bundle-path attacker leaves a FIFO with no writer and pins the refresh path indefinitely without ever approaching the cap. - FR-010: an authority with no declared baseline — possible only for the operator authority, which FR-009b explicitly permits — has no rollback floor to enforce, and none is invented. The rollback proceeds and the install is marked degraded while that bundle is active, which is already FR-009b's treatment. Refusing the rollback would break FR-010 for a configuration FR-009b allows; assuming a zero floor would claim a guarantee that does not exist. - SC-006: the freshness metric is scoped to publisher-issued corpora and its derivation is stated — age comes from a server-side join of the reported version/sequence to the publisher's own release dates. Installs on an operator-built corpus (the `other` bucket) are not ageable and form their own bucket rather than being folded into the ratio. ## Notes on the rejected remedy The reviewer's suggested fix for SC-006 was to report `generated_at`/age in the heartbeat. That is exactly the near-unique value FR-021's schema-v8 backstop exists to keep out of the payload, and it would buy nothing the version/sequence join does not already give, so the derivation was specified instead. ## Testing - Docs-only change; no Go sources touched. --- specs/101-tpa-db/spec.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 6228caaba..f9c07fb5a 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -101,7 +101,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Anti-downgrade state lost** (data directory wiped/reset): the persisted watermark resets to the FR-009b seeded baseline — NOT to zero, so a wiped install still refuses any publisher-signed bundle older than the corpus its own binary ships. For the operator authority the same protection holds only if the operator declared a config-resident baseline; without one, operator-authority anti-downgrade does not survive the wipe and the install runs degraded while an operator-signed bundle is active, by FR-009b, rather than pretending to a guarantee it does not have. What a wipe does lose is the FR-009 signature-ratchet state and any FR-010 pin/deny-list (both data-dir resident), so a wiped install accepts an unsigned drop again. This is accepted — the ratchet is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. - **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). - **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. -- **Huge or adversarial database** (thousands of rules, pathological regex) **or an adversarial sidecar**: the sidecar byte cap (read before it is parsed, since verification's own inputs come from it), plus the bundle size and rule-count ceilings, are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. +- **Huge or adversarial database** (thousands of rules, pathological regex) **or an adversarial sidecar**: the sidecar byte cap (read before it is parsed, since verification's own inputs come from it), the regular-file requirement and read deadline that stop a FIFO or stalled stream from pinning the refresh path, plus the bundle size and rule-count ceilings, are enforced at load; every pattern remains RE2 (linear-time, no catastrophic backtracking by construction); a candidate exceeding ceilings is rejected whole, keeping last-known-good. - **Post-activation re-scan storms**: the re-scan is off the hot path, rate-limited, runs once per activation over cached metadata only (no upstream reconnects), and produces at most one review finding per tool per bundle activation — no notification storm. - **Signature removed from a newer bundle** (false-positive retired): tools previously held by it are NOT auto-approved by refresh; the held state persists for human review (state changes only flow toward review, never silently toward approval). @@ -128,8 +128,8 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. -- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. -- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). +- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. +- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be required to be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read — and every candidate read (local files and fetch response bodies alike) MUST run under an enforceable I/O deadline whose expiry releases the single-flight slot and records a rejection reason, so no candidate can pin the refresh path. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. **Seed corpus** @@ -177,7 +177,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. - **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a **deterministic** negative control (FR-016b): removing a designated control signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. -- **SC-006**: Fleet freshness is answerable from telemetry: for opted-in installs, the distribution of active database version/sequence/source is reportable, enabling the metric "% of active installs on a database ≤ 30 days old" — a metric that is impossible to compute today. +- **SC-006**: Fleet freshness is answerable from telemetry: for opted-in installs, the distribution of active database version/sequence/source is reportable, enabling the metric "% of active installs on a **publisher-issued** database ≤ 30 days old" — a metric that is impossible to compute today. Age MUST be derived server-side by joining the reported publisher-issued version/sequence to the publication date the publisher already knows for its own releases, NOT by adding a `generated_at` or age field to the heartbeat: a build timestamp is precisely the kind of near-unique value FR-021's backstop exists to keep out of the payload, and it would undo that protection to buy nothing the join does not already give. Installs on an operator-built corpus (reported in the `other` bucket per FR-021) are consequently NOT ageable and MUST be reported as their own unaged bucket rather than silently folded into either side of the ratio — the metric's denominator is publisher-corpus installs. - **SC-007**: A newly published signature reaches detection within one daily refresh cycle on a fetch-enabled install (and immediately on manual drop), measured end to end: publish → active → the new signature's fixture fires. - **SC-008**: With a database at the design load ceilings (≥ 500 regex rules), p95 scan latency over a 100-tool server changes by less than 10% versus the 6-signature baseline, and bundle load-plus-verify completes under one second on commodity hardware. - **SC-009**: After activating a database containing a new signature, an already-approved tool whose cached description matches it is surfaced for review within the same refresh cycle, with zero approvals auto-revoked and zero servers auto-quarantined by the refresh itself. From 3979f29f79fe1d16381a575cfb01bd915e0a7a44 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 06:57:16 +0300 Subject: [PATCH 11/20] docs(specs): name the baseline-less exemption and bucket operator sequences (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode / gpt-5.6-sol) round 5 on specs/101-tpa-db/spec.md. Round 5 reported no remaining adversary-reachable security hole; both findings were internal contradictions, verified against the document before editing. ## Changes - FR-009b, the Sequence Watermark entity, and SC-002: the baseline-less operator authority is now named as an explicit exemption everywhere the document made a universal claim. FR-009b permits an operator authority with no declared baseline (marking the install degraded), while the same requirement opened with "every watermark MUST be SEEDED", the entity said "Never empty", and SC-002 called FR-009a the "sole exemption" — an implementation could not satisfy all four at once. There are exactly two exemptions and both are degraded-coverage states. - FR-021: `sequence` is reported only when `bundle_version` resolves to a publisher-issued value; in the `other` bucket it is omitted. Bucketing the version string while transmitting a raw operator-chosen sequence defeats the purpose of the bucket, since an operator-built corpus can number itself with a unix timestamp or private build number and be just as fleet-identifying. SC-006 already scopes freshness to publisher corpora, so no analysis is lost. ## Testing - Docs-only change; no Go sources touched. - Round 5 is the per-PR cross-model review cap (CLAUDE.md escalation trigger 4); these two fixes are not themselves re-reviewed. --- specs/101-tpa-db/spec.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index f9c07fb5a..3bc600403 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -126,7 +126,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a case — when it is serving as a fallback BELOW a retained signed watermark, i.e. when the install has evidence a newer signed corpus should be active. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. - **FR-008**: The full validate-before-activate order MUST be: sidecar byte cap → parse sidecar → bundle byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's ceilings MUST therefore be split across the pipeline rather than applied as one stage: the sidecar cap comes first because the sidecar is read before any signature exists to check, bundle byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. -- **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. +- **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything — with exactly one exception, carved out at the end of this requirement: an operator authority configured without a baseline, which has no seed available and is therefore run as explicitly degraded rather than as a silent zero. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be required to be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read — and every candidate read (local files and fetch response bodies alike) MUST run under an enforceable I/O deadline whose expiry releases the single-flight slot and records a rejection reason, so no candidate can pin the refresh path. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). @@ -151,7 +151,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Observability, telemetry & docs** - **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signing authority and trust epoch, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, stale-epoch replay, unsigned-refused-by-ratchet, authority-not-active, rollback-target mismatch, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. -- **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. +- **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. **The same reasoning MUST be applied to `sequence`, not only to `bundle_version`**: an operator picks their own corpus's sequence values freely — a unix timestamp or a private build number is the obvious choice — so a raw integer sequence from an operator-built corpus is every bit the near-unique fleet identifier the `other` bucket was introduced to suppress, and bucketing the version while transmitting the sequence verbatim would defeat it. `sequence` MUST therefore be reported only when `bundle_version` resolves to a publisher-issued value (a known, low-cardinality release line); when the version falls into the `other` bucket the sequence MUST be omitted. Nothing analytic is lost: SC-006 already scopes fleet-freshness to publisher-issued corpora and reports operator-built ones as an unaged bucket. - **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. ### Key Entities *(include if feature involves data)* @@ -162,7 +162,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Signing Authority**: the grouping anti-downgrade state is keyed by (FR-004a) — one publisher authority spanning all publisher keys including those mid-rotation, and a separate operator authority for operator-configured keys. Each holds its own watermark so neither can lock the other out. Exactly one authority is **active** at a time, declared in config (FR-004b, default publisher); artifacts of a non-active authority are refused, so per-authority isolation cannot be turned into an authority-swap by whoever can write the bundle path. - **Signature (TPA record)**: one cataloged campaign/technique — `TPA-YYYY-NNNN` id, detectors, category, level, confidence, tier intent (gating vs non-gating), provenance reference, license. - **Eval Sample Pair**: the labeled gated-malicious sample(s) plus category-matched hard-negative(s) a gating signature must contribute to the frozen eval dataset. -- **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. Never empty — it is seeded from a trusted baseline (the embedded bundle's sequence, for the publisher authority) per FR-009b, so "no state yet" is not an accept-anything state. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving in a binary release. +- **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. Never empty for any authority that has a baseline — it is seeded from a trusted one (the embedded bundle's sequence, for the publisher authority) per FR-009b, so "no state yet" is not an accept-anything state. The single exception is an operator authority configured without a baseline, which FR-009b permits precisely because no binary-trusted seed exists for it; there the install carries no anti-downgrade guarantee and is marked degraded while such a bundle is active rather than being presented as fully trusted. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving in a binary release. - **Publication Channel**: the stable public location of versioned signed artifact pairs with a "latest" reference; consumed by the Spec 087 fetch or by manual download. - **Post-Activation Re-scan**: the one-shot, off-hot-path evaluation of cached approved tool metadata against a newly-activated database, emitting review findings only. - **Active Database / Last-Known-Good**: unchanged from Spec 086/087 — the serving bundle and the fail-closed fallback target. @@ -172,7 +172,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes - **SC-001**: 100% of tampered candidates are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. "Tampered" is defined precisely, because the signature covers the bundle bytes and NOT the sidecar's own encoding: any altered bundle byte, any altered signature value, any altered sidecar identity field (algorithm or key id, whose agreement with the manifest FR-003 already requires), and any mismatched bundle/sidecar pair. Byte changes to the sidecar that alter neither the signature value nor an identity field (whitespace, key order) are explicitly NOT required to be rejected — a detached signature cannot detect them — so the criterion is stated in terms of the fields that carry meaning rather than an unachievable "every byte of the sidecar". -- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. The sole exemption is the FR-009a embedded-default availability fallback, which serves as degraded coverage rather than counting as a downgrade. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. +- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. There are exactly two exemptions, and both are degraded-coverage states rather than silent ones: the FR-009a embedded-default availability fallback, and the baseline-less operator authority of FR-009b (SC-002a fixture (c)), which has no seed to compare against and so carries no anti-downgrade guarantee to claim. Neither counts as a downgrade; both MUST surface as degraded. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. - **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) with the operator authority declared active (FR-004b), an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so after the declaration is switched back the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A fixture (b2) covers the declaration itself: the same operator-signed bundle dropped while the PUBLISHER authority is declared active is refused as "authority not active" with the publisher bundle still serving, so the choice of authority cannot be made by whoever writes the bundle path. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. From 544d7135c72f87629c2c24ed151d81806a06525f Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 07:09:21 +0300 Subject: [PATCH 12/20] docs(specs): correct the SC-002a fixture count after adding (b2) (101-tpa-db) --- specs/101-tpa-db/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 3bc600403..51ddbefde 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -173,7 +173,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **SC-001**: 100% of tampered candidates are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. "Tampered" is defined precisely, because the signature covers the bundle bytes and NOT the sidecar's own encoding: any altered bundle byte, any altered signature value, any altered sidecar identity field (algorithm or key id, whose agreement with the manifest FR-003 already requires), and any mismatched bundle/sidecar pair. Byte changes to the sidecar that alter neither the signature value nor an identity field (whitespace, key order) are explicitly NOT required to be rejected — a detached signature cannot detect them — so the criterion is stated in terms of the fields that carry meaning rather than an unachievable "every byte of the sidecar". - **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. There are exactly two exemptions, and both are degraded-coverage states rather than silent ones: the FR-009a embedded-default availability fallback, and the baseline-less operator authority of FR-009b (SC-002a fixture (c)), which has no seed to compare against and so carries no anti-downgrade guarantee to claim. Neither counts as a downgrade; both MUST surface as degraded. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. -- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by two fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) with the operator authority declared active (FR-004b), an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so after the declaration is switched back the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A fixture (b2) covers the declaration itself: the same operator-signed bundle dropped while the PUBLISHER authority is declared active is refused as "authority not active" with the publisher bundle still serving, so the choice of authority cannot be made by whoever writes the bundle path. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. +- **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by the following fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) with the operator authority declared active (FR-004b), an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so after the declaration is switched back the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A fixture (b2) covers the declaration itself: the same operator-signed bundle dropped while the PUBLISHER authority is declared active is refused as "authority not active" with the publisher bundle still serving, so the choice of authority cannot be made by whoever writes the bundle path. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. - **SC-005**: The scan-eval CI gate (recall ≥ 0.90 gated categories, hard-negative FP ≤ 0.05) passes with the full seed corpus embedded AND actually scored — i.e. the gate run registers the bundle check and maps every new campaign category (FR-016a), proven by a **deterministic** negative control (FR-016b): removing a designated control signature from the bundle must turn the gate red. Without that control the criterion is vacuous, since today's gate would pass with the bundle entirely absent. A deliberately-regressing corpus change is then blocked twice independently: at CI (cannot merge) and at activation (self-check rejects it) — both demonstrated by fixture. From 82d544ddf49888d80509259499b37e44e3409c50 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 07:21:07 +0300 Subject: [PATCH 13/20] docs(specs): align FR-007 and FR-009a with the two-exemption rule (101-tpa-db) --- specs/101-tpa-db/spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 51ddbefde..05cc70b39 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -123,11 +123,11 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Loader & activation** - **FR-006**: For any candidate carrying a signature, verification MUST run over the exact raw candidate bytes BEFORE those bytes are parsed as JSON or any pattern is compiled; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). The bytes that are verified MUST be the same bytes that are subsequently parsed (read once into memory, verify, then parse that buffer) — the file MUST NOT be re-read from disk after verification, or a writer racing the loader could swap content between verify and parse. -- **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a case — when it is serving as a fallback BELOW a retained signed watermark, i.e. when the install has evidence a newer signed corpus should be active. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. +- **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a cases — when it is serving as a fallback BELOW the retained publisher watermark, or when the active operator authority's previously-activated bundle has gone missing, i.e. when the install has evidence that a corpus it should be running is newer than, or not covered by, the embedded one. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. - **FR-008**: The full validate-before-activate order MUST be: sidecar byte cap → parse sidecar → bundle byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's ceilings MUST therefore be split across the pipeline rather than applied as one stage: the sidecar cap comes first because the sidecar is read before any signature exists to check, bundle byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything — with exactly one exception, carved out at the end of this requirement: an operator authority configured without a baseline, which has no seed available and is therefore run as explicitly degraded rather than as a silent zero. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. -- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This below-watermark fallback is the single explicit exemption to SC-002. +- **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This fallback is one of SC-002's two explicit exemptions; the other is FR-009b's baseline-less operator authority. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be required to be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read — and every candidate read (local files and fetch response bodies alike) MUST run under an enforceable I/O deadline whose expiry releases the single-flight slot and records a rejection reason, so no candidate can pin the refresh path. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. From afff3cb7988c64284033d329fd9e61f5a83bd24f Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 13:37:59 +0300 Subject: [PATCH 14/20] docs(specs): scope pins per epoch, gate telemetry on provenance, fix exemption count (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode / gpt-5.6-sol) round 6 on specs/101-tpa-db/spec.md. All five findings verified against the document and the cited code before editing. - Edge Cases (threat model): the boundary is now explicit about the data directory — the bundle-path attacker may also be able to delete/reset the persisted security state (equivalent to the wipe), against which only the binary/config-anchored guarantees survive (seeded publisher baseline, config-resident operator baseline, require_signed_bundle, active-authority declaration); ratchet and pin/deny-list are best-effort. Chosen-value corruption of the state store is out of scope (that attacker is already past every mcpproxy guarantee). - Edge Cases + FR-009a: two stale 'single exemption to SC-002' statements now name both exemptions, matching SC-002's 'exactly two'. - US1 acceptance 3 + SC-002: 'lower sequence' is qualified everywhere as a same-authority, same-epoch comparison; a deliberate FR-004b authority switch is by definition not a downgrade and needs no exemption. - FR-010 + FR-005a: pin/deny-list entries are keyed (authority, epoch, sequence, fingerprint); an authority switch suspends (not deletes) the non-active authority's entries; an epoch bump retires prior-epoch entries so stale rollback state cannot block the key-compromise recovery bundle. - FR-021: 'resolves to a publisher-issued value' is a provenance test (embedded default or publisher-authority-verified), never a claimed-version-string match — an operator corpus can copy a publisher version string verbatim; the backstop also rejects sequence-present-with-version-other cross-field. - Docs-only change; no Go sources touched. - Round 6 of the raised (10-round) cross-model review cap. --- specs/101-tpa-db/spec.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 05cc70b39..c384a9126 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -46,7 +46,7 @@ An operator receives a TPA signature database out of band — a `scanner-bundle. 1. **Given** a candidate bundle whose detached signature verifies against a configured trust anchor and whose sequence is greater than the active bundle's, **When** refresh runs, **Then** the candidate proceeds through the existing validation pipeline and, on success, becomes active with its sequence persisted and `signature_verified=true` surfaced in bundle status. 2. **Given** a candidate whose bytes were altered after signing (signature mismatch), **When** refresh runs, **Then** the candidate is rejected BEFORE its JSON is parsed or any pattern compiled, the active bundle is unchanged, and a "signature verification failed" reason is recorded. -3. **Given** a validly-signed candidate whose sequence is lower than the highest sequence ever activated on this install, **When** refresh runs, **Then** it is refused as a downgrade, the active bundle is unchanged, and a "downgrade refused" reason names both sequences. +3. **Given** a validly-signed candidate whose sequence is lower than its OWN signing authority's watermark — the highest sequence that authority has ever activated on this install within the current trust epoch (FR-004a/FR-005a), never a cross-authority comparison — **When** refresh runs, **Then** it is refused as a downgrade, the active bundle is unchanged, and a "downgrade refused" reason names both sequences. 4. **Given** `require_signed_bundle` is enabled and a file-drop candidate has no signature sidecar, **When** refresh runs, **Then** the candidate is refused (fail-closed) with a "signature required" reason; with the setting disabled (default) and no signature-verified bundle ever having been active on this install, an unsigned drop is accepted through the existing Spec 086 validation path but surfaced as `signature_verified=false`, does NOT advance the sequence watermark, and marks coverage degraded so `scan`-mode auto-approval is suspended while it is active. 4a. **Given** an install that has already activated a signature-verified bundle, **When** an unsigned candidate is dropped (with `require_signed_bundle` still off), **Then** it is refused by the signature ratchet — signing cannot be stripped back off an install by whoever can write the bundle path — and an unsigned candidate claiming an arbitrarily high sequence never becomes the watermark, so it cannot lock out subsequent legitimate signed releases. 5. **Given** a network-fetched candidate (Spec 087 opt-in path), **When** it arrives, **Then** signature verification is ALWAYS mandatory regardless of `require_signed_bundle` — an unsigned or wrongly-signed fetched bundle is never activated (parity with Spec 087 FR-013/FR-014). @@ -93,11 +93,11 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Edge Cases - **Air-gapped host, forever**: every P1 behavior (verification, sequence check, rollback, activation self-check) is network-free; the trust anchors ship inside the binary; staleness is surfaced but never blocks scanning — an old database is degraded coverage, not an outage. -- **Threat-model boundary — attacker with config write**: FR-004 lets operator-configured keys join the trust set, so anyone who can write `mcp_config.json` can trust their own key and sign anything. This is explicitly OUT of the threat model and is not a regression: an attacker with config write can already disable quarantine outright or set a server's trust mode to `auto`. The attacker this design defends against is the one who can write the BUNDLE PATH or intercept the FETCH but not the config — which is precisely why the signature ratchet (FR-009) is keyed to install state rather than to the bundle path, why the active-authority declaration (FR-004b) and the operator baseline (FR-009b) are config-resident, and why `require_signed_bundle` is config-resident (none of them must be defeatable from the data dir alone). +- **Threat-model boundary — attacker with config write**: FR-004 lets operator-configured keys join the trust set, so anyone who can write `mcp_config.json` can trust their own key and sign anything. This is explicitly OUT of the threat model and is not a regression: an attacker with config write can already disable quarantine outright or set a server's trust mode to `auto`. The attacker this design defends against is the one who can write the BUNDLE PATH or intercept the FETCH but not the config — which is precisely why the signature ratchet (FR-009) is keyed to install state rather than to the bundle path, why the active-authority declaration (FR-004b) and the operator baseline (FR-009b) are config-resident, and why `require_signed_bundle` is config-resident (none of them must be defeatable from the data dir alone). The boundary MUST be read conservatively: because the bundle path may be configured to live inside the data directory, the design MUST NOT assume the bundle-path attacker cannot also delete or reset the data-directory persisted security state (watermark record, ratchet flag, pin/deny-list) — the worst such an attacker can achieve is exactly the "anti-downgrade state lost" wipe below, against which the guarantees that survive are the ones anchored outside the data dir: the FR-009b publisher baseline (re-seeded from the binary at every startup), the config-resident operator baseline, `require_signed_bundle`, and the FR-004b active-authority declaration. The ratchet and the pin/deny-list are data-dir resident and are therefore explicitly best-effort hardening against this attacker, never the sole defense. What the design DOES assume is that the attacker cannot substitute or corrupt the persisted state with CHOSEN values (e.g. rewrite the watermark to a lower number without triggering the seeded-baseline floor): the state store is the same integrity-protected local database that already holds approvals and quarantine state, and an attacker who can write arbitrary records into it is already past every guarantee mcpproxy makes, not just this feature's. - **Signature key rotation**: the trust anchor is a SET of publisher keys embedded per binary release. Rotation is a binary-release event: a release ships old+new keys, bundles are signed with the new key, the old key is removed in a later release. Keys are NEVER delivered over the network, and rotation is not done casually — every shipped binary pins its key set, so a hasty rotation strands older installs on unsigned-refresh only (same philosophy as the app-update signing keys). Key compromise → remove the key in the next release, **bump the trust epoch (FR-005a)**, and publish a bundle in the new epoch signed by the surviving key. The epoch bump is what makes recovery possible at all: a compromised key can sign at the maximum representable sequence, and against a monotonic watermark no surviving key could ever outbid that within the same epoch, so revocation alone would leave installs permanently unable to accept any future release. - **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade watermark still blocks sequence replay. - **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. -- **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted watermark does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the watermark retained and the situation surfaced, but per FR-009a as DEGRADED coverage, so an attacker who merely deletes the external bundle cannot use the weaker embedded corpus to obtain `scan`-mode auto-approvals. This is the one explicit exemption to SC-002. +- **Downgrade attack via the embedded default**: a fresh binary whose embedded bundle sequence is LOWER than a previously-activated external bundle's persisted watermark does not silently downgrade — the external candidate re-validates and wins by precedence; if only the embedded bundle is available, it serves (scanning must never stop) with the watermark retained and the situation surfaced, but per FR-009a as DEGRADED coverage, so an attacker who merely deletes the external bundle cannot use the weaker embedded corpus to obtain `scan`-mode auto-approvals. This is one of SC-002's exactly two exemptions (the other is FR-009b's baseline-less operator authority). - **Anti-downgrade state lost** (data directory wiped/reset): the persisted watermark resets to the FR-009b seeded baseline — NOT to zero, so a wiped install still refuses any publisher-signed bundle older than the corpus its own binary ships. For the operator authority the same protection holds only if the operator declared a config-resident baseline; without one, operator-authority anti-downgrade does not survive the wipe and the install runs degraded while an operator-signed bundle is active, by FR-009b, rather than pretending to a guarantee it does not have. What a wipe does lose is the FR-009 signature-ratchet state and any FR-010 pin/deny-list (both data-dir resident), so a wiped install accepts an unsigned drop again. This is accepted — the ratchet is best-effort hardening on top of signing, not the sole defense, and a wiped data dir already resets approvals. Operators who need this closed absolutely set `require_signed_bundle`, which is config-resident rather than data-dir-resident and therefore survives the wipe. - **Equal sequence, different bytes** (two distinct bundles claiming the same sequence): refused as suspicious unless it is the currently-active fingerprint (idempotent re-drop stays a no-op per Spec 087 FR-011). - **Clock skew**: no validity windows or expiry in v1 — sequence ordering, not wall-clock time, is the trust signal; `generated_at` is advisory freshness only. @@ -118,7 +118,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Choosing an operator-signed corpus over a higher-sequence publisher one is a deliberate operator decision (declared per FR-004b) and MUST NOT by itself be treated as a downgrade or as degraded coverage — the operator configured that key precisely to assert their own corpus — but the active authority MUST be visible in bundle status (FR-020) so the choice is never invisible. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. - **FR-004b**: The **active signing authority MUST be a config-resident declaration**, defaulting to the publisher authority, and MUST NOT be inferred solely from whichever key happens to verify a candidate. A candidate whose verifying key belongs to a non-active authority MUST be refused with an "authority not active" reason, exactly as any other failed candidate (last-known-good keeps serving). Without this, per-authority watermarks (FR-004a) open the very hole they were introduced to avoid: because each authority is compared only against its own counter, an attacker who can write the bundle path — squarely inside the threat model, unlike the config-write attacker — can replace a current publisher bundle at sequence 100 with ANY operator-signed artifact that merely clears the operator authority's FR-009b baseline, silently swapping the install onto an older or deliberately narrower corpus while it still reports `signature_verified=true` at FULL coverage. The publisher watermark cannot detect this, since the comparison never crosses namespaces, and no operator decision is involved — whoever writes the path picks the authority. Config-residency is what puts that choice out of the attacker's reach, matching `require_signed_bundle` (FR-007) and the operator baseline (FR-009b), which are config-resident for the same reason. Both key sets MAY remain configured simultaneously; only artifacts of the declared active authority activate, and switching authorities is a config edit, not a file drop. - **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure and the "do not rotate casually" operational stance. -- **FR-005a**: Each signing authority's trust set MUST carry a **trust epoch** — a monotonic per-authority integer that advances ONLY via a binary release — and every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. The epoch MUST be carried inside the SIGNED manifest (FR-002) and a candidate MUST be accepted only when its manifest epoch EQUALS the current binary-trusted epoch of the authority whose key verified it — never merely less-than-or-equal. Binding the epoch to the signed bytes and requiring exact equality is what stops the epoch bump from becoming a replay window of its own: if the epoch lived only in the binary, then after a bump-and-revoke the re-baselined watermark would start from zero, and an OLD, weaker bundle legitimately signed by a still-trusted surviving key would be silently reinterpreted as belonging to the new epoch, clear the fresh watermark, and activate with `signature_verified=true` and full coverage — reinstating exactly the blind spots the recovery release was published to close. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle whose manifest epoch matches a newly-bumped binary epoch MUST supersede the retained watermark and re-baseline it. A bundle whose manifest epoch is BELOW the current binary-trusted epoch MUST be refused as stale-epoch replay even when its signature verifies against a still-trusted key. Epoch advancement is safe to treat as authoritative because it arrives only inside a release binary, whose integrity is already the root of trust for the embedded key set (FR-004) — it is not attacker-reachable over the bundle or fetch path. The accepted consequence of exact-equality matching is that a binary still on the OLD epoch will refuse the new epoch's bundles and stop receiving database updates until it is itself updated. That is deliberate, not incidental: an epoch bump means a key was compromised, and an un-updated binary still trusts that compromised key, so freezing its database is safer than feeding it artifacts whose trust basis it cannot evaluate. It also matches the existing "a hasty rotation strands older installs" stance, and it MUST be surfaced as a stale-epoch degraded state (FR-020) rather than failing silently. +- **FR-005a**: Each signing authority's trust set MUST carry a **trust epoch** — a monotonic per-authority integer that advances ONLY via a binary release — and every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. The epoch MUST be carried inside the SIGNED manifest (FR-002) and a candidate MUST be accepted only when its manifest epoch EQUALS the current binary-trusted epoch of the authority whose key verified it — never merely less-than-or-equal. Binding the epoch to the signed bytes and requiring exact equality is what stops the epoch bump from becoming a replay window of its own: if the epoch lived only in the binary, then after a bump-and-revoke the re-baselined watermark would start from zero, and an OLD, weaker bundle legitimately signed by a still-trusted surviving key would be silently reinterpreted as belonging to the new epoch, clear the fresh watermark, and activate with `signature_verified=true` and full coverage — reinstating exactly the blind spots the recovery release was published to close. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle whose manifest epoch matches a newly-bumped binary epoch MUST supersede the retained watermark and re-baseline it; the bump likewise retires the authority's prior-epoch FR-010 pins and deny-list entries, so stale rollback state cannot block the recovery bundle (FR-010). A bundle whose manifest epoch is BELOW the current binary-trusted epoch MUST be refused as stale-epoch replay even when its signature verifies against a still-trusted key. Epoch advancement is safe to treat as authoritative because it arrives only inside a release binary, whose integrity is already the root of trust for the embedded key set (FR-004) — it is not attacker-reachable over the bundle or fetch path. The accepted consequence of exact-equality matching is that a binary still on the OLD epoch will refuse the new epoch's bundles and stop receiving database updates until it is itself updated. That is deliberate, not incidental: an epoch bump means a key was compromised, and an un-updated binary still trusts that compromised key, so freezing its database is safer than feeding it artifacts whose trust basis it cannot evaluate. It also matches the existing "a hasty rotation strands older installs" stance, and it MUST be surfaced as a stale-epoch degraded state (FR-020) rather than failing silently. **Loader & activation** @@ -128,7 +128,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything — with exactly one exception, carved out at the end of this requirement: an operator authority configured without a baseline, which has no seed available and is therefore run as explicitly degraded rather than as a silent zero. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This fallback is one of SC-002's two explicit exemptions; the other is FR-009b's baseline-less operator authority. -- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. +- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. **Pin and deny-list state MUST be scoped like the watermark it excepts**: each entry is keyed by (authority, trust epoch, sequence, fingerprint) and binds only while its authority is the active one (FR-004b) and its epoch equals the binary's current epoch for that authority. Switching the active authority SUSPENDS (does not delete) the non-active authority's pin and deny-list, so switching back restores them unchanged. A binary release that bumps an authority's trust epoch (FR-005a) RETIRES that authority's prior-epoch pins and deny-list entries, loudly logged: the epoch bump is the key-compromise recovery path and MUST NOT be blockable by stale pin state — a retained old-epoch pin could not re-activate its artifact anyway (the pinned manifest's epoch no longer matches, FR-005a), so honoring it could only strand the install on the embedded fallback while blocking the very recovery bundle the bump exists to deliver. - **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be required to be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read — and every candidate read (local files and fetch response bodies alike) MUST run under an enforceable I/O deadline whose expiry releases the single-flight slot and records a rejection reason, so no candidate can pin the refresh path. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. @@ -151,7 +151,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Observability, telemetry & docs** - **FR-020**: Bundle status on all existing surfaces (CLI security commands, REST security overview, Web UI) MUST additionally report sequence, signing authority and trust epoch, signature-verified state, and staleness (age since `generated_at`), alongside the existing version/fingerprint/source/load-error fields, and MUST show the last rejection reason (tamper, downgrade, stale-epoch replay, unsigned-refused-by-ratchet, authority-not-active, rollback-target mismatch, gate failure) so an operator can tell "current and verified" from "stuck on last-known-good and why". Where the active database has demoted the install to DEGRADED coverage (FR-007 unsigned-active, FR-009a embedded fallback), that state and its reason MUST be surfaced too — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install. -- **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. **The same reasoning MUST be applied to `sequence`, not only to `bundle_version`**: an operator picks their own corpus's sequence values freely — a unix timestamp or a private build number is the obvious choice — so a raw integer sequence from an operator-built corpus is every bit the near-unique fleet identifier the `other` bucket was introduced to suppress, and bucketing the version while transmitting the sequence verbatim would defeat it. `sequence` MUST therefore be reported only when `bundle_version` resolves to a publisher-issued value (a known, low-cardinality release line); when the version falls into the `other` bucket the sequence MUST be omitted. Nothing analytic is lost: SC-006 already scopes fleet-freshness to publisher-issued corpora and reports operator-built ones as an unaged bucket. +- **FR-021**: The existing anonymous TPA-scanner telemetry MUST gain the active bundle version, sequence, and source (and signature-verified state) so database adoption and freshness are measurable fleet-wide; the existing telemetry opt-out MUST fully cover these fields, and no new identifying data is introduced. This is NOT a drop-in addition: `internal/telemetry/anonymity.go`'s schema-v8 backstop hard-whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and requires every scalar to be a non-negative integer, so a string `bundle_version`/`source` or a boolean `signature_verified` would be rejected before transmit as "carries a key outside the whitelist". The design MUST therefore extend that backstop with explicitly TYPED entries (integer `sequence`, enum-bounded `source` ∈ {embedded, file, fetch}, boolean signature-verified) rather than relaxing it to accept arbitrary keys or free strings. `bundle_version` MUST be constrained to publisher-issued values (unrecognized/operator-built corpus versions reported as a single `other` bucket): the backstop exists precisely to stop a bespoke, near-unique string from acting as a fleet identifier, and a self-built corpus version would be exactly that. **The same reasoning MUST be applied to `sequence`, not only to `bundle_version`**: an operator picks their own corpus's sequence values freely — a unix timestamp or a private build number is the obvious choice — so a raw integer sequence from an operator-built corpus is every bit the near-unique fleet identifier the `other` bucket was introduced to suppress, and bucketing the version while transmitting the sequence verbatim would defeat it. `sequence` MUST therefore be reported only when `bundle_version` resolves to a publisher-issued value (a known, low-cardinality release line); when the version falls into the `other` bucket the sequence MUST be omitted. **"Resolves to a publisher-issued value" MUST be a provenance test, never a match on the claimed version string**: an operator-signed or unsigned corpus controls its own manifest and can copy a genuine publisher version string verbatim while carrying a unique sequence, which would sail through a string allowlist and transmit the fingerprinting sequence anyway. The verbatim version and the sequence MUST therefore be reported only when the active bundle's provenance is the publisher authority — the embedded default, or an external bundle signature-verified against a publisher key (FR-004) — and every other active bundle reports `other` with sequence omitted REGARDLESS of the version string it claims. The reporting layer is where authority is known, so the gate lives there; the anonymity backstop additionally enforces the cross-field invariant it can see on the payload alone — a heartbeat carrying a `sequence` while `bundle_version` is `other` is rejected before transmit. Nothing analytic is lost: SC-006 already scopes fleet-freshness to publisher-issued corpora and reports operator-built ones as an unaged bucket. - **FR-022**: Documentation MUST cover the database lifecycle end to end for operators (verify, drop, fetch, roll back, read status) and for contributors (author a signature, provenance/license rules, eval-sample requirements, how the gate blocks a bad corpus), updating the existing security-quarantine/tool-scanner docs. ### Key Entities *(include if feature involves data)* @@ -172,7 +172,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes - **SC-001**: 100% of tampered candidates are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. "Tampered" is defined precisely, because the signature covers the bundle bytes and NOT the sidecar's own encoding: any altered bundle byte, any altered signature value, any altered sidecar identity field (algorithm or key id, whose agreement with the manifest FR-003 already requires), and any mismatched bundle/sidecar pair. Byte changes to the sidecar that alter neither the signature value nor an identity field (whitespace, key order) are explicitly NOT required to be rejected — a detached signature cannot detect them — so the criterion is stated in terms of the fields that carry meaning rather than an unachievable "every byte of the sidecar". -- **SC-002**: A validly-signed but lower-sequence bundle is never activated by any non-rollback path (drop, fetch, restart) — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. There are exactly two exemptions, and both are degraded-coverage states rather than silent ones: the FR-009a embedded-default availability fallback, and the baseline-less operator authority of FR-009b (SC-002a fixture (c)), which has no seed to compare against and so carries no anti-downgrade guarantee to claim. Neither counts as a downgrade; both MUST surface as degraded. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. +- **SC-002**: A validly-signed bundle whose sequence is below its OWN signing authority's watermark for the current trust epoch is never activated by any non-rollback path (drop, fetch, restart) — "lower-sequence" is always a same-authority, same-epoch comparison per FR-004a/FR-005a, so a deliberate FR-004b authority switch followed by the newly-active authority's bundle clearing its own baseline is by definition not a downgrade and needs no exemption — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. There are exactly two exemptions, and both are degraded-coverage states rather than silent ones: the FR-009a embedded-default availability fallback, and the baseline-less operator authority of FR-009b (SC-002a fixture (c)), which has no seed to compare against and so carries no anti-downgrade guarantee to claim. Neither counts as a downgrade; both MUST surface as degraded. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. - **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by the following fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) with the operator authority declared active (FR-004b), an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so after the declaration is switched back the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A fixture (b2) covers the declaration itself: the same operator-signed bundle dropped while the PUBLISHER authority is declared active is refused as "authority not active" with the publisher bundle still serving, so the choice of authority cannot be made by whoever writes the bundle path. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. From 10ae3cf583a6170b1205dd365a28358af6e35d75 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 24 Aug 2026 21:14:13 +0300 Subject: [PATCH 15/20] docs(specs): define authority-switch serving semantics + operator epoch recovery (101-tpa-db) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode / gpt-5.6-sol) round 7 on specs/101-tpa-db/spec.md. All three findings verified against the document before editing. ## Changes - FR-004b: the active-authority declaration governs the SERVING bundle, not only future candidates. On a declaration change, an active external bundle of the now-inactive authority is deactivated (loudly logged) and the install falls back to the embedded default per FR-009a until a bundle of the newly active authority activates. Otherwise keep-last-known-good makes the switch advisory: a bundle-path attacker keeping the new authority's candidates invalid could perpetuate the old corpus at full coverage. Declaring the operator authority active does not by itself degrade coverage. - FR-009 + SC-002: the highest-ever watermark and the signature ratchet are explicitly claims about retained data-directory state — after a reset only the FR-009b seeded/config-resident baselines survive and the ratchet re-arms on the next signed activation. Fixtures must test the post-reset guarantees the threat model actually supports, resolving the contradiction with the 'Anti-downgrade state lost' edge case. - FR-005a + Key Entities: epoch advancement is split per trust root — binary release for the publisher epoch, config edit for the operator epoch (declared alongside the operator keys/baseline). A binary-only rule made operator-key compromise an unrecoverable permanent lockout, since no publisher release can coordinate a private operator authority's epoch and a replacement key inherits the same-epoch watermark. A config bump re-baselines from the config-declared baseline and retires prior-epoch operator pins/deny-list. ## Testing - Docs-only change; no Go sources touched. - Round 7 of the raised (10-round) cross-model review cap. --- specs/101-tpa-db/spec.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index c384a9126..14047ac1c 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -116,16 +116,16 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. A key id MUST be a canonical full-length fingerprint of the public key itself (not an operator-chosen label), the trust set MUST reject duplicate ids at build/config time, and verification MUST succeed only when the manifest key id, the sidecar key id, and the id of the trust-set key whose signature actually verified are all identical — a manifest naming one publisher while the bytes are signed by another MUST be a verification failure, not a cosmetic mismatch. - **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. These are two distinct **signing authorities**, and the design MUST keep them so: all publisher keys (including keys being rotated in or out, FR-005) belong to ONE publisher authority, while operator-configured keys form a separate operator authority. - **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Choosing an operator-signed corpus over a higher-sequence publisher one is a deliberate operator decision (declared per FR-004b) and MUST NOT by itself be treated as a downgrade or as degraded coverage — the operator configured that key precisely to assert their own corpus — but the active authority MUST be visible in bundle status (FR-020) so the choice is never invisible. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. -- **FR-004b**: The **active signing authority MUST be a config-resident declaration**, defaulting to the publisher authority, and MUST NOT be inferred solely from whichever key happens to verify a candidate. A candidate whose verifying key belongs to a non-active authority MUST be refused with an "authority not active" reason, exactly as any other failed candidate (last-known-good keeps serving). Without this, per-authority watermarks (FR-004a) open the very hole they were introduced to avoid: because each authority is compared only against its own counter, an attacker who can write the bundle path — squarely inside the threat model, unlike the config-write attacker — can replace a current publisher bundle at sequence 100 with ANY operator-signed artifact that merely clears the operator authority's FR-009b baseline, silently swapping the install onto an older or deliberately narrower corpus while it still reports `signature_verified=true` at FULL coverage. The publisher watermark cannot detect this, since the comparison never crosses namespaces, and no operator decision is involved — whoever writes the path picks the authority. Config-residency is what puts that choice out of the attacker's reach, matching `require_signed_bundle` (FR-007) and the operator baseline (FR-009b), which are config-resident for the same reason. Both key sets MAY remain configured simultaneously; only artifacts of the declared active authority activate, and switching authorities is a config edit, not a file drop. +- **FR-004b**: The **active signing authority MUST be a config-resident declaration**, defaulting to the publisher authority, and MUST NOT be inferred solely from whichever key happens to verify a candidate. A candidate whose verifying key belongs to a non-active authority MUST be refused with an "authority not active" reason, exactly as any other failed candidate (last-known-good keeps serving). Without this, per-authority watermarks (FR-004a) open the very hole they were introduced to avoid: because each authority is compared only against its own counter, an attacker who can write the bundle path — squarely inside the threat model, unlike the config-write attacker — can replace a current publisher bundle at sequence 100 with ANY operator-signed artifact that merely clears the operator authority's FR-009b baseline, silently swapping the install onto an older or deliberately narrower corpus while it still reports `signature_verified=true` at FULL coverage. The publisher watermark cannot detect this, since the comparison never crosses namespaces, and no operator decision is involved — whoever writes the path picks the authority. Config-residency is what puts that choice out of the attacker's reach, matching `require_signed_bundle` (FR-007) and the operator baseline (FR-009b), which are config-resident for the same reason. Both key sets MAY remain configured simultaneously; only artifacts of the declared active authority activate, and switching authorities is a config edit, not a file drop. **The declaration governs the SERVING bundle, not only future candidates**: when the declaration changes, an active external bundle belonging to the now-inactive authority MUST NOT keep serving as full coverage — it is deactivated (loudly logged, surfaced per FR-020) and the install falls back to the embedded default under FR-009a's rules until a bundle of the newly-active authority activates. Without this the switch is advisory only: fail-safe keep-last-known-good would leave the old authority's corpus serving indefinitely, and an attacker who keeps the new authority's candidates invalid (bundle-path writes are in scope) could perpetuate that state at full coverage — the operator's explicit change of trust would simply never take effect. Merely declaring the operator authority active does not by itself degrade coverage: until an operator bundle activates, the embedded default serves exactly as on a fresh install (FR-009a condition (b) concerns a previously-activated operator bundle that was lost, not one that never existed). - **FR-005**: Key rotation MUST be supported by trusting multiple keys simultaneously: a binary release MAY add a new key while retaining the old for at least one release cycle, and key removal is also a binary-release event. The design MUST document the compromise procedure and the "do not rotate casually" operational stance. -- **FR-005a**: Each signing authority's trust set MUST carry a **trust epoch** — a monotonic per-authority integer that advances ONLY via a binary release — and every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. The epoch MUST be carried inside the SIGNED manifest (FR-002) and a candidate MUST be accepted only when its manifest epoch EQUALS the current binary-trusted epoch of the authority whose key verified it — never merely less-than-or-equal. Binding the epoch to the signed bytes and requiring exact equality is what stops the epoch bump from becoming a replay window of its own: if the epoch lived only in the binary, then after a bump-and-revoke the re-baselined watermark would start from zero, and an OLD, weaker bundle legitimately signed by a still-trusted surviving key would be silently reinterpreted as belonging to the new epoch, clear the fresh watermark, and activate with `signature_verified=true` and full coverage — reinstating exactly the blind spots the recovery release was published to close. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle whose manifest epoch matches a newly-bumped binary epoch MUST supersede the retained watermark and re-baseline it; the bump likewise retires the authority's prior-epoch FR-010 pins and deny-list entries, so stale rollback state cannot block the recovery bundle (FR-010). A bundle whose manifest epoch is BELOW the current binary-trusted epoch MUST be refused as stale-epoch replay even when its signature verifies against a still-trusted key. Epoch advancement is safe to treat as authoritative because it arrives only inside a release binary, whose integrity is already the root of trust for the embedded key set (FR-004) — it is not attacker-reachable over the bundle or fetch path. The accepted consequence of exact-equality matching is that a binary still on the OLD epoch will refuse the new epoch's bundles and stop receiving database updates until it is itself updated. That is deliberate, not incidental: an epoch bump means a key was compromised, and an un-updated binary still trusts that compromised key, so freezing its database is safer than feeding it artifacts whose trust basis it cannot evaluate. It also matches the existing "a hasty rotation strands older installs" stance, and it MUST be surfaced as a stale-epoch degraded state (FR-020) rather than failing silently. +- **FR-005a**: Each signing authority's trust set MUST carry a **trust epoch** — a monotonic per-authority integer that advances only through that authority's own trust root: the PUBLISHER epoch advances ONLY via a binary release, while the OPERATOR epoch is config-resident, declared alongside the operator keys and baseline, and advances only by a config edit. The split is forced by the same recovery argument that motivates the epoch itself: a compromised OPERATOR key can sign the maximum representable sequence exactly as a publisher key can, and no binary release can coordinate an epoch bump for a private operator authority the publisher has never heard of — a binary-only rule would make operator-key compromise an unrecoverable permanent lockout of the operator authority (replacing the key in config does not help, since the new key inherits the same-epoch watermark). A config-resident operator epoch is sound for the same reason the operator baseline is (FR-009b): the operator authority's entire trust root — its keys, baseline, and the active-authority declaration — is already config-resident, so the epoch adds no new attack surface for the config-write attacker, who is out of the threat model. Bumping the operator epoch in config behaves exactly as a publisher binary bump: it re-baselines the operator watermark from the config-declared baseline (or marks the authority baseline-less-degraded per FR-009b) and retires prior-epoch operator pins and deny-list entries (FR-010). Every anti-downgrade comparison (FR-009) MUST be scoped `(authority, epoch, sequence)`, ordered by epoch first. The epoch MUST be carried inside the SIGNED manifest (FR-002) and a candidate MUST be accepted only when its manifest epoch EQUALS the current binary-trusted epoch of the authority whose key verified it — never merely less-than-or-equal. Binding the epoch to the signed bytes and requiring exact equality is what stops the epoch bump from becoming a replay window of its own: if the epoch lived only in the binary, then after a bump-and-revoke the re-baselined watermark would start from zero, and an OLD, weaker bundle legitimately signed by a still-trusted surviving key would be silently reinterpreted as belonging to the new epoch, clear the fresh watermark, and activate with `signature_verified=true` and full coverage — reinstating exactly the blind spots the recovery release was published to close. Without this, key-compromise recovery is impossible rather than merely painful: an attacker holding a compromised publisher key signs a bundle at the maximum representable sequence, and because the watermark is monotonic and no surviving key can ever exceed that sequence, the install is permanently unable to accept any future legitimate release — a one-shot permanent denial of updates, downgraded only to the FR-009a fallback. A binary release that revokes the compromised key MUST therefore also bump the epoch, and a validly-signed bundle whose manifest epoch matches a newly-bumped binary epoch MUST supersede the retained watermark and re-baseline it; the bump likewise retires the authority's prior-epoch FR-010 pins and deny-list entries, so stale rollback state cannot block the recovery bundle (FR-010). A bundle whose manifest epoch is BELOW the current binary-trusted epoch MUST be refused as stale-epoch replay even when its signature verifies against a still-trusted key. Epoch advancement is safe to treat as authoritative because it arrives only through the authority's own trust root — a release binary for the publisher epoch (the binary's integrity is already the root of trust for the embedded key set, FR-004) and `mcp_config.json` for the operator epoch (already the root of trust for the operator keys themselves) — never over the bundle or fetch path, which is the attacker-reachable surface. The accepted consequence of exact-equality matching is that a binary still on the OLD epoch will refuse the new epoch's bundles and stop receiving database updates until it is itself updated. That is deliberate, not incidental: an epoch bump means a key was compromised, and an un-updated binary still trusts that compromised key, so freezing its database is safer than feeding it artifacts whose trust basis it cannot evaluate. It also matches the existing "a hasty rotation strands older installs" stance, and it MUST be surfaced as a stale-epoch degraded state (FR-020) rather than failing silently. **Loader & activation** - **FR-006**: For any candidate carrying a signature, verification MUST run over the exact raw candidate bytes BEFORE those bytes are parsed as JSON or any pattern is compiled; a candidate failing verification MUST be rejected without further processing, keeping last-known-good (extends the Spec 086 fail-closed pipeline with a new first stage). The bytes that are verified MUST be the same bytes that are subsequently parsed (read once into memory, verify, then parse that buffer) — the file MUST NOT be re-read from disk after verification, or a writer racing the loader could swap content between verify and parse. - **FR-007**: Signing policy MUST be: network-fetched candidates ALWAYS require a valid signature; file-drop candidates are verified whenever a sidecar is present; a new `require_signed_bundle` setting (default off for back-compat with existing unsigned drops) makes the sidecar mandatory for file drops too; the embedded default needs no sidecar (its integrity rides on binary distribution integrity). Bundle status MUST always report whether the active bundle was signature-verified. While an unsigned EXTERNAL (file-dropped or fetched) bundle is active, the scan coverage the approval gates consume MUST be marked degraded so `scan`-mode auto-approval is suspended and changes route to human review — today `internal/security/scanner/inprocess.go` sets `coverageOK` from mere bundle *presence*, so without this an attacker-supplied unsigned corpus would keep silently auto-approving under weakened rules. This degradation rule applies to EXTERNAL candidates ONLY: the embedded default is binary-trusted (its integrity rides on the signed binary distribution, not on a sidecar) and MUST be full-coverage on an ordinary install, or every fresh install would ship with `scan` mode silently disabled. The embedded default is degraded only in the FR-009a cases — when it is serving as a fallback BELOW the retained publisher watermark, or when the active operator authority's previously-activated bundle has gone missing, i.e. when the install has evidence that a corpus it should be running is newer than, or not covered by, the embedded one. This requirement supersedes Spec 087 FR-014's precondition: because FR-004 makes publisher keys embedded in the binary part of the trust set, a fetch is no longer refused merely because the operator configured no key of their own; the "never activate an unverified fetched bundle" guarantee of Spec 087 FR-013/FR-014 is preserved unchanged. - **FR-008**: The full validate-before-activate order MUST be: sidecar byte cap → parse sidecar → bundle byte-size ceiling → signature over the raw bytes (per policy, FR-006) → parse manifest → rule-count ceiling → epoch match (FR-005a) → sequence/fingerprint anti-downgrade (FR-009) → version compatibility → compile-all-or-reject → runnable-rules > 0 → activation self-check (Spec 087 FR-008) → atomic activation. FR-011's ceilings MUST therefore be split across the pipeline rather than applied as one stage: the sidecar cap comes first because the sidecar is read before any signature exists to check, bundle byte size is measurable on raw bytes and so guards the verification step itself, while rule count is only knowable after parsing and so cannot precede it. The same reasoning applies to sequence, a manifest field necessarily read after parsing: what MUST precede parsing is signature verification, not the sequence or rule-count check — an unverified candidate's manifest is never trusted for any decision. Any failure at any stage MUST keep last-known-good and record a machine-readable reason and timestamp (surfaced via the existing bundle status), never an empty or partial rule set. -- **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. +- **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. Both "ever" clauses — the highest-ever watermark and the once-signed-always-signed ratchet — are claims about RETAINED data-directory state, not absolutes that survive its destruction: after a data-directory reset only the FR-009b seeded and config-resident baselines still hold, the ratchet re-arms on the next signed activation, and the accepted post-reset guarantees are exactly those of the "Anti-downgrade state lost" edge case — fixtures for FR-009 and SC-002 MUST test against those, not against a stronger guarantee the threat model disclaims. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything — with exactly one exception, carved out at the end of this requirement: an operator authority configured without a baseline, which has no seed available and is therefore run as explicitly degraded rather than as a silent zero. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This fallback is one of SC-002's two explicit exemptions; the other is FR-009b's baseline-less operator authority. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. **Pin and deny-list state MUST be scoped like the watermark it excepts**: each entry is keyed by (authority, trust epoch, sequence, fingerprint) and binds only while its authority is the active one (FR-004b) and its epoch equals the binary's current epoch for that authority. Switching the active authority SUSPENDS (does not delete) the non-active authority's pin and deny-list, so switching back restores them unchanged. A binary release that bumps an authority's trust epoch (FR-005a) RETIRES that authority's prior-epoch pins and deny-list entries, loudly logged: the epoch bump is the key-compromise recovery path and MUST NOT be blockable by stale pin state — a retained old-epoch pin could not re-activate its artifact anyway (the pinned manifest's epoch no longer matches, FR-005a), so honoring it could only strand the install on the embedded fallback while blocking the very recovery bundle the bump exists to deliver. @@ -158,11 +158,11 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **Signature Database (bundle + sidecar)**: the publishable unit — the deterministic compiled `scanner-bundle.json` plus a detached signature over its exact bytes. Identified by (bundle_version, sequence, fingerprint). - **Bundle Manifest metadata**: `bundle_version`/`schema_version` (existing), plus additive `generated_at`, `sequence`, publisher key id. -- **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. Each authority carries its own **trust epoch** (FR-005a), a monotonic integer that only a binary release can advance and that a candidate's signed manifest must match exactly. +- **Publisher Trust Anchor**: the set of publisher public keys embedded in a binary release, unioned with operator-configured keys; the only roots of trust for verification, never network-delivered. Each authority carries its own **trust epoch** (FR-005a), a monotonic integer advanced only through that authority's trust root (a binary release for the publisher, a config edit for the operator) and that a candidate's signed manifest must match exactly. - **Signing Authority**: the grouping anti-downgrade state is keyed by (FR-004a) — one publisher authority spanning all publisher keys including those mid-rotation, and a separate operator authority for operator-configured keys. Each holds its own watermark so neither can lock the other out. Exactly one authority is **active** at a time, declared in config (FR-004b, default publisher); artifacts of a non-active authority are refused, so per-authority isolation cannot be turned into an authority-swap by whoever can write the bundle path. - **Signature (TPA record)**: one cataloged campaign/technique — `TPA-YYYY-NNNN` id, detectors, category, level, confidence, tier intent (gating vs non-gating), provenance reference, license. - **Eval Sample Pair**: the labeled gated-malicious sample(s) plus category-matched hard-negative(s) a gating signature must contribute to the frozen eval dataset. -- **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. Never empty for any authority that has a baseline — it is seeded from a trusted one (the embedded bundle's sequence, for the publisher authority) per FR-009b, so "no state yet" is not an accept-anything state. The single exception is an operator authority configured without a baseline, which FR-009b permits precisely because no binary-trusted seed exists for it; there the install carries no anti-downgrade guarantee and is marked degraded while such a bundle is active rather than being presented as fully trusted. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving in a binary release. +- **Sequence Watermark**: the persisted highest-activated sequence, tracked per signing authority and scoped by trust epoch (FR-004a, FR-005a); the anti-downgrade/anti-replay state. Never empty for any authority that has a baseline — it is seeded from a trusted one (the embedded bundle's sequence, for the publisher authority) per FR-009b, so "no state yet" is not an accept-anything state. The single exception is an operator authority configured without a baseline, which FR-009b permits precisely because no binary-trusted seed exists for it; there the install carries no anti-downgrade guarantee and is marked degraded while such a bundle is active rather than being presented as fully trusted. It is MONOTONIC — rollback does not lower it (FR-010 expresses rollback as a pinned exception plus a deny-list instead), and the only thing that re-baselines it is a higher trust epoch arriving through the authority's trust root (a binary release for the publisher authority, a config edit for the operator authority — FR-005a). - **Publication Channel**: the stable public location of versioned signed artifact pairs with a "latest" reference; consumed by the Spec 087 fetch or by manual download. - **Post-Activation Re-scan**: the one-shot, off-hot-path evaluation of cached approved tool metadata against a newly-activated database, emitting review findings only. - **Active Database / Last-Known-Good**: unchanged from Spec 086/087 — the serving bundle and the fail-closed fallback target. @@ -172,7 +172,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Measurable Outcomes - **SC-001**: 100% of tampered candidates are rejected before parsing, with the last-known-good database still serving and the reason surfaced — demonstrated by fixture in CI. "Tampered" is defined precisely, because the signature covers the bundle bytes and NOT the sidecar's own encoding: any altered bundle byte, any altered signature value, any altered sidecar identity field (algorithm or key id, whose agreement with the manifest FR-003 already requires), and any mismatched bundle/sidecar pair. Byte changes to the sidecar that alter neither the signature value nor an identity field (whitespace, key order) are explicitly NOT required to be rejected — a detached signature cannot detect them — so the criterion is stated in terms of the fields that carry meaning rather than an unachievable "every byte of the sidecar". -- **SC-002**: A validly-signed bundle whose sequence is below its OWN signing authority's watermark for the current trust epoch is never activated by any non-rollback path (drop, fetch, restart) — "lower-sequence" is always a same-authority, same-epoch comparison per FR-004a/FR-005a, so a deliberate FR-004b authority switch followed by the newly-active authority's bundle clearing its own baseline is by definition not a downgrade and needs no exemption — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet — demonstrated by fixture in CI. There are exactly two exemptions, and both are degraded-coverage states rather than silent ones: the FR-009a embedded-default availability fallback, and the baseline-less operator authority of FR-009b (SC-002a fixture (c)), which has no seed to compare against and so carries no anti-downgrade guarantee to claim. Neither counts as a downgrade; both MUST surface as degraded. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. +- **SC-002**: A validly-signed bundle whose sequence is below its OWN signing authority's watermark for the current trust epoch is never activated by any non-rollback path (drop, fetch, restart) — "lower-sequence" is always a same-authority, same-epoch comparison per FR-004a/FR-005a, so a deliberate FR-004b authority switch followed by the newly-active authority's bundle clearing its own baseline is by definition not a downgrade and needs no exemption — including on a fresh install or after a data-directory wipe, where the FR-009b seeded baseline stands in for the absent persisted watermark — and an unsigned candidate never advances the watermark nor survives the signature ratchet (the ratchet claim holds while data-directory state is retained; after a wipe it re-arms on the next signed activation, per FR-009) — demonstrated by fixture in CI. There are exactly two exemptions, and both are degraded-coverage states rather than silent ones: the FR-009a embedded-default availability fallback, and the baseline-less operator authority of FR-009b (SC-002a fixture (c)), which has no seed to compare against and so carries no anti-downgrade guarantee to claim. Neither counts as a downgrade; both MUST surface as degraded. An explicit rollback succeeds, leaves an audit trail, does not lower the historical watermark, activates only the fingerprint the operator named (a substituted file at the same path is refused), and the deny-listed release it rolled away from is not re-activated by the following refresh cycle. - **SC-002a**: The watermark cannot be weaponized into a permanent update lockout, demonstrated by the following fixtures: (a) a bundle signed by a to-be-revoked key at the MAXIMUM representable sequence, after which a binary carrying a bumped trust epoch and the surviving key successfully activates a new release — proving key-compromise recovery is possible rather than merely documented; (a2) immediately after that bump, an OLD pre-bump bundle carrying the previous epoch and signed by the still-trusted surviving key is REFUSED as stale-epoch replay rather than clearing the re-baselined watermark — the epoch bump must not itself become a downgrade window; and (b) with the operator authority declared active (FR-004b), an operator-authority-signed bundle at a high sequence that leaves the publisher authority's watermark untouched, so after the declaration is switched back the next official publisher release still activates — proving the two signing authorities cannot lock each other out. A fixture (b2) covers the declaration itself: the same operator-signed bundle dropped while the PUBLISHER authority is declared active is refused as "authority not active" with the publisher bundle still serving, so the choice of authority cannot be made by whoever writes the bundle path. A third fixture (c) covers the operator authority after a data-directory wipe: with a config-resident baseline declared, an older operator-signed bundle is refused; with no baseline declared, it may activate but the install reports degraded coverage rather than full trust. - **SC-003**: An air-gapped install (zero network, ever) reaches full P1 capability: verified activation of a file-dropped signed database, downgrade refusal, and rollback, with behavior byte-identical to a connected install's file-drop path. - **SC-004**: The shipped seed corpus contains at least 25 signatures spanning at least 8 distinct public campaign/technique classes, every one carrying provenance + redistributable license, and every gating signature backed by its eval sample pair; the canonical fixture of each class produces a hard-tier finding naming its TPA id. From 2e0ceee89e02ad9562f4062ed2dbf25b57986ab6 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 26 Aug 2026 16:13:14 +0300 Subject: [PATCH 16/20] docs(specs): close cross-model review gaps in 101-tpa-db MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode/gpt-5.6) against the shipped 086/087 code found seven defects that survived verification against the repo: - Threat model called config.db "integrity-protected". It is BBolt: transactional and checksummed, not authenticated. Restated as a trust boundary and required SUBSTITUTION fixtures (prepared db with chosen watermark / cleared ratchet / forged pin), not just deletion/reset. - FR-010 authorized a watermark-bypassing rollback with the fingerprint "offered by bundle status history". That fingerprint is hex(sha256)[:12] — 48 bits — and no activation history exists (BundleInfo reports only the live artifact). Now: full SHA-256, compared byte-for-byte, plus a durable append-only activation record specified as new work. - FR-011 required candidates to "be REGULAR files" and to read under an "enforceable I/O deadline". A pathname stat is a substitution window, and Go's SetReadDeadline is unsupported on regular files (ErrNoDeadline) — the rule was unimplementable as written. Now: O_NOFOLLOW open + fstat on the descriptor, and a bounded budget enforced outside the read. - FR-001 ("the existing compiled bundle") and FR-012 ("each signature MUST carry provenance and a redistributable license") could not both hold: rules[] has no such fields. Added FR-001a — an additive signatures[] section keyed by TPA id, many-to-one to rules, with referential-integrity validation. - FR-001's "byte-identical rebuilds for a given corpus" contradicted FR-002 binding release metadata into the signed bytes. Determinism now scoped to (corpus, manifest metadata) with pinned inputs. - sequence had no numeric domain while SC-002a relies on a maximum. Now a canonical uint64 as a JSON decimal string, with explicit rejection rules. - FR-003 left the sidecar wire format undefined although it is the one artifact parsed before any signature is verified. Design must fix schema, encodings, required fields and rejection behaviour; Ed25519-only in v1. SC-010..SC-012 added so each new requirement is testable. --- specs/101-tpa-db/spec.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index 14047ac1c..f37cee381 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -93,7 +93,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), ### Edge Cases - **Air-gapped host, forever**: every P1 behavior (verification, sequence check, rollback, activation self-check) is network-free; the trust anchors ship inside the binary; staleness is surfaced but never blocks scanning — an old database is degraded coverage, not an outage. -- **Threat-model boundary — attacker with config write**: FR-004 lets operator-configured keys join the trust set, so anyone who can write `mcp_config.json` can trust their own key and sign anything. This is explicitly OUT of the threat model and is not a regression: an attacker with config write can already disable quarantine outright or set a server's trust mode to `auto`. The attacker this design defends against is the one who can write the BUNDLE PATH or intercept the FETCH but not the config — which is precisely why the signature ratchet (FR-009) is keyed to install state rather than to the bundle path, why the active-authority declaration (FR-004b) and the operator baseline (FR-009b) are config-resident, and why `require_signed_bundle` is config-resident (none of them must be defeatable from the data dir alone). The boundary MUST be read conservatively: because the bundle path may be configured to live inside the data directory, the design MUST NOT assume the bundle-path attacker cannot also delete or reset the data-directory persisted security state (watermark record, ratchet flag, pin/deny-list) — the worst such an attacker can achieve is exactly the "anti-downgrade state lost" wipe below, against which the guarantees that survive are the ones anchored outside the data dir: the FR-009b publisher baseline (re-seeded from the binary at every startup), the config-resident operator baseline, `require_signed_bundle`, and the FR-004b active-authority declaration. The ratchet and the pin/deny-list are data-dir resident and are therefore explicitly best-effort hardening against this attacker, never the sole defense. What the design DOES assume is that the attacker cannot substitute or corrupt the persisted state with CHOSEN values (e.g. rewrite the watermark to a lower number without triggering the seeded-baseline floor): the state store is the same integrity-protected local database that already holds approvals and quarantine state, and an attacker who can write arbitrary records into it is already past every guarantee mcpproxy makes, not just this feature's. +- **Threat-model boundary — attacker with config write**: FR-004 lets operator-configured keys join the trust set, so anyone who can write `mcp_config.json` can trust their own key and sign anything. This is explicitly OUT of the threat model and is not a regression: an attacker with config write can already disable quarantine outright or set a server's trust mode to `auto`. The attacker this design defends against is the one who can write the BUNDLE PATH or intercept the FETCH but not the config — which is precisely why the signature ratchet (FR-009) is keyed to install state rather than to the bundle path, why the active-authority declaration (FR-004b) and the operator baseline (FR-009b) are config-resident, and why `require_signed_bundle` is config-resident (none of them must be defeatable from the data dir alone). The boundary MUST be read conservatively: because the bundle path may be configured to live inside the data directory, the design MUST NOT assume the bundle-path attacker cannot also delete or reset the data-directory persisted security state (watermark record, ratchet flag, pin/deny-list) — the worst such an attacker can achieve is exactly the "anti-downgrade state lost" wipe below, against which the guarantees that survive are the ones anchored outside the data dir: the FR-009b publisher baseline (re-seeded from the binary at every startup), the config-resident operator baseline, `require_signed_bundle`, and the FR-004b active-authority declaration. The ratchet and the pin/deny-list are data-dir resident and are therefore explicitly best-effort hardening against this attacker, never the sole defense. What the design DOES assume is that the attacker cannot substitute or corrupt the persisted state with CHOSEN values (e.g. rewrite the watermark to a lower number without triggering the seeded-baseline floor): the state store is the same local BBolt database (`config.db`) that already holds approvals and quarantine state. That database is transactionally consistent and checksummed against corruption, NOT cryptographically authenticated: nothing in it resists an attacker who can substitute the file with a prepared one. The assumption is therefore a TRUST BOUNDARY, not a cryptographic property — an attacker who can write chosen records into `config.db` is already past every guarantee mcpproxy makes (approvals, tool-quarantine baselines, OAuth state), not just this feature's. The design MUST NOT be read as claiming the ratchet, watermark, pin or deny-list survive that attacker; the guarantees that do survive are only the ones anchored outside the data directory, enumerated above. Conformance fixtures MUST therefore cover state SUBSTITUTION (a prepared database carrying a chosen watermark, cleared ratchet, or forged pin/deny-list entry) and assert the outside-the-data-dir floors still hold — deletion and reset fixtures alone do not exercise this boundary. - **Signature key rotation**: the trust anchor is a SET of publisher keys embedded per binary release. Rotation is a binary-release event: a release ships old+new keys, bundles are signed with the new key, the old key is removed in a later release. Keys are NEVER delivered over the network, and rotation is not done casually — every shipped binary pins its key set, so a hasty rotation strands older installs on unsigned-refresh only (same philosophy as the app-update signing keys). Key compromise → remove the key in the next release, **bump the trust epoch (FR-005a)**, and publish a bundle in the new epoch signed by the surviving key. The epoch bump is what makes recovery possible at all: a compromised key can sign at the maximum representable sequence, and against a monotonic watermark no surviving key could ever outbid that within the same epoch, so revocation alone would leave installs permanently unable to accept any future release. - **Bundle signed by a formerly-trusted, now-removed key**: verification fails on binaries that dropped the key (fail-closed to last-known-good); older binaries still trusting it keep working — the downgrade watermark still blocks sequence replay. - **Tampered bundle / tampered sidecar / swapped pair** (valid signature belonging to a different bundle's bytes): all reduce to signature-verification failure before parse; never a crash, never partial load. @@ -111,9 +111,10 @@ The publisher cuts a new database release (new signature for a fresh campaign), **Database format & signing** -- **FR-001**: The publishable database artifact MUST be the existing compiled bundle (`scanner-bundle.json`, Scanner Bundle Contract) plus a detached signature sidecar over the exact bundle bytes. Bundle bytes MUST remain deterministic for a given corpus (byte-identical rebuilds), so the signature and the fingerprint are stable. -- **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing integer publish counter), a publisher key identifier, and the **trust epoch** the artifact was published under (FR-005a). Because these keys live inside the signed bytes, the epoch is cryptographically bound to the bundle and cannot be reinterpreted after the fact. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). `generated_at` MUST be derived from deterministic release metadata (the release commit's timestamp or `SOURCE_DATE_EPOCH`), NEVER from build wall-clock time — a wall-clock stamp would change the bundle bytes, fingerprint, and signature on every rebuild of an identical corpus and so would directly violate FR-001's byte-identical-rebuild guarantee. -- **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. A key id MUST be a canonical full-length fingerprint of the public key itself (not an operator-chosen label), the trust set MUST reject duplicate ids at build/config time, and verification MUST succeed only when the manifest key id, the sidecar key id, and the id of the trust-set key whose signature actually verified are all identical — a manifest naming one publisher while the bytes are signed by another MUST be a verification failure, not a cosmetic mismatch. +- **FR-001**: The publishable database artifact MUST be the existing compiled bundle (`scanner-bundle.json`, Scanner Bundle Contract) plus a detached signature sidecar over the exact bundle bytes. Bundle bytes MUST be deterministic for a given **(corpus, manifest metadata)** pair — rebuilding the same corpus with the same manifest inputs MUST be byte-identical, so the signature and the fingerprint are stable. Determinism is deliberately NOT claimed over the corpus alone: FR-002 binds release-identifying metadata (`generated_at`, `sequence`, key id, trust epoch) into the signed bytes, so two releases of an unchanged corpus legitimately differ in bytes, fingerprint and signature. The reproducibility check MUST therefore pin the manifest inputs (a fixed `SOURCE_DATE_EPOCH` and the release's sequence/key id/epoch) and assert byte-equality under those pinned inputs; asserting byte-equality across two different releases of the same corpus would fail by construction and MUST NOT be a gate. +- **FR-001a**: The bundle contract MUST gain a defined home for the per-signature metadata FR-012 requires. Today's compiled bundle carries `rules[]` entries keyed `category, confidence, detector, engine, flags, id, indicators, level, pattern, severity, target, type` and a `skipped[]` list — there is no provenance field and no license field anywhere in the artifact, so FR-001 ("the existing compiled bundle") and FR-012 ("each signature MUST carry provenance … and a redistributable license") cannot both hold as written. Design MUST resolve this by adding a `signatures[]` section keyed by the stable `TPA-YYYY-NNNN` id, carrying provenance, license, gating intent, and the ids of the `rules[]` entries that implement it — additive, so a v0.1 loader ignores it (contract §4 forward-compat). The relationship MUST be explicitly many-to-one (several rules may implement one TPA id) and the build MUST enforce referential integrity in both directions: every `signatures[]` entry references at least one existing rule id, every gating rule is referenced by exactly one signature entry, and TPA ids are unique. A bundle failing either check MUST be rejected whole, like any other contract violation. +- **FR-002**: The bundle manifest MUST gain additive metadata within the supported schema line: `generated_at` (RFC3339 build stamp — the loader already surfaces it), `sequence` (a strictly-increasing publish counter, canonically a `uint64` serialized as a JSON **decimal string** — never a bare JSON number, whose double-precision round-trip in generic consumers silently corrupts values above 2^53-1 and admits exponent notation; the loader MUST reject a value that is not a bare decimal string of 1–20 digits with no sign, leading zeros, whitespace or exponent, and MUST reject anything above 2^64-1, which also fixes the "MAXIMUM representable sequence" SC-002a exercises), a publisher key identifier, and the **trust epoch** the artifact was published under (FR-005a). Because these keys live inside the signed bytes, the epoch is cryptographically bound to the bundle and cannot be reinterpreted after the fact. These keys MUST be additive so existing v0.1 loaders ignore them (contract §4 forward-compat preserved). `generated_at` MUST be derived from deterministic release metadata (the release commit's timestamp or `SOURCE_DATE_EPOCH`), NEVER from build wall-clock time — a wall-clock stamp would change the bundle bytes, fingerprint, and signature on every rebuild of the SAME release, which is exactly the byte-equality FR-001 pins its reproducibility check to. (Two different releases of an unchanged corpus differing in bytes is expected and permitted; see FR-001.) +- **FR-003**: The signature scheme MUST be Ed25519 detached signatures (or an equivalent modern EdDSA scheme selected in design); the sidecar MUST carry the algorithm identifier and signing key id so the scheme can evolve without breaking verification of existing artifacts. **The sidecar wire format MUST be specified in design, not left to the implementation**, because it is the one artifact an independent publisher must be able to produce and every loader must reject identically: design MUST fix its own schema version key, the exact encodings of the algorithm identifier, key id and signature value, the complete set of required fields, and the deterministic rejection behaviour for a missing field, an unknown or duplicate field, a duplicate key, an unknown algorithm, and trailing bytes. v1 MUST verify Ed25519 and nothing else — an unknown algorithm identifier is a hard rejection, never a fallback or a skip — and the spec MUST ship byte-level valid and invalid sidecar fixtures so two implementations can be shown to agree on both acceptance and refusal. A key id MUST be a canonical full-length fingerprint of the public key itself (not an operator-chosen label), the trust set MUST reject duplicate ids at build/config time, and verification MUST succeed only when the manifest key id, the sidecar key id, and the id of the trust-set key whose signature actually verified are all identical — a manifest naming one publisher while the bytes are signed by another MUST be a verification failure, not a cosmetic mismatch. - **FR-004**: The set of trusted verification keys MUST be the union of (a) publisher public keys embedded in the binary at build time and (b) operator-configured keys (the Spec 087 FR-013 key). Keys MUST NOT be retrieved over the network, and verification MUST work fully offline. These are two distinct **signing authorities**, and the design MUST keep them so: all publisher keys (including keys being rotated in or out, FR-005) belong to ONE publisher authority, while operator-configured keys form a separate operator authority. - **FR-004a**: Anti-downgrade state MUST be tracked per signing authority, not in one global namespace. An operator maintaining a self-built corpus is an explicitly supported persona (air-gapped installs, private signature sets), and with a single shared counter their locally-signed bundle at sequence N would raise the one global watermark above the publisher's release line and permanently block official updates — or, symmetrically, a publisher release would block the operator's own corpus. Each authority MUST therefore carry its own watermark, and a candidate MUST be compared only against the watermark of the authority whose key verified it. Choosing an operator-signed corpus over a higher-sequence publisher one is a deliberate operator decision (declared per FR-004b) and MUST NOT by itself be treated as a downgrade or as degraded coverage — the operator configured that key precisely to assert their own corpus — but the active authority MUST be visible in bundle status (FR-020) so the choice is never invisible. Grouping all publisher keys into one authority is what keeps per-authority watermarks compatible with key rotation: a rotated-in key inherits the publisher authority's watermark rather than starting a fresh one. - **FR-004b**: The **active signing authority MUST be a config-resident declaration**, defaulting to the publisher authority, and MUST NOT be inferred solely from whichever key happens to verify a candidate. A candidate whose verifying key belongs to a non-active authority MUST be refused with an "authority not active" reason, exactly as any other failed candidate (last-known-good keeps serving). Without this, per-authority watermarks (FR-004a) open the very hole they were introduced to avoid: because each authority is compared only against its own counter, an attacker who can write the bundle path — squarely inside the threat model, unlike the config-write attacker — can replace a current publisher bundle at sequence 100 with ANY operator-signed artifact that merely clears the operator authority's FR-009b baseline, silently swapping the install onto an older or deliberately narrower corpus while it still reports `signature_verified=true` at FULL coverage. The publisher watermark cannot detect this, since the comparison never crosses namespaces, and no operator decision is involved — whoever writes the path picks the authority. Config-residency is what puts that choice out of the attacker's reach, matching `require_signed_bundle` (FR-007) and the operator baseline (FR-009b), which are config-resident for the same reason. Both key sets MAY remain configured simultaneously; only artifacts of the declared active authority activate, and switching authorities is a config edit, not a file drop. **The declaration governs the SERVING bundle, not only future candidates**: when the declaration changes, an active external bundle belonging to the now-inactive authority MUST NOT keep serving as full coverage — it is deactivated (loudly logged, surfaced per FR-020) and the install falls back to the embedded default under FR-009a's rules until a bundle of the newly-active authority activates. Without this the switch is advisory only: fail-safe keep-last-known-good would leave the old authority's corpus serving indefinitely, and an attacker who keeps the new authority's candidates invalid (bundle-path writes are in scope) could perpetuate that state at full coverage — the operator's explicit change of trust would simply never take effect. Merely declaring the operator authority active does not by itself degrade coverage: until an operator bundle activates, the embedded default serves exactly as on a fresh install (FR-009a condition (b) concerns a previously-activated operator bundle that was lost, not one that never existed). @@ -128,8 +129,8 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-009**: The system MUST persist the highest sequence ever activated — the **watermark**, scoped per signing authority and trust epoch per FR-004a/FR-005a — and refuse any candidate with a lower sequence, or an equal sequence with a different content fingerprint, as a downgrade/replay, unless an explicit operator rollback (FR-010) is in effect. **The watermark is a property of SIGNED artifacts only**: `sequence` lives in the manifest, so an unsigned candidate's sequence is attacker-chosen. An unsigned/unverified candidate MUST NOT advance the watermark (otherwise a single unsigned drop claiming a huge sequence permanently locks out every legitimate signed release), and once an install has ever activated a signature-verified bundle, unsigned candidates MUST be refused thereafter regardless of `require_signed_bundle` — a signature ratchet, so signing cannot be stripped back off an install by whoever can write the bundle path. Both "ever" clauses — the highest-ever watermark and the once-signed-always-signed ratchet — are claims about RETAINED data-directory state, not absolutes that survive its destruction: after a data-directory reset only the FR-009b seeded and config-resident baselines still hold, the ratchet re-arms on the next signed activation, and the accepted post-reset guarantees are exactly those of the "Anti-downgrade state lost" edge case — fixtures for FR-009 and SC-002 MUST test against those, not against a stronger guarantee the threat model disclaims. The embedded default is exempt from BOTH the watermark and the ratchet — it is the availability fallback of last resort and must never be locked out (scanning must never stop), which is exactly why FR-009a demotes it to degraded coverage instead of refusing it. - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything — with exactly one exception, carved out at the end of this requirement: an operator authority configured without a baseline, which has no seed available and is therefore run as explicitly degraded rather than as a silent zero. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This fallback is one of SC-002's two explicit exemptions; the other is FR-009b's baseline-less operator authority. -- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint (as offered by bundle status history), the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. **Pin and deny-list state MUST be scoped like the watermark it excepts**: each entry is keyed by (authority, trust epoch, sequence, fingerprint) and binds only while its authority is the active one (FR-004b) and its epoch equals the binary's current epoch for that authority. Switching the active authority SUSPENDS (does not delete) the non-active authority's pin and deny-list, so switching back restores them unchanged. A binary release that bumps an authority's trust epoch (FR-005a) RETIRES that authority's prior-epoch pins and deny-list entries, loudly logged: the epoch bump is the key-compromise recovery path and MUST NOT be blockable by stale pin state — a retained old-epoch pin could not re-activate its artifact anyway (the pinned manifest's epoch no longer matches, FR-005a), so honoring it could only strand the install on the embedded fallback while blocking the very recovery bundle the bump exists to deliver. -- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be required to be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read — and every candidate read (local files and fetch response bodies alike) MUST run under an enforceable I/O deadline whose expiry releases the single-flight slot and records a rejection reason, so no candidate can pin the refresh path. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). +- **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint, the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. **The rollback fingerprint MUST be the complete SHA-256 digest of the bundle bytes, compared byte-for-byte.** The abbreviated fingerprint the loader surfaces today is the first 12 hex characters of that digest (`internal/security/scanner/tpa_bundle_source.go` — `hex.EncodeToString(sum[:])[:12]`), i.e. 48 bits: adequate as a human-comparable identity label, but NOT as the authorization for the one path that deliberately bypasses the downgrade watermark, where the attacker chooses the substituted bytes and need only match a 48-bit prefix. Bundle status MUST therefore surface the full digest alongside the abbreviation, and the rollback interface MUST accept only the full digest. **Bundle status history is a new requirement, not an existing surface**: no activation history exists today (`BundleInfo` reports only the currently active artifact), so this spec MUST add a durable, append-only activation record — (authority, trust epoch, sequence, full SHA-256 digest, generated_at, activated_at, source) per activation, retained across restarts — because without it an operator has no trustworthy way to learn the digest of the known-good artifact they intend to revert to. That record is data-dir resident and therefore carries the same best-effort caveat as the ratchet: it makes a legitimate rollback possible, it does not by itself authorize one. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. **Pin and deny-list state MUST be scoped like the watermark it excepts**: each entry is keyed by (authority, trust epoch, sequence, fingerprint) and binds only while its authority is the active one (FR-004b) and its epoch equals the binary's current epoch for that authority. Switching the active authority SUSPENDS (does not delete) the non-active authority's pin and deny-list, so switching back restores them unchanged. A binary release that bumps an authority's trust epoch (FR-005a) RETIRES that authority's prior-epoch pins and deny-list entries, loudly logged: the epoch bump is the key-compromise recovery path and MUST NOT be blockable by stale pin state — a retained old-epoch pin could not re-activate its artifact anyway (the pinned manifest's epoch no longer matches, FR-005a), so honoring it could only strand the install on the embedded fallback while blocking the very recovery bundle the bump exists to deliver. +- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read. **This check MUST be made on the OPEN DESCRIPTOR, never on the pathname.** A `stat`-then-open sequence is itself a substitution window: the in-scope bundle-path attacker replaces the path between the two calls, the stat sees a regular file and the open lands on their FIFO. The loader MUST open the candidate with the platform's no-follow protection (`O_NOFOLLOW`; on Go, `os.OpenFile` with `syscall.O_NOFOLLOW`, or `os.Root`-scoped opens where the containing directory is itself trusted), `fstat` the resulting descriptor, refuse anything whose mode is not a regular file, and then read the size ceiling from that same descriptor — never re-resolving the path. **The time bound MUST NOT be specified as an I/O deadline on the file itself**: Go's `SetReadDeadline` is not supported for regular files and returns `ErrNoDeadline`, so a requirement phrased that way is unimplementable as written and would be silently dropped. The requirement is instead that every candidate read complete within a bounded wall-clock budget enforced OUTSIDE the read — a context-cancelled reader goroutine for local files (whose expiry releases the single-flight slot and records a rejection reason even if the blocked read itself cannot be interrupted), and `http.Client`/`context` deadlines plus a capped `io.LimitReader` for fetch response bodies. Design MUST name the concrete mechanism and its per-platform limits, including the case where a stalled network filesystem leaves the abandoned read goroutine parked: the slot is released and the refresh path stays available, which is the property being guaranteed. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. **Seed corpus** @@ -181,6 +182,9 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **SC-007**: A newly published signature reaches detection within one daily refresh cycle on a fetch-enabled install (and immediately on manual drop), measured end to end: publish → active → the new signature's fixture fires. - **SC-008**: With a database at the design load ceilings (≥ 500 regex rules), p95 scan latency over a 100-tool server changes by less than 10% versus the 6-signature baseline, and bundle load-plus-verify completes under one second on commodity hardware. - **SC-009**: After activating a database containing a new signature, an already-approved tool whose cached description matches it is surfaced for review within the same refresh cycle, with zero approvals auto-revoked and zero servers auto-quarantined by the refresh itself. +- **SC-010**: Rollback authorization is bound to the whole digest and to a real history: a fixture in which an attacker-substituted, publisher-signed, current-epoch bundle shares the operator-supplied abbreviated fingerprint (first 12 hex chars) but differs in the full SHA-256 is REFUSED as a rollback-target mismatch; and after N activations the activation record replays all N entries — (authority, epoch, sequence, full digest, generated_at, activated_at, source) — across a process restart, so the digest of a known-good artifact is recoverable without trusting the candidate file. +- **SC-011**: Candidate opening is race-free and time-bounded, demonstrated by fixtures that (a) swap the candidate path for a FIFO between path resolution and open and are refused by the descriptor-level `fstat` check rather than blocking, (b) point the path at a symlink to a FIFO and are refused without following it, and (c) leave a never-written FIFO in place and see the single-flight refresh slot released within the bounded budget with a recorded rejection reason, so a subsequent legitimate database update still activates. +- **SC-012**: The persisted-state trust boundary is exercised by SUBSTITUTION, not only deletion: fixtures replace `config.db` with a prepared database carrying (a) a lowered watermark, (b) a cleared signature ratchet, and (c) a forged pin/deny-list entry, and in every case the guarantees anchored outside the data directory still hold — the FR-009b publisher baseline re-seeded from the binary, the config-resident operator baseline, `require_signed_bundle`, and the FR-004b active-authority declaration — while the install reports itself degraded. No fixture may assert that the ratchet, watermark, pin or deny-list themselves survive; they are best-effort against this attacker by construction. ## Assumptions From a6f0d58587c3256d21be98f3e8ed18157f6ed6db Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 26 Aug 2026 16:37:28 +0300 Subject: [PATCH 17/20] docs(specs): put numbers on the 101-tpa-db load ceilings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FR-011 deferred every threshold to "design" — the bundle byte size, the rule count, the sidecar cap ("sized to its fixed small structure") and the I/O deadline. Each of those is a rejection boundary, and a rejection boundary with no number cannot be tested: no fixture can be built one byte or one rule over a value that does not exist yet, so the requirement would have shipped unfalsifiable. It is also not a tightening of something already there. Today the loader has no ceiling of any kind: tpa_bundle_source.go reads the candidate with a bare os.ReadFile(path) — no size check and no file-type check — and loadBundleCheck applies no rule-count limit. Numbers now in the FR: bundle <= 8 MiB, rules <= 2000 (SC-008 exercises >= 500, so four more corpus generations fit without a spec change), sidecar <= 4 KiB, read budget 5 s per candidate. At the ceiling loads, one over is rejected whole. SC-010a drives both sides of every boundary and asserts the last-known-good corpus keeps serving. --- specs/101-tpa-db/spec.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/101-tpa-db/spec.md b/specs/101-tpa-db/spec.md index f37cee381..a29eb53e5 100644 --- a/specs/101-tpa-db/spec.md +++ b/specs/101-tpa-db/spec.md @@ -130,7 +130,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **FR-009b**: Each authority's watermark MUST be SEEDED from a trusted baseline rather than starting empty, because an absent watermark accepts anything — with exactly one exception, carved out at the end of this requirement: an operator authority configured without a baseline, which has no seed available and is therefore run as explicitly degraded rather than as a silent zero. Otherwise the anti-downgrade guarantee simply does not exist at first use or after the data directory is reset: an attacker who can write the bundle path drops an OLD but genuinely publisher-signed, current-epoch bundle — say sequence 5 while the shipping release is at 100 — and with no watermark to compare against it activates as a fully signature-verified bundle at FULL coverage, silently reinstating every blind spot fixed between 5 and 100. Neither the FR-009 ratchet nor the FR-010 pin/deny-list helps here; both only bind once persisted state exists. Therefore: the **publisher** authority's watermark MUST be seeded from the sequence of the bundle embedded in the running binary — a candidate older than what the binary already ships is strictly worse than the fallback and MUST be refused — and this baseline MUST be re-applied on every startup, so wiping the data directory cannot lower it below the binary's own corpus. The **operator** authority has no binary-trusted baseline available, so its baseline MUST be declared explicitly in configuration alongside the operator key, and it MUST live in `mcp_config.json` rather than the data directory — that is what makes it durable across the wipe, since a baseline stored in resettable state provides no protection precisely when it is needed. A silent zero default MUST NOT be used: with a zero baseline the operator authority has NO anti-downgrade after state loss at all, and a bundle-path attacker — who is squarely inside the threat model, unlike the config-write attacker — could replay any older genuinely operator-signed bundle at full coverage on a fresh or wiped install. Therefore: if an operator key is configured WITHOUT a baseline, the system MUST treat operator-authority anti-downgrade as unsupported and mark coverage degraded (per FR-007) while an operator-signed bundle is active, rather than silently presenting it as fully trusted. The data-directory watermark then tracks activations above the configured baseline and may only raise the effective minimum, never lower it. The seeded baseline also makes the epoch re-baseline of FR-005a safe: after an epoch bump the new epoch's watermark starts at the authority's baseline, not at zero. - **FR-009a**: When no valid external candidate is available, the embedded default MUST still serve so scanning never stops, even if its sequence is below the watermark. Where the install has NO retained signed watermark and no missing previously-active external bundle (the ordinary fresh install), this is simply the normal full-coverage state. Two — and only two — conditions demote that fallback to degraded coverage (auto-approval suspended, as in FR-007's unsigned-external case): (a) the embedded default's sequence is below the **publisher** authority's retained watermark, or (b) the **operator** authority is the active one (FR-004b) and its previously-activated external bundle is no longer available or no longer valid. The comparison in (a) MUST be same-namespace: the embedded default is a publisher artifact, and sequences of different authorities are independent counters, so it MUST be compared ONLY against the publisher watermark and never against an operator watermark — a cross-authority numeric comparison is meaningless and would, for an operator corpus at sequence 900 against an embedded publisher corpus at 100, permanently suspend auto-approval on an install whose publisher fallback is perfectly current. Condition (b) is what a cross-authority comparison was reaching for, restricted to the case where the fallback cannot be shown to be at least as fresh as what was lost: an operator corpus is not comparable to the publisher's, so a missing one leaves the install with no evidence the embedded default covers it. It MUST NOT extend to a missing PUBLISHER external bundle, because there condition (a) already decides the question on the same axis — an embedded default at or above the publisher watermark is demonstrably no older than the bundle that went missing (the ordinary "external file removed after a binary upgrade shipped a newer corpus" case) and MUST stay full coverage rather than being degraded merely because an older external file once existed. In either case the retained watermark, the active authority, and the degraded reason MUST be surfaced, and it MUST NOT count as an anti-downgrade violation. This fallback is one of SC-002's two explicit exemptions; the other is FR-009b's baseline-less operator authority. - **FR-010**: An explicit rollback action MUST let an operator revert to the embedded default or a named bundle file: the target is re-validated (signature policy and self-check still apply), activated despite a lower sequence, and the override is loudly logged and visible in bundle status. **The rollback target MUST be identified by content fingerprint, not by pathname alone.** Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path hands the adversary the choice of what the bypass lands on: with a legitimate bundle at sequence 100 active and the operator intending to revert to a known-good sequence 90, an attacker who can write the bundle path — in scope, unlike the config-write attacker — swaps the named file for a genuinely publisher-signed, current-epoch bundle at sequence 5 before the rollback reads it. It passes signature, epoch, compilation, and plausibly the self-check; the watermark that would normally refuse it is switched off by the rollback itself; and the result is then PINNED, so the downgrade persists across refresh cycles. FR-006's read-once rule closes the verify/parse race but not substitution before the read. Therefore the operator MUST supply the expected fingerprint, the path is only a hint for where to find those bytes, and a loaded artifact whose fingerprint differs MUST be refused as a rollback-target mismatch rather than activated. **The rollback fingerprint MUST be the complete SHA-256 digest of the bundle bytes, compared byte-for-byte.** The abbreviated fingerprint the loader surfaces today is the first 12 hex characters of that digest (`internal/security/scanner/tpa_bundle_source.go` — `hex.EncodeToString(sum[:])[:12]`), i.e. 48 bits: adequate as a human-comparable identity label, but NOT as the authorization for the one path that deliberately bypasses the downgrade watermark, where the attacker chooses the substituted bytes and need only match a 48-bit prefix. Bundle status MUST therefore surface the full digest alongside the abbreviation, and the rollback interface MUST accept only the full digest. **Bundle status history is a new requirement, not an existing surface**: no activation history exists today (`BundleInfo` reports only the currently active artifact), so this spec MUST add a durable, append-only activation record — (authority, trust epoch, sequence, full SHA-256 digest, generated_at, activated_at, source) per activation, retained across restarts — because without it an operator has no trustworthy way to learn the digest of the known-good artifact they intend to revert to. That record is data-dir resident and therefore carries the same best-effort caveat as the ratchet: it makes a legitimate rollback possible, it does not by itself authorize one. A rollback target MUST additionally still be at or above **its own authority's** FR-009b baseline — the publisher baseline for a publisher-signed target, the config-declared operator baseline for an operator-signed one. The comparison MUST NOT be made against the publisher baseline in both cases: sequences of different authorities are independent counters (FR-004a), so measuring an operator target at sequence 9 against a publisher embedded corpus at 100 would refuse every legitimate operator rollback. Where an authority has NO declared baseline — possible only for the operator authority, which FR-009b explicitly permits to run without one — there is no floor to enforce and none is invented: the rollback proceeds and the install is marked degraded for exactly as long as that operator-signed bundle is active, which is already FR-009b's treatment of a baseline-less operator authority. Neither refusing the rollback (which would break FR-010's guarantee for a configuration FR-009b allows) nor assuming a zero floor (which would claim an anti-downgrade guarantee that does not exist) is acceptable. The embedded default is the one target always permitted, since it IS the publisher baseline and is the availability fallback of last resort. Rollback MUST NOT lower the historical high-watermark: the watermark is monotonic and normal activation MUST still require exceeding it. Rollback is instead expressed as a **pinned exception** — the operator pins the rolled-back artifact by fingerprint, and the (sequence, fingerprint) being rolled away from is deny-listed so the very release the operator just rejected cannot be re-activated by the next refresh cycle (and neither can any intermediate release between the pin and the watermark). Clearing the pin returns the install to normal watermark-governed activation. **Pin and deny-list state MUST be scoped like the watermark it excepts**: each entry is keyed by (authority, trust epoch, sequence, fingerprint) and binds only while its authority is the active one (FR-004b) and its epoch equals the binary's current epoch for that authority. Switching the active authority SUSPENDS (does not delete) the non-active authority's pin and deny-list, so switching back restores them unchanged. A binary release that bumps an authority's trust epoch (FR-005a) RETIRES that authority's prior-epoch pins and deny-list entries, loudly logged: the epoch bump is the key-compromise recovery path and MUST NOT be blockable by stale pin state — a retained old-epoch pin could not re-activate its artifact anyway (the pinned manifest's epoch no longer matches, FR-005a), so honoring it could only strand the install on the embedded fallback while blocking the very recovery bundle the bump exists to deliver. -- **FR-011**: The loader MUST enforce load ceilings (maximum bundle byte size and maximum rule count, fixed in design) and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read. **This check MUST be made on the OPEN DESCRIPTOR, never on the pathname.** A `stat`-then-open sequence is itself a substitution window: the in-scope bundle-path attacker replaces the path between the two calls, the stat sees a regular file and the open lands on their FIFO. The loader MUST open the candidate with the platform's no-follow protection (`O_NOFOLLOW`; on Go, `os.OpenFile` with `syscall.O_NOFOLLOW`, or `os.Root`-scoped opens where the containing directory is itself trusted), `fstat` the resulting descriptor, refuse anything whose mode is not a regular file, and then read the size ceiling from that same descriptor — never re-resolving the path. **The time bound MUST NOT be specified as an I/O deadline on the file itself**: Go's `SetReadDeadline` is not supported for regular files and returns `ErrNoDeadline`, so a requirement phrased that way is unimplementable as written and would be silently dropped. The requirement is instead that every candidate read complete within a bounded wall-clock budget enforced OUTSIDE the read — a context-cancelled reader goroutine for local files (whose expiry releases the single-flight slot and records a rejection reason even if the blocked read itself cannot be interrupted), and `http.Client`/`context` deadlines plus a capped `io.LimitReader` for fetch response bodies. Design MUST name the concrete mechanism and its per-platform limits, including the case where a stalled network filesystem leaves the abandoned read goroutine parked: the slot is released and the refresh path stays available, which is the property being guaranteed. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). +- **FR-011**: The loader MUST enforce load ceilings and reject a candidate exceeding them as a whole; all patterns remain RE2-only. **The ceilings are numbers in this spec, not a design deferral**, because every one of them is a rejection boundary a fixture has to drive one unit past — a ceiling described only as "fixed in design" cannot be tested, and today none exists at all: `internal/security/scanner/tpa_bundle_source.go` reads the candidate with a bare `os.ReadFile(path)` (no size check, no type check), and `loadBundleCheck` (`tpa_bundle.go`) applies no rule-count limit. The values are: **bundle ≤ 8 MiB**, **rules ≤ 2000** (SC-008 exercises ≥ 500, so this leaves headroom for four more corpus generations without a spec change), **sidecar ≤ 4 KiB**, and a **read budget of 5 s per candidate** (local file or fetch body). A candidate at exactly the ceiling MUST load; one byte or one rule over MUST be rejected whole, with the reason recorded in bundle status. Design MAY tighten these but MUST NOT loosen them without amending this FR. **The sidecar needs its own, much smaller ceiling and a bounded read**, because it is the one artifact that must be read and parsed BEFORE any signature has been verified (its algorithm identifier, key id, and signature value are the inputs to verification), which puts it outside the protection FR-006 gives the bundle. Without it the in-scope bundle-path attacker simply writes a multi-gigabyte or never-terminating sidecar next to an otherwise acceptable bundle and exhausts memory or stalls the refresh before anything can be rejected — a denial of the scanner's own update path. The sidecar MUST therefore be read through a hard byte cap sized to its fixed small structure and rejected on overrun, before it is parsed, on both the file-drop and fetch paths. A byte cap alone is not sufficient, because it bounds bytes consumed but not time spent: the same attacker replaces the sidecar (or bundle) path with a FIFO and never writes to it, so the read blocks forever without ever approaching the cap and Spec 087's single-flight refresh stays occupied, blocking every legitimate database update while the loader waits. Both local candidate artifacts MUST therefore be REGULAR files — a FIFO, device, socket, or a symlink resolving to one is refused outright, not read. **This check MUST be made on the OPEN DESCRIPTOR, never on the pathname.** A `stat`-then-open sequence is itself a substitution window: the in-scope bundle-path attacker replaces the path between the two calls, the stat sees a regular file and the open lands on their FIFO. The loader MUST open the candidate with the platform's no-follow protection (`O_NOFOLLOW`; on Go, `os.OpenFile` with `syscall.O_NOFOLLOW`, or `os.Root`-scoped opens where the containing directory is itself trusted), `fstat` the resulting descriptor, refuse anything whose mode is not a regular file, and then read the size ceiling from that same descriptor — never re-resolving the path. **The time bound MUST NOT be specified as an I/O deadline on the file itself**: Go's `SetReadDeadline` is not supported for regular files and returns `ErrNoDeadline`, so a requirement phrased that way is unimplementable as written and would be silently dropped. The requirement is instead that every candidate read complete within a bounded wall-clock budget enforced OUTSIDE the read — a context-cancelled reader goroutine for local files (whose expiry releases the single-flight slot and records a rejection reason even if the blocked read itself cannot be interrupted), and `http.Client`/`context` deadlines plus a capped `io.LimitReader` for fetch response bodies. Design MUST name the concrete mechanism and its per-platform limits, including the case where a stalled network filesystem leaves the abandoned read goroutine parked: the slot is released and the refresh path stays available, which is the property being guaranteed. A database at the ceilings MUST NOT measurably regress the scan hot path (see SC-008). - **FR-011a**: Activation MUST be crash-consistent, not merely atomic in memory. Spec 087 FR-010 makes the rule-set swap atomic with respect to concurrent scans; that says nothing about persisted state, and a process death between "activated" and "watermark written" reopens exactly the replay window FR-009 exists to close (while writing the watermark first can leave the install with only an older recoverable bundle). The activation record — active fingerprint, sequence, per-authority watermarks and their epochs, the fingerprint each watermark was set by, signature-verified state, any FR-010 pin, and the FR-010 deny-list — MUST be persisted as ONE transaction, and the recoverable last-known-good bundle MUST be durable before the in-memory pointer is published. The deny-list and watermark fingerprint are load-bearing members of that transaction, not bookkeeping: a crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator had just rejected, silently undoing the rollback. Startup, refresh, hot-reload, and rollback MUST all serialize through that same transaction, so any crash leaves the install on a consistent (bundle, watermark) pair. **Seed corpus** @@ -183,6 +183,7 @@ The publisher cuts a new database release (new signature for a fresh campaign), - **SC-008**: With a database at the design load ceilings (≥ 500 regex rules), p95 scan latency over a 100-tool server changes by less than 10% versus the 6-signature baseline, and bundle load-plus-verify completes under one second on commodity hardware. - **SC-009**: After activating a database containing a new signature, an already-approved tool whose cached description matches it is surfaced for review within the same refresh cycle, with zero approvals auto-revoked and zero servers auto-quarantined by the refresh itself. - **SC-010**: Rollback authorization is bound to the whole digest and to a real history: a fixture in which an attacker-substituted, publisher-signed, current-epoch bundle shares the operator-supplied abbreviated fingerprint (first 12 hex chars) but differs in the full SHA-256 is REFUSED as a rollback-target mismatch; and after N activations the activation record replays all N entries — (authority, epoch, sequence, full digest, generated_at, activated_at, source) — across a process restart, so the digest of a known-good artifact is recoverable without trusting the candidate file. +- **SC-010a**: Every FR-011 ceiling is exercised at its boundary: a bundle of exactly 8 MiB and one of 8 MiB + 1 byte, a corpus of exactly 2000 rules and one of 2001, a sidecar of exactly 4 KiB and one of 4 KiB + 1 byte, and a candidate whose read exceeds 5 s — the first of each pair activates, the second is rejected whole with its reason in bundle status, and the last-known-good corpus keeps serving throughout. - **SC-011**: Candidate opening is race-free and time-bounded, demonstrated by fixtures that (a) swap the candidate path for a FIFO between path resolution and open and are refused by the descriptor-level `fstat` check rather than blocking, (b) point the path at a symlink to a FIFO and are refused without following it, and (c) leave a never-written FIFO in place and see the single-flight refresh slot released within the bounded budget with a recorded rejection reason, so a subsequent legitimate database update still activates. - **SC-012**: The persisted-state trust boundary is exercised by SUBSTITUTION, not only deletion: fixtures replace `config.db` with a prepared database carrying (a) a lowered watermark, (b) a cleared signature ratchet, and (c) a forged pin/deny-list entry, and in every case the guarantees anchored outside the data directory still hold — the FR-009b publisher baseline re-seeded from the binary, the config-resident operator baseline, `require_signed_bundle`, and the FR-004b active-authority declaration — while the install reports itself degraded. No fixture may assert that the ratchet, watermark, pin or deny-list themselves survive; they are best-effort against this attacker by construction. From cb619fdd3c242d616e68b9ebd7f6fc41f793f1ca Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 26 Aug 2026 17:03:44 +0300 Subject: [PATCH 18/20] docs(specs): 101-tpa-db implementation plan Phase 0 + Phase 1 for the tpa-db epic: research.md (R1-R11 findings, D1-D11 decisions), plan.md, data-model.md, three contracts, quickstart.md. The research pass changed the shape of the plan in three ways worth calling out, each because the tree does not look like the documents describe it: - Spec 087's plan names a package internal/security/bundle that does not exist. What shipped (086) lives in internal/security/scanner/tpa_bundle_source.go and tpa_bundle.go. Reading 087's plan as a description of the tree is the fastest way to write tasks against wrong paths, so R1 says so explicitly and the plan extends the real files. Only verification earns a new leaf package (internal/security/bundlesig), on the toolsig argument: a pure fixture-testable wire contract an independent publisher must reproduce. - Spec 087 is specified but NOT implemented - no bundle package, no security .bundle config, no refresh ticker. FR-018 forbids inventing a second refresh lifecycle, and there is no first one to insert stages into. D11 therefore cuts scope at the network boundary: this feature delivers the file-drop path end to end and leaves fetch to 087. SC-003 (air-gapped install reaches full P1) is satisfied entirely by that path, which is the evidence the cut is right - but SC-007's fetch half is consequently not demonstrable by this feature alone, and that is recorded as needing the maintainer's assent rather than discovered by a task. - The eval gate cannot see the database at all. gateChecks() returns only the four built-in checks; scanner.BundleCheck, which production DOES append, is absent and scan-eval never loads a bundle. FR-016/SC-005 would pass today with the corpus entirely absent, so teaching the gate to score the bundle (D8) is a prerequisite for growing the corpus, not a parallel task. Decisions resolving the spec's two remaining clarification markers: publication goes to GitHub Releases on the corpus repo, keeping the artifact and the gate that must precede it in one workflow (D1); require_signed_bundle stays opt-in indefinitely, because the FR-009 ratchet already covers every install that has seen a signed bundle and a default flip would break only the air-gapped self-built-corpus persona (D2). Contracts carry the parts FR-003 and FR-001a explicitly refuse to leave to the implementation: a five-key sidecar with a total, reason-bearing rejection grammar and byte-level accept/reject fixtures, and an additive signatures[] section whose referential integrity is enforced in BOTH directions - the reverse direction (every gating rule named by exactly one signature) being the one that stops a gating rule shipping with no provenance, licence or eval evidence. --- CLAUDE.md | 2 +- ROADMAP.md | 2 + .../contracts/bundle-signatures-section.md | 87 +++++++++ .../contracts/bundle-status-surface.md | 80 ++++++++ specs/101-tpa-db/contracts/sidecar-format.md | 79 ++++++++ specs/101-tpa-db/data-model.md | 123 ++++++++++++ specs/101-tpa-db/plan.md | 163 ++++++++++++++++ specs/101-tpa-db/quickstart.md | 148 ++++++++++++++ specs/101-tpa-db/research.md | 182 ++++++++++++++++++ 9 files changed, 865 insertions(+), 1 deletion(-) create mode 100644 specs/101-tpa-db/contracts/bundle-signatures-section.md create mode 100644 specs/101-tpa-db/contracts/bundle-status-surface.md create mode 100644 specs/101-tpa-db/contracts/sidecar-format.md create mode 100644 specs/101-tpa-db/data-model.md create mode 100644 specs/101-tpa-db/plan.md create mode 100644 specs/101-tpa-db/quickstart.md create mode 100644 specs/101-tpa-db/research.md diff --git a/CLAUDE.md b/CLAUDE.md index a83115580..2bd9d46a0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -158,6 +158,6 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/ - **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md). ## Recent Changes +- 101-tpa-db: Go 1.25.5 module toolchain (`go.mod`), backend-only + stdlib only for the new work — `crypto/ed25519`, `crypto/sha256`, `syscall` (no ed25519 usage in the tree before this); existing bbolt (state), zap, Cobra. **No new dependencies.** - 098-tools-preflight: Added Go 1.24 module toolchain (repo builds with local Go 1.25) + existing only — chi (httpapi), bbolt (storage), Bleve (index), zap (logging), Cobra (CLI), swaggo/swag v2 (contract regen). **No new dependencies.** - 097-stored-scripts: Added Go 1.25 (os.Root/Root.ReadFile available — R1) + stdlib only (os.Root). **No new dependencies.** -- 096-batched-call-tools: Added Go 1.24 module toolchain (repo builds with local Go 1.25) + existing only — goja (sandbox), mark3labs/mcp-go (tool surface), zap. **No new dependencies.** diff --git a/ROADMAP.md b/ROADMAP.md index 4643c721d..69a2d7a7a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -801,3 +801,5 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [097-stored-scripts](./specs/097-stored-scripts/) | `in-flight` | 13/14 (93%) | | [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 26/33 (79%) | | [099-describe-check-mode](./specs/099-describe-check-mode/) | `in-flight` | 9/10 (90%) | +| [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | +| [101-tpa-db](./specs/101-tpa-db/) | — | — | diff --git a/specs/101-tpa-db/contracts/bundle-signatures-section.md b/specs/101-tpa-db/contracts/bundle-signatures-section.md new file mode 100644 index 000000000..7ac2f3604 --- /dev/null +++ b/specs/101-tpa-db/contracts/bundle-signatures-section.md @@ -0,0 +1,87 @@ +# Contract: Bundle `signatures[]` Section + +**Feature**: `101-tpa-db` · Resolves FR-001a. + +## 1. Why this exists + +FR-001 says the publishable artifact is "the existing compiled bundle". FR-012 says "each signature MUST carry provenance … and a redistributable license". **Both cannot hold today**, because the shipped `scanner-bundle.json` has nowhere to put either: + +```text +top level : bundle_version, generated_from, rules, schema_version, signature_count, skipped +rules[] : category, confidence, detector, engine, flags, id, indicators, + level, pattern, severity, target, type +``` + +No provenance. No licence. No campaign identity — `signature_count` counts *rules*, so SC-004's "at least 25 signatures spanning at least 8 distinct classes" is not even expressible. + +## 2. Shape + +A new **additive** top-level array. A v0.1 loader ignores unknown top-level keys, so contract §4 forward-compat is preserved (FR-001a). + +```json +{ + "signatures": [ + { + "id": "TPA-2026-0001", + "category": "instruction_injection", + "gating": true, + "provenance": { + "source": "https://invariantlabs.ai/blog/mcp-github-vulnerability", + "published": "2026-01-14", + "note": "hidden block coaxing ~/.ssh/id_rsa" + }, + "license": "CC-BY-4.0", + "rule_ids": ["r-hidden-important-block", "r-hidden-html-comment"] + } + ] +} +``` + +| Field | Type | Rules | +|---|---|---| +| `id` | string | `^TPA-[0-9]{4}-[0-9]{4}$`, unique across the bundle | +| `category` | string | the campaign/technique class; must have a `categoryCheck` entry in the gate (FR-016a) | +| `gating` | bool | `true` ⇒ must emit hard-tier signals and must ship eval evidence (FR-013/FR-014) | +| `provenance.source` | URL | the public disclosure that motivated the signature | +| `provenance.published` | `YYYY-MM-DD` | | +| `provenance.note` | string | optional, human context | +| `license` | SPDX id | from the redistributable allowlist below | +| `rule_ids` | array | ≥1, each naming an existing `rules[]` entry | + +**Licence allowlist** (redistributable, FR-012): `CC0-1.0`, `CC-BY-4.0`, `CC-BY-SA-4.0`, `MIT`, `Apache-2.0`. Anything else is refused at corpus build — including "no licence stated", which is the common case for a pattern lifted from a blog post and is precisely what FR-012 exists to stop shipping. + +## 3. Relationship to `rules[]` + +**Many-to-one, explicitly.** Several rules may implement one signature — a hidden-instruction class already needs distinct patterns for `` blocks, HTML comments and zero-width runs. One rule belongs to at most one signature. + +```text +signatures[] ──1─────*── rules[] + TPA id pattern +``` + +## 4. Referential integrity — enforced at build, in both directions + +A bundle failing any of these is **rejected whole**, exactly like any other contract violation: + +1. every `signatures[]` entry names ≥1 rule id that exists in `rules[]`; +2. every rule with gating intent is named by **exactly one** signature entry — not zero (an ungoverned gating rule has no provenance and no eval evidence) and not two (ambiguous campaign attribution); +3. `id` values are unique; +4. `license` is in the allowlist; +5. every `gating: true` signature has its FR-013 eval pair in the frozen dataset — ≥1 labeled gated-malicious sample and ≥1 category-matched hard negative following the `hn__*` naming the dataset validator enforces. + +Direction 2 is the one a naive implementation drops, and it is the one that matters: checking only "signatures point at real rules" lets a gating rule ship with no campaign record at all, which is the state the corpus is in today. + +## 5. Interaction with the eval gate + +`gating: true` is not self-certifying. FR-016a's `categoryCheck` map decides whether a category is *enforced*: `gatedCategory()` enforces a category only when its mapped check id is registered, so **a new class whose category is missing from that map is tallied and reported but excluded from `gatedMalicious`/`OverallRecall`** — its misses can never fail the gate. Adding a signature class therefore means adding its `categoryCheck` entry in the same change, or the coverage is decorative. + +(Its false positives still count either way, since `FPRate` is computed over the whole hard-negative set regardless of category mapping. So an unmapped class can only ever hurt — never help — the gate outcome. That asymmetry is the tell.) + +## 6. Counting for SC-004 + +- **signatures** = `len(signatures)` — must be ≥ 25 +- **classes** = distinct `category` — must be ≥ 8 +- every signature carries provenance + licence (rule 4 above) +- every `gating: true` signature has its eval pair (rule 5) + +Note `signature_count` (existing, top-level) counts **rules** and keeps that meaning. Renaming it would break the v0.1 line for no gain; SC-004 counts `signatures[]`. diff --git a/specs/101-tpa-db/contracts/bundle-status-surface.md b/specs/101-tpa-db/contracts/bundle-status-surface.md new file mode 100644 index 000000000..e69cf09d0 --- /dev/null +++ b/specs/101-tpa-db/contracts/bundle-status-surface.md @@ -0,0 +1,80 @@ +# Contract: Bundle Status Surface + +**Feature**: `101-tpa-db` · Resolves FR-020. + +The operator question this surface must answer is not "what version am I on" but **"am I current and verified, or stuck on last-known-good — and why?"** Today's `BundleInfo` cannot answer the second half at all. + +## 1. Fields + +Existing fields keep their meaning and their names. `Fingerprint` stays the 12-hex short form (it is load-bearing in current operator output and in `logBundle`); the full digest is **added beside it**, not substituted. + +| Field | Existing | Purpose | +|---|---|---| +| `bundle_version`, `schema_version`, `source`, `generated_at`, `load_error` | ✅ | unchanged | +| `runnable_rules`, `skipped_rules`, `declared_skipped` | ✅ | unchanged — un-evaluated coverage, never clean coverage | +| `fingerprint` | ✅ | 12-hex short form, human-comparable | +| `full_digest` | **new** | complete SHA-256; the **only** value rollback accepts (FR-010) | +| `sequence` | **new** | decimal string, as published | +| `authority` | **new** | `publisher` \| `operator` — which authority is serving | +| `trust_epoch` | **new** | the epoch it was published under | +| `signature_verified` | **new** | bool | +| `staleness_days` | **new** | derived from `generated_at` | +| `coverage_degraded` | **new** | bool | +| `degraded_reason` | **new** | enum, §3 | +| `last_rejection` | **new** | `{reason, at}`, §2 — retained across a successful load so "what did it refuse, and when" survives | + +## 2. Rejection reasons + +The vocabulary an operator reads to tell a tamper attempt from a stale epoch from their own misconfiguration: + +| Reason | Means | +|---|---| +| `tamper` | signature did not verify over the raw bytes | +| `downgrade` | sequence below this authority's watermark for this epoch, or equal sequence with a different fingerprint | +| `stale_epoch_replay` | manifest epoch below the binary-trusted epoch — a validly-signed but superseded artifact | +| `unsigned_refused_by_ratchet` | this install has activated a signed bundle before; unsigned drops are refused thereafter | +| `authority_not_active` | verified against a real key of the **non-active** authority (FR-004b) | +| `rollback_target_mismatch` | the loaded artifact's full digest ≠ the digest the operator named | +| `gate_failure` | activation self-check (Spec 087 FR-008) rejected the candidate | +| `ceiling_exceeded` | over one of FR-011's four limits (which one is in the message) | +| `not_regular_file` | candidate was a FIFO/device/socket, or a symlink to one | +| `read_timeout` | the 5 s read budget expired; the single-flight slot was released | +| `sidecar_*`, `key_*`, `signature_invalid`, `sequence_mismatch` | the sidecar grammar — see [sidecar-format.md](sidecar-format.md) §3 | + +**Every one of these keeps last-known-good serving.** There is no rejection path that leaves the scanner with an empty or partial rule set. + +## 3. Degraded-coverage reasons + +`coverage_degraded` suspends `scan`-mode auto-approval, routing changes to human review. It **must** be visible: a silently suspended auto-approval is indistinguishable from a quiet install, which is the failure this field exists to prevent. + +| `degraded_reason` | Condition | +|---|---| +| `unsigned_external_active` | an unsigned file-dropped/fetched bundle is active (FR-007) | +| `embedded_below_watermark` | the embedded default is serving and its sequence is below the **publisher** watermark (FR-009a a) | +| `operator_bundle_missing` | the operator authority is active and its previously-activated external bundle is gone or invalid (FR-009a b) | +| `operator_authority_unbaselined` | an operator-signed bundle is active under an authority configured without a baseline (FR-009b) | + +**Not degraded** — the two cases a careless implementation gets wrong: + +- an ordinary fresh install serving the embedded default with no retained watermark and no missing external bundle. The embedded default is binary-trusted; degrading here would ship every fresh install with `scan` mode silently disabled. +- an embedded default **at or above** the publisher watermark after an external file was removed. It is demonstrably no older than what went missing — the ordinary "external file removed after a binary upgrade shipped a newer corpus" case. + +## 4. Surfaces + +Same field set everywhere; three renderings. + +| Surface | Rendering | +|---|---| +| **CLI** (`mcpproxy security …`) | human table + `-o json`/`yaml`; the full digest shown in full (it is the rollback input) | +| **REST** (security overview) | the JSON object above; `make swagger` regenerated | +| **Web UI** (Security page) | short fingerprint with the full digest on demand; degraded state as a banner with its reason, never a silent flag | + +The Web UI rule follows the house preference already established for status surfaces: **mark only the states that need action**, and always say *why* rather than showing a bare badge. + +## 5. Activation history + +`(authority, trust_epoch, sequence, full digest, generated_at, activated_at, source)` per activation, append-only, retained across restarts. + +Exposed read-only. It exists because rollback is authorized by full digest and there is otherwise **no trustworthy way to learn the digest of the known-good artifact you want to revert to** — reading it from the candidate file would defeat the purpose, since the file is what the attacker controls. + +Data-dir resident, so it carries the same caveat as the ratchet: it makes a legitimate rollback *possible*; it does not *authorize* one. diff --git a/specs/101-tpa-db/contracts/sidecar-format.md b/specs/101-tpa-db/contracts/sidecar-format.md new file mode 100644 index 000000000..3980e320e --- /dev/null +++ b/specs/101-tpa-db/contracts/sidecar-format.md @@ -0,0 +1,79 @@ +# Contract: Signature Sidecar Format v1 + +**Feature**: `101-tpa-db` · Resolves FR-003's "the sidecar wire format MUST be specified in design, not left to the implementation". + +This is the one artifact an independent publisher must be able to produce and every loader must reject **identically**. It is also the only artifact read and parsed *before* any signature has been verified — its `algorithm`, `key_id` and `signature` are the inputs to verification — which is why the grammar below is total and why the byte cap (FR-011: **4 KiB**) applies before parsing. + +## 1. Encoding + +A single JSON object. UTF-8. No BOM. Trailing newline optional and ignored; **any other trailing bytes are a rejection**. + +File naming: `.sig` beside the bundle (e.g. `scanner-bundle.json.sig`). + +## 2. Fields — exactly these five, no others + +```json +{"sidecar_version":"1","algorithm":"ed25519","key_id":"3f0a…64hex","signature":"MEUCIQ…","sequence":"101"} +``` + +| Field | Type | Accepted values | +|---|---|---| +| `sidecar_version` | string | exactly `"1"` | +| `algorithm` | string | exactly `"ed25519"` | +| `key_id` | string | exactly 64 lowercase hex chars — SHA-256 of the 32-byte raw Ed25519 public key | +| `signature` | string | standard base64 **with** padding, decoding to exactly 64 bytes | +| `sequence` | string | `^[1-9][0-9]{0,19}$`, value ≤ 2^64−1 | + +`key_id` is a fingerprint **of the key itself**, never an operator-chosen label (FR-003). That is what makes "the manifest names one publisher while the bytes are signed by another" detectable rather than cosmetic. + +`sequence` is duplicated from the manifest deliberately: it is the one manifest field whose disagreement is cheap to detect *before* parsing the thing the signature protects. + +## 3. Rejection grammar — total, deterministic, and reason-bearing + +Every row is a hard rejection. There is no leniency, no fallback and no skip anywhere in this table. Each rejection records its own reason (surfaced via FR-020's `LastRejection`), because a conformance test that only asserts "failed" cannot show two implementations agree. + +| # | Condition | Reason code | +|---|---|---| +| 1 | file exceeds 4 KiB (checked **before** parsing) | `sidecar_too_large` | +| 2 | not a single JSON object / trailing bytes | `sidecar_malformed` | +| 3 | a required key is missing | `sidecar_missing_field` | +| 4 | any key outside the five | `sidecar_unknown_field` | +| 5 | a key appears more than once | `sidecar_duplicate_field` | +| 6 | any value has the wrong JSON type | `sidecar_type` | +| 7 | `sidecar_version != "1"` | `sidecar_version_unsupported` | +| 8 | `algorithm != "ed25519"` | `sidecar_algorithm_unsupported` | +| 9 | `key_id` not 64 lowercase hex | `sidecar_key_id_malformed` | +| 10 | `signature` not valid padded base64, or not 64 bytes decoded | `sidecar_signature_malformed` | +| 11 | `sequence` fails the grammar or exceeds 2^64−1 | `sidecar_sequence_malformed` | +| 12 | `key_id` not in the active authority's trust set | `key_not_trusted` | +| 13 | signature does not verify over the raw bundle bytes | `signature_invalid` | +| 14 | manifest `key_id` ≠ sidecar `key_id` ≠ verifying key's id | `key_id_mismatch` | +| 15 | manifest `sequence` ≠ sidecar `sequence` | `sequence_mismatch` | + +Row 8 deserves emphasis: **an unknown algorithm identifier is a hard rejection, never a fallback to Ed25519 and never a skip.** The field exists so the scheme can evolve without breaking verification of *existing* artifacts — a v2 loader will accept both; a v1 loader accepts one. + +Rows 5 and 6 need a strict decoder. Go's `encoding/json` silently accepts duplicate keys (last wins) and ignores unknown fields unless told otherwise — use `Decoder.DisallowUnknownFields` for row 4, an explicit duplicate scan for row 5, and `Decoder.More()` after the object for row 2. + +## 4. What the signature does and does not cover + +The signature is over **the exact raw bundle bytes**, not over the sidecar. Therefore: + +- **Covered**: every byte of the bundle. Any alteration → row 13. +- **Covered by agreement rules**: the sidecar's identity fields, because FR-003 requires manifest/sidecar/verifying-key id agreement → row 14. +- **NOT covered**: the sidecar's own encoding. Whitespace and key-order changes that alter neither the signature value nor an identity field are **not required to be rejected** — a detached signature cannot detect them. + +SC-001 is written against exactly this boundary, which is why it enumerates four tamper classes rather than claiming "every byte of the sidecar". + +## 5. Fixtures (shipped in `internal/security/bundlesig/testdata/`) + +**Accept** — one canonical valid pair (`valid.json` + `valid.json.sig`) plus: + +| Fixture | Property demonstrated | +|---|---| +| `accept-min-sequence` | `sequence: "1"` | +| `accept-max-sequence` | `sequence: "18446744073709551615"` (2^64−1) — the value SC-002a fixture (a) signs | +| `accept-trailing-newline` | a trailing `\n` is ignored | + +**Reject** — one fixture per numbered row above, named `reject-`. Each asserts the **reason code**, not merely that loading failed. + +Two implementations agreeing on the accept set is not conformance. Agreeing on the reject set, reason by reason, is. diff --git a/specs/101-tpa-db/data-model.md b/specs/101-tpa-db/data-model.md new file mode 100644 index 000000000..65acd4be3 --- /dev/null +++ b/specs/101-tpa-db/data-model.md @@ -0,0 +1,123 @@ +# Phase 1 Data Model: tpa-db + +**Feature**: `101-tpa-db` · **Plan**: [plan.md](plan.md) · **Decisions**: [research.md](research.md) + +Three stores, deliberately separated by who can write them and what survives a data-directory wipe. That separation *is* the security model — every FR that says "config-resident" is choosing store 2 over store 3 for exactly that reason. + +| # | Store | Written by | Survives data-dir wipe | Trust role | +|---|---|---|---|---| +| 1 | **Binary** (`go:embed`) | a release build | yes | publisher keys, publisher epoch, embedded corpus + its sequence (the FR-009b publisher baseline) | +| 2 | **`mcp_config.json`** | the operator | yes | operator keys, operator baseline, operator epoch, active-authority declaration, `require_signed_bundle` | +| 3 | **`config.db`** (BBolt) | the running proxy | **no** | watermarks, ratchet, pin, deny-list, activation history — all best-effort (plan Complexity Tracking row 1) | + +--- + +## 1. Bundle manifest additions (inside the signed bytes) + +Additive to the Scanner Bundle Contract v0.1 line, so a v0.1 loader ignores them (FR-002). + +| Field | Type | Rules | +|---|---|---| +| `generated_at` | RFC3339 string | derived from release metadata (`SOURCE_DATE_EPOCH` or the release commit), **never** build wall-clock | +| `sequence` | **decimal string** | `^[1-9][0-9]{0,19}$`, value ≤ 2^64−1. Never a bare JSON number — double-precision round-trip corrupts >2^53−1 and admits exponent notation (FR-002) | +| `key_id` | 64 lowercase hex | canonical fingerprint of the signing public key; must equal the sidecar's `key_id` and the verifying trust-set key's id (FR-003) | +| `trust_epoch` | integer ≥ 0 | the authority's epoch this artifact was published under; must **equal** the binary-trusted epoch of the verifying authority (FR-005a) | +| `signatures[]` | array | see [contracts/bundle-signatures-section.md](contracts/bundle-signatures-section.md) (FR-001a/D4) | + +**Why these live inside the signed bytes**: the epoch especially. If it lived only in the binary, a bump-and-revoke would let an old bundle signed by a surviving key be silently reinterpreted as belonging to the new epoch, clear the re-baselined watermark and activate at full coverage — reinstating the blind spots the recovery release existed to close (FR-005a). + +## 2. Sidecar + +Separate file, never signed by itself. Five keys, total rejection grammar — [contracts/sidecar-format.md](contracts/sidecar-format.md). + +## 3. Signing authority + +```go +type Authority string // "publisher" | "operator" +``` + +Not a key set — a **namespace for anti-downgrade state** (FR-004a). All publisher keys, including ones mid-rotation, belong to one publisher authority precisely so a rotated-in key inherits the existing watermark instead of starting a fresh one. + +| Property | Publisher | Operator | +|---|---|---| +| Keys | embedded in the binary (store 1) | config (store 2) | +| Epoch advanced by | a binary release | a config edit | +| Baseline (FR-009b) | the embedded corpus's sequence, **re-applied every startup** | declared in config; **may be absent** → authority runs explicitly degraded | +| Active by default | yes (FR-004b) | no | + +Exactly one authority is **active** at a time, declared in config. A candidate whose verifying key belongs to a non-active authority is refused `authority_not_active`. Without that declaration, per-authority watermarks open the hole they were introduced to close: whoever writes the bundle path would choose the authority, and any operator-signed artifact clearing the operator baseline would swap the install onto a narrower corpus while still reporting `signature_verified=true` at full coverage. + +**The declaration governs the serving bundle, not only future candidates.** On a change, an active external bundle of the now-inactive authority is deactivated and the install falls back to the embedded default under FR-009a's rules. + +## 4. Per-authority persisted state (BBolt, store 3) + +One record per authority in the new `TPABundleStateBucket`, plus one history list. **All members of one `Update` transaction** (D6/FR-011a). + +```go +type authorityState struct { + Authority Authority + Epoch uint64 // the epoch these values are scoped to + Watermark uint64 // highest sequence ever activated for (authority, epoch) + WatermarkSetBy string // full SHA-256 of the bundle that set it + SignatureRatchet bool // has a signature-verified bundle ever activated + Pin *pinEntry + DenyList []denyEntry // keyed (authority, epoch, sequence, fingerprint) +} +``` + +**Scoping is `(authority, epoch, sequence)`, ordered by epoch first** (FR-005a). Every comparison is same-authority and same-epoch: a cross-authority numeric comparison is meaningless — an operator corpus at sequence 900 says nothing about a publisher corpus at 100 — and making one would permanently suspend auto-approval on an install whose publisher fallback is perfectly current (FR-009a). + +**`WatermarkSetBy` and `DenyList` are load-bearing transaction members, not bookkeeping.** A crash that preserved a rollback while losing the deny-list would let the next refresh immediately re-activate the exact release the operator just rejected — silently undoing the rollback (FR-011a). + +### Watermark lifecycle + +```text +seed from baseline (every startup, publisher) ──► raised by each higher-sequence activation + ▲ │ + │ rollback does NOT lower it + epoch bump re-baselines (expressed as pin + deny-list instead) + and retires prior-epoch pins/deny-list +``` + +Never empty for any authority that has a baseline — an absent watermark accepts anything, which is the whole point of FR-009b. The single exception is a baseline-less operator authority, which is run **explicitly degraded** rather than as a silent zero. + +## 5. Activation history (append-only, store 3) + +One record per activation, retained across restarts (FR-010): + +`(authority, trust_epoch, sequence, full SHA-256 digest, generated_at, activated_at, source)` + +**This is new.** `BundleInfo` reports only the currently active artifact, so without a history an operator has no trustworthy way to learn the digest of the known-good artifact they intend to revert to — and rollback authorization is by full digest. Being data-dir resident, it carries the same best-effort caveat as the ratchet: it makes a legitimate rollback *possible*, it does not *authorize* one. + +## 6. `BundleInfo` additions (FR-020) + +Existing fields keep their meaning. `Fingerprint` stays the 12-hex short form — it is load-bearing in current operator output — and a full digest is **added** beside it rather than widening it (R3). + +| New field | Purpose | +|---|---| +| `FullDigest` | complete SHA-256; the only value rollback accepts | +| `Sequence` | decimal string, as published | +| `Authority`, `TrustEpoch` | which authority is serving, under which epoch | +| `SignatureVerified` | bool | +| `StalenessDays` | derived from `generated_at` | +| `CoverageDegraded`, `DegradedReason` | FR-007/FR-009a — a silently suspended `scan`-mode auto-approval is otherwise indistinguishable from a quiet install | +| `LastRejection` | `{reason, at}` — one of `tamper`, `downgrade`, `stale_epoch_replay`, `unsigned_refused_by_ratchet`, `authority_not_active`, `rollback_target_mismatch`, `gate_failure`, `ceiling_exceeded`, `not_regular_file`, `read_timeout` | + +## 7. Signature (TPA record) + +The catalog unit SC-004 counts: `TPA-YYYY-NNNN` id, category, gating intent, detector(s), level, confidence, provenance reference, licence, and the `rule_ids` implementing it. Many-to-one to `rules[]` — one campaign class routinely needs several patterns (a hidden-instruction signature already implies distinct rules for `` blocks, HTML comments and zero-width runs). + +## 8. Coverage state (the derived value that matters most) + +`coverageOK` today is `bundlePresent && ChecksFailed == 0` (R5). It becomes: + +```text +degraded ⟸ an unsigned EXTERNAL bundle is active (FR-007) + ∨ the embedded default is serving AND its sequence is below + the PUBLISHER watermark (FR-009a a) + ∨ the operator authority is active AND its previously-activated + external bundle is gone or invalid (FR-009a b) + ∨ a baseline-less operator authority's bundle is active (FR-009b) +``` + +**And explicitly NOT degraded** in the two cases a careless reading would catch: an ordinary fresh install on the embedded default, and an embedded default at or above the publisher watermark after an external file was removed. A false degrade silently suspends `scan`-mode auto-approval fleet-wide, which is why both negatives are named here and tested in the plan's Test Strategy. diff --git a/specs/101-tpa-db/plan.md b/specs/101-tpa-db/plan.md new file mode 100644 index 000000000..25c22707d --- /dev/null +++ b/specs/101-tpa-db/plan.md @@ -0,0 +1,163 @@ +# Implementation Plan: Versioned, Signed, Offline-First TPA Signature Database + +**Branch**: `101-tpa-db` | **Date**: 2026-08-26 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `/specs/101-tpa-db/spec.md` (roadmap P1 epic `tpa-db`) + +## Summary + +Make the TPA signature database a first-class security artifact: a **signed, sequence-versioned publishable pair** (the existing `scanner-bundle.json` + a detached Ed25519 sidecar), a **verify-before-parse loader** with real ceilings and per-authority anti-downgrade state, an **explicit fingerprint-authorized rollback**, a **seed corpus of ≥25 cataloged campaign signatures** with provenance and licences, and the **observability** to tell "current and verified" from "stuck on last-known-good, and why". + +Technical approach, grounded in the code: + +- **Verification is a new leaf package, everything else extends what shipped.** `internal/security/bundlesig/` holds sidecar parsing and Ed25519 verification as pure functions over bytes — no I/O, no globals, no `scanner` import — because that is the one piece with an independent wire contract two implementations must agree on byte-for-byte (D3/D10). The loader, ceilings, activation and status work extend `internal/security/scanner/tpa_bundle_source.go` and `tpa_bundle.go`, where Spec 086 actually shipped them. **Spec 087's plan names a package `internal/security/bundle` that does not exist** (research.md R1); reading it as a description of the tree is the fastest way to write tasks against wrong paths. +- **The load pipeline gains a first stage and grows ceilings throughout.** FR-008's order is not a preamble to the existing pipeline but interleaved with it, because the checks become knowable at different points: sidecar cap → parse sidecar → bundle byte ceiling → **signature over the raw bytes** → parse manifest → rule-count ceiling → epoch match → sequence/fingerprint → version compat → compile-all-or-reject → runnable>0 → self-check → activate. Today's `loadBundleFromFile` does `os.ReadFile` with no cap, no file-type check and no deadline (R2), so FR-011's numbers — bundle ≤ 8 MiB, rules ≤ 2000, sidecar ≤ 4 KiB, 5 s read budget — are new code at three different points, not one guard. +- **Candidate opening is descriptor-based, not path-based.** `os.OpenFile(..., O_NOFOLLOW)` → `fstat` the descriptor → refuse non-regular → read the ceiling from that same `FileInfo`, never re-resolving the path (D5). The time bound is a context-cancelled reader goroutine, **not** `SetReadDeadline`, which returns `ErrNoDeadline` on regular files and would be silently dropped. +- **Anti-downgrade state is per signing authority and one BBolt transaction.** A new `TPABundleStateBucket` (`internal/storage/models.go`) holds, per authority: watermark, its epoch, the fingerprint that set it, the signature ratchet, the FR-010 pin and deny-list — plus an append-only activation history. All written in a single `Update`, with the last-known-good bundle bytes `fsync`ed before the commit and the in-memory pointer published after it (D6), so no crash yields a new bundle with an old watermark. +- **`BundleInfo` gains a full digest rather than a widened one.** Today's `Fingerprint` is `hex(sha256)[:12]` — 48 bits (R3), fine as a human label and load-bearing in existing output, but not as authorization for the one path that bypasses the watermark. Rollback accepts only the full digest, and status surfaces both. +- **`coverageOK` is where the trust story meets the approval gates.** `inprocess.go` currently computes it from mere bundle presence (R5). FR-007's unsigned-external degradation and FR-009a's fallback demotion both land on that one expression — the narrowest and most safety-critical edit in the feature, and the one whose regression is silent (a quietly suspended `scan`-mode auto-approval looks exactly like a quiet install, which is why FR-020 requires surfacing it). +- **The gate must be taught to see the database before the corpus grows.** `cmd/scan-eval` never loads a bundle and `gateChecks()` omits `BundleCheck` entirely (R6), so FR-016/SC-005 would pass with the corpus absent. `--bundle` + registration through the **production** loader (D8), `categoryCheck` entries for every new class, and a deterministic negative control whose gated-malicious set is one control sample alone (D9). +- **Telemetry needs typed backstop entries, not a relaxation.** `scanV8TPAScanner` whitelists four integer keys and rejects anything else before transmit (R7). Four typed entries plus one payload-level cross-field rule (`sequence` present while `bundle_version == "other"` is a violation), with the provenance gate in the reporting layer where authority is known (D7). +- **Scope stops at the network boundary.** Spec 087 is specified but not implemented (R8) and FR-018 forbids inventing a second refresh lifecycle, so this feature delivers the file-drop path end to end and leaves fetch to 087 (D11). SC-003's air-gapped install reaches full P1 capability on that path alone, which is the evidence the cut is in the right place. + +## Technical Context + +**Language/Version**: Go 1.25.5 module toolchain (`go.mod`). Backend-only; no frontend, tray, or Swift changes beyond status fields already rendered generically. +**Primary Dependencies**: **stdlib only for the new work** — `crypto/ed25519`, `crypto/sha256`, `encoding/json`, `syscall`. No `ed25519` usage exists in the tree today (R10). Existing: `go.etcd.io/bbolt` (state), `go.uber.org/zap`, Cobra (CLI). **No new dependencies.** +**Storage**: one new BBolt bucket (`TPABundleStateBucket`) in the existing database; no schema migration, no Bleve change. +**Testing**: `go test -race ./internal/...` (locally `internal/server` needs the CI skip regex); fixture-driven conformance tests for the sidecar format; `cmd/scan-eval --gate` in CI (`.github/workflows/eval.yml`). +**Target Platform**: Linux/macOS/Windows core binary, personal + server editions — pure `internal/`, edition-agnostic. `O_NOFOLLOW` is POSIX-only; Windows gets the same descriptor-level guarantee via `os.Root`-scoped opens (D5). +**Project Type**: Single Go project (core server) + one new leaf package. +**Performance Goals**: SC-008 — a database at the ceilings (≥500 RE2 rules) moves p95 scan latency <10% versus the 6-signature baseline, and load-plus-verify completes under one second. Verification is one Ed25519 check over ≤8 MiB; the cost is compilation, which is bounded by the rule ceiling. +**Constraints**: verify-before-parse (FR-006) is an ordering constraint on every candidate path; all patterns stay RE2-only; the embedded default is exempt from watermark and ratchet (FR-009) so scanning never stops. +**Scale/Scope**: ~5 packages — `internal/security/bundlesig` (new), `internal/security/scanner`, `internal/storage`, `internal/config`, `internal/telemetry` — plus `cmd/scan-eval`, `cmd/mcpproxy` (CLI status/rollback), the corpus, docs, and the CI signing job. + +## Constitution Check + +*GATE: must pass before Phase 0 and be re-checked after Phase 1.* — Constitution v1.1.0. + +| Principle | Assessment | +|-----------|------------| +| **I. Performance at Scale** | PASS. BM25 search and indexing untouched. Verification is one signature check per candidate load — not per scan. The rule ceiling (2000) is what bounds compile cost, and SC-008 measures the hot path directly rather than asserting it. | +| **II. Actor-Based Concurrency** | PASS. The activation swap is Spec 087 FR-010's existing atomic in-memory swap; this feature adds durable state written under a single BBolt transaction on the same path. No new goroutine except D5's bounded reader, which is scoped to one candidate read and abandoned rather than joined — stated as a residual, not hidden. | +| **III. Configuration-Driven Architecture** | PASS. New config: operator keys, operator baseline, operator epoch, active-authority declaration, `require_signed_bundle`. All config-resident **by design, not convenience** — FR-004b, FR-009b and FR-005a each turn on the config-write attacker being out of the threat model while the bundle-path attacker is in it. | +| **IV. Security by Default** | PASS, and this is the feature's whole subject. Verify-before-parse; ceilings before verification's own inputs; per-authority watermarks so neither authority can lock the other out; a config-resident active-authority declaration so file-drop cannot choose the authority; full-digest rollback authorization; a signature ratchet; degraded coverage rather than silent full trust whenever the trust basis is weaker than it looks. The one place the design accepts less than the spec's literal text is recorded in Complexity Tracking. | +| **V. Test-Driven Development** | PASS. Every SC has a named fixture; the sidecar format ships byte-level accept/reject fixtures (D3) so conformance is testable by construction, and D9's control asserts the gate itself is not vacuous. | +| **VI. Documentation Hygiene** | PASS. FR-022 is explicit and in scope: operator lifecycle (verify, drop, roll back, read status) and contributor rules (author a signature, provenance/licence, eval samples, how the gate blocks a bad corpus). | + +**Result**: PASS. One new leaf package, one new BBolt bucket, no new dependency. Re-checked after Phase 1 design: unchanged. + +## Key Decisions (normative for tasks) + +Resolved in [research.md](research.md); recorded here as the plan of record. + +| # | Decision | Where resolved | +|---|----------|----------------| +| D1 | Publication channel = **GitHub Releases on the corpus repo**, immutable per-sequence asset names, `latest` resolved via the Releases API (not a mutable file) | R-D1 | +| D2 | `require_signed_bundle` stays **opt-in indefinitely** — no scheduled default flip; the FR-009 ratchet already covers every install that has seen a signed bundle | R-D2 | +| D3 | Sidecar = one-line JSON, exactly five keys (`sidecar_version`, `algorithm`, `key_id`, `signature`, `sequence`), total deterministic rejection, Ed25519-only in v1 | R-D3, contracts/sidecar-format.md | +| D4 | `signatures[]` additive top-level section keyed by TPA id, many-to-one to `rules[]`, bidirectional referential integrity enforced at build, licence from an SPDX allowlist | R-D4, contracts/bundle-signatures-section.md | +| D5 | Read budget = `O_NOFOLLOW` open + descriptor `fstat` + ceiling from that `FileInfo`; time bound is a context-cancelled reader goroutine, never `SetReadDeadline` | R-D5 | +| D6 | Activation state = one new `TPABundleStateBucket`, one BBolt `Update`; LKG bytes `fsync`ed before commit, in-memory pointer published after | R-D6 | +| D7 | Telemetry = four **typed** backstop entries + the cross-field rule (`sequence` with `bundle_version=="other"` → reject); provenance gate in the reporting layer | R-D7 | +| D8 | `scan-eval --bundle` loads through the **production** loader and registers `BundleCheck`; same path as the activation self-check | R-D8 | +| D9 | Negative control = one signature whose sample no built-in check flags, evaluated over a corpus whose gated-malicious set is that sample alone; no-overlap asserted in CI | R-D9 | +| D10 | Structure = new leaf `internal/security/bundlesig/` for verification only; everything else extends `internal/security/scanner/` | R-D10 | +| D11 | Scope = **file-drop path end to end**; fetch stays Spec 087's. SC-007's fetch half is not demonstrable by this feature alone | R-D11 | + +## Project Structure + +### Documentation (this feature) + +```text +specs/101-tpa-db/ +├── spec.md # Feature spec (normative) +├── plan.md # This file +├── research.md # Phase 0: R1–R11 findings + D1–D11 decisions +├── data-model.md # Phase 1: state records, manifest additions, entity relations +├── quickstart.md # Phase 1: sign, drop, verify, roll back, read status +├── contracts/ +│ ├── sidecar-format.md # D3 wire format + accept/reject fixtures +│ ├── bundle-signatures-section.md # D4 signatures[] schema + integrity rules +│ └── bundle-status-surface.md # FR-020 fields across CLI / REST / Web UI +└── tasks.md # Phase 2 (/speckit.tasks) +``` + +### Source Code (repository root) — REAL paths + +```text +internal/ +├── security/ +│ ├── bundlesig/ # NEW leaf package (D10) — pure, no I/O, no globals +│ │ ├── sidecar.go # D3 parse + total rejection grammar +│ │ ├── verify.go # Ed25519 verify over raw bundle bytes; key-id agreement (FR-003) +│ │ └── testdata/ # byte-level valid + invalid sidecar fixtures +│ └── scanner/ +│ ├── tpa_bundle_source.go # candidate open (O_NOFOLLOW + fstat, D5); sidecar cap; +│ │ # verify-before-parse (FR-006, read once, never re-read); +│ │ # BundleInfo gains full digest, sequence, authority, epoch, +│ │ # signature_verified, staleness, last rejection reason (FR-020) +│ ├── tpa_bundle.go # rule-count ceiling; signatures[] parse + integrity (D4); +│ │ # sequence grammar (FR-002 decimal string, ≤ 2^64-1) +│ ├── tpa_bundle_state.go # NEW: per-authority watermark/ratchet/pin/deny-list/history +│ │ # read+write through storage (D6); epoch scoping (FR-005a) +│ ├── tpa_bundle_rollback.go # NEW: FR-010 full-digest rollback, pin + deny-list +│ ├── inprocess.go # coverageOK: degrade on unsigned-external and on the two +│ │ # FR-009a fallback conditions ONLY (R5) — the narrowest edit +│ └── bundled/scanner-bundle.json # seed corpus grows to ≥25 signatures / ≥8 classes (SC-004) +├── storage/ +│ ├── models.go # TPABundleStateBucket const +│ └── tpa_bundle_state.go # NEW: single-Update transaction over that bucket (D6) +├── config/ +│ └── config.go # SecurityConfig: operator keys, operator baseline, operator +│ # epoch, active-authority declaration, require_signed_bundle +└── telemetry/ + └── anonymity.go # scanV8TPAScanner: four typed entries + cross-field rule (D7) +cmd/ +├── scan-eval/ +│ ├── main.go # --bundle flag (D8) +│ └── gate.go # register BundleCheck; categoryCheck entries per new class; +│ # D9 control + its no-overlap assertion +└── mcpproxy/ # security status output (FR-020) + rollback command (FR-010) +.github/workflows/ +├── eval.yml # D2 job now scores the bundle (D8) +└── (corpus repo) # gate-then-sign-then-publish (D1) — not this repo +docs/ # FR-022 operator + contributor lifecycle +``` + +**Structure Decision**: one new leaf package for verification, everything else in place. `bundlesig` earns its own package on the Spec-085 `toolsig` argument — a pure, fixture-testable contract with an independent consumer (any publisher tool) — which does **not** apply to the loader, the state, or the status surface. + +## Test Strategy + +**Fixture-first, because almost every requirement here is a rejection boundary.** A rejection you cannot drive one unit past is not tested. + +- **Sidecar conformance** (D3): byte-level accept and reject fixtures in `bundlesig/testdata/` — missing key, unknown key, duplicate key, wrong type, trailing bytes, non-hex `key_id`, wrong-length signature, unknown algorithm, `sidecar_version != "1"`, malformed `sequence`. Each asserts a *specific* rejection reason, not merely failure, so two implementations can be compared on refusal as well as acceptance. +- **Verify-before-parse** (FR-006/SC-001): a candidate whose bytes are altered after the digest is computed must be refused **before** any pattern compiles — asserted by a compile counter, not by output alone. Plus the four SC-001 tamper classes (bundle byte, signature value, sidecar identity field, mismatched pair) and the explicit non-requirement (sidecar whitespace/key-order changes are NOT required to be rejected). +- **Ceilings** (SC-010a): both sides of every boundary — 8 MiB and 8 MiB+1, 2000 rules and 2001, 4 KiB and 4 KiB+1, and a read exceeding 5 s — with last-known-good serving throughout. +- **Race-free opening** (SC-011): FIFO swapped in between path resolution and open; symlink-to-FIFO; a never-written FIFO whose slot is released within the budget so a later legitimate update still activates. +- **Anti-downgrade** (SC-002/SC-002a): same-authority same-epoch comparisons; the max-sequence lockout and its epoch-bump recovery; the post-bump stale-epoch replay refusal (a2); the two-authority isolation (b); the authority-not-active refusal (b2); and the baseline-less operator authority activating **degraded** rather than fully trusted (c). +- **Rollback** (SC-010): the 12-hex-prefix collision fixture — an attacker-substituted, publisher-signed, current-epoch bundle sharing the abbreviated fingerprint but differing in the full digest — refused as a target mismatch; plus N-activation history replay across a process restart. +- **Crash consistency** (FR-011a): a fault injected between LKG persist, state commit and pointer publish, asserting every recovery lands on a consistent (bundle, watermark) pair, and specifically that a preserved rollback never loses its deny-list. +- **State substitution** (SC-012): a prepared `config.db` with a lowered watermark, a cleared ratchet and a forged pin — in each case the outside-the-data-dir anchors still hold and the install reports degraded. **No fixture may assert the ratchet/watermark/pin survive**; they are best-effort against this attacker by construction (R11). +- **Coverage degradation** (FR-007/FR-009a): the four states that must degrade, and — equally important — the two that must **not**: an ordinary fresh install on the embedded default, and an embedded default at or above the publisher watermark after an external file was removed. A false degrade silently disables `scan`-mode auto-approval fleet-wide. +- **Gate non-vacuity** (SC-005/D9): the control signature's removal turns the gate red with exit `exitGateBreach`, plus the standalone no-overlap assertion that keeps the control honest as built-in checks grow. +- **Telemetry backstop** (D7): each typed entry accepted; a free-string version rejected; the cross-field violation rejected; and an operator-built corpus reporting `other` with `sequence` omitted **even when it claims a genuine publisher version string**. +- **Air-gapped parity** (SC-003): the file-drop path with the network stack unavailable produces byte-identical behavior to a connected install's file-drop path. + +## Documentation & wiring checklist (Constitution VI) + +- `docs/features/security-quarantine.md` + the tool-scanner docs: database lifecycle, degraded-coverage states and what each means. +- New operator doc: verify, drop, roll back, read status; the air-gapped walkthrough. +- New contributor doc: author a signature, provenance/licence rules, eval-sample requirements, how the gate blocks a bad corpus. +- `docs/configuration.md`: operator keys, baseline, epoch, active authority, `require_signed_bundle`. +- `make swagger` after the status-surface change. +- `CLAUDE.md`: security-model line + Recent Changes entry. + +## Complexity Tracking + +| Violation | Why Needed | Simpler Alternative Rejected Because | +|-----------|------------|-------------------------------------| +| **Persisted anti-downgrade state is best-effort against an attacker who can substitute `config.db`** (spec §Threat-model boundary; SC-012) | The bundle path may be configured inside the data directory, so the in-scope bundle-path attacker may also reach the state store. BBolt is transactional and checksummed, **not** authenticated (R11). | Authenticating the state with a key outside the writable data directory means a second key-management story — key location, rotation, and its own compromise path — to protect state whose loss already degrades safely to the FR-009b seeded baseline and the config-resident anchors. The spec accepts this explicitly and SC-012 tests the boundary rather than a guarantee that does not exist. **The plan's obligation is to never present the ratchet, watermark, pin or deny-list as surviving that attacker** — and to keep the anchors that do survive genuinely outside the data dir. | +| **A read blocked on a stalled filesystem leaves a goroutine parked** (D5) | Go cannot interrupt a blocked read on a regular file; `SetReadDeadline` returns `ErrNoDeadline`, which is why FR-011 forbids specifying the bound that way. | Every alternative that truly kills the read is a subprocess or an OS-specific `aio` path — a large, platform-divergent surface to reclaim one goroutine. The property that matters is *the single-flight refresh slot is released and a rejection reason recorded*, so a legitimate update still activates; that is delivered, tested (SC-011c), and stated rather than implied. | +| **SC-007 is only half demonstrable by this feature** (D11) | SC-007 measures publish → active → fixture fires "within one daily refresh cycle on a fetch-enabled install". There is no refresh cycle: Spec 087 is unimplemented (R8), and FR-018 forbids building a second one here. | Implementing the fetch loop here would either duplicate 087 or absorb it, and absorbing it makes this feature the whole epic. The manual-drop half of SC-007 ("and immediately on manual drop") **is** demonstrated end to end, and SC-003 shows the air-gapped install reaches full P1 capability on the file-drop path alone. **This needs the maintainer's assent at the tasks stage**, because it means one success criterion carries forward rather than closing with this feature. | + +No other deviations: no new dependency, one new leaf package the verification contract justifies, one new BBolt bucket the crash-consistency requirement names. diff --git a/specs/101-tpa-db/quickstart.md b/specs/101-tpa-db/quickstart.md new file mode 100644 index 000000000..ac10283f1 --- /dev/null +++ b/specs/101-tpa-db/quickstart.md @@ -0,0 +1,148 @@ +# Quickstart: tpa-db + +**Feature**: `101-tpa-db` · Sign a corpus, drop it, verify it activated, roll it back, read the status. + +Everything here works with **no network** — that is SC-003, not a footnote. The fetch path is Spec 087's (plan D11). + +## 0. Build + +```bash +go build -o mcpproxy ./cmd/mcpproxy +``` + +Use an isolated instance for all of this — a high port and a scratch `--data-dir` **and** `--config`. The watermark, ratchet and pin are per-data-directory, so testing against your real one leaves state behind that is deliberately hard to undo. + +```bash +SCRATCH=$(mktemp -d) +cat > "$SCRATCH/config.json" <<'EOF' +{ "listen": "127.0.0.1:18701", "api_key": "dev", "mcpServers": [] } +EOF +``` + +## 1. Generate a signing key + +```bash +go run ./cmd/tpa-db keygen --out "$SCRATCH/keys" +# writes keys/operator.key (private, 0600) and keys/operator.pub +# prints the key id: the lowercase-hex SHA-256 of the 32-byte raw public key +``` + +The **key id is a fingerprint of the key**, never a label you choose (FR-003). That is what makes "the manifest names one publisher, the bytes are signed by another" detectable rather than cosmetic. + +## 2. Sign a bundle + +```bash +go run ./cmd/tpa-db sign \ + --bundle internal/security/scanner/bundled/scanner-bundle.json \ + --key "$SCRATCH/keys/operator.key" \ + --sequence 1 \ + --epoch 0 \ + --out "$SCRATCH/scanner-bundle.json" +# writes scanner-bundle.json and scanner-bundle.json.sig +``` + +`sign` refuses a bundle that fails the eval gate — FR-016 means a bundle that cannot pass is never signed, so the gate runs here rather than only in CI. + +Inspect the sidecar; it is one line, five keys, nothing else ([contract](contracts/sidecar-format.md)): + +```bash +cat "$SCRATCH/scanner-bundle.json.sig" | jq . +``` + +## 3. Configure the operator authority and drop the bundle + +An operator-signed bundle needs three config decisions, all of them **config-resident on purpose**: the config-write attacker is out of the threat model, the bundle-path attacker is in it. + +```json +{ + "security": { + "tpa_bundle_path": "/scanner-bundle.json", + "tpa_active_authority": "operator", + "tpa_operator_keys": [""], + "tpa_operator_baseline": 1, + "tpa_operator_epoch": 0, + "require_signed_bundle": true + } +} +``` + +Declaring `tpa_operator_baseline` matters more than it looks. Without it the authority runs **explicitly degraded** rather than silently trusted — with no baseline there is no anti-downgrade at all after a data-directory wipe, and an attacker who can write the bundle path could replay any older genuinely operator-signed bundle at full coverage (FR-009b). + +Start the proxy; the file watcher picks the bundle up without a restart. + +```bash +./mcpproxy serve --config "$SCRATCH/config.json" --data-dir "$SCRATCH/data" --log-level=debug +``` + +## 4. Verify it activated + +```bash +./mcpproxy security bundle -o json --config "$SCRATCH/config.json" --data-dir "$SCRATCH/data" | jq . +``` + +Expect `signature_verified: true`, `authority: "operator"`, `trust_epoch: 0`, `sequence: "1"`, a `full_digest`, `coverage_degraded: false`. + +**If `coverage_degraded` is true, read `degraded_reason` before anything else** — it distinguishes "you forgot the baseline" (`operator_authority_unbaselined`) from "the bundle is unsigned" (`unsigned_external_active`) from "we fell back to the embedded default" (`embedded_below_watermark`). A degraded install has `scan`-mode auto-approval suspended, which is otherwise invisible. + +## 5. Watch a downgrade be refused + +Sign the same corpus at a lower sequence and drop it: + +```bash +go run ./cmd/tpa-db sign --bundle … --sequence 1 --epoch 0 --out "$SCRATCH/old.json" +cp "$SCRATCH/old.json"* "$SCRATCH/" # overwrite the configured path +``` + +The sequence-1 artifact is now at or below the watermark, so it is refused and the current bundle keeps serving: + +```bash +./mcpproxy security bundle -o json … | jq '.last_rejection' +# { "reason": "downgrade", "at": "…" } +``` + +Nothing about this is fatal — every rejection keeps last-known-good. That is the point: a refused candidate must never leave the scanner with an empty or partial rule set. + +## 6. Roll back — by digest, not by path + +Read the history to learn the digest you want: + +```bash +./mcpproxy security bundle history -o json … | jq '.[] | {sequence, full_digest, activated_at}' +``` + +Then roll back naming **the full digest**: + +```bash +./mcpproxy security bundle rollback \ + --path "$SCRATCH/old.json" \ + --digest … +``` + +The path is a *hint about where to find those bytes*; the digest is the authorization. Rollback is the one path that deliberately bypasses the watermark, so identifying its target by an attacker-writable path would hand the attacker the choice of what the bypass lands on. Supply the wrong digest and it is refused `rollback_target_mismatch` — try it, it is one command: + +```bash +./mcpproxy security bundle rollback --path "$SCRATCH/old.json" --digest 0000… # refused +``` + +**The 12-hex short fingerprint is not accepted here.** It is 48 bits — fine as a human label, not as authorization for a watermark bypass where the attacker chooses the substituted bytes. + +After a rollback the artifact is **pinned** and the release you rolled away from is **deny-listed**, so the next refresh cycle cannot re-activate the very thing you just rejected. Clear the pin to return to normal watermark-governed activation: + +```bash +./mcpproxy security bundle unpin … +``` + +## 7. Air-gapped check (SC-003) + +Steps 1–6 with the network unavailable behave identically. That is the criterion — the file-drop path is not a degraded mode, it is the primary one, and the fetch path adds convenience rather than capability. + +## 8. Contributor loop + +```bash +go run ./cmd/tpa-db validate --bundle # schema + signatures[] integrity + licences +go run ./cmd/scan-eval --gate --bundle # recall ≥ 0.90 gated, hard-negative FP ≤ 0.05 +``` + +`validate` enforces the [signatures[] contract](contracts/bundle-signatures-section.md) in both directions — every signature names a real rule, **and** every gating rule is named by exactly one signature. The second direction is the one that is easy to skip and the one that matters: without it a gating rule can ship with no provenance, no licence and no eval evidence. + +`--gate` is the same code path activation runs (plan D8), so a bundle that passes here is one that will activate — the CI layer and the activation layer cannot drift apart. diff --git a/specs/101-tpa-db/research.md b/specs/101-tpa-db/research.md new file mode 100644 index 000000000..c3d519a9f --- /dev/null +++ b/specs/101-tpa-db/research.md @@ -0,0 +1,182 @@ +# Phase 0 Research: Versioned, Signed, Offline-First TPA Signature Database + +**Feature**: `101-tpa-db` · **Spec**: [spec.md](spec.md) · **Date**: 2026-08-26 + +Two kinds of entry below. **D**ecisions resolve a `[NEEDS CLARIFICATION]` marker or a "MUST be specified in design" instruction from the spec. **R**esults are mechanical findings about the tree as it stands — each one is a fact a task will trip over if the plan does not state it. + +--- + +## Mechanical findings (the tree as it is today) + +### R1 — The loader is in `internal/security/scanner/`, not the package Spec 087's plan named + +Spec 087's plan proposes a new package `internal/security/bundle` owning the file lifecycle. **That package does not exist.** What shipped (Spec 086) lives in `internal/security/scanner/`: + +| File | Role | +|---|---| +| `tpa_bundle_source.go` | `ConfigureBundle(path, logger)`, `BundleStatus() BundleInfo`, `loadEmbeddedBundle`, `loadBundleFromFile`, `loadBundleWithInfo`, and the `storeBundle`/`snapshotBundle` global-state pair | +| `tpa_bundle.go` | `rawBundle`/`rawRule`/`rawSkip` wire types, `bundleMetadata`, `loadBundleCheck`, `checkBundleVersion`, `BundleCheck` (the `detect.Check`) | +| `registry_bundled.go` | embedded default via `go:embed` | +| `bundled/scanner-bundle.json` | the shipped corpus | + +**Consequence for the plan**: this feature extends those files rather than creating 087's package. Reading 087's plan as a description of the current tree is the single most likely source of wrong file paths in tasks — treat 087's *spec* as normative for the refresh lifecycle and its *plan* as superseded on structure. + +### R2 — Candidate reads are completely unbounded today + +`tpa_bundle_source.go` reads a candidate with a bare `os.ReadFile(path)` carrying a `//nolint:gosec` comment that reasons "operator-configured path, same trust level as `mcp_config.json`". There is **no size check, no file-type check, no deadline, and no re-read protection**. `loadBundleCheck` (`tpa_bundle.go`) applies **no rule-count limit**. FR-011's ceilings are therefore entirely new code, not a tightening — and the existing nolint rationale is exactly the assumption FR-011 overturns, since the bundle path is in the threat model while the config path is not. That comment must be updated, not just the code beside it. + +### R3 — `BundleInfo.Fingerprint` is 48 bits + +`tpa_bundle_source.go` computes `Fingerprint: hex.EncodeToString(sum[:])[:12]` — the first 12 hex characters of the SHA-256. FR-010 requires the **full** digest for rollback authorization, so `BundleInfo` needs a second field rather than a widened one: the short form is load-bearing in existing operator-facing output and in `logBundle`. + +### R4 — The bundle has nowhere to put per-signature metadata + +Top-level keys of the shipped `scanner-bundle.json`: `bundle_version`, `generated_from`, `rules`, `schema_version`, `signature_count`, `skipped`. A `rules[]` entry is keyed `category, confidence, detector, engine, flags, id, indicators, level, pattern, severity, target, type`. **No provenance field, no license field, anywhere.** This is what FR-001a exists to fix, and it means FR-012's "each signature MUST carry provenance and a redistributable license" cannot be satisfied by any rearrangement of today's shape. + +### R5 — `coverageOK` is derived from mere bundle presence + +`internal/security/scanner/inprocess.go`: `coverageOK = bundlePresent && result.Coverage.ChecksFailed == 0`. Nothing consults signature state, authority, or watermark. FR-007's degradation rule and FR-009a's fallback demotion both land on this one expression, which makes it the narrowest and most safety-critical edit in the feature. + +### R6 — The eval gate cannot see the bundle at all + +`cmd/scan-eval/gate.go`'s `gateChecks()` returns only the four built-in `detect.Check`s; `scanner.BundleCheck` — which production **does** append in `inprocess.go` — is absent, and `scan-eval` never loads a bundle. `categoryCheck` maps a category to a check id, and `gatedCategory()` enforces a category **only** when its mapped check id is registered. So today FR-016/SC-005 would pass with the corpus entirely absent. This is FR-016a, and it is a prerequisite for the seed-corpus work rather than a parallel task: growing the corpus while the gate cannot score it is how a vacuous pass ships. + +### R7 — The telemetry backstop rejects strings and booleans by construction + +`internal/telemetry/anonymity.go` `scanV8TPAScanner` whitelists the `tpa_scanner` sub-object to `{scans_completed, scans_failed, scans_with_findings, findings}` and rejects anything else with `"carries a key outside the whitelist"`. A string `bundle_version`/`source` or a boolean `signature_verified` is refused **before transmit**. FR-021's typed extension is therefore a change to the backstop itself, and the cross-field invariant FR-021 names (`sequence` present while `bundle_version` is `other` → reject) is enforceable right there on the payload. + +### R8 — Spec 087 is specified but not implemented + +No `internal/security/bundle`, no `security.bundle` config section, no refresh ticker. `SecurityConfig` carries exactly one bundle-related field, `TPABundlePath` (`config.go:2751`), plus the `MCPPROXY_TPA_BUNDLE_PATH` env override resolved in `EffectiveTPABundlePath`. **Sequencing consequence**: FR-018 forbids this feature from inventing a second refresh lifecycle, but there is no first one to insert stages into. The spec's own Assumptions anticipate this — US1's signature/sequence verification is reachable through the existing Spec-086 `ConfigureBundle` hot-reload/file-drop path — so the plan phases the fetch-dependent parts behind 087 and delivers the file-drop path first. + +### R9 — Persistence has a home but no bucket + +BBolt buckets are declared in `internal/storage/models.go` (`UpstreamsBucket`, `ToolApprovalBucket`, `MetaBucket`, …). FR-011a's one-transaction requirement maps cleanly onto a single BBolt `Update` over a new bucket; nothing in the current schema needs to change. + +### R10 — Ed25519 is stdlib and unused here + +No `crypto/ed25519` import exists in `internal/` or `cmd/` (the only `ed25519` matches are SSH-key *path strings* in the sensitive-data detectors). Verification is stdlib — **no new dependency**, consistent with the repo's stated stance. + +### R11 — The `config.db` trust boundary is a boundary, not a guarantee + +BBolt is transactionally consistent and checksummed against corruption; it is not authenticated. Spec §Threat-model boundary already says so. Every persisted item this feature adds — watermark, ratchet, pin, deny-list, activation history — inherits that. The plan must never present them as surviving a prepared-database substitution (SC-012 asserts exactly this). + +--- + +## Decisions + +### D1 — Publication channel: **GitHub Releases on the corpus repo** (resolves FR-017's marker) + +**Decision**: publish artifact pairs as GitHub Release assets on the tpa-db corpus repository, with immutable per-release asset names (`scanner-bundle-.json` + `.sig`) and a `latest` reference resolved through the Releases API rather than a mutable file. + +**Rationale**: the alternative (a static path under mcpproxy.app) puts corpus publication on the Cloudflare Pages deploy pipeline, which is already the site's critical path and whose failure mode is silent — [the appcast feed has gone stale this way before](../../docs/). Releases give immutable asset URLs, a publication timestamp SC-006's server-side age join needs, and a signing job that lives in the same repo as the corpus it signs, so FR-016's "never signed or published without a green gate" is one workflow rather than a cross-repo handoff. + +**Alternatives considered**: mcpproxy.app static path — rejected for the silent-failure coupling above and because it separates the artifact from the gate that must precede it. A dedicated CDN — rejected as unjustified operational surface for a file fetched at most daily per install. + +**Consequence**: `latest` is a resolution step, not a URL, so the fetch path (087) must handle an API response rather than a fixed href. Note this in the 087 handoff. + +### D2 — `require_signed_bundle` stays **opt-in indefinitely** (resolves the Assumptions marker) + +**Decision**: no scheduled default flip. The setting stays default-off; the ratchet does the work. + +**Rationale**: the spec's own note narrows this almost to nothing — FR-009's ratchet means any install that has ever activated a signed bundle already refuses unsigned drops, and FR-007 suspends auto-approval while an unsigned bundle is active. A default flip would therefore change behavior **only** for installs that have never seen a signed bundle, which is precisely the air-gapped/self-built-corpus persona FR-004a and FR-009b exist to support. Flipping the default would break that persona to harden a case the ratchet already hardens. + +**Alternatives considered**: flip after two release cycles — rejected: it buys nothing the ratchet does not already provide and breaks a supported persona. Flip with an escape hatch — rejected as the same thing with more config. + +**Revisit trigger**: if telemetry (SC-006) ever shows a material population running unsigned external bundles long-term, that is evidence for a flip — and it is the evidence a future decision would need. Not now. + +### D3 — Sidecar wire format (resolves FR-003's "MUST be specified in design") + +**Decision**: a JSON object, one line, with exactly five required keys and no others. Full grammar and fixtures in [contracts/sidecar-format.md](contracts/sidecar-format.md). + +```json +{"sidecar_version":"1","algorithm":"ed25519","key_id":"<64 lowercase hex>","signature":"","sequence":""} +``` + +- `sidecar_version` — exactly `"1"` in v1; any other value is a hard rejection (not a fallback). +- `algorithm` — exactly `"ed25519"`. **An unknown algorithm is a hard rejection, never a skip.** +- `key_id` — the canonical full-length fingerprint of the public key: lowercase hex SHA-256 of the 32-byte raw Ed25519 public key. Not an operator-chosen label (FR-003). +- `signature` — standard base64 with padding, decoding to exactly 64 bytes. +- `sequence` — the decimal-string form of FR-002's counter, duplicated here so a mismatch against the signed manifest is detectable **before** the bundle is parsed. + +Rejection is deterministic and total: a missing key, an unknown key, a duplicate key (JSON with repeated names), a wrong type, trailing bytes after the object, a `key_id` that is not 64 lowercase hex, a `signature` that does not decode to 64 bytes, or a `sequence` failing FR-002's grammar. **No leniency anywhere** — this is the artifact an independent publisher must reproduce exactly. + +**Rationale**: JSON because the bundle is JSON and the repo already parses it strictly; five fields because each one is consumed before verification and nothing else is; `sequence` duplicated because it is the one manifest field whose disagreement with the sidecar is cheap to detect early and is otherwise only visible after parsing the thing the signature protects. + +**Alternatives considered**: a binary/`minisign`-style format — rejected as a second parser and a second toolchain for publishers, for no property JSON lacks here. Embedding the signature in the bundle — rejected outright: it makes the signed bytes self-referential. + +### D4 — `signatures[]` section shape (resolves FR-001a) + +**Decision**: an additive top-level `signatures[]` array keyed by TPA id, with rules referencing signatures many-to-one. Full schema in [contracts/bundle-signatures-section.md](contracts/bundle-signatures-section.md). + +```json +{"signatures":[{"id":"TPA-2026-0001","category":"instruction_injection","gating":true, + "provenance":{"source":"https://…","published":"2026-01-14"}, + "license":"CC-BY-4.0","rule_ids":["r-hidden-important-block"]}]} +``` + +Build-time referential integrity, enforced in **both** directions and rejected whole on failure: every `signatures[]` entry names ≥1 existing rule id; every rule with `gating: true` is named by exactly one signature entry; TPA ids are unique; licenses come from a fixed allowlist of redistributable SPDX ids. + +**Rationale**: additive keeps contract §4 forward-compat, so a v0.1 loader ignores it (FR-001a). Many-to-one because a campaign class routinely needs several patterns — one hidden-instruction signature already implies distinct rules for `` blocks, HTML comments and zero-width runs. Keying by TPA id rather than by rule is what makes SC-004's "at least 25 signatures spanning 8 classes" a countable property. + +**Alternatives considered**: per-rule `provenance`/`license` fields — rejected: it duplicates the record across every rule implementing one campaign and gives no place to hang gating intent or the class identity SC-004 counts. + +### D5 — Read budget mechanism (resolves FR-011's "design MUST name the concrete mechanism") + +**Decision**: three distinct mechanisms, because one does not fit all three cases. + +| Case | Mechanism | +|---|---| +| Local file open | `os.OpenFile(path, os.O_RDONLY\|syscall.O_NOFOLLOW, 0)`, then `f.Stat()` on the **descriptor**, refuse unless `Mode().IsRegular()`, then `io.ReadFull` bounded by the ceiling read from that same `FileInfo` | +| Local file read time | a goroutine performing the read, selected against a `context.WithTimeout(5s)`; on expiry the **slot is released and a rejection reason recorded** while the goroutine is abandoned | +| Fetch body | `http.Client` timeout + request context + `io.LimitReader(body, ceiling+1)` | + +The abandoned-goroutine case is stated as an accepted residual, not hidden: a read blocked on a stalled network filesystem cannot be interrupted in Go, so the guarantee is *the refresh path stays available*, not *the goroutine dies*. `O_NOFOLLOW` is unavailable on Windows; there the no-follow property comes from `os.Root`-scoped opens where the containing directory is trusted, and the descriptor-level `fstat` check is unchanged. + +**Rationale**: FR-011 already rules out `SetReadDeadline` (unsupported on regular files — returns `ErrNoDeadline`). The descriptor-level check is the only form that closes the stat/open substitution window. + +### D6 — Crash-consistent activation state = **one BBolt bucket, one `Update`** (resolves FR-011a) + +**Decision**: a new `TPABundleStateBucket` in `internal/storage/models.go` holding one record per authority plus one activation-history list, written in a single BBolt `Update` transaction. The recoverable last-known-good bundle bytes are `fsync`ed to the data directory **before** the transaction commits and before the in-memory pointer is published. + +Ordering, normatively: persist LKG bytes → commit the state transaction → publish the in-memory pointer. A crash before the commit leaves the old (bundle, watermark) pair; a crash after leaves the new one; there is no interleaving that yields a new bundle with an old watermark or vice versa. + +**Rationale**: BBolt gives the single-transaction property for free and is already the durable store for approvals and quarantine — the state this record sits beside. R9 confirms no schema change is needed. The deny-list and the watermark's setting-fingerprint are members of the same record precisely because FR-011a names the crash that splits them. + +### D7 — Telemetry: typed whitelist entries + a payload-level cross-field invariant (resolves FR-021) + +**Decision**: extend `scanV8TPAScanner` with four **typed** entries rather than relaxing it: integer `sequence`, enum-bounded `source ∈ {embedded, file, fetch}`, boolean `signature_verified`, and enum-bounded `bundle_version` (publisher release line, or the literal `other`). Add one cross-field rule the backstop can enforce on the payload alone: **`sequence` present while `bundle_version == "other"` is a violation.** + +The provenance gate lives in the reporting layer, where authority is known: verbatim version + sequence are reported only when the active bundle's provenance is the publisher authority (the embedded default, or an external bundle signature-verified against a publisher key). Every other bundle reports `other` with `sequence` omitted **regardless of the version string it claims** — a string allowlist alone is defeated by an operator corpus copying a genuine publisher version verbatim. + +**Rationale**: R7 shows the backstop rejects the naive addition outright, and FR-021 explicitly forbids relaxing it to free strings. Typed entries keep the backstop's purpose (no near-unique value reaches the payload) while making the new fields expressible. + +### D8 — `scan-eval` loads the bundle through the **production** loader (resolves FR-016a) + +**Decision**: `cmd/scan-eval` gains a `--bundle ` flag defaulting to the embedded default, loads it via `scanner`'s existing loader (never a parallel parser), and registers the resulting `BundleCheck` in `gateChecks()`. The same loading path is what the Spec-087 activation self-check runs. + +**Rationale**: FR-016a names the shared path explicitly, and the reason is drift: two parsers would let CI and activation disagree while SC-005 claims they are independent checks of the same artifact. Sharing the loader makes them independent *runs* of one implementation, which is the property actually wanted. + +**Consequence**: `gateChecks()` stops being a pure function of the built-in set, so its "MUST mirror" doc comment needs rewriting rather than deleting. + +### D9 — Control signature for the negative control (resolves FR-016b) + +**Decision**: designate one control signature whose canonical malicious sample is verified to be hard-flagged by **no** built-in check and by no other bundle signature, and evaluate the control over a corpus whose gated-malicious set is **that sample alone**, so removal drives recall to 0 and `decide()` fails unambiguously. + +The no-overlap property is asserted in CI as its own test, so a later built-in check gaining coverage of the control sample turns the control **red** rather than silently re-vacuating it. + +**Rationale**: FR-016b spells out both vacuity paths. The "restrict to the control's category" shortcut is specifically rejected there — ten gated samples in a category land on exactly 0.90 after one miss, and `decide()` fails only on `< minRecall`. + +### D10 — Structure: extend `internal/security/scanner/`, add `internal/security/bundlesig/` for verification only + +**Decision**: the loader, ceilings, activation and status work extend the existing `scanner` files (R1). Signature verification and sidecar parsing go in a new leaf package `internal/security/bundlesig/` — pure functions over bytes, no I/O, no global state, no scanner imports. + +**Rationale**: verification is the one piece with a genuinely independent contract (D3's format), it is where an independent implementation would be compared byte-for-byte, and a leaf package is what makes the fixture-driven conformance tests cheap. Everything else is already `scanner`'s job and moving it would be churn. Deliberately **not** 087's `internal/security/bundle` — that name describes a lifecycle that shipped elsewhere. + +### D11 — Phasing against unimplemented Spec 087 (resolves the R8 sequencing problem) + +**Decision**: this feature delivers the **file-drop path end to end** (format, signing, verification, ceilings, watermark, rollback, status, seed corpus, gate) and stops at the network boundary. The fetch path stays 087's, and this spec's FR-008 stages are written so 087 inserts them without a second lifecycle. + +**Rationale**: FR-018 forbids a second refresh lifecycle, and R8 shows there is no first one. Building the fetch loop here would either duplicate 087 or absorb it — both worse than shipping the artifact and trust story that 087 then consumes. SC-003 (air-gapped install reaches full P1 capability) is satisfied entirely by the file-drop path, which is the strongest signal that this is the right cut. + +**Consequence**: SC-007 ("within one daily refresh cycle on a fetch-enabled install") is **not** fully demonstrable by this feature alone; its manual-drop half is. Say so in the plan rather than letting a task discover it. From 78c683c38764b0dd3f3e016467bd957f0a85e112 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 26 Aug 2026 18:22:22 +0300 Subject: [PATCH 19/20] docs(specs): close cross-model review findings in the 101-tpa-db plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (opencode/gpt-5.6) of the plan, research and contracts, plus Go semantics checked empirically rather than asserted. Three P1s. **open(2) blocks on a FIFO before fstat can see it.** D5 specified O_NOFOLLOW + descriptor fstat + a context-cancelled reader, which looks airtight and is not: opening a writer-less FIFO blocks inside open itself, before fstat, before the reader goroutine, before any timeout exists to observe it. SC-011(c) — the never-written FIFO whose slot must be released within the budget — could not pass by construction. O_NONBLOCK is now part of the open and is cleared once fstat has confirmed a regular file, so an ordinary read keeps blocking semantics. **The last-known-good bundle needed content addressing, not just an fsync.** D6 said "persist LKG bytes, commit, publish" and claimed a pre-commit crash leaves the old (bundle, watermark) pair. False when the bytes land at a fixed filename: overwriting before the commit leaves NEW bytes under OLD state, the same split the requirement forbids, from the other side. LKG files are now tpa-lkg/.json referenced by digest from the state record, so writing a new one never destroys the old; plus the directory fsync (a file fsync does not make its directory entry durable) and startup reconciliation of orphans. **Releasing the single-flight slot leaked an fd per refresh.** Found independently on both sides. Abandoned readers are now capped at one per path. Also corrected: - os.Root does NOT give the no-follow property on Windows — it blocks path escape but still follows symlinks resolving inside the root. Either handle-based opens with reparse-point controls, or the weaker guarantee written down. The candidate open is now build-tagged POSIX/Windows files, since syscall.O_NOFOLLOW does not compile there at all. - GitHub Release assets are mutable (delete and re-upload under the same name), so "immutable per-release asset names" asserted something the platform does not provide. Content-addressed names + a signed release index, cacheable at a stable URL with ETag and refresh jitter, since a fleet behind one NAT can exhaust the unauthenticated API limit in lockstep. - D2 is REOPENED rather than answered. "The ratchet does the work" describes installs that are already safe; the ratchet only arms after a first signed activation, so fresh installs are exactly the unprotected population. The recommendation is default-on with an explicit opt-out, but that breaks existing 086 unsigned droppers on upgrade, which is a maintainer's call, not mine. - The loader is now source-neutral from day one — Candidate{bytes, source} — so spec 087's fetch path cannot fork the verifier, which is the one component that must never exist twice. - SC-007's fetch clause moves to 087 in spec.md as an amendment task. A success criterion no fixture can exercise is a spec defect; assent does not make it testable. - Telemetry validates the whole tuple, not one direction, and carries schema_version + sequence instead of a per-release enum a shipped binary cannot know (which would classify every new corpus as "other" until the binary updates). Contract fixes, with the Go claims verified against Go 1.25 rather than assumed: - Rejection rows now have explicit first-match-wins precedence; they overlap ({"foo":1} is both missing-field and unknown-field) and without ordering two conforming implementations report different reasons. - Duplicate keys need their own scan. Measured: {"algorithm":"ed25519", "algorithm":"rsa"} decodes with err=nil and algorithm=="rsa", and DisallowUnknownFields does NOT catch it. On the field that pins the algorithm. - base64 must be canonical and strict. Measured: StdEncoding accepts embedded newlines and an 88-char string that does not round-trip; Strict() rejects it. - sequence dropped from the sidecar. Its stated rationale — detectable "before parsing" — was wrong, since detecting a mismatch requires parsing the manifest. It bought no pre-parse check and added an inconsistency state. - "Gating" is now mechanical (hard tier) rather than intent-prose: rules[] has no gating field, so the reverse integrity rule was unenforceable and a hard-tier rule could dodge eval evidence via a gating:false signature. - Every gating category must resolve to a registered categoryCheck, enforced at build. The contract explained that hazard and then did not check it. - last_rejection gains a detail field it was already referencing. --- .../contracts/bundle-signatures-section.md | 9 ++- .../contracts/bundle-status-surface.md | 4 +- specs/101-tpa-db/contracts/sidecar-format.md | 34 +++++---- specs/101-tpa-db/plan.md | 25 ++++--- specs/101-tpa-db/research.md | 74 ++++++++++++++++--- 5 files changed, 104 insertions(+), 42 deletions(-) diff --git a/specs/101-tpa-db/contracts/bundle-signatures-section.md b/specs/101-tpa-db/contracts/bundle-signatures-section.md index 7ac2f3604..89257343f 100644 --- a/specs/101-tpa-db/contracts/bundle-signatures-section.md +++ b/specs/101-tpa-db/contracts/bundle-signatures-section.md @@ -64,12 +64,15 @@ signatures[] ──1─────*── rules[] A bundle failing any of these is **rejected whole**, exactly like any other contract violation: 1. every `signatures[]` entry names ≥1 rule id that exists in `rules[]`; -2. every rule with gating intent is named by **exactly one** signature entry — not zero (an ungoverned gating rule has no provenance and no eval evidence) and not two (ambiguous campaign attribution); +2. every **hard-tier** rule is named by exactly one `gating: true` signature entry — not zero (an ungoverned gating rule has no provenance and no eval evidence) and not two (ambiguous campaign attribution). + + **"Gating" must be mechanical, because `rules[]` has no such field.** Its keys are `category, confidence, detector, engine, flags, id, indicators, level, pattern, severity, target, type` — nothing that says "this rule gates approvals". Left as prose about intent, the check is unenforceable and a hard-tier rule can evade attribution entirely, or be attached to a `gating: false` signature and thereby dodge the FR-013 eval-evidence requirement while still driving the approval gate at runtime. The build MUST therefore derive it from `level` — a rule is gating **iff** it emits at the hard tier, which is exactly what FR-014 says gating means and exactly what the eval gate and the `scan`-mode approval gates score — and MUST reject any bundle where a hard-tier rule's owning signature has `gating: false`. `signatures[].gating` is then a declaration that is *checked* against the rules it names, not an independent switch; 3. `id` values are unique; 4. `license` is in the allowlist; -5. every `gating: true` signature has its FR-013 eval pair in the frozen dataset — ≥1 labeled gated-malicious sample and ≥1 category-matched hard negative following the `hn__*` naming the dataset validator enforces. +5. every `gating: true` signature has its FR-013 eval pair in the frozen dataset — ≥1 labeled gated-malicious sample and ≥1 category-matched hard negative following the `hn__*` naming the dataset validator enforces; +6. every `gating: true` signature's `category` resolves to a **registered** `categoryCheck` entry in the gate. §5 below explains why an unmapped category is scored vacuously; that explanation is not enforcement, so this rule is what actually stops one shipping. A bundle naming a category the gate cannot enforce is rejected at build, not merely noted. -Direction 2 is the one a naive implementation drops, and it is the one that matters: checking only "signatures point at real rules" lets a gating rule ship with no campaign record at all, which is the state the corpus is in today. +Direction 2 is the one a naive implementation drops, and it is the one that matters: checking only "signatures point at real rules" lets a gating rule ship with no campaign record at all, which is the state the corpus is in today. Rule 6 is the same failure one level up — a campaign record that exists but that the gate silently declines to score. ## 5. Interaction with the eval gate diff --git a/specs/101-tpa-db/contracts/bundle-status-surface.md b/specs/101-tpa-db/contracts/bundle-status-surface.md index e69cf09d0..64c08c8f8 100644 --- a/specs/101-tpa-db/contracts/bundle-status-surface.md +++ b/specs/101-tpa-db/contracts/bundle-status-surface.md @@ -21,7 +21,7 @@ Existing fields keep their meaning and their names. `Fingerprint` stays the 12-h | `staleness_days` | **new** | derived from `generated_at` | | `coverage_degraded` | **new** | bool | | `degraded_reason` | **new** | enum, §3 | -| `last_rejection` | **new** | `{reason, at}`, §2 — retained across a successful load so "what did it refuse, and when" survives | +| `last_rejection` | **new** | `{reason, detail, at}`, §2 — retained across a successful load so "what did it refuse, and when" survives. `detail` is a short human string carrying the part the enum cannot (which ceiling was exceeded and by how much, which key id was untrusted, which epoch was expected); it is never parsed, and `reason` is the only field anything branches on | ## 2. Rejection reasons @@ -36,7 +36,7 @@ The vocabulary an operator reads to tell a tamper attempt from a stale epoch fro | `authority_not_active` | verified against a real key of the **non-active** authority (FR-004b) | | `rollback_target_mismatch` | the loaded artifact's full digest ≠ the digest the operator named | | `gate_failure` | activation self-check (Spec 087 FR-008) rejected the candidate | -| `ceiling_exceeded` | over one of FR-011's four limits (which one is in the message) | +| `ceiling_exceeded` | over one of FR-011's four limits — which limit, and the observed value, in `detail` | | `not_regular_file` | candidate was a FIFO/device/socket, or a symlink to one | | `read_timeout` | the 5 s read budget expired; the single-flight slot was released | | `sidecar_*`, `key_*`, `signature_invalid`, `sequence_mismatch` | the sidecar grammar — see [sidecar-format.md](sidecar-format.md) §3 | diff --git a/specs/101-tpa-db/contracts/sidecar-format.md b/specs/101-tpa-db/contracts/sidecar-format.md index 3980e320e..3ecd085e1 100644 --- a/specs/101-tpa-db/contracts/sidecar-format.md +++ b/specs/101-tpa-db/contracts/sidecar-format.md @@ -10,10 +10,10 @@ A single JSON object. UTF-8. No BOM. Trailing newline optional and ignored; **an File naming: `.sig` beside the bundle (e.g. `scanner-bundle.json.sig`). -## 2. Fields — exactly these five, no others +## 2. Fields — exactly these four, no others ```json -{"sidecar_version":"1","algorithm":"ed25519","key_id":"3f0a…64hex","signature":"MEUCIQ…","sequence":"101"} +{"sidecar_version":"1","algorithm":"ed25519","key_id":"3f0a…64hex","signature":"<88 canonical base64 chars>"} ``` | Field | Type | Accepted values | @@ -21,38 +21,43 @@ File naming: `.sig` beside the bundle (e.g. `scanner-bundle.jso | `sidecar_version` | string | exactly `"1"` | | `algorithm` | string | exactly `"ed25519"` | | `key_id` | string | exactly 64 lowercase hex chars — SHA-256 of the 32-byte raw Ed25519 public key | -| `signature` | string | standard base64 **with** padding, decoding to exactly 64 bytes | -| `sequence` | string | `^[1-9][0-9]{0,19}$`, value ≤ 2^64−1 | +| `signature` | string | **exactly 88 characters** of canonical RFC 4648 standard base64 ending `==`, decoding to exactly 64 bytes | `key_id` is a fingerprint **of the key itself**, never an operator-chosen label (FR-003). That is what makes "the manifest names one publisher while the bytes are signed by another" detectable rather than cosmetic. -`sequence` is duplicated from the manifest deliberately: it is the one manifest field whose disagreement is cheap to detect *before* parsing the thing the signature protects. +**`sequence` is deliberately NOT duplicated here.** An earlier draft carried it, reasoning that a manifest disagreement would be "cheap to detect before parsing". That rationale was wrong: detecting a disagreement requires parsing the manifest, which happens only *after* verification, so the duplicate bought no pre-parse check and added a whole inconsistency state — plus two rejection rows — for nothing. The sequence lives in the signed manifest, where it is authenticated, and nowhere else. + +**Canonical base64 is a requirement, not a formality.** Go's default `base64.StdEncoding` is lenient in two ways that both break the "two implementations reject identically" property this contract exists to provide: it silently accepts **embedded newlines** inside the value, and it accepts non-canonical trailing bits — an 88-character string that decodes without error yet does **not** round-trip to itself. Verified against Go 1.25: `StdEncoding.DecodeString` returns `err=nil` for both, while `StdEncoding.Strict().DecodeString` rejects the non-canonical case at the offending byte. Implementations MUST therefore use the strict decoder **and** assert that re-encoding the decoded bytes reproduces the input exactly. ## 3. Rejection grammar — total, deterministic, and reason-bearing Every row is a hard rejection. There is no leniency, no fallback and no skip anywhere in this table. Each rejection records its own reason (surfaced via FR-020's `LastRejection`), because a conformance test that only asserts "failed" cannot show two implementations agree. +**Rows are evaluated in the numerical order below and the FIRST match wins.** The conditions deliberately overlap — `{"foo":1}` satisfies both "a required key is missing" and "a key outside the four" — so without a stated precedence two conforming implementations could report different reasons for the same input, which is exactly the divergence the reason codes exist to rule out. + | # | Condition | Reason code | |---|---|---| | 1 | file exceeds 4 KiB (checked **before** parsing) | `sidecar_too_large` | | 2 | not a single JSON object / trailing bytes | `sidecar_malformed` | | 3 | a required key is missing | `sidecar_missing_field` | -| 4 | any key outside the five | `sidecar_unknown_field` | +| 4 | any key outside the four | `sidecar_unknown_field` | | 5 | a key appears more than once | `sidecar_duplicate_field` | | 6 | any value has the wrong JSON type | `sidecar_type` | | 7 | `sidecar_version != "1"` | `sidecar_version_unsupported` | | 8 | `algorithm != "ed25519"` | `sidecar_algorithm_unsupported` | | 9 | `key_id` not 64 lowercase hex | `sidecar_key_id_malformed` | -| 10 | `signature` not valid padded base64, or not 64 bytes decoded | `sidecar_signature_malformed` | -| 11 | `sequence` fails the grammar or exceeds 2^64−1 | `sidecar_sequence_malformed` | -| 12 | `key_id` not in the active authority's trust set | `key_not_trusted` | -| 13 | signature does not verify over the raw bundle bytes | `signature_invalid` | -| 14 | manifest `key_id` ≠ sidecar `key_id` ≠ verifying key's id | `key_id_mismatch` | -| 15 | manifest `sequence` ≠ sidecar `sequence` | `sequence_mismatch` | +| 10 | `signature` is not exactly 88 canonical base64 chars, does not decode strictly, does not round-trip, or is not 64 bytes decoded | `sidecar_signature_malformed` | +| 11 | `key_id` not in the active authority's trust set | `key_not_trusted` | +| 12 | signature does not verify over the raw bundle bytes | `signature_invalid` | +| 13 | manifest `key_id` ≠ sidecar `key_id` ≠ verifying key's id | `key_id_mismatch` | Row 8 deserves emphasis: **an unknown algorithm identifier is a hard rejection, never a fallback to Ed25519 and never a skip.** The field exists so the scheme can evolve without breaking verification of *existing* artifacts — a v2 loader will accept both; a v1 loader accepts one. -Rows 5 and 6 need a strict decoder. Go's `encoding/json` silently accepts duplicate keys (last wins) and ignores unknown fields unless told otherwise — use `Decoder.DisallowUnknownFields` for row 4, an explicit duplicate scan for row 5, and `Decoder.More()` after the object for row 2. +Rows 2, 4 and 5 each need their own mechanism, and one of them is easy to get wrong. Verified against Go 1.25: + +- **Row 4** (unknown key): `Decoder.DisallowUnknownFields()` — works as expected. +- **Row 5** (duplicate key): needs an **explicit scan**. `encoding/json` silently accepts duplicates and takes the LAST one, and `DisallowUnknownFields` does **not** catch them. Measured: `{"algorithm":"ed25519","algorithm":"rsa"}` decodes with `err == nil` and `algorithm == "rsa"`. On a field whose whole job is to pin the algorithm, that is the difference between a hard rejection and quietly reading a different value than a reviewer would. +- **Row 2** (trailing bytes): plain `json.Unmarshal` already rejects them — `invalid character '{' after top-level value` — so it is the simplest correct choice. If a streaming `Decoder` is used instead, decode once and then require the next `Decode` to return `io.EOF`; `Decoder.More()` also reports `true` for trailing bytes and garbage and `false` for trailing whitespace, but the `io.EOF` form states the intent directly and does not depend on how `More()` treats the top level. ## 4. What the signature does and does not cover @@ -70,9 +75,8 @@ SC-001 is written against exactly this boundary, which is why it enumerates four | Fixture | Property demonstrated | |---|---| -| `accept-min-sequence` | `sequence: "1"` | -| `accept-max-sequence` | `sequence: "18446744073709551615"` (2^64−1) — the value SC-002a fixture (a) signs | | `accept-trailing-newline` | a trailing `\n` is ignored | +| `accept-all-zero-signature` | the 88-char canonical encoding of 64 zero bytes — pins the canonical form itself | **Reject** — one fixture per numbered row above, named `reject-`. Each asserts the **reason code**, not merely that loading failed. diff --git a/specs/101-tpa-db/plan.md b/specs/101-tpa-db/plan.md index 25c22707d..00faaa7da 100644 --- a/specs/101-tpa-db/plan.md +++ b/specs/101-tpa-db/plan.md @@ -11,13 +11,13 @@ Technical approach, grounded in the code: - **Verification is a new leaf package, everything else extends what shipped.** `internal/security/bundlesig/` holds sidecar parsing and Ed25519 verification as pure functions over bytes — no I/O, no globals, no `scanner` import — because that is the one piece with an independent wire contract two implementations must agree on byte-for-byte (D3/D10). The loader, ceilings, activation and status work extend `internal/security/scanner/tpa_bundle_source.go` and `tpa_bundle.go`, where Spec 086 actually shipped them. **Spec 087's plan names a package `internal/security/bundle` that does not exist** (research.md R1); reading it as a description of the tree is the fastest way to write tasks against wrong paths. - **The load pipeline gains a first stage and grows ceilings throughout.** FR-008's order is not a preamble to the existing pipeline but interleaved with it, because the checks become knowable at different points: sidecar cap → parse sidecar → bundle byte ceiling → **signature over the raw bytes** → parse manifest → rule-count ceiling → epoch match → sequence/fingerprint → version compat → compile-all-or-reject → runnable>0 → self-check → activate. Today's `loadBundleFromFile` does `os.ReadFile` with no cap, no file-type check and no deadline (R2), so FR-011's numbers — bundle ≤ 8 MiB, rules ≤ 2000, sidecar ≤ 4 KiB, 5 s read budget — are new code at three different points, not one guard. -- **Candidate opening is descriptor-based, not path-based.** `os.OpenFile(..., O_NOFOLLOW)` → `fstat` the descriptor → refuse non-regular → read the ceiling from that same `FileInfo`, never re-resolving the path (D5). The time bound is a context-cancelled reader goroutine, **not** `SetReadDeadline`, which returns `ErrNoDeadline` on regular files and would be silently dropped. -- **Anti-downgrade state is per signing authority and one BBolt transaction.** A new `TPABundleStateBucket` (`internal/storage/models.go`) holds, per authority: watermark, its epoch, the fingerprint that set it, the signature ratchet, the FR-010 pin and deny-list — plus an append-only activation history. All written in a single `Update`, with the last-known-good bundle bytes `fsync`ed before the commit and the in-memory pointer published after it (D6), so no crash yields a new bundle with an old watermark. +- **Candidate opening is descriptor-based, non-blocking, and platform-split.** `os.OpenFile(..., O_NOFOLLOW|O_NONBLOCK)` → `fstat` the descriptor → refuse non-regular → clear `O_NONBLOCK` → read the ceiling from that same `FileInfo`, never re-resolving the path (D5). `O_NONBLOCK` is not defensive: opening a writer-less FIFO **blocks inside `open(2)`**, before `fstat` and before any timeout exists to observe it, so without it SC-011(c) cannot pass by construction. The time bound is a context-cancelled reader goroutine, **not** `SetReadDeadline` (`ErrNoDeadline` on regular files), and abandoned readers are capped at one per path so releasing the slot cannot leak an fd per refresh. Windows gets its own build-tagged implementation and an explicitly **weaker** guarantee — `os.Root` blocks path escape but still follows symlinks resolving inside the root. +- **Anti-downgrade state is per signing authority, one BBolt transaction, and content-addressed on disk.** A new `TPABundleStateBucket` (`internal/storage/models.go`) holds, per authority: watermark, its epoch, the fingerprint that set it, the signature ratchet, the FR-010 pin and deny-list — plus an append-only activation history, all in a single `Update`. Last-known-good bundles are stored as `tpa-lkg/.json` and referenced by digest from that record, so writing a new one never destroys the old and no crash can pair new bytes with old state — the failure a fixed `last-known-good.json` filename would reintroduce. Write-temp → fsync file → rename → **fsync the directory** (a file fsync does not make its directory entry durable) → commit → publish; startup reconciles unreferenced files (D6). - **`BundleInfo` gains a full digest rather than a widened one.** Today's `Fingerprint` is `hex(sha256)[:12]` — 48 bits (R3), fine as a human label and load-bearing in existing output, but not as authorization for the one path that bypasses the watermark. Rollback accepts only the full digest, and status surfaces both. - **`coverageOK` is where the trust story meets the approval gates.** `inprocess.go` currently computes it from mere bundle presence (R5). FR-007's unsigned-external degradation and FR-009a's fallback demotion both land on that one expression — the narrowest and most safety-critical edit in the feature, and the one whose regression is silent (a quietly suspended `scan`-mode auto-approval looks exactly like a quiet install, which is why FR-020 requires surfacing it). - **The gate must be taught to see the database before the corpus grows.** `cmd/scan-eval` never loads a bundle and `gateChecks()` omits `BundleCheck` entirely (R6), so FR-016/SC-005 would pass with the corpus absent. `--bundle` + registration through the **production** loader (D8), `categoryCheck` entries for every new class, and a deterministic negative control whose gated-malicious set is one control sample alone (D9). - **Telemetry needs typed backstop entries, not a relaxation.** `scanV8TPAScanner` whitelists four integer keys and rejects anything else before transmit (R7). Four typed entries plus one payload-level cross-field rule (`sequence` present while `bundle_version == "other"` is a violation), with the provenance gate in the reporting layer where authority is known (D7). -- **Scope stops at the network boundary.** Spec 087 is specified but not implemented (R8) and FR-018 forbids inventing a second refresh lifecycle, so this feature delivers the file-drop path end to end and leaves fetch to 087 (D11). SC-003's air-gapped install reaches full P1 capability on that path alone, which is the evidence the cut is in the right place. +- **Scope stops at the network boundary, behind a source-neutral seam.** Spec 087 is specified but not implemented (R8) and FR-018 forbids inventing a second refresh lifecycle, so this feature delivers the file-drop path end to end and leaves fetch to 087 (D11). SC-003's air-gapped install reaches full P1 capability on that path alone. The cut is only safe because the pipeline takes **bounded bytes + source metadata**, not a path: file-drop and 087's fetch are both thin adapters over one verifier, so the one component that must never exist twice cannot be forked. SC-007's fetch clause moves to 087 **in spec.md** rather than shipping permanently half-met. ## Technical Context @@ -53,16 +53,16 @@ Resolved in [research.md](research.md); recorded here as the plan of record. | # | Decision | Where resolved | |---|----------|----------------| | D1 | Publication channel = **GitHub Releases on the corpus repo**, immutable per-sequence asset names, `latest` resolved via the Releases API (not a mutable file) | R-D1 | -| D2 | `require_signed_bundle` stays **opt-in indefinitely** — no scheduled default flip; the FR-009 ratchet already covers every install that has seen a signed bundle | R-D2 | +| D2 | **OPEN — maintainer's call.** The ratchet only arms *after* a first signed activation, so "no flip" leaves fresh installs accepting an attacker's unsigned drop. Recommendation: default-on for external candidates with an explicit opt-out, accepting a loud breaking change for existing 086 unsigned droppers | R-D2 | | D3 | Sidecar = one-line JSON, exactly five keys (`sidecar_version`, `algorithm`, `key_id`, `signature`, `sequence`), total deterministic rejection, Ed25519-only in v1 | R-D3, contracts/sidecar-format.md | | D4 | `signatures[]` additive top-level section keyed by TPA id, many-to-one to `rules[]`, bidirectional referential integrity enforced at build, licence from an SPDX allowlist | R-D4, contracts/bundle-signatures-section.md | -| D5 | Read budget = `O_NOFOLLOW` open + descriptor `fstat` + ceiling from that `FileInfo`; time bound is a context-cancelled reader goroutine, never `SetReadDeadline` | R-D5 | -| D6 | Activation state = one new `TPABundleStateBucket`, one BBolt `Update`; LKG bytes `fsync`ed before commit, in-memory pointer published after | R-D6 | -| D7 | Telemetry = four **typed** backstop entries + the cross-field rule (`sequence` with `bundle_version=="other"` → reject); provenance gate in the reporting layer | R-D7 | +| D5 | Read budget = `O_NOFOLLOW\|O_NONBLOCK` open + descriptor `fstat` + ceiling from that `FileInfo`; context-cancelled reader goroutine, never `SetReadDeadline`; abandoned readers capped at one per path; Windows build-tagged and explicitly weaker | R-D5 | +| D6 | Activation state = one new `TPABundleStateBucket`, one BBolt `Update`; LKG bundles **content-addressed** (`tpa-lkg/.json`) and referenced by digest; fsync file **and directory**, commit, publish; startup reconciles orphans | R-D6 | +| D7 | Telemetry = four **typed** backstop entries + **whole-tuple** validation (not one direction); `schema_version` + sequence instead of a per-release enum the binary cannot know; provenance gate still in the reporting layer, but the backstop no longer depends on it | R-D7 | | D8 | `scan-eval --bundle` loads through the **production** loader and registers `BundleCheck`; same path as the activation self-check | R-D8 | | D9 | Negative control = one signature whose sample no built-in check flags, evaluated over a corpus whose gated-malicious set is that sample alone; no-overlap asserted in CI | R-D9 | | D10 | Structure = new leaf `internal/security/bundlesig/` for verification only; everything else extends `internal/security/scanner/` | R-D10 | -| D11 | Scope = **file-drop path end to end**; fetch stays Spec 087's. SC-007's fetch half is not demonstrable by this feature alone | R-D11 | +| D11 | Scope = **file-drop path end to end** behind a source-neutral `Candidate{bytes, source}` pipeline so 087 cannot fork the verifier; SC-007's fetch clause moves to 087 in spec.md | R-D11 | ## Project Structure @@ -92,7 +92,10 @@ internal/ │ │ ├── verify.go # Ed25519 verify over raw bundle bytes; key-id agreement (FR-003) │ │ └── testdata/ # byte-level valid + invalid sidecar fixtures │ └── scanner/ -│ ├── tpa_bundle_source.go # candidate open (O_NOFOLLOW + fstat, D5); sidecar cap; +│ ├── candidate_open_posix.go # NEW (//go:build !windows): O_NOFOLLOW|O_NONBLOCK + fstat (D5) +│ ├── candidate_open_windows.go # NEW: handle-based open; weaker guarantee, documented (D5) +│ ├── tpa_candidate.go # NEW: source-neutral Candidate{bytes, source} pipeline (D11) +│ ├── tpa_bundle_source.go # file-drop ADAPTER over that pipeline; sidecar cap; │ │ # verify-before-parse (FR-006, read once, never re-read); │ │ # BundleInfo gains full digest, sequence, authority, epoch, │ │ # signature_verified, staleness, last rejection reason (FR-020) @@ -158,6 +161,8 @@ docs/ # FR-022 operator + contributor lifecyc |-----------|------------|-------------------------------------| | **Persisted anti-downgrade state is best-effort against an attacker who can substitute `config.db`** (spec §Threat-model boundary; SC-012) | The bundle path may be configured inside the data directory, so the in-scope bundle-path attacker may also reach the state store. BBolt is transactional and checksummed, **not** authenticated (R11). | Authenticating the state with a key outside the writable data directory means a second key-management story — key location, rotation, and its own compromise path — to protect state whose loss already degrades safely to the FR-009b seeded baseline and the config-resident anchors. The spec accepts this explicitly and SC-012 tests the boundary rather than a guarantee that does not exist. **The plan's obligation is to never present the ratchet, watermark, pin or deny-list as surviving that attacker** — and to keep the anchors that do survive genuinely outside the data dir. | | **A read blocked on a stalled filesystem leaves a goroutine parked** (D5) | Go cannot interrupt a blocked read on a regular file; `SetReadDeadline` returns `ErrNoDeadline`, which is why FR-011 forbids specifying the bound that way. | Every alternative that truly kills the read is a subprocess or an OS-specific `aio` path — a large, platform-divergent surface to reclaim one goroutine. The property that matters is *the single-flight refresh slot is released and a rejection reason recorded*, so a legitimate update still activates; that is delivered, tested (SC-011c), and stated rather than implied. | -| **SC-007 is only half demonstrable by this feature** (D11) | SC-007 measures publish → active → fixture fires "within one daily refresh cycle on a fetch-enabled install". There is no refresh cycle: Spec 087 is unimplemented (R8), and FR-018 forbids building a second one here. | Implementing the fetch loop here would either duplicate 087 or absorb it, and absorbing it makes this feature the whole epic. The manual-drop half of SC-007 ("and immediately on manual drop") **is** demonstrated end to end, and SC-003 shows the air-gapped install reaches full P1 capability on the file-drop path alone. **This needs the maintainer's assent at the tasks stage**, because it means one success criterion carries forward rather than closing with this feature. | +| **SC-007's fetch clause moves to Spec 087** (D11) | SC-007 measures publish → active → fixture fires "within one daily refresh cycle on a fetch-enabled install". There is no refresh cycle: Spec 087 is unimplemented (R8), and FR-018 forbids building a second one here. | Implementing the fetch loop here would either duplicate 087 or absorb it, and absorbing it makes this feature the whole epic. But leaving the criterion in place and half-met is not the alternative: a success criterion no fixture can exercise is a **spec defect**, and maintainer assent does not make it testable. So this is an amendment task against spec.md, not an accepted residual — the fetch clause moves to 087, the manual-drop clause ("and immediately on manual drop") stays here where it is demonstrated end to end. Recorded in this table only because it changes what the spec promises. | + +| **The Windows candidate open gives a weaker guarantee than POSIX** (D5) | `O_NOFOLLOW` does not exist on Windows, and `os.Root` — the obvious substitute — blocks path *escape* but still follows a symlink resolving inside the root, so a final symlink to a regular file passes `fstat` cleanly. | Reaching parity means a handle-based open with reparse-point controls (`FILE_FLAG_OPEN_REPARSE_POINT`), which is a real amount of platform-specific code for a path the operator already controls. Design may take that route; what is NOT acceptable is implying parity. If the weaker guarantee is accepted, it is written down: on Windows the candidate path trusts its containing directory and carries symlink-substitution risk POSIX does not. | No other deviations: no new dependency, one new leaf package the verification contract justifies, one new BBolt bucket the crash-consistency requirement names. diff --git a/specs/101-tpa-db/research.md b/specs/101-tpa-db/research.md index c3d519a9f..6e4c5f120 100644 --- a/specs/101-tpa-db/research.md +++ b/specs/101-tpa-db/research.md @@ -67,23 +67,35 @@ BBolt is transactionally consistent and checksummed against corruption; it is no ### D1 — Publication channel: **GitHub Releases on the corpus repo** (resolves FR-017's marker) -**Decision**: publish artifact pairs as GitHub Release assets on the tpa-db corpus repository, with immutable per-release asset names (`scanner-bundle-.json` + `.sig`) and a `latest` reference resolved through the Releases API rather than a mutable file. +**Decision**: publish artifact pairs as GitHub Release assets on the tpa-db corpus repository, with **content-addressed** asset names (`scanner-bundle-.json` + `.sig`) plus a **signed release index** mapping sequence → digest, and a `latest` pointer inside that index. + +**"Immutable per-release asset names" would have been wrong.** GitHub Release assets are mutable: a maintainer (or a compromised token) can delete an asset and upload a different one under the same name, and releases and tags can be edited after publication. A sequence-bearing filename asserts immutability the platform does not provide. Content-addressing makes the name a checksum instead of a promise, so a substituted asset simply fails to match the digest the index — itself signed — names. Publication automation MUST additionally refuse to replace an existing asset, and historical artifacts needed for FR-010 rollback MUST be archived independently of the Release that first carried them. **Rationale**: the alternative (a static path under mcpproxy.app) puts corpus publication on the Cloudflare Pages deploy pipeline, which is already the site's critical path and whose failure mode is silent — [the appcast feed has gone stale this way before](../../docs/). Releases give immutable asset URLs, a publication timestamp SC-006's server-side age join needs, and a signing job that lives in the same repo as the corpus it signs, so FR-016's "never signed or published without a green gate" is one workflow rather than a cross-repo handoff. **Alternatives considered**: mcpproxy.app static path — rejected for the silent-failure coupling above and because it separates the artifact from the gate that must precede it. A dedicated CDN — rejected as unjustified operational surface for a file fetched at most daily per install. -**Consequence**: `latest` is a resolution step, not a URL, so the fetch path (087) must handle an API response rather than a fixed href. Note this in the 087 handoff. +**Consequence for the 087 handoff**: `latest` resolves through the signed index, not a fixed href — and the index must be reachable as a **cacheable object at a stable URL**, not only through the Releases API. Relying on the API alone adds an availability dependency and a rate limit that a fleet shares: unauthenticated GitHub API calls are limited per source IP, so installs behind one NAT doing a synchronized daily refresh can exhaust it and all fail together. 087 MUST therefore specify conditional requests (`ETag`/`If-None-Match`), refresh jitter so a fleet does not tick in lockstep, and explicit behavior when the index is unreachable or rate-limited (keep last-known-good, record the reason, do not treat it as a verification failure). ### D2 — `require_signed_bundle` stays **opt-in indefinitely** (resolves the Assumptions marker) -**Decision**: no scheduled default flip. The setting stays default-off; the ratchet does the work. +**Decision**: **OPEN — needs the maintainer's call.** The draft answer was "no flip, the ratchet does the work". Cross-model review showed that reasoning is incomplete, and the honest resolution is a back-compat judgement I should not make silently. + +**Why the draft rationale fails.** It argued that a flip "would change behavior only for installs that have never seen a signed bundle". That is true — and it is the wrong conclusion, because **those installs are exactly the unprotected population**. The ratchet only arms after a first signed activation; before that, an attacker who can write the bundle path (in scope) can drop an unsigned corpus and have it activate. Saying the ratchet "does the work" describes installs that are already safe. + +**The real trade is back-compat, and it is not free either way.** -**Rationale**: the spec's own note narrows this almost to nothing — FR-009's ratchet means any install that has ever activated a signed bundle already refuses unsigned drops, and FR-007 suspends auto-approval while an unsigned bundle is active. A default flip would therefore change behavior **only** for installs that have never seen a signed bundle, which is precisely the air-gapped/self-built-corpus persona FR-004a and FR-009b exist to support. Flipping the default would break that persona to harden a case the ratchet already hardens. +| | Default OFF (draft) | Default ON with opt-out (review's proposal) | +|---|---|---| +| Fresh install, attacker writes bundle path | unsigned corpus **activates** (degraded coverage, but serving) | refused | +| Existing 086 user with an unsigned file drop, upgrading | keeps working | **breaks** until they set the flag | +| Air-gapped / self-built corpus | works out of the box | works after one explicit opt-out | -**Alternatives considered**: flip after two release cycles — rejected: it buys nothing the ratchet does not already provide and breaks a supported persona. Flip with an escape hatch — rejected as the same thing with more config. +Unsigned file drops are a shipped 086 capability (`security.tpa_bundle_path`), so default-on is a **user-visible breaking change on upgrade** — loud rather than silent (the rejection reason is surfaced per FR-020), but breaking. -**Revisit trigger**: if telemetry (SC-006) ever shows a material population running unsigned external bundles long-term, that is evidence for a flip — and it is the evidence a future decision would need. Not now. +**Recommendation**: flip to default-on for EXTERNAL candidates with an explicit `require_signed_bundle: false` opt-out, and call it out in the release notes. The failure it prevents is silent and security-relevant; the failure it causes is loud and self-describing, and that asymmetry is the usual reason to prefer the breaking change. + +**This is a spec-level decision** — it resolves an Assumptions marker in spec.md — so it is recorded here as a recommendation with its cost, not applied unilaterally. Tasks MUST NOT begin the config work until it is settled. ### D3 — Sidecar wire format (resolves FR-003's "MUST be specified in design") @@ -127,25 +139,54 @@ Build-time referential integrity, enforced in **both** directions and rejected w | Case | Mechanism | |---|---| -| Local file open | `os.OpenFile(path, os.O_RDONLY\|syscall.O_NOFOLLOW, 0)`, then `f.Stat()` on the **descriptor**, refuse unless `Mode().IsRegular()`, then `io.ReadFull` bounded by the ceiling read from that same `FileInfo` | +| Local file open | `os.OpenFile(path, os.O_RDONLY\|syscall.O_NOFOLLOW\|syscall.O_NONBLOCK, 0)`, then `f.Stat()` on the **descriptor**, refuse unless `Mode().IsRegular()`, clear `O_NONBLOCK` on the surviving regular file, then `io.ReadFull` bounded by the ceiling read from that same `FileInfo` | | Local file read time | a goroutine performing the read, selected against a `context.WithTimeout(5s)`; on expiry the **slot is released and a rejection reason recorded** while the goroutine is abandoned | | Fetch body | `http.Client` timeout + request context + `io.LimitReader(body, ceiling+1)` | -The abandoned-goroutine case is stated as an accepted residual, not hidden: a read blocked on a stalled network filesystem cannot be interrupted in Go, so the guarantee is *the refresh path stays available*, not *the goroutine dies*. `O_NOFOLLOW` is unavailable on Windows; there the no-follow property comes from `os.Root`-scoped opens where the containing directory is trusted, and the descriptor-level `fstat` check is unchanged. +**`O_NONBLOCK` is load-bearing, not defensive.** Opening a FIFO for reading with no writer attached **blocks in `open(2)` itself** — before `fstat`, before the reader goroutine, before any timeout can observe anything. Without it the whole descriptor-based design is defeated by the very fixture it exists to pass: SC-011(c)'s never-written FIFO would park the loader inside `open`, and the 5 s budget could not release the single-flight slot because nothing has started measuring yet. With `O_NONBLOCK` the open returns immediately, `fstat` sees a FIFO, and the candidate is refused `not_regular_file` without a single byte read. The flag is then cleared for the regular-file case, so an ordinary read keeps blocking semantics and `io.ReadFull` behaves normally rather than spinning on `EAGAIN`. + +The abandoned-goroutine case is stated as an accepted residual, not hidden: a read blocked on a stalled network filesystem cannot be interrupted in Go, so the guarantee is *the refresh path stays available*, not *the goroutine dies*. **Windows is a weaker guarantee, stated rather than claimed away.** `O_NOFOLLOW` does not exist there, and `os.Root` is **not** a substitute: it prevents a path from escaping the root, but a symlink resolving *within* the root is still followed, and a final symlink to a regular file passes `fstat` cleanly. So the no-follow property is genuinely not delivered on Windows by that mechanism. Design MUST either open by handle with reparse-point controls (`FILE_FLAG_OPEN_REPARSE_POINT`) and inspect the opened handle, or record explicitly that the Windows candidate path trusts its containing directory and carries symlink-substitution risk that POSIX does not. Silently implying parity is the one option that is not acceptable. + +**This needs build-tagged files, and the plan's file list must say so.** `syscall.O_NOFOLLOW` and `syscall.O_NONBLOCK` do not compile on Windows, so the candidate open is a platform-split helper — `candidate_open_posix.go` (`//go:build !windows`) and `candidate_open_windows.go` — each with its own test. A single `syscall`-using file would simply fail the Windows build. + +**The residual must be bounded, or releasing the slot converts a stall into a leak.** Releasing the single-flight slot lets the next cycle start, which opens the same path again — so a path that stays stalled accumulates one parked goroutine and one file descriptor per refresh, indefinitely (a daily tick makes that slow, not harmless). Design MUST therefore cap concurrently-abandoned readers for a given candidate path at a small number (1 is sufficient: a second stall on the same path means the first has not returned) and, while that cap is reached, refuse the candidate immediately with the existing `read_timeout` reason instead of opening it again. That keeps the availability property — the refresh path is never *blocked* — while making the fd/goroutine cost O(1) rather than O(refreshes). Fixture: SC-011(c) extended to run a second and third refresh cycle against the same never-written FIFO and assert the process holds no more than one parked reader for it. **Rationale**: FR-011 already rules out `SetReadDeadline` (unsupported on regular files — returns `ErrNoDeadline`). The descriptor-level check is the only form that closes the stat/open substitution window. ### D6 — Crash-consistent activation state = **one BBolt bucket, one `Update`** (resolves FR-011a) -**Decision**: a new `TPABundleStateBucket` in `internal/storage/models.go` holding one record per authority plus one activation-history list, written in a single BBolt `Update` transaction. The recoverable last-known-good bundle bytes are `fsync`ed to the data directory **before** the transaction commits and before the in-memory pointer is published. +**Decision**: a new `TPABundleStateBucket` in `internal/storage/models.go` holding one record per authority plus one activation-history list, written in a single BBolt `Update` transaction. The recoverable last-known-good bundle bytes are made durable **before** the transaction commits and before the in-memory pointer is published. + +**LKG files are CONTENT-ADDRESSED, and that is what makes the ordering safe.** An earlier draft said "persist LKG bytes → commit → publish" and claimed a pre-commit crash leaves the old (bundle, watermark) pair. That is false if the new bytes land at a fixed filename: overwriting or renaming onto `last-known-good.json` before the commit means a pre-commit crash leaves **new bytes under old state** — the same split the requirement forbids, reached from the other side. + +The protocol is therefore: + +```text +write /tpa-lkg/.json.tmp +fsync that file +rename to /tpa-lkg/.json # atomic; readers never see a partial file +fsync /tpa-lkg/ # NOT optional — see below +commit the BBolt state record, which REFERENCES the digest +publish the in-memory pointer +``` + +Because the filename **is** the digest, writing a new LKG never destroys the old one, so a crash at any point leaves every previously-committed (state → bundle) pair intact and resolvable. The state record is the only thing that says which digest is current, and it becomes durable in one transaction. + +**The directory `fsync` is a separate step and is not optional.** `fsync` on a newly created file flushes its *contents*; it does not make the *directory entry* naming it durable. Skip it and a crash can leave the state record committed — BBolt fsyncs its own file on commit — while the bundle it references has no name on disk. -Ordering, normatively: persist LKG bytes → commit the state transaction → publish the in-memory pointer. A crash before the commit leaves the old (bundle, watermark) pair; a crash after leaves the new one; there is no interleaving that yields a new bundle with an old watermark or vice versa. +**Recovery must reconcile orphans**, since content-addressing trades a lost-file failure for an unreferenced-file one: on startup, delete any `tpa-lkg/*.json` no state record references (a crash between rename and commit), and if the referenced digest is *missing*, fall back to the embedded default under FR-009a rather than starting with no corpus. + +Note BBolt's own durability needs no help here: it fsyncs on commit by default, and `NoSync` must stay off on this path. **Rationale**: BBolt gives the single-transaction property for free and is already the durable store for approvals and quarantine — the state this record sits beside. R9 confirms no schema change is needed. The deny-list and the watermark's setting-fingerprint are members of the same record precisely because FR-011a names the crash that splits them. ### D7 — Telemetry: typed whitelist entries + a payload-level cross-field invariant (resolves FR-021) -**Decision**: extend `scanV8TPAScanner` with four **typed** entries rather than relaxing it: integer `sequence`, enum-bounded `source ∈ {embedded, file, fetch}`, boolean `signature_verified`, and enum-bounded `bundle_version` (publisher release line, or the literal `other`). Add one cross-field rule the backstop can enforce on the payload alone: **`sequence` present while `bundle_version == "other"` is a violation.** +**Decision**: extend `scanV8TPAScanner` with four **typed** entries rather than relaxing it: integer `sequence`, enum-bounded `source ∈ {embedded, file, fetch}`, boolean `signature_verified`, and a bounded `bundle_version`. Enforce the complete allowed **tuple** at the backstop, not a single one-directional rule. + +**`bundle_version` cannot be both verbatim and a compiled enum.** A binary shipped today cannot enumerate release lines published after it, so a static enum would classify every *new* corpus as `other` until the binary itself updates — defeating the point of an independently updatable database, and making SC-006's freshness metric blind to exactly the installs that are current. Accepting an arbitrary signed string is not bounded at all. The resolution is a **normalization rule independent of any individual release**: transmit the corpus's `schema_version` (a small, slow-moving set the binary does know) plus the `sequence`, and let the server side join sequence → publication date from what the publisher already knows. The near-unique free string never leaves the install. + +**The cross-field rule must be bidirectional.** "`sequence` present while `bundle_version == "other"` is a violation" catches one direction and leaves three holes: a publisher-looking version with no sequence, a publisher-looking version from `source: file` that never verified, and `signature_verified: false` alongside publisher provenance. The backstop MUST validate the whole tuple: `(publisher schema_version, sequence)` appear **together** and **only** when `source == "embedded"`, or `source ∈ {file, fetch}` **and** `signature_verified == true`; every other combination MUST be `other` with `sequence` omitted. The reporting layer still applies the provenance gate — it is the only layer that knows which authority verified — but the backstop no longer depends on the reporting layer being correct, which is the entire reason a backstop exists. The provenance gate lives in the reporting layer, where authority is known: verbatim version + sequence are reported only when the active bundle's provenance is the publisher authority (the embedded default, or an external bundle signature-verified against a publisher key). Every other bundle reports `other` with `sequence` omitted **regardless of the version string it claims** — a string allowlist alone is defeated by an operator corpus copying a genuine publisher version verbatim. @@ -179,4 +220,13 @@ The no-overlap property is asserted in CI as its own test, so a later built-in c **Rationale**: FR-018 forbids a second refresh lifecycle, and R8 shows there is no first one. Building the fetch loop here would either duplicate 087 or absorb it — both worse than shipping the artifact and trust story that 087 then consumes. SC-003 (air-gapped install reaches full P1 capability) is satisfied entirely by the file-drop path, which is the strongest signal that this is the right cut. -**Consequence**: SC-007 ("within one daily refresh cycle on a fetch-enabled install") is **not** fully demonstrable by this feature alone; its manual-drop half is. Say so in the plan rather than letting a task discover it. +**The loader MUST be source-neutral from day one, or 087 will fork it.** Cutting scope at the network boundary is only safe if the seam is in the right place. A file-oriented loader (path in, bundle out) forces 087 to either duplicate verification or invent temp-file semantics so a fetched body can be fed to a path-shaped API — and a duplicated verifier is the one piece of this feature that must never exist twice. Design MUST therefore define ONE candidate pipeline taking **bounded raw bytes plus source metadata**: + +```text +Candidate{ BundleBytes, SidecarBytes []byte; Source (embedded|file|fetch); Origin string } + → verify → parse → ceilings → epoch → sequence → compile → self-check → activate +``` + +The file-drop path becomes a thin adapter (open, fstat, bounded read → `Candidate`), and 087's fetch path becomes a second adapter (`http` + `io.LimitReader` → `Candidate`) that shares every stage after it. The ceilings then apply identically to both, which FR-011 requires anyway. + +**Consequence for SC-007**: its fetch half is not demonstrable by this feature. Rather than leaving a success criterion permanently half-met — which is a spec defect, not a plan caveat, and no amount of assent makes it testable — **spec.md MUST be amended to move the fetch-cycle clause of SC-007 to Spec 087**, leaving the manual-drop clause here where it is fully demonstrable. That edit is a task, not a footnote. From 55497c00b3c8a2b9123b0cbd201e11b9c1cc15ea Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Thu, 27 Aug 2026 08:37:31 +0300 Subject: [PATCH 20/20] =?UTF-8?q?docs(specs):=20lock=20D2=20=E2=80=94=20re?= =?UTF-8?q?quire=5Fsigned=5Fbundle=20stays=20default-off?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maintainer's call, against the recommendation to flip it on. Recorded with the trade that was put, so a future reader sees the reasoning rather than only the outcome: default-off keeps every existing 086 unsigned file-drop working across the upgrade, and the population a flip would protect (installs that have never activated a signed bundle) is the same one that has not adopted signing at all. Consequence now written into both documents: FR-007's coverage degradation and FR-020's surfacing of it are the ONLY thing between a fresh install and an attacker-supplied unsigned corpus. They stop being polish — shipping the loader without the degraded-state surfacing would ship the risk without the mitigation. Revisit trigger is SC-006 telemetry, which makes the question measurable. --- specs/101-tpa-db/plan.md | 2 +- specs/101-tpa-db/research.md | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/specs/101-tpa-db/plan.md b/specs/101-tpa-db/plan.md index 00faaa7da..ebbf752f8 100644 --- a/specs/101-tpa-db/plan.md +++ b/specs/101-tpa-db/plan.md @@ -53,7 +53,7 @@ Resolved in [research.md](research.md); recorded here as the plan of record. | # | Decision | Where resolved | |---|----------|----------------| | D1 | Publication channel = **GitHub Releases on the corpus repo**, immutable per-sequence asset names, `latest` resolved via the Releases API (not a mutable file) | R-D1 | -| D2 | **OPEN — maintainer's call.** The ratchet only arms *after* a first signed activation, so "no flip" leaves fresh installs accepting an attacker's unsigned drop. Recommendation: default-on for external candidates with an explicit opt-out, accepting a loud breaking change for existing 086 unsigned droppers | R-D2 | +| D2 | **LOCKED: `require_signed_bundle` stays default-OFF.** Recommendation was to flip it on; maintainer chose off, keeping existing 086 unsigned file-drops working across the upgrade. Consequence: FR-007's degraded coverage + FR-020's surfacing are the only mitigation for a fresh install, so they are mandatory rather than polish | R-D2 | | D3 | Sidecar = one-line JSON, exactly five keys (`sidecar_version`, `algorithm`, `key_id`, `signature`, `sequence`), total deterministic rejection, Ed25519-only in v1 | R-D3, contracts/sidecar-format.md | | D4 | `signatures[]` additive top-level section keyed by TPA id, many-to-one to `rules[]`, bidirectional referential integrity enforced at build, licence from an SPDX allowlist | R-D4, contracts/bundle-signatures-section.md | | D5 | Read budget = `O_NOFOLLOW\|O_NONBLOCK` open + descriptor `fstat` + ceiling from that `FileInfo`; context-cancelled reader goroutine, never `SetReadDeadline`; abandoned readers capped at one per path; Windows build-tagged and explicitly weaker | R-D5 | diff --git a/specs/101-tpa-db/research.md b/specs/101-tpa-db/research.md index 6e4c5f120..163586c12 100644 --- a/specs/101-tpa-db/research.md +++ b/specs/101-tpa-db/research.md @@ -79,7 +79,7 @@ BBolt is transactionally consistent and checksummed against corruption; it is no ### D2 — `require_signed_bundle` stays **opt-in indefinitely** (resolves the Assumptions marker) -**Decision**: **OPEN — needs the maintainer's call.** The draft answer was "no flip, the ratchet does the work". Cross-model review showed that reasoning is incomplete, and the honest resolution is a back-compat judgement I should not make silently. +**Decision (LOCKED 2026-08-26, maintainer): `require_signed_bundle` stays DEFAULT-OFF.** The trade below was put to the maintainer with the recommendation to flip it on; the call is to keep it off for now. **Why the draft rationale fails.** It argued that a flip "would change behavior only for installs that have never seen a signed bundle". That is true — and it is the wrong conclusion, because **those installs are exactly the unprotected population**. The ratchet only arms after a first signed activation; before that, an attacker who can write the bundle path (in scope) can drop an unsigned corpus and have it activate. Saying the ratchet "does the work" describes installs that are already safe. @@ -93,9 +93,11 @@ BBolt is transactionally consistent and checksummed against corruption; it is no Unsigned file drops are a shipped 086 capability (`security.tpa_bundle_path`), so default-on is a **user-visible breaking change on upgrade** — loud rather than silent (the rejection reason is surfaced per FR-020), but breaking. -**Recommendation**: flip to default-on for EXTERNAL candidates with an explicit `require_signed_bundle: false` opt-out, and call it out in the release notes. The failure it prevents is silent and security-relevant; the failure it causes is loud and self-describing, and that asymmetry is the usual reason to prefer the breaking change. +**Recommendation as put** was to flip to default-on for EXTERNAL candidates with an explicit opt-out. **The maintainer chose default-off**, which keeps every existing 086 unsigned file-drop working across the upgrade — the breaking change was the whole cost of the alternative, and the population it would protect (installs that have never activated a signed bundle) is the same population that has not yet adopted signing at all. -**This is a spec-level decision** — it resolves an Assumptions marker in spec.md — so it is recorded here as a recommendation with its cost, not applied unilaterally. Tasks MUST NOT begin the config work until it is settled. +**What that obliges this spec to carry, so the residual is not silent**: while an unsigned EXTERNAL bundle is active, FR-007 already degrades coverage and suspends `scan`-mode auto-approval, and FR-020 requires that state and its reason to be surfaced. Those are now the ONLY thing standing between a fresh install and an attacker-supplied unsigned corpus, so they are not optional polish — a task that ships the loader without the degraded-state surfacing has shipped the risk without the mitigation. + +**Revisit trigger**: SC-006 telemetry showing a material population running unsigned external bundles long-term. That is the evidence a future flip would need, and it is now measurable by design. ### D3 — Sidecar wire format (resolves FR-003's "MUST be specified in design")