Skip to content

Commit 92dc64e

Browse files
authored
feat(security): Spec 101 US3 TPA bundle refresh lifecycle and publication channel (#1720)
Spec 101 User Story 3 together with Spec 087 User Stories 1 and 2: the TPA bundle refresh lifecycle and the publication channel. - Refresh lifecycle (internal/security/scanner/tpa_refresh.go). Each daily jittered cycle runs a file leg, then an optional fetch leg. Both feed the US1 Candidate pipeline, so the verifier is not forked. Concurrent triggers join one cycle, scheduling is one deadline held under the refresher's lock, and Runtime.Close joins the refresher. - Opt-in signed fetch (tpa_fetch_enabled, default off; tpa_fetch_url, no default URL; env MCPPROXY_TPA_FETCH_ENABLED). It supports the GitHub Releases latest shape and a direct shape. Fetched pairs must be signed. The transport is https-only (http only to loopback), redirects are capped, egress is checked at dial time, bytes and time are bounded, ETag and backoff are honoured, and only the host is logged. Network outcomes are kept apart from rejections. A fetched bundle survives reloads and restarts. - Activation self-check (tpa_selfcheck.go): the CI recall/FP gate over an embedded copy of the eval corpus. A failing candidate is refused as gate_failure and last-known-good keeps serving. - Freshness: BundleInfo gains last_checked_at, last_activated_at, stale, refresh and reprotect, on REST (swagger regenerated), the CLI and the Web UI. Doctor warns on a stale database, on being stuck on last-known-good, and on a fetch that is enabled but cannot run. New: mcpproxy security bundle refresh and POST /api/v1/security/bundle/refresh. - Re-protection after an activation. Held-evidence refresh, plus a cached-contract informational scan of approved toolsets pinned to the activated snapshot (no upstream connection, gated by auto_baseline_scan). RT-02 holds: nothing is approved, unquarantined, revoked or quarantined. - Telemetry schema 16: five integer refresh and re-protection counters inside the closed anonymity contract. - Specs: SC-007's fetch clause moves to Spec 087 as SC-009, and Spec 087 is marked implemented by Spec 101 US3. Tasks T300-T396 are ticked and roadmap tpa-db-refresh is done. - Docs: tpa-bundle-signing, security-quarantine, tool-scanner, configuration, security-commands, telemetry, quickstart, CLAUDE.md. No new dependencies. The fetch works today only with operator-signed bundles, because the publisher key set is empty and the corpus repository does not exist yet. Whichever of this PR and #1702 merges second must refresh bundled/selfcheck-corpus.json from specs/065-evaluation-foundation/datasets/detect_corpus_v1.json. Related #938
1 parent ba51774 commit 92dc64e

129 files changed

Lines changed: 19793 additions & 152 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,7 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/
146146
- **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md).
147147

148148
## Recent Changes
149+
- 101-tpa-db US3 + 087-tpa-daily-refresh (tpa-bundle-refresh): Go 1.26, Vue 3 Web UI — a refresh cycle (`internal/security/scanner/tpa_refresh.go`: file leg then optional fetch leg, jittered `tpa_refresh_interval` default 24h, single-flight joins, `Nudge` on `config.reloaded`, joined in `Runtime.Close`) feeds the same source-neutral `evaluateCandidate` pipeline; the opt-in fetch (`tpa_fetch_enabled` default off, `tpa_fetch_url` no compiled-in default, env `MCPPROXY_TPA_FETCH_ENABLED`; Releases-API `latest` or direct `<url>` + `<url>.sig`, signed-only via `evalEnv.MustSign`, https/loopback-only transport with dial-time egress checks, host-only logs, outcomes kept apart from `last_rejection`) works today only with operator-signed bundles (publisher key set still empty); the activation self-check (`tpa_selfcheck.go`, parity-tested port of the `cmd/scan-eval` gate, embedded `bundled/selfcheck-corpus.json`) refuses a candidate as `gate_failure`; fetched bundles survive reload/restart (`externalConfigured`) and disabling fetch keeps them serving; activation observers drive the cached-only re-protection sweep (`internal/server/scan_bundle_reprotect.go`, held-evidence leg + `Service.StartScanFromCachedContracts` gated by `auto_baseline_scan`, marker `storage.BundleReprotectState`, RT-02: never approves, unquarantines or revokes); `BundleInfo` gains `last_checked_at`, `last_activated_at`, `stale`, `refresh`, `reprotect`; surfaces `POST /api/v1/security/bundle/refresh`, `mcpproxy security bundle refresh`, `BundleDoctorWarnings`, telemetry schema 16 refresh counters. Docs: `docs/features/tpa-bundle-signing.md` "Daily refresh and optional fetch", `docs/features/security-quarantine.md` "Re-protection after a bundle activates". **No new dependencies.**
149150
- 101-tpa-db US1 (tpa-db-format): Go 1.26, Vue 3 Web UI — external TPA bundles are signed artifacts: `<bundle>.sig` four-key Ed25519 sidecar (`internal/security/bundlesig`, pure, contract `specs/101-tpa-db/contracts/sidecar-format.md`), verify-before-parse source-neutral `Candidate` pipeline (`scanner/tpa_candidate.go`, FR-008 order, ceilings 8 MiB / 2000 rules / 4 KiB / 5 s via `candidate_open_{unix,windows}.go` no-follow descriptor opens), per-authority (publisher/operator) epoch-scoped watermark + ratchet + pin/deny-list in BBolt `tpa_bundle_state` (one tx with the history row) and content-addressed LKG files `<datadir>/tpa-lkg/`, full-digest rollback, `coverageOK` + async scan-mode admission degrade on the four FR-007/009a/009b conditions. Config: `require_signed_bundle` (default off), `tpa_active_root`, `tpa_operator_signers`, `tpa_operator_baseline`, `tpa_operator_epoch`. Surfaces: `GET/POST /api/v1/security/bundle{,/history,/rollback,/unpin,/reload}`, `mcpproxy security bundle …`, `cmd/tpa-db keygen|sign|verify`, telemetry schema 15 bundle keys. Publisher key set ships EMPTY. Docs: `docs/features/tpa-bundle-signing.md`. **No new dependencies.**
150151
- 118-auto-rescan-on-change: Go 1.26, small Web UI change — after a tool definition changes (or a tool is added after the baseline) in `manual`/`scan` trust mode the in-process TPA scan of its current definition is recorded as hold evidence (`held_reason: awaiting_review` + `held_verdict: clean` for a clean verdict; findings/coverage as before) and a debounced informational server rescan (`scan_informational.go`, fed by `Runtime.AddToolDefinitionsChangedObserver`) brings the review scan line back to "current". Switch: `security.auto_rescan_on_change` (default on, env `MCPPROXY_AUTO_RESCAN_ON_CHANGE`) AND `auto_baseline_scan`. RT-02 untouched: a clean verdict never approves a change. Docs: `docs/features/security-quarantine.md` "Automatic informational baseline scan". **No new dependencies.**
151152
- quarantine-approval-pilot: Go 1.26, tests + CI only — independent policy model `internal/qapilot` (no production imports), deterministic scenario inventory S01–S26 + 896-cell dispatch table + REST authz table + counting-upstream dispatch check, check ledger (`QAPILOT_REPORT_DIR`) gated by `go run ./cmd/qapilot report`, versioned mutation set `tests/qapilot/mutations.json` run by `qapilot mutate` in disposable worktrees; workflow `.github/workflows/quarantine-pilot.yml` (3 PR checks + scheduled rapid exploration). Docs: `docs/development/quarantine-approval-pilot.md`. **No new dependencies.**

‎ROADMAP.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ graph LR
223223
<details>
224224
<summary>🔵 tpa-db: versioned TPA signature database for the offline scanner — In progress · P1</summary>
225225

226-
> Vision pillar 'feel protected': the deterministic detect engine (Spec 076/077) ships with built-in checks but no updatable knowledge of in-the-wild Tool Poisoning Attacks. Build a versioned, offline-first signature/pattern database (known TPA campaigns, malicious phrase corpora, IoC hashes) that the engine consumes — bundled with the binary, refreshable out-of-band, community-contributable, and guarded by the existing scan-eval recall/FP CI gate. SPEC STAGE: specs/101-tpa-db MERGED as PR #1028 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that); 2026-10-20: US1 (tpa-db-format: signed four-key Ed25519 sidecar, verify-before-parse loader, per-authority anti-downgrade state, full-digest rollback, coverage degrade) shipped as #1701; US2 corpus in a sibling session; US3 refresh waits on Spec 087.
226+
> Vision pillar 'feel protected': the deterministic detect engine (Spec 076/077) ships with built-in checks but no updatable knowledge of in-the-wild Tool Poisoning Attacks. Build a versioned, offline-first signature/pattern database (known TPA campaigns, malicious phrase corpora, IoC hashes) that the engine consumes — bundled with the binary, refreshable out-of-band, community-contributable, and guarded by the existing scan-eval recall/FP CI gate. SPEC STAGE: specs/101-tpa-db MERGED as PR #1028 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that); 2026-10-20: US1 (tpa-db-format: signed four-key Ed25519 sidecar, verify-before-parse loader, per-authority anti-downgrade state, full-digest rollback, coverage degrade) shipped as #1701; US2 corpus in a sibling session (#1702, open); 2026-10-22: US3 (tpa-db-refresh, implements Spec 087 US1+US2: daily file+optional signed fetch refresh, activation self-check, freshness surfaces, cached re-protection sweep) shipped as #1720. Epic stays in_progress until the corpus (US2) lands.
227227

228228
Spec: [101-tpa-db](./specs/101-tpa-db/)
229229

@@ -238,15 +238,15 @@ graph LR
238238
239239
classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
240240
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
241-
class tpa_db_format done;
242-
class tpa_db_corpus,tpa_db_refresh todo;
241+
class tpa_db_format,tpa_db_refresh done;
242+
class tpa_db_corpus todo;
243243
```
244244

245245
| Task | Status | Refs |
246246
| --- | --- | --- |
247247
| Signature DB format + loader (versioned, signed, bundled default) | 🟢 Done | #1701 |
248248
| Seed corpus: catalog known public TPA campaigns/patterns into the DB | ⚪ Todo | — |
249-
| Out-of-band refresh (offline-friendly: manual file drop + optional fetch), eval-gated | ⚪ Todo | — |
249+
| Out-of-band refresh (offline-friendly: manual file drop + optional fetch), eval-gated | 🟢 Done | #1720 |
250250

251251
</details>
252252

@@ -1008,7 +1008,7 @@ graph LR
10081008
| Web UI + macOS app UX audit | In progress | P0 | — | | |
10091009
| Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | 4/4 (100%) | [081-release-qa-gate](./specs/081-release-qa-gate/) | |
10101010
| Action log / transparency — info at a glance | In progress | P1 | — | | |
1011-
| tpa-db: versioned TPA signature database for the offline scanner | In progress | P1 | 30/30 (100%) | [101-tpa-db](./specs/101-tpa-db/) | |
1011+
| tpa-db: versioned TPA signature database for the offline scanner | In progress | P1 | 81/81 (100%) | [101-tpa-db](./specs/101-tpa-db/) | |
10121012
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | |
10131013
| Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | |
10141014
| Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | |
@@ -1158,7 +1158,7 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—`
11581158
| [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 28/33 (85%) |
11591159
| [099-describe-check-mode](./specs/099-describe-check-mode/) | `in-flight` | 9/10 (90%) |
11601160
| [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — |
1161-
| [101-tpa-db](./specs/101-tpa-db/) | `shipped` | 30/30 (100%) |
1161+
| [101-tpa-db](./specs/101-tpa-db/) | `shipped` | 81/81 (100%) |
11621162
| [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) |
11631163
| [103-token-bench](./specs/103-token-bench/) | `shipped` | 62/64 (97%) |
11641164
| [104-auto-routing-mode](./specs/104-auto-routing-mode/) | — | — |

‎cmd/mcpproxy/doctor_cmd.go‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -591,6 +591,16 @@ func outputDiagnosticsWithProfileChecks(diag map[string]interface{}, info map[st
591591
fmt.Println("⚠️ Runtime Warnings")
592592
fmt.Println("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
593593
for _, warningItem := range runtimeWarnings {
594+
// The daemon's management Doctor emits free-text warnings as
595+
// plain strings (config findings, audit counters, the
596+
// signature-bundle freshness warnings); the legacy path emits
597+
// structured objects. Both render here.
598+
if warningText, ok := warningItem.(string); ok {
599+
if warningText != "" {
600+
fmt.Printf(" • %s\n", warningText)
601+
}
602+
continue
603+
}
594604
if warningMap, ok := warningItem.(map[string]interface{}); ok {
595605
message := getStringField(warningMap, "message")
596606
severity := getStringField(warningMap, "severity")

‎cmd/mcpproxy/doctor_cmd_test.go‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -628,6 +628,46 @@ func TestOutputDiagnostics_WarningWithoutTitle(t *testing.T) {
628628
}
629629
}
630630

631+
// The daemon's management Doctor emits runtime_warnings as plain strings
632+
// (appData denial, config findings, audit counters and, from Spec 101 US3, the
633+
// signature-bundle freshness warnings registered via AddRuntimeWarningSource).
634+
// A string warning must render under "Runtime Warnings" and must suppress the
635+
// all-clear verdict; before this it was counted in total_issues but printed
636+
// nowhere.
637+
func TestOutputDiagnostics_StringRuntimeWarningRendersAndSuppressesAllClear(t *testing.T) {
638+
const warning = "TPA signature bundle is 81 days old (stale after 30): run `mcpproxy security bundle refresh`"
639+
diag := map[string]interface{}{
640+
"total_issues": 1,
641+
"runtime_warnings": []interface{}{warning},
642+
}
643+
644+
oldStdout := os.Stdout
645+
r, w, _ := os.Pipe()
646+
os.Stdout = w
647+
defer func() { os.Stdout = oldStdout }()
648+
649+
doctorOutput = "pretty"
650+
err := outputDiagnostics(diag, nil, nil, "", nil)
651+
652+
w.Close()
653+
var buf bytes.Buffer
654+
buf.ReadFrom(r)
655+
output := buf.String()
656+
657+
if err != nil {
658+
t.Errorf("outputDiagnostics() returned error: %v", err)
659+
}
660+
if !strings.Contains(output, "Runtime Warnings") {
661+
t.Error("Missing runtime warnings section header for a string warning")
662+
}
663+
if !strings.Contains(output, warning) {
664+
t.Errorf("A plain-string runtime warning must be printed, got:\n%s", output)
665+
}
666+
if strings.Contains(output, "All systems operational") {
667+
t.Error("A runtime warning must suppress the all-clear verdict")
668+
}
669+
}
670+
631671
func TestOutputDiagnostics_HighSeverityWarning(t *testing.T) {
632672
diag := map[string]interface{}{
633673
"total_issues": 1,

0 commit comments

Comments
 (0)