From a79130bcfab49afe0a58376a99b6cc8e952c0b08 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 21 Oct 2026 06:37:07 +0300 Subject: [PATCH 01/65] =?UTF-8?q?docs(specs):=20Spec=20101=20US3=20tasks?= =?UTF-8?q?=20=E2=80=94=20refresh=20lifecycle,=20fetch=20adapter,=20re-pro?= =?UTF-8?q?tect=20sweep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-validates Spec 087 and Spec 101 US3 against main @ 14cb288fc and appends the US3 task list (T300-T396, six Phase-1 groups) to specs/101-tpa-db/tasks.md. Spec 087 is marked implemented by Spec 101 US3: its internal/security/bundle package was never built; the refresh is a thin adapter over the US1 Candidate pipeline (D11). Related #938 Co-Authored-By: Claude Opus 5.5 --- ROADMAP.md | 4 +- specs/087-tpa-daily-refresh/spec.md | 3 +- specs/101-tpa-db/tasks.md | 397 +++++++++++++++++++++++++++- 3 files changed, 399 insertions(+), 5 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index bc60f6a8c..5e9b0941d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1008,7 +1008,7 @@ graph LR | Web UI + macOS app UX audit | In progress | P0 | — | | | | Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | 4/4 (100%) | [081-release-qa-gate](./specs/081-release-qa-gate/) | | | Action log / transparency — info at a glance | In progress | P1 | — | | | -| tpa-db: versioned TPA signature database for the offline scanner | In progress | P1 | 30/30 (100%) | [101-tpa-db](./specs/101-tpa-db/) | | +| tpa-db: versioned TPA signature database for the offline scanner | In progress | P1 | 30/81 (37%) | [101-tpa-db](./specs/101-tpa-db/) | | | Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | | | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | @@ -1158,7 +1158,7 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 28/33 (85%) | | [099-describe-check-mode](./specs/099-describe-check-mode/) | `in-flight` | 9/10 (90%) | | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | -| [101-tpa-db](./specs/101-tpa-db/) | `shipped` | 30/30 (100%) | +| [101-tpa-db](./specs/101-tpa-db/) | `in-flight` | 30/81 (37%) | | [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) | | [103-token-bench](./specs/103-token-bench/) | `shipped` | 62/64 (97%) | | [104-auto-routing-mode](./specs/104-auto-routing-mode/) | — | — | diff --git a/specs/087-tpa-daily-refresh/spec.md b/specs/087-tpa-daily-refresh/spec.md index 9c1ae762f..b05009af4 100644 --- a/specs/087-tpa-daily-refresh/spec.md +++ b/specs/087-tpa-daily-refresh/spec.md @@ -2,7 +2,8 @@ **Feature Branch**: `087-tpa-daily-refresh` **Created**: 2026-07-16 -**Status**: Draft +**Status**: Implemented by Spec 101 US3 (specs/101-tpa-db/tasks.md T300+) +**Implementation note (2026-10-21)**: `plan.md` names a package `internal/security/bundle` (and a nested `security.bundle` config block). Neither exists and neither will be built. Spec 101 US1 shipped the bundle pipeline as `internal/security/scanner/tpa_*.go` plus `internal/security/bundlesig`, with flat `security.tpa_*` config keys. Under Spec 101 D11 this feature is a thin adapter on that pipeline: the daily scheduler and the optional fetch build a source-neutral `Candidate` (`internal/security/scanner/tpa_candidate.go`) and hand it to the same `evaluateCandidate` verifier, which is never forked. The FR-to-task mapping and the amendments to this spec are listed in Spec 101 tasks.md, US3 section. Release awareness (User Story 3, FR-018, FR-019, SC-007) is already covered by Spec 079. **Input**: A once-per-day, non-blocking, offline-FIRST refresh that (a) updates the tpa-db signature bundle — bundled-with-binary default, manual file-drop override, and an OPTIONAL verified/signed network fetch, versioned against the tpa-db `SCHEMA_VERSION`, failing safe to last-known-good; and (b) checks for a new mcpproxy release and surfaces it, respecting the existing update-check/distribution mechanism (no auto-installer). ## User Scenarios & Testing *(mandatory)* diff --git a/specs/101-tpa-db/tasks.md b/specs/101-tpa-db/tasks.md index 983d1de47..6b8c5b170 100644 --- a/specs/101-tpa-db/tasks.md +++ b/specs/101-tpa-db/tasks.md @@ -1,7 +1,7 @@ # Tasks: Spec 101 tpa-db — User Story 1 (signed format + downgrade-proof loader) **Input**: `/specs/101-tpa-db/` (spec.md, plan.md, research.md, data-model.md, contracts/, quickstart.md) -**Scope of this file**: User Story 1 only, one PR against `main`. US2 (seed corpus, `signatures[]`, `cmd/scan-eval --bundle`, contributor docs) and US3 (publication channel, Spec 087 fetch/tick, post-activation re-scan) are listed at the end as **out of scope** so nothing is silently dropped. +**Scope of this file**: User Story 1 (merged, #1701) and User Story 3 (the section "User Story 3 — refresh lifecycle (Spec 087) + publication channel + adoption loop" at the end, tasks T300+), each one PR against `main`. US2 (seed corpus, `signatures[]`, `cmd/scan-eval --bundle`, contributor docs; sibling PR #1702) is **out of scope** for this file and listed in "Out of scope for this PR" so nothing is silently dropped. **TDD rule (constitution V)**: every `test` task is written first and must FAIL for the stated behavioural reason on a build that compiles (add a stub where a new symbol is needed — a missing symbol is a build failure, not a bite). Prove bite by reverting the code hunk, never by deleting symbols. @@ -105,4 +105,397 @@ plan.md was written 2026-08-26 against Go 1.25.5. Re-validated against `main` @ ## Out of scope for this PR - **US2 (sibling session)**: `signatures[]` section parsing and integrity (D4, `contracts/bundle-signatures-section.md`); the seed corpus in `internal/security/scanner/bundled/scanner-bundle.json` (**including stamping its `sequence`/`trust_epoch`/`generated_at`** — handed over); `cmd/scan-eval --bundle` + `BundleCheck` registration + `categoryCheck` entries + D9 negative control; `cmd/tpa-db validate` and gate-before-sign; contributor docs; `.github/workflows/eval.yml`. -- **US3 / Spec 087**: publication channel (D1), network fetch + daily tick + single-flight lifecycle, the real activation self-check, post-activation re-scan (FR-019), fleet-freshness dashboards (SC-006/SC-007 fetch clause), shipping the first publisher key in a release. +- **US3 / Spec 087**: now planned in "User Story 3 — refresh lifecycle (Spec 087) + publication channel + adoption loop" below (T300+): network fetch + daily tick + single-flight lifecycle, the real activation self-check, post-activation re-scan (FR-019), freshness status, telemetry v16. Still out of scope there: creating the publication channel itself (D1) and shipping the first publisher key in a release (see that section's "Out of scope" list). + +--- + +## User Story 3 — refresh lifecycle (Spec 087) + publication channel + adoption loop + +**Input**: `specs/087-tpa-daily-refresh/{spec,plan}.md`, `specs/101-tpa-db/{spec,plan,research,data-model,quickstart}.md`, the US1 sections above, US1 as merged (#1701), and the 2026-10-21 subsystem maps (pipeline, gate, lifecycle, surfaces, reprotect). +**Scope**: one PR against `main` that implements Spec 101 US3 together with Spec 087 US1 and US2. Spec 087 US3 (release awareness) is already delivered by Spec 079 (B23). +**Must not depend on or touch US2 (#1702)**: `internal/security/tpacorpus/**`, `cmd/scan-eval/**`, `internal/security/scanner/bundled/scanner-bundle.json`, `specs/065-evaluation-foundation/datasets/detect_corpus_v1.json`, `.gitattributes`, `.github/workflows/eval.yml`, `specs/065-evaluation-foundation/datasets/README.md`, `detect_corpus_test.go`, and the US2-added scanner symbols (`LoadBundleCheck`, `EmbeddedBundle`, `ParseBundleSignatures`, `Signature*`, `tpa_bundle_eval.go`, `tpa_bundle_signatures.go`). The only US2 coupling is the merge-order gate on the self-check corpus copy (B14), which is a checklist step, not a code dependency. +**TDD rule**: the same as US1. Every `test` task is written first and must FAIL for the behavioural reason it states, on a build that compiles. Wave 0 stubs **every** symbol any T3NN test references, each in the file that will hold its final implementation, so groups replace function bodies rather than move stubs. A missing symbol is a build failure, not a bite. Prove that a test bites by reverting the code hunk it covers, never by deleting symbols. +**Format**: `- [ ] T3NN [P?] test|impl|docs: ()`. `[P]` means the task can run in parallel with the other `[P]` tasks of the same wave; their files are disjoint. + +--- + +### 2026-10-21 re-validation delta + +Spec 087 plan.md is dated 2026-07-16 and Spec 101 plan.md 2026-08-26. Both were re-validated against `main` @ 14cb288fc, with US1 merged and US2 open. + +| # | Spec 087 plan / Spec 101 plan says | Tree today | US3 does instead | +|---|---|---|---| +| Δ15 | 087: new package `internal/security/bundle` (`loader.go`, `verify.go`, `refresh.go`, `fetch.go`, `selftest.go`) | Does not exist. US1 shipped the pipeline as `internal/security/scanner/tpa_*.go` plus the leaf package `internal/security/bundlesig`. Spec 101 D10/D11 retire the name. | No new package. The scheduler and the fetch adapter are new files in package `scanner` (`tpa_refresh.go`, `tpa_fetch.go`) and feed the existing source-neutral `evaluateCandidate` (`tpa_candidate.go`). The verifier is never forked (D11). | +| Δ16 | 087: nested config block `security.bundle{path, refresh, fetch.{enabled,url,public_key}}` | US1 uses flat `security.tpa_*` keys. `public_key` would be masked by `GET /api/v1/config` (the name contains KEY; Δ13). The trust set already exists (publisher anchors plus `tpa_operator_signers`). | Flat keys `tpa_refresh_interval`, `tpa_fetch_enabled`, `tpa_fetch_url`, `tpa_stale_after_days` (B1). No per-fetch key field. A fetched pair is verified against the US1 authority trust set. | +| Δ17 | 087: `selftest.go` reuses `cmd/scan-eval/gate.go` "verbatim"; 101 FR-016a(c): the activation self-check MUST run "the same candidate-loading path" as CI | `gate.go` is `package main` and cannot be imported. US2 is rewriting it (`gateChecks(bundle)`). | A gate port in `internal/security/scanner/tpa_selfcheck.go`, guarded by AST, corpus-byte and production-surface parity tests, without editing `cmd/scan-eval` (B14). This is a recorded **deviation** from FR-016a(c); the shared gate package is a follow-up. | +| Δ18 | 087: embedded self-check corpus is a "compact subset" | A subset changes the denominators and therefore the decision: 1/21 FP passes, 1/10 fails. | A full byte copy at `internal/security/scanner/bundled/selfcheck-corpus.json`, plus a parity test against `specs/065-…/detect_corpus_v1.json`. | +| Δ19 | 087: bundle status goes on `/api/v1/info` next to `update` | `/info` is `scopeFiltered` (agent tokens can read it). Bundle status is admin-only (`security_bundle.go:15-19`). | `/api/v1/info` is left alone. Status grows on `GET /api/v1/security/bundle` and `/security/overview`. Release status stays separate (087 FR-019). Flagged for maintainer sign-off. | +| Δ20 | 087: refresher "mirrors `updatecheck.Checker`" | The Checker has no Stop and is never joined. The refresher writes BBolt and `tpa-lkg/`. | Lifecycle copies the `telemetry.Service` Start/Stop/done pattern and is joined in `Runtime.Close` before `StopAsync` (B19). Only the generation-guard and seam patterns come from updatecheck. | +| Δ21 | 087: Go 1.24 | `go 1.26.0` | — | +| Δ22 | 101 plan D1: "immutable per-sequence asset names, latest via the Releases API". 101 research D1: content-addressed names plus a **signed release index**, a stable cacheable URL, ETag, jitter, rate-limit-is-not-verification. | The two contradict each other. The repo `smart-mcp-proxy/tpa-db` does not exist (404). The publisher signer set is empty. | Channel = GitHub Releases API `releases/latest` with per-sequence asset names used **as selection hints only**. Integrity comes from the signed pair plus the per-authority watermark (B4). Research D1's ETag, jitter and rate-limit rules are adopted in full (B6). A "direct" URL mode serves mirrors and air-gap staging. The signed release index is deferred (out of scope). | +| Δ23 | 101 tasks Δ2: "one `activationMu` serializes every run" | The identifier is `bundleRT.mu` (`tpa_bundle_runtime.go:86`), a non-reentrant `sync.Mutex`. It serializes but does not coalesce. | Network I/O happens outside `bundleRT.mu`. Coalescing is the refresher's own joinable in-flight cycle (B8). | +| Δ24 | 087 FR-013/014: "operator-configured trusted public key; no key → fail closed" | Trust comes from US1 FR-004/004b: the active root's trust set. The publisher set is empty, so with the default root `publisher` every fetched pair would be `key_not_trusted`. | When the active root's trust set is empty, the fetch issues **zero requests** and reports `disabled/no_trusted_signer`, surfaced in status and doctor (B10, B16). Today a fetch works only with `tpa_active_root=operator` plus `tpa_operator_signers` (plus `tpa_operator_baseline` for non-degraded coverage). | +| Δ25 | 087 US1: "once per day mcpproxy notices the dropped file" | No periodic file read and no watcher. Only config apply, REST/CLI reload and startup run the pipeline. | The daily cycle runs a **file leg** (the existing path read, network-free, always on) followed by an optional **fetch leg** (B7). | +| Δ26 | (implicit) the fetched bundle persists like a file drop | **Hazard A**: with `tpa_bundle_path==""`, `fallbackLocked` (:469), `canSkipLocked` (:425) and `restorePinLocked` (:518) treat external bundles as removed. A fetched bundle would be deactivated at the next `config.reloaded` and at every restart, and reconcile would delete its LKG files. | Those three guards use `externalConfigured(policy, state)`; the file read, the file-candidate evaluation and the file-leg `loadErr` stay Path-only (B9). This is the first failing test (T320). | +| Δ27 | US1 hook `bundleSelfCheck` applies to every evaluation mode (nil, so not observable) | Called in `compileAndGate` for `evalNormal`, `evalLKG` (startup restore, pin restore) and `evalRollback`, under `bundleRT.mu`. | Self-check runs for `evalNormal` and `evalRollback` only and is skipped for `evalLKG` (B13). The hook is installed from production wiring, not `init()`, and a server-level test proves it is installed (T365). | +| Δ28 | 101 FR-019: "surface as review findings" | `GetReviewQueue` (`review.go:196`) and attention kinds ignore approved-only servers. `Service.StartScan` connects to upstreams (`EnsureConnected`, live `tools/list`) and schedules Pass 2. | A new cached-contracts scan entry. Findings appear in the scan summary, report, badge, overview, activity and the per-tool review verdict on the server's review payload (`GetServerReview`, `reviewToolScanVerdict`). A held-evidence refresh covers held tools. A new attention kind is deferred pending a maintainer decision (B17, B18). | +| Δ29 | 087 FR-016: env override for every refresh setting | US1 stance: trust config has no env alias (A9). An env-supplied URL would let the environment redirect the fetch. | Only `MCPPROXY_TPA_FETCH_ENABLED` exists (bidirectional, read inside the accessor, never persisted). The URL stays config-resident (B2). | +| Δ30 | 087 edge case: "an in-flight refresh completes against the settings it started with" | — | An in-flight download finishes, but its candidate is offered under the **live** policy. If any fetch or trust setting changed, it is discarded as `superseded` (B8). Flagged for maintainer sign-off; 087 spec.md is amended to match. | +| Δ31 | 087 FR-016: "refresh enable/interval" | — | There is **no refresh-enable key**: the file leg is network-free, idempotent and always on; only its interval is configurable. The fetch leg has its own enable (B1). | + +### Design decisions (zero-interruption assumptions) + +- **B1. Config keys.** All keys are flat under `security`, follow the US1 naming, and contain none of the masked substrings KEY, AUTH, TOKEN, PASS, CERT or SECRET. + - `tpa_refresh_interval`: `config.Duration`, `swaggertype:"string"`. Absent or `0` means 24h. The runtime clamps to at least 1h; a test seam goes lower. + - `tpa_fetch_enabled`: plain `bool`, default **false**. + - `tpa_fetch_url`: string, default **empty**. + - `tpa_stale_after_days`: `*int`. Absent means 30; `0` disables the stale flag and the doctor stale warning. + - Shape checks run in a new `tpaRefreshErrors()` appended to `ValidateDetailed` (write-time only), never in `validateDetailedCore` (boot-fatal). It refuses: interval under 1h; a scheme other than https unless the host is a loopback literal or `localhost`; userinfo, query or fragment; fetch enabled with an empty URL; fetch enabled with `tpa_active_root=operator` and an empty `tpa_operator_signers`; negative stale days. Messages never echo the URL. The publisher-root-with-empty-built-in-set case cannot be seen by `internal/config` (it must not import scanner), so it is reported at runtime (`no_trusted_signer`, B10) and by doctor (B16). + - A malformed value at boot is logged and treated as "fetch off". + - `DetectConfigChanges` needs no edit, because the fields sit inside `SecurityConfig`, which is compared with `jsonEqual`; a hot-reload test is still added (T311). +- **B2. Env override.** `MCPPROXY_TPA_FETCH_ENABLED=true|false` is read inside `IsTPAFetchEnabled()`, the same pattern as `EffectiveTPABundlePath`. It is not mirrored into `process_overrides.go`, so `SaveConfig` never persists it. Enabling via env still fetches only the config-resident URL, verified against config-resident keys, so the environment cannot redirect the fetch. There is no env alias for the URL, interval or stale threshold. `MCPPROXY_DISABLE_AUTO_UPDATE` does **not** silence the bundle fetch (087 FR-018/FR-019 keep the two distinct). +- **B3. No working default URL.** An empty `tpa_fetch_url` means the fetch leg is off (`disabled/no_url`, zero requests). No URL is compiled into the binary until a release ships the first publisher key and the corpus repo exists. Docs give the intended channel URL shape as an example only. +- **B4. Channel shape and integrity.** The mode is detected from the URL **path only**, so `httptest` can exercise both modes. + - **Releases mode**: the path matches `^/repos/[^/]+/[^/]+/releases/latest$`. GET the JSON with `Accept: application/vnd.github+json`, `User-Agent: mcpproxy/`, a 1 MiB cap and `If-None-Match`. Select exactly one asset named `^scanner-bundle-([0-9]{1,20})\.json$` with a sibling `.sig`. Zero or several candidate pairs, or a missing `.sig` asset, give `channel_invalid`. Download both through `browser_download_url` with `Accept: application/octet-stream`. + - **Direct mode** (any other path): GET the URL as the bundle and `URL + ".sig"` as the sidecar, with `If-None-Match` on the bundle. + - The naming matches the file-drop convention (`` + `.sig`, `tpa_bundle_runtime.go:871`), so an air-gapped operator drops the downloaded pair as-is under any one name pair. + - The filename sequence is an **unsigned hint**, used only to skip a download when it is at or below the active root's watermark (`unchanged`). It is never trusted for activation. + - Integrity rests only on what US1 already enforces: Ed25519 over the raw bundle bytes; key-id, epoch and sequence agreement; the per-authority watermark and the equal-sequence-different-digest refusal; the deny-list; the self-check. A substituted or replayed asset therefore fails verification (`tamper`) or is a replay (`downgrade`). A withheld asset (freeze) is surfaced by staleness (B15). This is why research D1's signed index can be deferred without weakening activation. +- **B5. Candidate construction and signed-only enforcement.** + - Bodies are read through `io.LimitReader(MaxBundleBytes+1)` and `io.LimitReader(MaxSidecarBytes+1)`. Over-limit bytes are handed to the pipeline as they are, so the existing `ceiling_exceeded` stage rejects them, with no second ceiling implementation. + - The candidate is `Candidate{Bundle, Sidecar, SidecarPresent, Source: BundleSourceFetch, Path: "://"}`. The path is never the full URL. + - Signed-only is enforced **per evaluation**, never by mutating the policy: a new `evalEnv.MustSign bool` is consulted in `unsigned()` alongside `policy.RequireSigned` and is set only for the offered evaluation. A policy copy with `RequireSigned=true` would also be passed to fallback step b and deactivate a serving unsigned file-drop bundle (`tpa_candidate.go:196`, `tpa_bundle_runtime.go:472`). + - In direct mode, a `404` on `.sig` becomes `SidecarPresent=false` plus `MustSign`, so the pipeline itself emits `signature_required`. Other `.sig` HTTP failures are `fetch_unavailable/http_status`. In Releases mode a missing `.sig` asset is `channel_invalid`. +- **B6. Network-failure semantics.** These outcomes are a **non-verification** refresh status, never a `last_rejection`, and they never touch the watermark: network error, timeout, non-2xx, `403`/`429`, `X-RateLimit-Remaining: 0`, `Retry-After`, DNS failure. They report `fetch_unavailable` with reason `network_error|timeout|http_status|rate_limited`, and LKG keeps serving. + - `304` gives `not_modified`. ETags are cached in memory per URL (restart-scoped). + - Rate-limited: the next attempt is `max(next tick, reset or Retry-After)`, capped at 24h. A plain failure retries at the next tick, with no exponential stretch beyond the interval (087 FR-015). + - `consecutive_failures` counts `fetch_unavailable` and `channel_invalid` cycles and resets on `activated`, `unchanged`, `not_modified` and `rejected`. + - Logs are deduplicated on `(outcome, reason)`: log on change, or once per 24h while the outcome persists. + - Jitter: the first cycle fires at a uniform random point in `[30s, 5m]` after Start, each later one at `interval ± 10%`. The timer is re-armed after each cycle completes (a single `time.Timer`, monotonic clock), so a suspended laptop or a forward clock jump yields **one** cycle on wake, never a catch-up burst, and a backward wall-clock jump cannot wedge it. Rand and clock are injectable. +- **B7. Precedence and per-cycle noise.** + - A cycle runs the **file leg** (the existing path run, only when `tpa_bundle_path` is set), then the **fetch leg** (only when B10 allows it). + - The fetch leg calls `runLocked(policy, log, runOptions{offered: &c})`, which evaluates **only** the offered candidate as the fresh one. On refusal the current serving bundle stays, via fallback step b with `Reuse`. The leg never re-reads the file and never writes the file-leg `loadErr`; its refusal goes to new `selection.offeredRej/offeredDigest`, recorded through `recordRejectionLocked` (deduplicated). + - The "same as active" and negative-cache comparisons happen **inside** the offered branch of `runLocked`, under `bundleRT.mu`, before `evaluateCandidate`: bytes against `rt.active.cand` (the `canSkipLocked` compare) and the digest pair against a mutex-guarded cache of the last refused fetched pair. A match gives `unchanged` with zero compiles and zero self-checks. Nothing reads `rt.state`/`rt.active` unlocked. + - The pre-download hint skip reads per-authority watermarks from a small snapshot published under `bundleRT.mu` by `publishLocked` (`atomic.Pointer`), never `rt.state` and never `BundleStatus()`. + - A stale file below a fetched watermark is refused once (deduplicated), only on the file leg, and reported as the file leg's `load_error`; it does not mask fetch status. + - Both candidates valid and above the watermark: the file wins first, the fetch wins at the next cycle. Monotone, no flapping. + - A deny-listed or pinned-around remote release (after `rollback`) is refused by the pipeline as `deny_listed`/`downgrade`, then hits the negative cache on later cycles: quiet, no reactivation, no observer fire (B16 quiet set). A strictly newer signed release activates and clears the pin (US1 A4). +- **B8. Single-flight and hot reload.** + - The refresher holds one in-flight cycle. A manual refresh, a timer tick or a config nudge arriving during a cycle **joins** it and receives that cycle's result: one download, one activation per distinct candidate (087 "two triggers race"). `reload` and `config.reloaded` runs still serialize on `bundleRT.mu` and stay idempotent, so tick + reload + refresh yields one activation per distinct candidate. + - A generation counter is bumped on any change to the fetch settings (enabled, URL) **or** to any input of the offered-run policy fingerprint (active root, operator signers/baseline/epoch, `require_signed_bundle`). A finished download is offered only if the generation is unchanged; otherwise it reports `superseded`. An interval change re-arms the timer. A `Nudge` after `Stop` began is a no-op. + - The HTTP fetch runs outside `bundleRT.mu` with its own context: a 20s total per-cycle deadline (under the CLI's hard 30s, `security_cmd.go:2542`) and a 10s response-header timeout, so a trickled body is cut by the cycle deadline. +- **B9. Fetched bundles survive reloads and restarts.** + - `externalConfigured(policy *bundlePolicy, st *TPABundleState) bool` = `policy.Path != "" || policy.FetchEnabled || (st.Active != nil && st.Active.Source == BundleSourceFetch)`. It is a function, because `bundlePolicy` holds config only. + - Sites: `canSkipLocked` (:425; with `Path==""` it skips iff the serving external bundle still matches the recorded Active), `fallbackLocked` steps b/c (:469), `restorePinLocked` (:518). The file read (:260), the file-candidate evaluation in `chooseLocked` (:447) and the file-leg `loadErr` in `finishLocked` (:338) stay `policy.Path != ""`. + - Disabling the fetch therefore **keeps** the last verified fetched bundle serving; dropping silently to the weaker embedded corpus would be an unrequested FR-009a degrade. `security bundle rollback --embedded` is the explicit way back. Rollback to a **prior fetched** release needs that file in the `tpa_bundle_path` directory (`confineRollbackPath`); documented. + - `FetchEnabled` and the URL are added to `policy.Fingerprint`. The interval and stale threshold are not (B15). +- **B10. When the fetch leg may issue requests.** Only when all hold: `IsTPAFetchEnabled()`; the URL is non-empty and valid; the active root's trust set is non-empty (publisher anchors, or `tpa_operator_signers` when the root is `operator`); the state store is loadable (not `loadFailed`). Otherwise the leg reports `disabled` with reason `fetch_off|no_url|invalid_url|no_trusted_signer|state_unavailable`. SC-005 and 087 FR-012 are asserted as **zero requests**. +- **B11. Transport and egress.** + - https is required; http is allowed only when the configured host is a loopback literal or `localhost`. + - Redirects: at most 5 hops; a hop may not downgrade https → http. Hops may cross hosts, because integrity comes from the signature and GitHub asset downloads redirect off `api.github.com`. + - A dial-time `net.Dialer.Control` always refuses link-local (169.254/16 including metadata, fe80::/10), unspecified and multicast addresses. It refuses loopback and RFC1918/ULA/CGNAT unless the configured host itself resolved to that class (a LAN mirror), so a public URL cannot redirect inward. + - When `HTTP(S)_PROXY` is set, the same predicate is applied to the resolved target host at the application layer. + - The predicates are written minimally in `tpa_fetch.go`; extracting a shared leaf from `internal/registries/ssrf.go` is a follow-up. +- **B12. Host-only logging.** The host is computed once with `u.Hostname()`. Every zap field, status field and telemetry value uses only that host. `*url.Error` is unwrapped and only `urlErr.Err` plus the host is logged; redirect hops are logged by host. A test asserts the URL path, query and redirect targets never appear in logs, `BundleInfo`, `last_rejection` or error strings. +- **B13. Activation self-check.** + - `internal/security/scanner/tpa_selfcheck.go` defines `activationSelfCheck(*BundleCheck) error` and `EnableActivationSelfCheck()`. The latter assigns `bundleSelfCheck` and is called once from production wiring in `internal/server/server.go`, next to `configureTPABundleRuntime`, so US1 fixtures and `clearBundleRuntime` keep the nil default. A server-level test proves the hook is non-nil after construction (T365). + - The check builds its own engine: `detect.NewEngine(detect.Options{Checks: append(selfCheckBuiltins(), cand)})` over each entry's tool plus peers. It never calls `BundleStatus`, `defaultBundleCheck`, `activeBundleSnapshot` or `ScanToolMetadataVerdict` (deadlock through the lazy install under `bundleRT.mu`). + - The decision is identical to the CI gate. Built-ins: UnicodeHidden, Shadowing, PayloadDecoded, PhraseInjection. Category map: the five HEAD entries plus `strings.HasPrefix(cat,"tpa_") → "tpa.bundle"`. Rule: `recall < 0.90 || hard-negative FP > 0.05 → reject`, same float expressions. + - Fail-closed additions: either vacuity guard (gated malicious == 0, hard negatives == 0), corpus parse/empty, and `Coverage.ChecksFailed > 0`. A package-level corpus override seam (`selfCheckCorpus func() []byte`) lets tests feed vacuous corpora. + - The rejection detail is terse: counts and corpus ids only, never candidate pattern text. + - It runs for `evalNormal` and `evalRollback` and is skipped for `evalLKG` (restore of a bundle that already passed), avoiding deactivation at upgrade and the startup cost. The embedded default bypasses the hook as today and is guarded by a unit test. + - Operator-signed bundles are **not** exempt. + - The recall leg is knowingly inert on the 57-entry HEAD corpus until US2's `tpa_*` samples land; the test logs this loudly. + - Fixture consequence: any test bundle that runs through the real self-check (T391) must keep at most 1 false positive over the 21 HEAD hard negatives (2/21 fails), so its added signature uses a narrow, fixture-specific pattern. +- **B14. Self-check corpus location and drift guard.** A byte copy at `internal/security/scanner/bundled/selfcheck-corpus.json`, embedded with `//go:embed`. Guards: + - (a) byte parity with `specs/065-…/detect_corpus_v1.json` after `\r\n → \n`; the failure message prints the `cp` command; + - (b) go/parser parity with `cmd/scan-eval/gate.go` and `main.go`, read-only, **tolerant to both shapes** (`gateChecks()`, `gateChecks(bundle)`, `builtinGateChecks()`), keyed on `checks.X` composite literals and `categoryCheck` string pairs: built-in list, pairs, `tpa.bundle` only on `tpa_` keys, `ThreatLevelDangerous` in `scanEntryFlagged`, flag defaults 0.90 and 0.05; + - (c) identical per-entry hard-flags between `baselineEngineChecks()` and the self-check built-ins; + - (d) an optional differential `go run ./cmd/scan-eval --gate`, skipped when `go` is absent; + - (e) non-empty provenance and licence fields on every corpus entry (087 FR-021). + - Test names use the `TestSelfCheck*` prefix (off eval.yml's `-run` regex). + - **Merge-order gate (T396 checklist)**: whichever of US2/US3 merges second refreshes the copy. If US3 merges second, it rebases and runs the `cp` in the same PR. If US3 merges first, the PR body and a comment on #1702 tell US2 to run the `cp`; guard (a) is red on US2's branch by design until then. +- **B15. Freshness fields and settings.** All new `BundleInfo` fields are pointer or omitempty: + - `last_checked_at *time.Time`: end of any pipeline run, including `canSkipLocked` no-ops. + - `last_activated_at *time.Time`: `state.Active.ActivatedAt` when the Active digest equals the serving digest; nil for embedded. + - `stale bool` and `stale_after_days *int`: computed at **read time** next to `StalenessDays`. A future `generated_at` is never stale and never a negative age. + - `refresh *BundleRefreshStatus{fetch_enabled, fetch_host, interval_seconds, last_trigger, last_outcome, last_reason, last_attempt_at, consecutive_failures, next_check_at}`. + - `reprotect *BundleReprotectStatus{state: idle|running|suspended|failed, reason, last_completed_at, servers_scanned, servers_with_hits}`, set from the server through `scanner.SetReprotectStatus` (B17). + - Display settings (stale threshold, interval, fetch enabled, fetch host) live in a small struct under its own mutex next to `histMu`, refreshed by `configureWithPolicy` on **every** apply (before `canSkipLocked`) and by `Nudge`. A threshold edit therefore takes effect on the next read without rerunning the pipeline. Refresh status and `last_checked_at` live in the same struct and are merged into `BundleStatus()`/`statusFromSnapshot()` copies, so cycles never churn the published (check, info) pair. + - Trigger vocabulary (closed): `startup` (the construction-time `ConfigureBundleFromConfig`), `config` (a `config.reloaded` re-apply), `timer`, `manual`. + - Outcome vocabulary (closed): `activated`, `unchanged`, `not_modified`, `rejected` (detail in `last_rejection`), `fetch_unavailable`, `channel_invalid`, `superseded`, `disabled`. + - `last_rejection` keeps meaning "a candidate was refused" and stays in memory, restart-scoped (US1 behaviour). +- **B16. Doctor.** The pure function `scanner.BundleDoctorWarnings(info BundleInfo, now time.Time) []string` (`tpa_bundle_doctor.go`) is registered once with `mgmtService.AddRuntimeWarningSource`. It returns at most three short strings with no digest, path, URL or host: + - (1) **stale**: `stale` AND (source is `file` or `fetch`, OR fetch enabled). A stale embedded corpus on an install that never configured a refresh shows in status only; binary upgrades are Spec 079's nudge. + - (2) **stuck on last-known-good**: `last_rejection` present, newer than `last_activated_at` (or process start for embedded), reason outside the quiet set {`downgrade`, `deny_listed`}; or `refresh.consecutive_failures ≥ 3`. + - (3) **fetch configured but inactive**: fetch enabled and the leg is `disabled` with `no_url|invalid_url|no_trusted_signer` (087 edge case "no key configured: a clear configuration error is surfaced"). + - These warnings **intentionally** feed the doctor verdict, the MCP doctor tool and the telemetry `runtime_warnings` pass/fail bit: each is an actionable state on an install that configured a refresh. Calling the source on a never-published runtime lazily installs the embedded default without holding `bundleRT.mu` first; a test proves it does not hang. +- **B17. Re-protection sweep (FR-019, SC-009, RT-02).** + - Trigger: `scanner.AddBundleActivationObserver(fn) (unsubscribe func())`. It fires only when a history row was written (activation or rollback), **after** publish, dispatched on a goroutine outside `bundleRT.mu`, with an in-flight counter tests can drain. Never on an LKG restore, an embedded fallback, an idempotent rerun or a refusal. Payload: digest, sequence, authority, source, rollback flag, previous digest. + - **Leg (i), held-evidence refresh, always runs.** For every server with `pending` or `changed` records, **including quarantined servers**: `Runtime.RefreshHeldScanEvidence(server)` takes the tool-approval lock and re-runs `scanToolEvidence` plus `recordScanHold` on the stored `Current*`. Evidence only: Status, ApprovedHash and ApprovedBy are never written; no unquarantine, no approve. A signature removed by a newer bundle leaves the tool held, with verdict clean `awaiting_review` (Spec 101 edge case). + - **Leg (ii), cached informational scan, gated by `security.auto_baseline_scan`** (the existing informational-scan kill switch, env included). For every **non-quarantined** server with at least one `approved` record, any trust mode: `Service.StartScanFromCachedContracts(ctx, server, digest)` builds `ToolDefinitions` from the approved records' cached contracts and peers from the StateView, with `ScanContext.SourceMethod="tool_definitions_only"`, accurate `ToolContractDigests`, scanner `tpa-descriptions`, `SourceDir=""`. It never calls `IsConnected`, `EnsureConnected` or `GetServerTools`, never schedules Pass 2, and is excluded from `scans_completed`/`scans_with_findings`. Quarantined servers are skipped, so `maybeAutoApproveScanSettled` and `maybeCaptureReviewDefinitions` (both quarantined-only) can never act on a reprotect job. + - With the switch off, leg (ii) does not run and `reprotect.state=suspended`, `reason=auto_baseline_scan_off` is shown on every status surface. This dependency is recorded in the Spec 101 FR-019 amendment and flagged for maintainer sign-off; no new config key. + - Pacing: serial under `infoScanRunMu`, a 2s pause between servers, busy re-arm (`ErrScanInProgress` re-arms without spending a failure), at most 3 attempts per server, one job per (server, digest). + - Storm bounds: one sweep emits at most one finding per tool (structural, `detect.aggregate`), at most one activity record per server, and a bounded number of SSE events; a rollback is one more history row and therefore one more sweep; the same digest re-fired yields nothing. + - Persistence: `storage.BundleReprotectState{FullDigest, CompletedAt}` is written only when a sweep completes with zero failures. A startup catch-up after `setSecurityScanner` re-runs the sweep when the marker differs from the active digest (covers the construction-time activation, a crash mid-sweep and a binary upgrade that changes the embedded corpus). + - Shutdown: handler goroutines are tracked by a `WaitGroup` on the Server and bound to `informationalScanContext()`, with `ctx.Err()` checked before every store write. `Server.Shutdown` (`server.go:1354`) unsubscribes and waits for them before `s.runtime.Close()` (:1408). + - stdio mode has no scanner `Service`, so leg (ii) is inert there (documented). Leg (i) runs wherever a Runtime exists. + - The report lists all current hard hits; "new" is not diffed against the previous bundle. The new signature's TPA id is in the finding's signals. +- **B18. Where reprotect findings surface.** The existing scan summary, report, server-list badge, Security overview, activity record, and the per-tool `scan_verdict` in the server review payload (`GetServerReview`, `reviewToolScanVerdict`, which uses findings when the cached scan covers the tool's current contract). SC-009's "surfaced for review" is asserted on that review payload (T391). No new attention or review-queue kind: an approved-only server does not enter `/review` or Home attention. That is recorded as an **open maintainer decision** (needs the Spec 109 parity matrix, Swift and Web), not as a silently weakened MUST. +- **B19. Lifecycle ownership.** + - `scanner.NewBundleRefresher(BundleRefresherOptions{Clock, Rand, HTTPClient, Transport, Resolver, ConfigProvider, Logger, Version})`, installed with `scanner.SetBundleRefresher` (package-level, replaceable slot). + - Constructed in `internal/server/server.go` right after `rt.SetVersion`, registered with `Runtime.SetBundleRefresher`, started in `StartBackgroundInitialization` with `r.appCtx` on every transport, including dev builds (no semver gate). + - Stopped and joined in `Runtime.Close` before `storageManager.StopAsync`; `Close` also clears the scanner slot if it still points at this instance. A Start after Stop is a no-op (telemetry pattern). + - `RefreshBundle` checks the slot's refresher is running; otherwise it falls back to `ReloadBundle`. + - Hot reload: the existing `config.reloaded` listener's `reapplyScannerSecurityConfig` calls `refresher.Nudge(sec)`, non-blocking. The refresher reads live `SecurityConfig` through its provider every cycle and never takes `Runtime.mu`. +- **B20. Manual refresh.** + - `mcpproxy security bundle refresh [-o json|yaml]` → `POST /api/v1/security/bundle/refresh` (`requireServerOp(auth.ServerOpBundleManage)`, agent-denied) → `Service.RefreshBundle(ctx)` → `scanner.RefreshBundle(ctx, logger)`: one joined cycle (file leg plus fetch leg), trigger `manual`. Synchronous 200 with `BundleInfo`; the outcome is in `refresh`. + - `reload` stays as it is: the local path only, never network. A test asserts zero HTTP requests on `reload` with fetch enabled. + - The Web UI gets no "Check now" button. +- **B21. Telemetry (schema 16).** Five non-negative integer counters on the `tpa_scanner` object through `tpaScannerScalarKeys`, **not** through the typed `tpaBundleKeys` whole-tuple group: `bundle_refresh_cycles`, `bundle_refresh_activations`, `bundle_refresh_rejections`, `bundle_refresh_unavailable`, `bundle_reprotect_hits` (servers whose reprotect scan had at least one hard finding). + - Reset on accept; wired through `Record*On`, `RegistrySnapshot`, `TPAScannerStats`, `Snapshot`, `Drain`, `HasPendingCounters`, `Reset` and `isZero`. + - `internal/telemetry` must not import scanner (`telemetry/tpa_bundle.go:5-11`). The class is a telemetry-local string enum `telemetry.TPABundleRefreshClass` in `internal/telemetry/tpa_bundle_refresh.go`. The producer `scanner.SetBundleRefreshObserver(func(RefreshOutcomeClass))` carries only scanner's own enum, and `internal/server/server.go` maps it to the telemetry enum. + - `internal/server/tpa_bundle_telemetry.go` maps `source=fetch`, which `anonymity.go` already allows. The provenance gate stays: an operator-signed fetch reports `other` with no sequence. + - The heartbeat never carries age, `generated_at`, `last_checked_at`, URL, host, reason strings or digests (FR-021, SC-006). The telemetry opt-out suppresses the counters like every other key. + - The server-side freshness join (SC-006) lives in the receiver and is out of scope; T394 verifies the receiver accepts v16 and the new counters, live, before and after deploy (memory: verify-telemetry-in-browser). +- **B22. No new dependencies.** Stdlib `net/http` and `crypto` only. +- **B23. Spec 087 US3 (release awareness) is covered by Spec 079 and not touched.** `internal/updatecheck` already checks daily (24h default), surfaces `update` on `/api/v1/info`, the tray and the CLI, and respects the opt-outs. 087 FR-019's "distinct items" holds structurally: release status is on `/info`, bundle status on `/security/bundle`. 087 FR-018, FR-019, SC-007 and US3 AS1-AS3 map to Spec 079 with no US3 task. + +### Spec amendments + +Done in the PR that appends this section: the Spec 087 `spec.md` Status line and its short note. The rest is T300. + +- **Spec 101 `spec.md`**: + - SC-007 keeps only the manual-drop clause, with a pointer "fetch-cycle clause: Spec 087 SC-009". + - FR-017's `[NEEDS CLARIFICATION]` becomes "Assumed (zero-interruption), pending maintainer confirmation: GitHub Releases on the corpus repo; per-sequence names are selection hints; integrity from the signed pair and watermark; signed index deferred (tasks B4). The publication half is not implemented until the corpus repo and first publisher key exist." + - FR-016a(c) gains "Deviation (tasks Δ17/B14): the activation self-check is a parity-tested port in package scanner, because `cmd/scan-eval` is `package main`; a shared gate package is a follow-up." + - FR-019 gains "Review surface: the per-tool scan verdict on the server review payload and the server's scan summary (tasks B18). A dedicated attention/review-queue kind for approved tools is an open maintainer decision. The cached scan leg runs only while `auto_baseline_scan` is on; when off, status shows reprotect suspended (tasks B17)." +- **Spec 087 `spec.md`**: + - A "Mapping note" with the 087 table below; US3, FR-018/019 and SC-007 are "covered by Spec 079, unchanged". + - Add **SC-009** (moved from Spec 101 SC-007): "A newly published signature reaches detection within one daily refresh cycle on a fetch-enabled install, measured end to end: publish → active → the new signature's fixture fires." + - FR-013/FR-014: the trust set is Spec 101 FR-004/004b; no per-fetch key; an empty active trust set means zero requests plus a status/doctor configuration error. + - FR-016: no refresh-enable key (the file leg is network-free and always on); env override covers fetch enable only. + - Edge case "Hot-reload flips…": in-flight candidates under changed settings are `superseded`. + - FR-017: surfaces are `/security/bundle`, `/security/overview`, CLI, doctor and Web UI, not `/info`. +- **Spec 087 `plan.md`**: banner "Superseded by specs/101-tpa-db/tasks.md US3 (Δ15–Δ31). The package, config block and file layout below were never built." + +**Spec 087 FR/SC → task map** + +| Spec 087 | Tasks | +|---|---| +| FR-001, FR-002, FR-005, FR-006, FR-007 | US1 (shipped) | +| FR-003 | T320, T321 | +| FR-004 | T327 (daily file pickup, manual refresh with fetch off), T370–T373 (manual surface) | +| FR-008 | T323, T340–T343, T365 (production wiring), T391 | +| FR-009 | T321, T325, T327 | +| FR-010 | T327 (single-flight), T329 (atomic, non-blocking) | +| FR-011 | US1, plus T321 (fetch leg idempotence) | +| FR-012 | T326 | +| FR-013, FR-014 | T310, T321, T326, T374 | +| FR-015 | T324, T325, T327 | +| FR-016 | T310–T313, T327 (hot reload) | +| FR-017 | T322, T370–T377 | +| FR-018, FR-019, SC-007, US3 | Spec 079 (B23) | +| FR-020, FR-022 | US1 (unchanged); FR-022 re-run in T393 | +| FR-021 | T341 (provenance guard e) | +| SC-001, SC-008 | US1 / CI (unchanged) | +| SC-002 | T327 | +| SC-003 | T321, T325, T391 | +| SC-004 | T340, T342, T391 | +| SC-005 | T326, T391 | +| SC-006 | T329 | +| SC-009 (new) | T391 | + +**Spec 101 FR/SC → task map (US3 part)** + +| Spec 101 | Tasks | +|---|---| +| FR-016a(c) | T340–T343 (deviation recorded, Δ17) | +| FR-017 consumer half | T324, T391 | +| FR-017 publication half | Out of scope (repo missing, publisher key set empty) | +| FR-018 | B7, B8; T327 | +| FR-019 | T361–T366, T391 | +| FR-020 | T322, T370–T377 | +| FR-021 | T350–T352 | +| FR-022 | T392 | +| SC-006 | Heartbeat tuple: US1; counters: T350; server-side age join: out of scope (receiver), checked by T394 | +| SC-007 | Manual-drop clause: US1 + T327; fetch clause: moved to 087 SC-009 → T391 | +| SC-009 | T362, T366, T391 | + +--- + +## Wave 0: shared contract and spec amendments (sequential, one owner, no behaviour) + +- [ ] T300 docs: apply the remaining amendments above to `specs/101-tpa-db/spec.md` (SC-007, FR-016a(c), FR-017, FR-019), `specs/087-tpa-daily-refresh/spec.md` (mapping note, SC-009, FR-013/014/016/017, edge case) and `specs/087-tpa-daily-refresh/plan.md` (superseded banner). +- [ ] T301 impl (stubs only, each in its final file; builds; all existing tests green): + - `internal/config/config.go`: `SecurityConfig.TPARefreshInterval`, `TPAFetchEnabled`, `TPAFetchURL`, `TPAStaleAfterDays` (json/mapstructure tags); nil-safe accessors returning defaults `EffectiveTPARefreshInterval`, `IsTPAFetchEnabled` (no env yet), `EffectiveTPAFetchURL`, `EffectiveTPAStaleAfterDays`; const `EnvTPAFetchEnabled`; `tpaRefreshErrors()` returning nil, already appended in `ValidateDetailed`. + - `internal/security/scanner/tpa_bundle_source.go`: `BundleSourceFetch` and the new `BundleInfo` fields (B15). + - `internal/security/scanner/tpa_bundle_policy.go`: `bundlePolicy.FetchEnabled`, `FetchURL`, and `externalConfigured(policy, st)` returning `policy.Path != ""`. + - `internal/security/scanner/tpa_bundle_runtime.go`: `runOptions.offered *Candidate` (ignored), `selection.offeredRej/offeredDigest`, and `runOffered(c Candidate, log) BundleInfo` that delegates to a normal run. + - `internal/security/scanner/tpa_candidate.go`: `evalEnv.MustSign` (ignored). + - New `internal/security/scanner/tpa_refresh_status.go`: `BundleRefreshStatus`, `BundleReprotectStatus`, outcome/reason/trigger constants, `RefreshOutcomeClass`, `SetReprotectStatus` (no-op). + - New `internal/security/scanner/tpa_refresh.go`: `BundleRefresher`, `BundleRefresherOptions` (Clock, Rand, HTTPClient, Transport, Resolver, ConfigProvider, Logger, Version, IntervalOverride), `NewBundleRefresher`, `SetBundleRefresher`, `RefreshBundle(ctx, logger) BundleInfo` (falls back to `ReloadBundle`), `Start`, `Stop`, `Nudge`, `Running`, all no-op. + - New `internal/security/scanner/tpa_fetch.go`: `fetchResult`, `fetchCandidate(ctx, settings) fetchResult` returning `disabled`, `egressAllowed(ip, configuredClass)` returning true. + - New `internal/security/scanner/tpa_bundle_observer.go`: `BundleActivation`, `AddBundleActivationObserver` (no-op unsubscribe), `SetBundleRefreshObserver`, `drainBundleObserversForTest`. + - New `internal/security/scanner/tpa_selfcheck.go`: `EnableActivationSelfCheck()` no-op, `activationSelfCheck` returning nil, `selfCheckCorpus` seam. + - New `internal/security/scanner/tpa_bundle_doctor.go`: `BundleDoctorWarnings` returning nil. + - `internal/security/scanner/service.go`: `Service.RefreshBundle(ctx)` (delegates) and `Service.StartScanFromCachedContracts` returning `errors.New("not implemented")`. + - `internal/httpapi/security_scanner.go`: `RefreshBundle` on `SecurityController`; mock in `internal/httpapi/security_bundle_test.go`. + - `internal/server/server.go`: `StartScanFromCachedContracts` on the `securityScannerService` interface; fake in `internal/server/scan_admission_test.go`. + - New `internal/server/scan_bundle_reprotect.go`: `handleBundleActivation(BundleActivation)`, `runBundleReprotectCatchUp(ctx)`, `bundleReprotectWG`, all no-op. + - New `internal/telemetry/tpa_bundle_refresh.go`: `TPABundleRefreshClass` constants, `RecordTPABundleRefreshOn(reg, class)` and `RecordTPABundleReprotectHitOn(reg)` no-op. + - `internal/runtime/runtime.go`: field `bundleRefresher`, `SetBundleRefresher`, and a `closeOrderHook func(step string)` test seam. New `internal/runtime/tool_quarantine_held_refresh.go`: `RefreshHeldScanEvidence(server) error` returning nil. + - `internal/storage/models.go` + `internal/storage/manager.go`: `BundleReprotectState`, `LoadBundleReprotectState`/`SaveBundleReprotectState` returning zero/nil. +- [ ] T302 verify (Wave 0 exit): `go build ./... && go build -tags server -o /dev/null ./cmd/mcpproxy && go vet ./... && go vet -tags server ./... && go test -count=1 -run '^$' ./...` (compile-only). Each Wave 1 group's first commit (its tests) must compile against these stubs; `go vet` on the group's packages is the per-group check. + +## Wave 1: Phase-1 groups (parallel worktrees; disjoint files; each group writes its tests first) + +### G1: Config (087 FR-013/014/016, B1, B2) +- [ ] T310 [P] test: create `internal/config/tpa_refresh_config_test.go`. Defaults (24h, fetch off, empty URL, stale 30); accessor nil-safety on a zero `Config`; env `MCPPROXY_TPA_FETCH_ENABLED` true/false overrides config and is not persisted by a save round-trip; `ValidateDetailed` rejects interval < 1h, `http://` on a non-loopback host, userinfo, query, fragment, fetch enabled with an empty URL, fetch enabled with root `operator` and no signers, negative stale days, with messages that never echo the URL; `Validate()` (boot) accepts all of them; explicit `tpa_stale_after_days: 0` survives a JSON round-trip. Fails on the stubs, which ignore env and validation. +- [ ] T311 [P] test: in `internal/oauth/configview_tpa_test.go` and `internal/oauth/configview_tpa_scope_test.go`, assert the four new keys survive `RedactedConfig` verbatim (a plain https URL is not masked) and round-trip GET→PATCH. In `internal/config/zero_value_preservation_test.go`, add the `numericOmitemptyAllowlist` entry `Config.Security.TPARefreshInterval` with a reason. In `internal/runtime/config_hotreload_test.go`, extend `TestDetectConfigChanges_TPATrustRoot` (or a sibling) so each new key reports `security` changed. No change to the nullable-bool fixture (no new `*bool`). +- [ ] T312 [P] impl: in `internal/config/config.go`, `tpaRefreshErrors()`, the env read inside `IsTPAFetchEnabled`, and the clamp in `EffectiveTPARefreshInterval`. +- [ ] T313 [P] docs: rows for the four keys and the env var, with the "no working default URL" and "fetch needs an operator root today" notes, in `docs/configuration.md` (sample JSON, table, prose). + +### G2: Runtime core — external-source persistence, offered candidate, fetch adapter, scheduler, status (087 FR-003/004/009–015/017, B4–B12, B15, B19) +- [ ] T320 test: `internal/security/scanner/tpa_refresh_runtime_test.go`, the Hazard A regression. `newRTFixture` with `TPABundlePath=""`, `TPAActiveRoot=operator`, `TPAOperatorBaseline=u64p(1)`, fetch enabled; offer an operator-signed seq 2 candidate (`Source=fetch`) through `runOffered`. Assert it activates with one history row, then: (a) `ConfigureBundleFromConfig` again keeps it serving, `Active` unchanged, LKG files present; (b) `f.restart()` restores it from LKG; (c) disabling fetch keeps it serving (B9); (d) `rollback --embedded` returns to embedded, and the next offer of the rolled-away digest reports `deny_listed`/`unchanged` quietly (one log line total), with no reactivation and no observer fire. Fails today: the `Path != ""` guards deactivate the bundle and reconcile deletes its LKG. +- [ ] T321 test: same file, offered-run semantics. + - The offered run never reads the file and never sets the file-leg `load_error`; its refusal is recorded once in `last_rejection`. + - Direct-mode unsigned (`SidecarPresent=false`, `MustSign`) is refused as `signature_required` with `require_signed_bundle=false`. + - An unsigned file-drop bundle is serving; a refused fetch offer leaves it serving (no `deactivated`). + - Over-limit bodies give `ceiling_exceeded`. + - An offered candidate equal to the active bytes gives `unchanged` with zero compiles (`countCompiles`); a refused pair offered again gives `unchanged` with zero compiles and no second log. + - Fetch active plus a stale file: the file leg logs one `downgrade`; the fetched bundle keeps serving; fetch status is not masked. + - Equal sequence, different digest on the fetch leg: `downgrade`. + - A byte-identical fetched pair while the file copy is active: source stays `file`, no activation, no observer fire. + - Pinned after rollback, a strictly newer seq offered: activates and clears the pin (US1 A4). + - Operator fetch with no `tpa_operator_baseline`: activates degraded (`operator_authority_unbaselined`). + - Parity (US3 AS1): the same signed pair via `runOffered` and via file drop gives the identical (digest, sequence, authority, signature_verified, coverage) tuple, differing only in source. + - `BundleInfo.Source == "fetch"`, `Path` is host-only. + - `policy.Fingerprint` changes when `FetchEnabled` or the URL changes, and not when the interval or stale threshold changes. +- [ ] T322 test: `internal/security/scanner/tpa_refresh_status_test.go`. `last_checked_at` advances on a `canSkipLocked` no-op; `last_activated_at` equals `Active.ActivatedAt` for an external bundle and is nil for embedded; `stale`/`stale_after_days` at read time with an injected clock (29/30/31 days, threshold 0 never stale, `generated_at` in the future never stale and no negative age); a `tpa_stale_after_days` edit applied through `ConfigureBundleFromConfig` changes `stale` on the next read without a pipeline rerun; `last_trigger` is `startup` for the first configure and `config` for a re-apply; refresh status merges into both `BundleStatus()` and `statusFromSnapshot()` without republishing (the `storeBundle` count is unchanged). +- [ ] T323 test: `internal/security/scanner/tpa_candidate_selfcheck_mode_test.go`, `withSelfCheck` counting calls: `evalNormal` 1, `evalRollback` 1, startup LKG restore 0, pin restore 0. Fails today: the hook runs in every mode. +- [ ] T324 test: `internal/security/scanner/tpa_fetch_test.go` (httptest, loopback http). Releases mode: one `scanner-bundle-.json` plus `.sig` → candidate; zero, two, or a missing `.sig` asset → `channel_invalid`; asset seq at or below the watermark → `unchanged` with no download; ETag sent and `304` → `not_modified` with no download; the JSON body is capped at 1 MiB (oversize → `channel_invalid`). Direct mode: bundle plus `URL+".sig"`, `If-None-Match` on the bundle, `.sig` 404 → `SidecarPresent=false`. `User-Agent` and `Accept` present. +- [ ] T325 test: `internal/security/scanner/tpa_fetch_egress_test.go`. + - `https` → `http` redirect refused; more than 5 hops refused. + - Redirect from a public-configured host to loopback or link-local refused through the dial-control seam (injected resolver); link-local and metadata always refused; with `HTTPS_PROXY` set and an inward-resolving target, refused at the application layer. + - Status mapping: `403`/`429`/`Retry-After`/`X-RateLimit-Remaining: 0` → `fetch_unavailable/rate_limited` and `next_check_at` moves to the reset (cap 24h); timeout → `timeout`; DNS failure → `network_error`; a trickled body (headers fast, body slow, fake clock) is cut by the 20s cycle deadline → `timeout`. None writes `last_rejection` or changes the watermark. + - Host-only logging: a URL whose path is `/s3cr3t-path` never appears in zap observer logs, `BundleInfo`, `last_rejection` or error strings. +- [ ] T326 test: `internal/security/scanner/tpa_fetch_gate_test.go`. With fetch enabled, **zero requests** for: empty URL; invalid URL; root `publisher` with the empty publisher set (`no_trusted_signer`); root `operator` with empty `tpa_operator_signers`; a `loadFailed` store; and fetch disabled across a full simulated day of cycles. +- [ ] T327 test: `internal/security/scanner/tpa_refresh_scheduler_test.go`, fake clock and rand. + - Cadence: the first cycle lands in `[30s, 5m]`; later cycles at `interval ± 10%`; a 10-day forward jump yields exactly one cycle; a backward jump does not wedge the timer; an interval change via `Nudge` re-arms it. + - File leg (087 US1, FR-004, SC-002): a valid signed file dropped after `Start` activates on the next cycle with no restart (`source=file`, `last_trigger=timer`); `RefreshBundle` with fetch disabled activates a dropped file, returns the new `BundleInfo`, and issues zero HTTP requests; a refused dropped file sets `last_rejection` and the previous bundle keeps serving. + - Single-flight: a manual `RefreshBundle` during an in-flight cycle joins it (one download, one activation); tick + `ReloadBundle` + `RefreshBundle` racing yield one activation per distinct candidate. + - Mid-flight `Nudge` that changes the URL, disables fetch, changes the active root or operator signers, or flips `tpa_bundle_path` → `superseded`, no activation; a `Nudge` while `Stop` is in progress is a no-op. + - `Stop` cancels in-flight HTTP and returns only after the goroutine exits; a later `Start` is a no-op; `RefreshBundle` on a stopped refresher falls back to `ReloadBundle`; `SetBundleRefresher(nil)` clears the slot. + - `ReloadBundle` issues zero HTTP requests with fetch enabled (B20). + - `consecutive_failures` increments on `fetch_unavailable`/`channel_invalid` and resets on `not_modified`, `unchanged`, `activated` and `rejected`; after a plain failure `next_check_at` equals the next jittered cycle. + - Failure logs are deduplicated per (outcome, reason); the refresh observer receives exactly one class per cycle. +- [ ] T328 test: `internal/security/scanner/tpa_bundle_observer_test.go`. The activation observer fires once per history row (activation and rollback), not on an idempotent rerun, LKG restore, refusal or embedded fallback; it runs outside `bundleRT.mu` (an observer calling `BundleStatus()` does not deadlock; 10s timeout); unsubscribe stops delivery; `drainBundleObserversForTest` waits for in-flight dispatches. +- [ ] T329 test: `internal/security/scanner/tpa_refresh_concurrency_test.go` (087 FR-010, SC-006). Scans run in a loop through `ScanToolMetadataVerdict` while (a) the fetch leg is blackholed and (b) an activation runs with a gated, deliberately slow self-check hook. Every result is drawn from exactly the old or the new digest, and scan latency during the run stays within a relative factor of a steady-state baseline measured in the same test. The latency assertion is skipped under `-race`; the old-or-new assertion always runs. +- [ ] T330 impl: + - `internal/security/scanner/tpa_bundle_policy.go`: fetch fields, `externalConfigured`, fingerprint. + - `internal/security/scanner/tpa_bundle_runtime.go`: offered-only branch with the under-lock unchanged/negative-cache compare; `offeredRej` routing; the three guard sites (B9); `buildInfoLocked` source from `sv.cand.Source`; `last_checked_at`/`last_activated_at`; the watermark snapshot; observer fire after publish; display-settings refresh in `configureWithPolicy`. + - `internal/security/scanner/tpa_candidate.go`: `MustSign` in `unsigned()`; self-check only for `evalNormal` and `evalRollback`; comment updated. + - `internal/security/scanner/tpa_bundle_source.go`: read-time stale fields and the status merge. + - `internal/security/scanner/tpa_bundle_rollback.go`: observer fire on rollback. + - `internal/security/scanner/tpa_bundle_observer.go`, `internal/security/scanner/tpa_refresh_status.go`. +- [ ] T331 impl: + - `internal/security/scanner/tpa_fetch.go`: mode detection, bounded reads, ETag cache, rate-limit parsing, redirect policy, dial-control and proxy-target predicate, host-only error unwrapping. + - `internal/security/scanner/tpa_refresh.go`: scheduler, jitter, re-armed timer, generation counter, joinable in-flight cycle, file leg then fetch leg, Start/Stop/done, `RefreshBundle`, `Nudge`, `Running`, refresh observer. + +### G3: Activation self-check (087 FR-008/FR-021, SC-004, 101 FR-016a(c), B13, B14) +- [ ] T340 [P] test: `internal/security/scanner/tpa_selfcheck_test.go`, one-rule regex `tool_description` bundles over the embedded corpus: catch-all `(?s).+` rejects (FP 21/21); `(?i)\bthe\b` rejects; `(?i)system prompt` rejects (2/21); `(?i)password|secret|token|credential` passes (1/21, boundary); `ZZZ-NEVER` passes and logs "recall leg inert" when the corpus has no `tpa_*` malicious entries, or rejects when it has some; the embedded default passes (`loadEmbeddedBundleCheck`). Through the `selfCheckCorpus` seam: a corpus with no gated malicious, one with no hard negatives, and an unparsable one all reject; `ChecksFailed > 0` rejects. The detail contains no candidate pattern text. Fails on the stub, which accepts everything. +- [ ] T341 [P] test: `internal/security/scanner/tpa_selfcheck_parity_test.go`, B14 guards (a), (b) tolerant to both gate.go shapes, (c), (e) and optional (d). Fails until the corpus copy exists. `TestSelfCheck*` names only. +- [ ] T342 [P] test: `internal/security/scanner/tpa_selfcheck_pipeline_test.go`, after `EnableActivationSelfCheck()` (restored on cleanup): a catch-all file drop through `ConfigureBundle(path)` gives `last_rejection.reason == gate_failure` while embedded keeps serving; the embedded bytes written as a file activate as `source=file`. Cost is asserted as "at most one self-check per distinct candidate" (counted), not by wall time. Real self-checks in the suite are kept to a handful. (The fetch-sourced catch-all case is in T391, since it needs G2.) +- [ ] T343 [P] impl: `internal/security/scanner/tpa_selfcheck.go` (port per B13; `selfCheck*` identifiers only; no `bundleRT`/`activeBundle*` access) and `internal/security/scanner/bundled/selfcheck-corpus.json` (byte copy of `specs/065-evaluation-foundation/datasets/detect_corpus_v1.json`, provenance and licence fields intact). + +### G4: Telemetry v16 (101 FR-021, SC-006, B21) +- [ ] T350 [P] test: + - `internal/telemetry/tpa_bundle_refresh_test.go` and `internal/telemetry/anonymity_test.go`: `SchemaVersion == 16`; the five counters accepted as non-negative ints and rejected as strings, negatives or floats; counters alone emit `tpa_scanner` (the `isZero` gap); `Drain`/`Reset` zero them and `HasPendingCounters` sees them; with telemetry disabled no counter or `tpa_scanner` key is sent; the heartbeat never contains age, URL, host or reason keys. + - Bump every schema-15 pin to 16, found by `grep -rnE 'schema_version.*15|SchemaVersion.*15|want 15|\b15\b' internal/telemetry/*_test.go` (includes `payload_v2_test.go:92`'s message); these fail until T351 bumps the constant. + - `internal/server/tpa_bundle_telemetry_test.go`: `source=fetch` with a publisher-verified bundle reports `fetch`; an operator-signed fetch reports `other` with no sequence. + - Fails today: counters are unknown, the schema is 15, and fetch maps to `file`. +- [ ] T351 [P] impl: `internal/telemetry/anonymity.go` (append to `tpaScannerScalarKeys`); `internal/telemetry/registry.go` (fields, Record/RecordOn, snapshot, drain, pending, reset); `internal/telemetry/tpa_scanner.go` (struct fields and `isZero`); `internal/telemetry/tpa_bundle_refresh.go` (real recorders over the telemetry-local enum); `internal/telemetry/telemetry.go` (`SchemaVersion = 16` plus changelog); `internal/server/tpa_bundle_telemetry.go` (fetch mapping). +- [ ] T352 [P] docs: `docs/features/telemetry.md`: backfill the v15 `bundle_*` rows and add the v16 counter rows. + +### G5: Server and runtime wiring, plus the re-protect sweep (101 FR-019, SC-009, B13, B16, B17, B19) +- [ ] T360 [P] test: `internal/runtime/bundle_refresher_lifecycle_test.go`. `SetBundleRefresher` then `StartBackgroundInitialization` starts it; `Close` joins it **before** `StopAsync` (via `closeOrderHook`) and clears the scanner slot; a `Close` racing a late `Start` leaves no goroutine (goroutine counter). +- [ ] T361 [P] test: `internal/runtime/tool_quarantine_held_refresh_test.go`, with the `rescanRuntime` fixtures of `tool_quarantine_auto_rescan_test.go`. `RefreshHeldScanEvidence` updates `HeldVerdict`/`HeldSignals` on `pending` and `changed` records from the newly published bundle (including a pending record whose stored verdict `tool_quarantine.go:1199` would skip); it never changes `Status`, `ApprovedHash`, `ApprovedBy` or `Quarantined`; a signature removed by a newer bundle leaves the record held with verdict `clean`/`awaiting_review`; approved records are untouched. +- [ ] T362 [P] test: `internal/server/scan_bundle_reprotect_test.go`, built from `newRescanTestServer` and `fakeSecurityScanner`, calling `handleBundleActivation` with a synthetic `BundleActivation` (no G2 dependency). + - One `StartScanFromCachedContracts` per non-quarantined server with approved records; `RefreshHeldScanEvidence` for servers with held records, quarantined included; quarantined servers never get a scan job. + - The same digest re-fired produces no job; a rollback activation produces exactly one more sweep. + - Busy servers re-arm without spending a failure; failures cap at 3. + - `auto_baseline_scan=false`: no scan jobs, leg (i) still runs, and `scanner.SetReprotectStatus` receives `suspended/auto_baseline_scan_off`. + - No `EnsureConnected`, `RestartServer` or `GetServerTools`; no `ApproveServer` or `ApproveTools`. + - Storm bounds: one sweep over N servers → at most one activity record per server and a bounded number of SSE events. + - Startup catch-up runs when `BundleReprotectState.FullDigest` differs from the active digest and is skipped when equal; the marker is written only on a zero-failure completion. + - `Shutdown` during a running sweep: the observer is unsubscribed, the sweep goroutine exits, and no store write happens after `runtime.Close` begins. +- [ ] T363 [P] test: `internal/security/scanner/service_cached_scan_test.go`. `StartScanFromCachedContracts` builds a job from the ingested approved contracts with `SourceMethod "tool_definitions_only"` and accurate `ToolContractDigests`; calls the server-info provider's `IsConnected`, `EnsureConnected` and `GetServerTools` **zero** times; schedules no Pass 2 with deep scan on; is not counted in `scans_completed`; a server with zero records reports unscanned, never clean. +- [ ] T364 [P] test: `internal/storage/bundle_reprotect_state_test.go`. Round trip; absent → zero value without creating the bucket; a corrupt value → error, not panic; the record is not touched by server purge. +- [ ] T365 [P] test: `internal/server/tpa_bundle_wiring_test.go`. After server construction: `bundleSelfCheck` is non-nil; a `config.reloaded` event reaches the refresher's `Nudge` (fake refresher); the doctor warning source is registered, and calling it on a fresh, never-published runtime returns within the timeout; the refresh observer increments the telemetry counters; the activation observer is registered at construction and unsubscribed on `Shutdown`. Fails on the W0 no-op wiring. +- [ ] T366 [P] test: `internal/server/scan_bundle_reprotect_rt02_test.go` (real runtime, real tool-approval store, real scanner `Service`, real `EventTypeSecurityScanSettled` handler). A non-quarantined server holds approved and held (`pending`, `changed`) tools; an unsigned file-drop bundle that removes the held tools' signature is activated with `ConfigureBundle`, then `handleBundleActivation` runs. Assert the held tools stay `pending`/`changed` with `ApprovedHash` unchanged, the server stays unquarantined, no `ApproveTools`/unquarantine/`ApproveServer` is called, and `maybeAutoApproveScanSettled`/`maybeCaptureReviewDefinitions` take no action on the cached job's settle event. +- [ ] T367 [P] impl: `internal/runtime/runtime.go` (refresher field, setter, `Close` join and slot clear), `internal/runtime/lifecycle.go` (Start in `StartBackgroundInitialization`), `internal/runtime/tool_quarantine_held_refresh.go` (`RefreshHeldScanEvidence`). +- [ ] T368 [P] impl: `internal/security/scanner/service.go`: `RefreshBundle` and `StartScanFromCachedContracts` (reuses `withDivergentIngestedContracts`, `toolDefinitionHashes`, `allServerToolsProvider`; a callback adapter that never starts Pass 2 and is marked non-telemetry). +- [ ] T369 [P] impl: + - `internal/storage/models.go` and `internal/storage/manager.go`: real `BundleReprotectState` Load/Save, modelled on `BaselineSweepState`. + - `internal/server/scan_bundle_reprotect.go`: handler, serial sweep, catch-up, `WaitGroup`, `SetReprotectStatus` calls. + - `internal/server/server.go`: `EnableActivationSelfCheck()` next to `configureTPABundleRuntime`; refresher construction after `rt.SetVersion`; `Nudge` in `reapplyScannerSecurityConfig`; refresh observer → telemetry enum mapping; activation observer registration next to `registerToolChangeRescan`; catch-up next to `go s.runBaselineSweep(ctx)`; unsubscribe and `WaitGroup` wait in `Shutdown` before `s.runtime.Close()`; `AddRuntimeWarningSource(func() []string { return scanner.BundleDoctorWarnings(scanner.BundleStatus(), time.Now()) })`. + - Run `scripts/check-upstream-write-sites.sh` and `scripts/check-no-unquarantine-callers.sh` (and their `.test.sh`) in the group. + +### G6: Surfaces — REST, CLI, doctor and Web UI (101 FR-020, 087 FR-017, B15, B16, B20) +- [ ] T370 [P] test: `internal/httpapi/security_bundle_test.go`: `POST /api/v1/security/bundle/refresh` in the `bundleRoutes` table (agent token 403, admin 200 with `BundleInfo` including `refresh`, unauth 401); `GET /api/v1/security/bundle` and `GET /api/v1/security/overview` carry `last_checked_at`, `last_activated_at`, `stale`, `refresh` and `reprotect` from the mock. `internal/httpapi/agent_token_gating_test.go`: `bundle-refresh` in `mutatingServerRoutes`. The tenant walk covers the route automatically. Fails today with 404 and missing fields. +- [ ] T371 [P] impl: `internal/httpapi/server.go` (route next to `/bundle/reload` with `requireServerOp(auth.ServerOpBundleManage)`), `internal/httpapi/security_bundle.go` (`handleRefreshSecurityBundle` with swag annotations; GET `/security/bundle` and refresh annotated as `contracts.SuccessResponse{data=scanner.BundleInfo}`). +- [ ] T372 [P] test: `cmd/mcpproxy/security_bundle_cmd_test.go`. `security bundle refresh` POSTs `/bundle/refresh`; human output shows `Last checked`, `Last activated`, `Refresh: (), next check …`, `Fetch host` when fetch is enabled, `Re-protection: `, and a `STALE` line when `stale`; JSON/YAML pass the fields through; embedded with no refresh shows no stale line and no host line. Fails on the missing subcommand and lines. +- [ ] T373 [P] impl: `cmd/mcpproxy/security_cmd.go`: `newSecurityBundleRefreshCmd` registered next to reload (help text updated) and `signatureBundleLines` extended; one edit covers status, overview and mutation output. +- [ ] T374 [P] test: `internal/security/scanner/tpa_bundle_doctor_test.go`, the B16 truth table: stale external; stale embedded with fetch on/off; a stuck rejection newer than activation; the quiet set; ≥3 consecutive fetch failures; fetch enabled with `no_url`/`invalid_url`/`no_trusted_signer`; strings contain no hex digest, `/` path or host. `cmd/mcpproxy/doctor_cmd_test.go`: a runtime warning renders under Runtime Warnings and suppresses "All systems operational". +- [ ] T375 [P] impl: `internal/security/scanner/tpa_bundle_doctor.go`. +- [ ] T376 [P] test: `frontend/tests/unit/signature-bundle-refresh.spec.ts`: `formatSignatureBundle` emits "Last checked", "Last activated", "Refresh" and "Re-protection" rows next to Age; `refreshProblem` is non-null only for a B16-style condition and `degraded` stays coverage-only; `SignatureBundlePanelBody.vue` renders the rows; the panel has no mutation button and no `/unquarantine` or approve references. Fails on the current formatter. +- [ ] T377 [P] impl: `frontend/src/utils/signatureBundle.ts` (interface fields, rows, `refreshProblem`, outcome labels) and `frontend/src/components/security/SignatureBundlePanelBody.vue`. + +## Wave 2: integration, docs, verification (sequential; after G1–G6 merge into the US3 branch) + +- [ ] T390 impl: `make swagger` regenerates `oas/swagger.yaml` and `oas/docs.go` (config keys, refresh route, `BundleInfo`/`BundleRefreshStatus`/`BundleReprotectStatus` schemas); then `make swagger-verify`. +- [ ] T391 test: `internal/server/tpa_bundle_fetch_e2e_test.go`, end to end with production wiring (real self-check via `EnableActivationSelfCheck`, real refresher, real activation observer). A loopback httptest serves Releases-API JSON whose `browser_download_url`s point back at it, with an operator-signed seq baseline+1 pair adding one narrow, fixture-specific signature (≤1 FP over the 21 hard negatives, B13). + - Fetch enabled, interval seam 100ms: activates within one cycle with `source=fetch`, `signature_verified=true`, `generated_at`, refresh `activated`; the signature's fixture fires through `ScanToolMetadataVerdict` (087 SC-009, 101 SC-007 fetch clause). + - Re-protection (101 SC-009): an approved tool whose cached description matches the new signature gets a cached scan with a `dangerous` summary naming `tpa..*`, and its `scan_verdict` in `GetServerReview` reflects the finding; the record stays `approved`, ApprovedHash unchanged; the server is not quarantined; the fake upstream gets zero calls; `bundle_reprotect_hits` is 1. + - A catch-all fetched pair: `gate_failure`, the previous bundle keeps serving (087 SC-004 at activation). + - A tampered pair: `tamper`, LKG serving, no sweep fired. + - The httptest server stopped: `fetch_unavailable`, bundle unchanged. + - An unsigned direct-mode body: `signature_required`. +- [ ] T392 docs: `docs/features/tpa-bundle-signing.md` ("Daily refresh and optional fetch": keys, operator-root prerequisite, channel shapes, air-gapped drop of the downloaded pair, host-only logs, outcomes vs rejections, `refresh` vs `reload`, rollback for fetch-only installs, status-field and command table rows); `docs/features/security-quarantine.md` ("Re-protection after a bundle activates": cached-only, RT-02 invariants, `auto_baseline_scan` dependency, stdio limitation); `docs/features/tool-scanner.md` (the self-check gate); `docs/cli/security-commands.md` (`bundle refresh`); `specs/101-tpa-db/quickstart.md` (optional fetch step against a local mirror); `CLAUDE.md` Recent Changes. No new docs page, so no docs-site allowlist edit. +- [ ] T393 verify: `./scripts/test-api-e2e.sh`; `go test -race ./internal/security/... ./internal/config/... ./internal/telemetry/... ./internal/runtime/... ./internal/storage/...`; the CLAUDE.md `-tags server -skip` race run over `internal/server`, `httpapi`, `storage`; `go test -tags server ./internal/serveredition/... -race`; golangci-lint v2 bare and with `--build-tags server`; `scripts/check-upstream-write-sites.sh` and `scripts/check-no-unquarantine-callers.sh`; `cd frontend && npm run test:unit`; the Web UI Playwright sweep per `docs/development/web-ui-verification.md`. +- [ ] T394 verify: telemetry receiver and dashboard (repo-external). Before merge, confirm the receiver accepts schema 16 and the five counters (staging or a dry-run payload) and load the live dashboard before and after deploy. If the receiver needs a change, it is a separate receiver PR that blocks this merge. The server-side SC-006 age join stays out of scope. +- [ ] T395 review: review workflow, then cross-model review (≤ 10 rounds; chunked, file-named briefs). The concurrency (`tpa_refresh.go`, runtime offered branch), egress (`tpa_fetch.go`) and RT-02 (`scan_bundle_reprotect.go`, held refresh) diffs go to the hard-problem reviewer. +- [ ] T396 docs: open the PR (`Related #` issue refs, one per issue). The body checklist: (1) the B14 merge-order gate (copy refreshed here if US2 merged first; otherwise a comment on #1702); (2) the empty publisher key set and "fetch works today only with operator-signed bundles"; (3) zero-interruption assumptions needing maintainer sign-off: Δ30 `superseded`, Δ19 status off `/info`, B9 keep-on-disable, B13 `evalLKG` skip, B17 `auto_baseline_scan` gating, B18 no attention kind, Δ17 FR-016a(c) deviation, Δ22/FR-017 channel. Tick this file. + +--- + +### Out of scope (recorded so nothing is silently dropped) + +- Creating the `smart-mcp-proxy/tpa-db` corpus repository, its release and signing workflow, and the "refuse to replace an existing asset" automation (research D1): the publication half of 101 FR-017. +- Shipping the first publisher key or bumping `publisherTrustEpoch` (US1 A1). This is a binary-release event. +- Research D1's **signed release index**, content-addressed asset names and a stable cacheable index URL. US3's integrity does not depend on them (B4); they become necessary for freeze or selection protection with expiry. +- A compiled-in default fetch URL (B3). +- A dedicated attention or review-queue kind for "approved tool now matches signature X" (open maintainer decision; needs the Spec 109 parity matrix, Swift and Web; B18). Diffing "new" against the previous bundle's hits (B17). +- The receiver-side SC-006 freshness join (publication date × reported version/sequence); only its acceptance of v16 is checked (T394). +- A Web UI "Check now" button and an SSE event for bundle changes; panels keep the 30s poll. +- Persisting `last_rejection`, refresh status or ETags across restarts (restart-scoped; re-established by the first jittered cycle). +- Extracting a shared SSRF leaf package from `internal/registries` (B11 duplicates the predicate minimally). +- Moving the gate into a shared importable package used by both `cmd/scan-eval` and the scanner (needs `cmd/scan-eval` edits after US2); until then Δ17 is a recorded deviation from 101 FR-016a(c). +- Re-running `RefreshPrompts` on activation (an existing drop-gate; needs its own decision). +- Spec 087 US3 / FR-018 / FR-019 / SC-007 release awareness: already delivered by Spec 079 (B23). +- Settings UI or native catalogue entries for the new keys (consistent with the existing `tpa_*` keys: config file and CLI only). + +### Phase 1 groups + +| Group | Tasks | Files owned (exclusive in Wave 1) | Depends on | +|---|---|---|---| +| W0 shared contract | T300–T302 | `specs/101-tpa-db/spec.md`, `specs/087-tpa-daily-refresh/{spec,plan}.md`; the stub hunks and new stub files listed in T301 | — | +| G1 Config | T310–T313 | `internal/config/config.go` (validation and accessors), `internal/config/tpa_refresh_config_test.go`, `internal/config/zero_value_preservation_test.go`, `internal/oauth/configview_tpa_test.go`, `internal/oauth/configview_tpa_scope_test.go`, `internal/runtime/config_hotreload_test.go`, `docs/configuration.md` | W0 | +| G2 Runtime core and fetch | T320–T331 | `internal/security/scanner/{tpa_bundle_runtime.go, tpa_bundle_policy.go, tpa_bundle_source.go, tpa_candidate.go, tpa_bundle_rollback.go, tpa_bundle_observer.go, tpa_refresh_status.go, tpa_refresh.go, tpa_fetch.go}` and new tests `tpa_refresh_runtime_test.go`, `tpa_refresh_status_test.go`, `tpa_candidate_selfcheck_mode_test.go`, `tpa_fetch_test.go`, `tpa_fetch_egress_test.go`, `tpa_fetch_gate_test.go`, `tpa_refresh_scheduler_test.go`, `tpa_bundle_observer_test.go`, `tpa_refresh_concurrency_test.go` | W0 (env-driven cases use G1's accessor behaviour only after merge; inside G2 they set config fields directly) | +| G3 Self-check | T340–T343 | `internal/security/scanner/{tpa_selfcheck.go, tpa_selfcheck_test.go, tpa_selfcheck_parity_test.go, tpa_selfcheck_pipeline_test.go}`, `internal/security/scanner/bundled/selfcheck-corpus.json` | W0 | +| G4 Telemetry v16 | T350–T352 | `internal/telemetry/{anonymity.go, registry.go, tpa_scanner.go, telemetry.go, tpa_bundle_refresh.go}` and their `*_test.go` (incl. every schema-pin file), `internal/server/tpa_bundle_telemetry{,_test}.go`, `docs/features/telemetry.md` | W0 | +| G5 Wiring and re-protect | T360–T369 | `internal/runtime/{runtime.go, lifecycle.go, tool_quarantine_held_refresh.go}` and their new tests, `internal/security/scanner/service.go` and `service_cached_scan_test.go`, `internal/storage/{models.go, manager.go, bundle_reprotect_state_test.go}`, `internal/server/{server.go, scan_bundle_reprotect.go, scan_admission_test.go, scan_bundle_reprotect_test.go, scan_bundle_reprotect_rt02_test.go, tpa_bundle_wiring_test.go}` | W0 (T362/T366 call the handler directly; the observer-driven end to end is T391) | +| G6 Surfaces | T370–T377 | `internal/httpapi/{server.go, security_bundle.go, security_bundle_test.go, agent_token_gating_test.go}`, `cmd/mcpproxy/{security_cmd.go, security_bundle_cmd_test.go, doctor_cmd_test.go}`, `internal/security/scanner/tpa_bundle_doctor{,_test}.go`, `frontend/src/utils/signatureBundle.ts`, `frontend/src/components/security/SignatureBundlePanelBody.vue`, `frontend/tests/unit/signature-bundle-refresh.spec.ts` | W0 | + +Wave 2 (T390–T396) runs after all six groups merge into the US3 branch: `oas/swagger.yaml` and `oas/docs.go` are regenerated once there, and T391 is the first test that needs G2, G3 and G5 together. From 1bc7de3b1e5d32f3eff793d4ef39a924aa5be868 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Wed, 21 Oct 2026 06:51:46 +0300 Subject: [PATCH 02/65] =?UTF-8?q?feat(security):=20Spec=20101=20US3=20wave?= =?UTF-8?q?=200=20=E2=80=94=20refresh=20contract=20stubs=20and=20spec=20am?= =?UTF-8?q?endments?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit T300: amend Spec 101 spec.md (SC-007, FR-016a(c), FR-017, FR-019), Spec 087 spec.md (mapping note, SC-009, FR-013/014/016/017, hot-reload edge case, release-awareness pointers) and Spec 087 plan.md (superseded banner). T301: stub every symbol the US3 tests reference, each in the file that will hold its final implementation, so the six Wave 1 groups replace function bodies instead of moving stubs. No behaviour changes: config keys and accessors (defaults only), BundleInfo freshness fields, BundleSourceFetch, externalConfigured (path-only), runOffered (delegates to a normal run), evalEnv.MustSign (ignored), refresher/fetch/observer/self-check/doctor stubs, Service.RefreshBundle and StartScanFromCachedContracts, the SecurityController and securityScannerService methods with their fakes, reprotect handler stubs, telemetry recorders, runtime refresher slot and RefreshHeldScanEvidence, and the BundleReprotectState storage pair. Also allowlists Config.Security.TPARefreshInterval in the zero-value preservation test so existing tests stay green (0 and absent both mean 24h). Co-Authored-By: Claude Sonnet 5.5 --- internal/config/config.go | 91 +++++++++++ .../config/zero_value_preservation_test.go | 1 + internal/httpapi/security_bundle_test.go | 7 + internal/httpapi/security_scanner.go | 5 +- internal/runtime/runtime.go | 25 +++ .../runtime/tool_quarantine_held_refresh.go | 20 +++ internal/security/scanner/service.go | 19 +++ .../security/scanner/tpa_bundle_doctor.go | 14 ++ .../security/scanner/tpa_bundle_observer.go | 55 +++++++ .../security/scanner/tpa_bundle_policy.go | 21 +++ .../security/scanner/tpa_bundle_runtime.go | 36 ++++- .../security/scanner/tpa_bundle_source.go | 28 +++- internal/security/scanner/tpa_candidate.go | 8 + internal/security/scanner/tpa_fetch.go | 112 +++++++++++++ internal/security/scanner/tpa_refresh.go | 148 ++++++++++++++++++ .../security/scanner/tpa_refresh_status.go | 125 +++++++++++++++ internal/security/scanner/tpa_selfcheck.go | 31 ++++ internal/server/scan_admission_test.go | 10 ++ internal/server/scan_bundle_reprotect.go | 38 +++++ internal/server/server.go | 10 ++ internal/storage/manager.go | 19 +++ internal/storage/models.go | 15 ++ internal/telemetry/tpa_bundle_refresh.go | 38 +++++ specs/087-tpa-daily-refresh/plan.md | 2 + specs/087-tpa-daily-refresh/spec.md | 34 ++-- specs/101-tpa-db/spec.md | 8 +- 26 files changed, 902 insertions(+), 18 deletions(-) create mode 100644 internal/runtime/tool_quarantine_held_refresh.go create mode 100644 internal/security/scanner/tpa_bundle_doctor.go create mode 100644 internal/security/scanner/tpa_bundle_observer.go create mode 100644 internal/security/scanner/tpa_fetch.go create mode 100644 internal/security/scanner/tpa_refresh.go create mode 100644 internal/security/scanner/tpa_refresh_status.go create mode 100644 internal/security/scanner/tpa_selfcheck.go create mode 100644 internal/server/scan_bundle_reprotect.go create mode 100644 internal/telemetry/tpa_bundle_refresh.go diff --git a/internal/config/config.go b/internal/config/config.go index 6a5d68541..8d9fe4265 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -2112,6 +2112,14 @@ func IsValidTrustMode(s string) bool { // loader's env pass. const EnvTPABundlePath = "MCPPROXY_TPA_BUNDLE_PATH" +// EnvTPAFetchEnabled is the environment override for the optional TPA bundle +// network fetch (`security.tpa_fetch_enabled`, Spec 087 FR-016 / Spec 101 US3 +// B2). It is read inside SecurityConfig.IsTPAFetchEnabled, bidirectional +// ("true"/"1" force on, "false"/"0" force off), and is never persisted. It +// enables only the config-resident URL, verified against config-resident keys, +// so the environment cannot redirect the fetch. +const EnvTPAFetchEnabled = "MCPPROXY_TPA_FETCH_ENABLED" + // EnvAutoBaselineScan is the environment kill-switch for the automatic, // informational Pass-1 baseline scan (`security.auto_baseline_scan`). Like the // bundle path, precedence is enforced in the accessor @@ -2351,6 +2359,7 @@ func (c *Config) ValidateDetailed() []ValidationError { errors = append(errors, c.forwardHeadersErrors()...) errors = append(errors, c.oauthEndpointOverrideErrors()...) errors = append(errors, c.tpaTrustRootErrors()...) + errors = append(errors, c.tpaRefreshErrors()...) if _, err := ValidateProfiles(c); err != nil { errors = append(errors, ValidationError{Field: "profiles", Message: err.Error()}) } @@ -2418,6 +2427,17 @@ func (c *Config) tpaTrustRootErrors() []ValidationError { return errors } +// tpaRefreshErrors validates the Spec 087 / Spec 101 US3 refresh keys under +// `security` (tpa_refresh_interval, tpa_fetch_enabled, tpa_fetch_url, +// tpa_stale_after_days). Write-time only, like tpaTrustRootErrors: it is +// appended to ValidateDetailed and never reached from the boot-fatal +// validateDetailedCore. Messages never echo the URL. +// +// STUB (Spec 101 US3 wave 0): the real shape checks land with T312. +func (c *Config) tpaRefreshErrors() []ValidationError { + return nil +} + // ValidateServerForLoad runs the per-server checks config.Load applies at boot // (the stdio command-required check, url-required for HTTP protocols, protocol // and name validity, ...) against a single server and returns only the errors @@ -3318,6 +3338,30 @@ type SecurityConfig struct { // watermark and retires prior-epoch pins. Absent means 0. TPAOperatorEpoch *uint64 `json:"tpa_operator_epoch,omitempty" mapstructure:"tpa-operator-epoch"` + // --- Spec 087 / Spec 101 US3: daily refresh and optional fetch --- + // + // Flat keys, none containing a masked substring (KEY, AUTH, TOKEN, PASS, + // CERT, SECRET), so GET /api/v1/config returns them verbatim. The trust set + // a fetched pair is verified against is the Spec 101 one above; there is no + // per-fetch key. + + // TPARefreshInterval is the cadence of the refresh cycle (the network-free + // file leg, then the optional fetch leg). Absent or 0 means 24h; the runtime + // clamps to at least 1h. There is no refresh-enable key: the file leg is + // idempotent and always on. + TPARefreshInterval Duration `json:"tpa_refresh_interval,omitempty" mapstructure:"tpa-refresh-interval" swaggertype:"string"` + // TPAFetchEnabled opts in to the network fetch leg. Default false; the + // environment override is MCPPROXY_TPA_FETCH_ENABLED (EnvTPAFetchEnabled). + TPAFetchEnabled bool `json:"tpa_fetch_enabled" mapstructure:"tpa-fetch-enabled"` + // TPAFetchURL is the channel URL: a GitHub Releases "latest" API URL or a + // direct bundle URL. Empty (the default) means the fetch leg is off; no URL + // is compiled into the binary. + TPAFetchURL string `json:"tpa_fetch_url,omitempty" mapstructure:"tpa-fetch-url"` + // TPAStaleAfterDays is the age in days after which the active bundle reads + // as stale. Absent means 30; an explicit 0 disables the stale flag and the + // doctor stale warning. + TPAStaleAfterDays *int `json:"tpa_stale_after_days,omitempty" mapstructure:"tpa-stale-after-days"` + // DeepScan is the opt-in "deep scan" layer (Spec 077 US3). It subsumes the // deprecated top-level scanner_fetch_package_source / scanner_disable_no_new_privileges // keys (migrated on load) and gates the heavy Docker-based scanners + source @@ -3459,6 +3503,53 @@ func (sc *SecurityConfig) EffectiveTPAOperatorEpoch() uint64 { return *sc.TPAOperatorEpoch } +// Defaults for the Spec 087 / Spec 101 US3 refresh keys. +const ( + // DefaultTPARefreshInterval is the refresh cadence when none is configured. + DefaultTPARefreshInterval = 24 * time.Hour + // DefaultTPAStaleAfterDays is the stale threshold when none is configured. + DefaultTPAStaleAfterDays = 30 +) + +// EffectiveTPARefreshInterval returns the configured refresh cadence, 24h when +// absent or non-positive. Nil-safe. The runtime applies its own 1h floor. +// +// STUB (wave 0): the clamp lands with T312. +func (sc *SecurityConfig) EffectiveTPARefreshInterval() time.Duration { + if sc == nil || sc.TPARefreshInterval <= 0 { + return DefaultTPARefreshInterval + } + return sc.TPARefreshInterval.Duration() +} + +// IsTPAFetchEnabled reports whether the optional TPA bundle network fetch is +// on. Default false; nil-safe. The environment override +// (EnvTPAFetchEnabled) is read here, like EffectiveTPABundlePath, once T312 +// lands. +// +// STUB (wave 0): no env override yet. +func (sc *SecurityConfig) IsTPAFetchEnabled() bool { + return sc != nil && sc.TPAFetchEnabled +} + +// EffectiveTPAFetchURL returns the configured fetch URL, "" when none. +// Nil-safe. +func (sc *SecurityConfig) EffectiveTPAFetchURL() string { + if sc == nil { + return "" + } + return sc.TPAFetchURL +} + +// EffectiveTPAStaleAfterDays returns the stale threshold in days: 30 when +// absent, 0 (never stale) when explicitly zero. Nil-safe. +func (sc *SecurityConfig) EffectiveTPAStaleAfterDays() int { + if sc == nil || sc.TPAStaleAfterDays == nil { + return DefaultTPAStaleAfterDays + } + return *sc.TPAStaleAfterDays +} + // IsAutoBaselineScanEnabled reports whether mcpproxy may run the automatic, // informational Pass-1 baseline scan (new-server admission scan + the one-shot // baseline sweep). Default is ENABLED: a nil SecurityConfig, or an unset diff --git a/internal/config/zero_value_preservation_test.go b/internal/config/zero_value_preservation_test.go index 2bb8bffb8..fc05d4390 100644 --- a/internal/config/zero_value_preservation_test.go +++ b/internal/config/zero_value_preservation_test.go @@ -157,6 +157,7 @@ var numericOmitemptyAllowlist = map[string]string{ "Config.Observability.UsagePersistInterval": "doc comment: 'Default 30s' — 0 and absent both resolve to it", "Config.Security.ScanTimeoutDefault": "0 and absent both take the built-in scan timeout", "Config.Security.IntegrityCheckInterval": "0 and absent both take the built-in check interval", + "Config.Security.TPARefreshInterval": "EffectiveTPARefreshInterval resolves <= 0 to the same 24h an absent key gets; there is deliberately no off switch (the file leg is always on)", // Code execution: every one of these is validated as "0 means use default" // (Validate: 'must be between 1 and 600000 milliseconds (or 0 for default)'). diff --git a/internal/httpapi/security_bundle_test.go b/internal/httpapi/security_bundle_test.go index 860256e2f..783820fe9 100644 --- a/internal/httpapi/security_bundle_test.go +++ b/internal/httpapi/security_bundle_test.go @@ -2,6 +2,7 @@ package httpapi import ( "bytes" + "context" "encoding/json" "errors" "net/http" @@ -28,6 +29,7 @@ type bundleMock struct { unpinErr error unpinCalls int reloadCalls int + refreshCalls int } func (m *mockSecurityController) BundleStatus() scanner.BundleInfo { return m.bundle.info } @@ -52,6 +54,11 @@ func (m *mockSecurityController) ReloadBundle() scanner.BundleInfo { return m.bundle.info } +func (m *mockSecurityController) RefreshBundle(context.Context) scanner.BundleInfo { + m.bundle.refreshCalls++ + return m.bundle.info +} + const bundleFullDigest = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" func sampleBundleInfo() scanner.BundleInfo { diff --git a/internal/httpapi/security_scanner.go b/internal/httpapi/security_scanner.go index ac0934827..524feb5e7 100644 --- a/internal/httpapi/security_scanner.go +++ b/internal/httpapi/security_scanner.go @@ -58,12 +58,15 @@ type SecurityController interface { GetScanReportByJobID(ctx context.Context, jobID string) (*scanner.AggregatedReport, error) // TPA signature bundle (Spec 101): status, activation history and the - // operator mutations (explicit rollback, unpin, reload). + // operator mutations (explicit rollback, unpin, reload, refresh). BundleStatus() scanner.BundleInfo BundleHistory() ([]scanner.TPABundleActivation, error) RollbackBundle(req scanner.BundleRollbackRequest) (scanner.BundleInfo, error) UnpinBundle() (scanner.BundleInfo, error) ReloadBundle() scanner.BundleInfo + // RefreshBundle runs one joined refresh cycle (file leg then optional + // fetch leg) and returns the resulting status (Spec 087 FR-004, B20). + RefreshBundle(ctx context.Context) scanner.BundleInfo } // SetSecurityController configures the security scanner controller on the server. diff --git a/internal/runtime/runtime.go b/internal/runtime/runtime.go index 79adf83a5..0f0b87d54 100644 --- a/internal/runtime/runtime.go +++ b/internal/runtime/runtime.go @@ -41,6 +41,7 @@ import ( "github.com/smart-mcp-proxy/mcpproxy-go/internal/runtime/supervisor" "github.com/smart-mcp-proxy/mcpproxy-go/internal/secret" "github.com/smart-mcp-proxy/mcpproxy-go/internal/security" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/security/scanner" "github.com/smart-mcp-proxy/mcpproxy-go/internal/server/tokens" "github.com/smart-mcp-proxy/mcpproxy-go/internal/shellwrap" "github.com/smart-mcp-proxy/mcpproxy-go/internal/storage" @@ -192,6 +193,17 @@ type Runtime struct { updateChecker *updatecheck.Checker // Background version checking telemetryService *telemetry.Service // Anonymous usage telemetry (Spec 036) + // bundleRefresher is the TPA bundle refresh scheduler (Spec 087 / Spec 101 + // US3 B19), registered by the server and started in + // StartBackgroundInitialization. Close stops and joins it before + // storageManager.StopAsync. Guarded by r.mu. + bundleRefresher *scanner.BundleRefresher + // closeOrderHook, when non-nil, is called with a step name at the ordered + // points of Close ("bundle_refresher_stopped", "storage_stop_async") so a + // test can assert the shutdown order. Test seam only. + //nolint:unused // used by the T360 shutdown-order test + closeOrderHook func(step string) + // Spec 080 (US3): pre-churn snapshot. prechurnStore owns the BBolt // shutdown marker + last_error_code record; previousShutdown is the // outcome of the PREVIOUS process instance, derived exactly once in New @@ -3569,6 +3581,19 @@ func (r *Runtime) RefreshOAuthToken(serverName string) error { return nil } +// SetBundleRefresher registers the TPA bundle refresher so the runtime owns its +// lifecycle: StartBackgroundInitialization starts it with the app context and +// Close stops and joins it (Spec 087 / Spec 101 US3 B19). Call it once during +// server construction, before StartBackgroundInitialization; nil clears it. +// +// STUB (Spec 101 US3 wave 0): only records the refresher; the start and the +// Close join land with T367. +func (r *Runtime) SetBundleRefresher(refresher *scanner.BundleRefresher) { + r.mu.Lock() + defer r.mu.Unlock() + r.bundleRefresher = refresher +} + // SetVersion initializes the update checker with the given version. // This should be called once during server startup with the build version. func (r *Runtime) SetVersion(version string) { diff --git a/internal/runtime/tool_quarantine_held_refresh.go b/internal/runtime/tool_quarantine_held_refresh.go new file mode 100644 index 000000000..77fdf2fd0 --- /dev/null +++ b/internal/runtime/tool_quarantine_held_refresh.go @@ -0,0 +1,20 @@ +package runtime + +// RefreshHeldScanEvidence re-runs the in-process TPA scan over the stored +// current definition of every pending or changed tool record of one server and +// refreshes only the held-scan evidence (HeldVerdict, HeldSignals), so a newly +// activated bundle's verdict is visible on tools that are already held for +// review (Spec 101 FR-019 leg (i), tasks B17). It takes the tool-approval lock +// and works for quarantined servers too. +// +// It is evidence only: Status, ApprovedHash, ApprovedBy and the server's +// quarantine flag are never written, and nothing is approved or unquarantined +// (RT-02). A signature removed by a newer bundle leaves the tool held, with +// verdict clean and reason awaiting_review. +// +// STUB (Spec 101 US3 wave 0): returns nil without touching anything until +// T367. +func (r *Runtime) RefreshHeldScanEvidence(server string) error { + _ = server + return nil +} diff --git a/internal/security/scanner/service.go b/internal/security/scanner/service.go index 5d2cd5958..00427b8ca 100644 --- a/internal/security/scanner/service.go +++ b/internal/security/scanner/service.go @@ -470,6 +470,25 @@ func (s *Service) ReloadBundle() BundleInfo { return ReloadBundle(s.logger) } +// RefreshBundle runs one joined refresh cycle (the network-free file leg, then +// the optional fetch leg) and returns the resulting status; without a running +// refresher it behaves as ReloadBundle (Spec 087 FR-004, Spec 101 US3 B20). +func (s *Service) RefreshBundle(ctx context.Context) BundleInfo { + return RefreshBundle(ctx, s.logger) +} + +// StartScanFromCachedContracts starts the post-activation re-protection scan of +// one server (Spec 101 FR-019, tasks B17): an informational job over the +// tool contracts discovery already ingested for its approved tools, labelled +// with the activated bundle's digest. It never connects to the upstream, never +// schedules Pass 2 and is excluded from the scan counters. +// +// STUB (Spec 101 US3 wave 0): returns an error until T368. +func (s *Service) StartScanFromCachedContracts(ctx context.Context, serverName, bundleDigest string) (*ScanJob, error) { + _, _, _ = ctx, serverName, bundleDigest + return nil, errors.New("not implemented") +} + // isBaselineScanner reports whether a scanner id belongs to the deterministic // in-process baseline (Spec 077). An unknown id is treated as a deep scanner so // its failure is attributed to the deep-scan layer, never the baseline. diff --git a/internal/security/scanner/tpa_bundle_doctor.go b/internal/security/scanner/tpa_bundle_doctor.go new file mode 100644 index 000000000..a921fc8cd --- /dev/null +++ b/internal/security/scanner/tpa_bundle_doctor.go @@ -0,0 +1,14 @@ +package scanner + +import "time" + +// BundleDoctorWarnings turns the bundle status into at most three short doctor +// warnings: stale, stuck on last-known-good, and fetch configured but inactive +// (tasks B16). The strings carry no digest, path, URL or host. It is a pure +// function of its arguments so the truth table is testable without a runtime. +// +// STUB (Spec 101 US3 wave 0): returns nil until T375. +func BundleDoctorWarnings(info BundleInfo, now time.Time) []string { + _, _ = info, now + return nil +} diff --git a/internal/security/scanner/tpa_bundle_observer.go b/internal/security/scanner/tpa_bundle_observer.go new file mode 100644 index 000000000..55daaef6f --- /dev/null +++ b/internal/security/scanner/tpa_bundle_observer.go @@ -0,0 +1,55 @@ +package scanner + +import "time" + +// Bundle observers (Spec 101 US3 tasks B17, B21). + +// BundleActivation is the payload handed to activation observers after a +// history row was written (an activation or a rollback). It carries no bundle +// bytes and no patterns. +type BundleActivation struct { + // Digest is the full SHA-256 of the newly serving bundle. + Digest string + // Sequence is the decimal-string manifest sequence ("" when none). + Sequence string + // Authority is the signing authority ("publisher" or "operator"). + Authority string + // Source is BundleSourceFile or BundleSourceFetch. + Source string + // Rollback is true when the activation was an explicit rollback. + Rollback bool + // PreviousDigest is the digest that served before ("" for the embedded + // default or a first activation). + PreviousDigest string + // ActivatedAt is the history row's timestamp. + ActivatedAt time.Time +} + +// AddBundleActivationObserver registers fn to be called, on its own goroutine +// and outside bundleRT.mu, after every activation or rollback that wrote a +// history row. It never fires on an LKG restore, an embedded fallback, an +// idempotent rerun or a refusal. The returned function unsubscribes. +// +// STUB (Spec 101 US3 wave 0): fn is never called and the unsubscribe is a +// no-op until T330. +func AddBundleActivationObserver(fn func(BundleActivation)) (unsubscribe func()) { + _ = fn + return func() {} +} + +// SetBundleRefreshObserver registers fn to receive exactly one +// RefreshOutcomeClass per refresh cycle. A nil fn clears it. internal/server +// maps the class to the telemetry enum. +// +// STUB (Spec 101 US3 wave 0): fn is never called until T331. +func SetBundleRefreshObserver(fn func(RefreshOutcomeClass)) { + _ = fn +} + +// drainBundleObserversForTest blocks until every in-flight activation-observer +// dispatch has returned, so a test can assert on its effects. +// +// STUB (Spec 101 US3 wave 0): returns immediately until T330. +// +//nolint:unused // referenced by the T328 tests +func drainBundleObserversForTest() {} diff --git a/internal/security/scanner/tpa_bundle_policy.go b/internal/security/scanner/tpa_bundle_policy.go index 3595a187f..7cfb6e136 100644 --- a/internal/security/scanner/tpa_bundle_policy.go +++ b/internal/security/scanner/tpa_bundle_policy.go @@ -31,6 +31,13 @@ type bundlePolicy struct { OperatorBaselineSet bool OperatorEpoch uint64 + // FetchEnabled and FetchURL mirror security.tpa_fetch_enabled (with its + // environment override) and security.tpa_fetch_url. They are part of the + // policy fingerprint once T330 lands: a change to either supersedes an + // in-flight download (tasks B8, B9). + FetchEnabled bool + FetchURL string + // PublisherEpoch and PublisherBaseline are the binary's anchors, captured // when the policy is built so that one run sees one consistent view. PublisherEpoch uint64 @@ -68,6 +75,8 @@ func policyFor(sec *config.SecurityConfig, path string) *bundlePolicy { OperatorBaseline: opBase, OperatorBaselineSet: opBaseSet, OperatorEpoch: sec.EffectiveTPAOperatorEpoch(), + FetchEnabled: sec.IsTPAFetchEnabled(), + FetchURL: sec.EffectiveTPAFetchURL(), PublisherEpoch: publisherTrustEpoch, PublisherBaseline: pubBase, Warnings: warnings, @@ -103,6 +112,18 @@ func (p *bundlePolicy) fingerprint(keys []bundlesig.Key) string { return hex.EncodeToString(h.Sum(nil)) } +// externalConfigured reports whether an external (non-embedded) bundle source +// is configured or already serving: the configured path, an enabled fetch, or a +// recorded fetched Active bundle (tasks B9, Hazard A). It is a function rather +// than a policy method because bundlePolicy holds configuration only; the +// recorded Active record lives in the committed state st (nil-safe). +// +// STUB (Spec 101 US3 wave 0): behaves as the path-only guard it replaces +// (policy.Path != ""). T330 widens it for fetched bundles. +func externalConfigured(policy *bundlePolicy, _ *TPABundleState) bool { + return policy.Path != "" +} + // currentEpoch returns the trusted epoch of auth (FR-005a). An unknown // authority has epoch 0 and no records. func (p *bundlePolicy) currentEpoch(auth string) uint64 { diff --git a/internal/security/scanner/tpa_bundle_runtime.go b/internal/security/scanner/tpa_bundle_runtime.go index 28aff61bd..0b6ace743 100644 --- a/internal/security/scanner/tpa_bundle_runtime.go +++ b/internal/security/scanner/tpa_bundle_runtime.go @@ -192,6 +192,23 @@ func ReloadBundle(logger *zap.Logger) BundleInfo { return statusFromSnapshot() } +// runOffered runs the pipeline with c as the freshly offered candidate (the +// fetch adapter's entry point, tasks B7): it evaluates ONLY c, never re-reads +// the configured file, and on refusal leaves the serving bundle in place. It +// returns the resulting status. +// +// STUB (Spec 101 US3 wave 0): delegates to a normal run and ignores c; T330 +// adds the offered-only branch. +// +//nolint:unused // called by the refresher's fetch leg once T331 lands +func runOffered(c Candidate, logger *zap.Logger) BundleInfo { + log := bundleLogger(logger) + bundleRT.mu.Lock() + defer bundleRT.mu.Unlock() + bundleRT.runLocked(bundleRT.policyOrDefaultLocked(), log, runOptions{offered: &c}) + return statusFromSnapshot() +} + // BundleCoverageDegraded reports whether the active bundle demotes the install to // degraded coverage, and why. The scan-mode admission gates consult it: while it // is true auto-approval is suspended and changes route to human review. @@ -221,6 +238,12 @@ type runOptions struct { // force skips the idempotence short-circuit (unpin re-validates the serving // bundle even though no input file changed). force bool + // offered, when non-nil, is the single freshly offered candidate (fetch + // leg). The run evaluates only it as the fresh one (tasks B7). + // + // STUB (Spec 101 US3 wave 0): ignored until T330. + //nolint:unused // read by the offered-only branch once T330 lands + offered *Candidate } // selection is what a run chose to serve and why. @@ -245,6 +268,13 @@ type selection struct { // restored is the Active record put back from the active root's rollback // pin (restorePinLocked); serving is then that pin's bundle. restored *TPABundleActiveRecord + // offeredRej is the refusal of the offered (fetch) candidate and + // offeredDigest the candidate's digest; they are kept apart from attempt + // so a fetch refusal never sets the file leg's load_error (tasks B7). + //nolint:unused // written by the offered-only branch once T330 lands + offeredRej *BundleRejection + //nolint:unused // written by the offered-only branch once T330 lands + offeredDigest string } // runLocked is one pipeline run (design §6). The caller holds rt.mu. @@ -422,7 +452,7 @@ func (rt *bundleRuntime) canSkipLocked(policy *bundlePolicy, cand *Candidate) bo if !rt.clean || rt.policy == nil || rt.policy.Fingerprint != policy.Fingerprint || rt.loadErr != "" { return false } - if policy.Path == "" { + if !externalConfigured(policy, rt.state) { return rt.active == nil } if cand == nil || rt.active == nil { @@ -466,7 +496,7 @@ func (rt *bundleRuntime) chooseLocked(policy *bundlePolicy, nst *TPABundleState, // fallbackLocked fills sel.serving from steps b, c and d. func (rt *bundleRuntime) fallbackLocked(policy *bundlePolicy, nst *TPABundleState, sel *selection, log *zap.Logger) { sel.serving, sel.fromPath = nil, false - if policy.Path != "" { + if externalConfigured(policy, nst) { switch { case rt.active != nil: acc, rej := evaluateCandidate(rt.active.cand, evalEnv{Policy: policy, State: nst, Mode: evalLKG, Reuse: rt.active.acc}) @@ -515,7 +545,7 @@ func (rt *bundleRuntime) fallbackLocked(policy *bundlePolicy, nst *TPABundleStat // switching back must restore the pinned bundle unchanged rather than leave the // install on the embedded default while the pin still binds. func (rt *bundleRuntime) restorePinLocked(policy *bundlePolicy, nst *TPABundleState, sel *selection, log *zap.Logger, now time.Time) { - if policy.Path == "" || rt.dataDir == "" || rt.loadFailed || sel.keepActive || sel.serving == nil || !sel.serving.embedded { + if !externalConfigured(policy, nst) || rt.dataDir == "" || rt.loadFailed || sel.keepActive || sel.serving == nil || !sel.serving.embedded { return } a := nst.Authorities[policy.ActiveRoot] diff --git a/internal/security/scanner/tpa_bundle_source.go b/internal/security/scanner/tpa_bundle_source.go index ecaeb78fd..c7a2a17cd 100644 --- a/internal/security/scanner/tpa_bundle_source.go +++ b/internal/security/scanner/tpa_bundle_source.go @@ -17,6 +17,10 @@ const ( // BundleSourceFile is a corpus read from the configured filesystem path // (security.tpa_bundle_path / MCPPROXY_TPA_BUNDLE_PATH). BundleSourceFile = "file" + // BundleSourceFetch is a corpus fetched from the configured channel + // (security.tpa_fetch_url, Spec 087 / Spec 101 US3). It reaches the same + // verifier as a file drop; only the Candidate's origin differs. + BundleSourceFetch = "fetch" ) // BundleInfo is the operator-facing status of the TPA signature corpus the @@ -25,7 +29,7 @@ const ( // running, and how old are they?" — the bundle was embed-only and its single // load report went to an unconfigured global logger. type BundleInfo struct { - // Source is BundleSourceEmbedded or BundleSourceFile. + // Source is BundleSourceEmbedded, BundleSourceFile or BundleSourceFetch. Source string `json:"source"` // Path is the configured filesystem path, when Source is "file". Path string `json:"path,omitempty"` @@ -90,6 +94,28 @@ type BundleInfo struct { // across a later successful load so "what did it refuse, and when" // survives. LastRejection *BundleRejection `json:"last_rejection,omitempty"` + + // --- Spec 087 / Spec 101 US3 freshness fields (tasks B15) --- + // + // All pointer or omitempty so a status that never refreshed reads exactly + // as before. + + // LastCheckedAt is the end of the most recent pipeline run, including + // no-op runs the idempotence short-circuit skipped. + LastCheckedAt *time.Time `json:"last_checked_at,omitempty"` + // LastActivatedAt is when the serving external bundle was activated; nil + // for the embedded default. + LastActivatedAt *time.Time `json:"last_activated_at,omitempty"` + // Stale and StaleAfterDays are computed at read time next to + // StalenessDays; a future generated_at is never stale. StaleAfterDays nil + // means the flag is not evaluated. + Stale bool `json:"stale,omitempty"` + StaleAfterDays *int `json:"stale_after_days,omitempty"` + // Refresh is the refresh-cycle status (nil before any cycle or setting is + // known). Reprotect is the post-activation re-protection sweep status, + // set from the server through SetReprotectStatus. + Refresh *BundleRefreshStatus `json:"refresh,omitempty"` + Reprotect *BundleReprotectStatus `json:"reprotect,omitempty"` } // BundleRejection is one refused candidate (FR-020). diff --git a/internal/security/scanner/tpa_candidate.go b/internal/security/scanner/tpa_candidate.go index d7bf45650..3b6a29a63 100644 --- a/internal/security/scanner/tpa_candidate.go +++ b/internal/security/scanner/tpa_candidate.go @@ -66,6 +66,14 @@ type evalEnv struct { // every policy- and state-dependent stage (signature, authority, epoch, // deny-list, watermark) still runs in full. Reuse *acceptedBundle + // MustSign makes THIS evaluation refuse an unsigned candidate as + // signature_required without touching the policy (tasks B5): the offered + // fetch run sets it, so a policy copy with RequireSigned=true never reaches + // fallback step b and a serving unsigned file-drop bundle is not + // deactivated. + // + // STUB (Spec 101 US3 wave 0): ignored until T330. + MustSign bool } // acceptedBundle is a candidate that passed every stage. diff --git a/internal/security/scanner/tpa_fetch.go b/internal/security/scanner/tpa_fetch.go new file mode 100644 index 000000000..9e63f7f26 --- /dev/null +++ b/internal/security/scanner/tpa_fetch.go @@ -0,0 +1,112 @@ +package scanner + +import ( + "context" + "net" + "net/http" + "time" +) + +// The fetch adapter (Spec 087 FR-003/FR-009 to FR-015, Spec 101 US3 tasks +// B4 to B12). It turns the configured channel into a source-neutral Candidate +// and hands it to the existing verifier; it never verifies, parses or compiles +// a bundle itself. The real implementation lands with T331. + +// FetchResolver is the DNS seam of the fetch adapter, satisfied by +// *net.Resolver. Tests inject a fake to exercise the dial-time egress rules +// without the network. +type FetchResolver interface { + LookupIPAddr(ctx context.Context, host string) ([]net.IPAddr, error) +} + +// fetchSettings is everything one fetch attempt needs, captured by the +// refresher from the live configuration before the (lock-free) network I/O +// starts. The field set may grow with T331; the refresher is its only builder. +// +//nolint:unused // built by the refresher once T331 lands +type fetchSettings struct { + // Enabled mirrors security.tpa_fetch_enabled (with its env override). + Enabled bool + // URL is the configured channel URL. It is never logged or reported; only + // its host is (B12). + URL string + // Version is the mcpproxy version for the User-Agent header. + Version string + // Policy is the live bundle policy: its active root's trust set decides + // whether the leg may issue requests at all (B10). + Policy *bundlePolicy + // StateUnavailable is true while the anti-downgrade state store could not + // be read (B10 state_unavailable). + StateUnavailable bool + // Watermark returns the active root's published watermark snapshot for the + // pre-download hint skip (B4, B7); nil means no hint skip. + Watermark func(authority string) (uint64, bool) + // ETag is the validator cached for URL from a previous cycle ("" if none). + ETag string + // Client, Resolver and Now are the injected seams; nil selects the + // defaults built by NewBundleRefresher. + Client *http.Client + Resolver FetchResolver + Now func() time.Time +} + +// fetchResult is the outcome of one fetch attempt. Outcome and Reason use the +// RefreshOutcome* and RefreshReason* vocabularies; Candidate is set only when +// a pair was downloaded and should be offered to the pipeline. +// +//nolint:unused // produced by fetchCandidate once T331 lands +type fetchResult struct { + Outcome string + Reason string + // Candidate is the downloaded pair (Source BundleSourceFetch, Path + // "://"); nil for every non-offer outcome. + Candidate *Candidate + // ETag is the validator to remember for the next cycle. + ETag string + // RetryAt, when non-zero, is the earliest time the next attempt may run + // (rate limiting); the refresher caps it at 24h. + RetryAt time.Time +} + +// fetchCandidate performs one fetch attempt against the configured channel and +// returns what to do with it. It issues zero requests unless every B10 +// precondition holds. +// +// STUB (Spec 101 US3 wave 0): reports disabled without any request. +// +//nolint:unused // called by the refresher once T331 lands +func fetchCandidate(ctx context.Context, settings fetchSettings) fetchResult { + _, _ = ctx, settings + return fetchResult{Outcome: RefreshOutcomeDisabled, Reason: RefreshReasonFetchOff} +} + +// egressClass classifies the address the configured host resolved to, for the +// dial-time egress predicate (B11). +// +//nolint:unused // used by egressAllowed once T331 lands +type egressClass int + +//nolint:unused // classes are assigned by the dial control once T331 lands +const ( + // egressPublic: the configured host resolved to a public address. + egressPublic egressClass = iota + // egressLoopback: the configured host is a loopback literal or localhost. + egressLoopback + // egressPrivate: the configured host resolved to RFC1918, ULA or CGNAT + // space (a LAN mirror). + egressPrivate +) + +// egressAllowed reports whether a connection (including a redirect hop) to ip +// is permitted given the class of the configured host: link-local (169.254/16, +// fe80::/10, metadata), unspecified and multicast addresses are always refused; +// loopback and private ranges only when the configured host itself resolved to +// that class (B11). +// +// STUB (Spec 101 US3 wave 0): permits everything until T331. +// +//nolint:unused // called by the dial-time control once T331 lands +func egressAllowed(ip net.IP, configuredClass egressClass) bool { + _, _ = ip, configuredClass + return true +} diff --git a/internal/security/scanner/tpa_refresh.go b/internal/security/scanner/tpa_refresh.go new file mode 100644 index 000000000..1ad7efbb2 --- /dev/null +++ b/internal/security/scanner/tpa_refresh.go @@ -0,0 +1,148 @@ +package scanner + +import ( + "context" + "net/http" + "sync" + "time" + + "go.uber.org/zap" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" +) + +// The refresh scheduler (Spec 087 FR-003, FR-004, FR-009 to FR-015; Spec 101 +// US3 tasks B6 to B8, B19, B20). One BundleRefresher per process runs a cycle +// per interval: the network-free file leg, then the optional fetch leg. Every +// trigger (timer tick, manual refresh, config nudge) that arrives during a +// cycle joins it. The real scheduler lands with T331. + +// RefreshClock is the time seam of the scheduler. Tests inject a fake whose +// timers fire on demand; production uses the real clock (nil Clock). +type RefreshClock interface { + Now() time.Time + // NewTimer returns a timer that fires once after d. + NewTimer(d time.Duration) RefreshTimer +} + +// RefreshTimer is the timer half of RefreshClock, shaped like *time.Timer. +type RefreshTimer interface { + // C delivers the fire time. + C() <-chan time.Time + Stop() bool + Reset(d time.Duration) bool +} + +// BundleRefresherOptions are the construction-time seams of a BundleRefresher. +// Every field is optional; the zero value selects the production defaults. +type BundleRefresherOptions struct { + // Clock replaces the real clock and timers. + Clock RefreshClock + // Rand returns a uniform value in [0,1) for jitter. + Rand func() float64 + // HTTPClient, when set, is used as is; otherwise one is built from + // Transport and the egress rules. + HTTPClient *http.Client + // Transport replaces the base transport of the built client. + Transport http.RoundTripper + // Resolver replaces DNS for the dial-time egress control. + Resolver FetchResolver + // ConfigProvider returns the live security config each cycle. The + // refresher never takes Runtime.mu to read it. + ConfigProvider func() *config.SecurityConfig + // Logger is the injected logger (nil => no-op). + Logger *zap.Logger + // Version is the mcpproxy version for the User-Agent header. + Version string + // IntervalOverride is a test seam that replaces the configured interval + // and bypasses the 1h floor; zero means "use the config". + IntervalOverride time.Duration +} + +// BundleRefresher owns the refresh lifecycle. Start/Stop follow the +// telemetry.Service pattern: Stop cancels and joins, a Start after Stop is a +// no-op. +type BundleRefresher struct { + opts BundleRefresherOptions + + mu sync.Mutex + running bool +} + +// NewBundleRefresher builds a refresher; it does nothing until Start. +func NewBundleRefresher(opts BundleRefresherOptions) *BundleRefresher { + return &BundleRefresher{opts: opts} +} + +// Start begins the cycle loop, bound to ctx. A second Start, or a Start after +// Stop, is a no-op. +// +// STUB (Spec 101 US3 wave 0): no-op until T331. +func (r *BundleRefresher) Start(ctx context.Context) { _ = ctx } + +// Stop cancels any in-flight cycle and returns only after the loop goroutine +// exited. Idempotent. +// +// STUB (Spec 101 US3 wave 0): no-op until T331. +func (r *BundleRefresher) Stop() {} + +// Nudge tells the refresher the security config changed: it re-reads the +// fetch and trust settings, bumps the generation (superseding an in-flight +// download whose inputs changed) and re-arms the timer. Non-blocking; a Nudge +// after Stop began is a no-op. +// +// STUB (Spec 101 US3 wave 0): no-op until T331. +func (r *BundleRefresher) Nudge(sec *config.SecurityConfig) { _ = sec } + +// Running reports whether the loop goroutine is alive. +// +// STUB (Spec 101 US3 wave 0): always false until T331. +func (r *BundleRefresher) Running() bool { + r.mu.Lock() + defer r.mu.Unlock() + return r.running +} + +// Refresh runs one cycle now (trigger "manual"), joining an in-flight cycle if +// there is one, and returns the resulting status. +// +// STUB (Spec 101 US3 wave 0): falls back to ReloadBundle until T331. +func (r *BundleRefresher) Refresh(ctx context.Context, logger *zap.Logger) BundleInfo { + _ = ctx + return ReloadBundle(logger) +} + +var ( + refresherSlotMu sync.RWMutex + refresherSlot *BundleRefresher +) + +// SetBundleRefresher installs r as the process's refresher, the one +// RefreshBundle consults. nil clears the slot. +func SetBundleRefresher(r *BundleRefresher) { + refresherSlotMu.Lock() + defer refresherSlotMu.Unlock() + refresherSlot = r +} + +// CurrentBundleRefresher returns the installed refresher, or nil. The runtime +// uses it on Close to clear the slot only if it still points at its own +// instance. +func CurrentBundleRefresher() *BundleRefresher { + refresherSlotMu.RLock() + defer refresherSlotMu.RUnlock() + return refresherSlot +} + +// RefreshBundle is the manual refresh (`security bundle refresh`, B20): one +// joined cycle (file leg plus fetch leg), trigger "manual". When no running +// refresher is installed it falls back to ReloadBundle, so a manual refresh +// still picks up a dropped file. +// +// STUB (Spec 101 US3 wave 0): always falls back to ReloadBundle until T331. +func RefreshBundle(ctx context.Context, logger *zap.Logger) BundleInfo { + if r := CurrentBundleRefresher(); r != nil && r.Running() { + return r.Refresh(ctx, logger) + } + return ReloadBundle(logger) +} diff --git a/internal/security/scanner/tpa_refresh_status.go b/internal/security/scanner/tpa_refresh_status.go new file mode 100644 index 000000000..c082bc6c3 --- /dev/null +++ b/internal/security/scanner/tpa_refresh_status.go @@ -0,0 +1,125 @@ +package scanner + +import "time" + +// Refresh and re-protection status (Spec 087 FR-017, Spec 101 US3 tasks B15, +// B17). The vocabularies below are CLOSED: a surface may branch on them. + +// Refresh trigger vocabulary (BundleRefreshStatus.LastTrigger). +const ( + // RefreshTriggerStartup is the construction-time ConfigureBundleFromConfig. + RefreshTriggerStartup = "startup" + // RefreshTriggerConfig is a config.reloaded re-apply. + RefreshTriggerConfig = "config" + // RefreshTriggerTimer is the refresher's periodic tick. + RefreshTriggerTimer = "timer" + // RefreshTriggerManual is `security bundle refresh` / POST .../bundle/refresh. + RefreshTriggerManual = "manual" +) + +// Refresh outcome vocabulary (BundleRefreshStatus.LastOutcome). +const ( + // RefreshOutcomeActivated: a new bundle was verified and activated. + RefreshOutcomeActivated = "activated" + // RefreshOutcomeUnchanged: the candidate equals the serving bundle, or was + // already refused (negative cache); nothing was compiled. + RefreshOutcomeUnchanged = "unchanged" + // RefreshOutcomeNotModified: the channel answered 304. + RefreshOutcomeNotModified = "not_modified" + // RefreshOutcomeRejected: the pipeline refused the candidate; the detail + // is in BundleInfo.LastRejection. + RefreshOutcomeRejected = "rejected" + // RefreshOutcomeFetchUnavailable: a non-verification failure (network, + // timeout, HTTP status, rate limit). Never a last_rejection. + RefreshOutcomeFetchUnavailable = "fetch_unavailable" + // RefreshOutcomeChannelInvalid: the channel answered but is malformed + // (zero or several candidate assets, missing sidecar asset, oversize JSON). + RefreshOutcomeChannelInvalid = "channel_invalid" + // RefreshOutcomeSuperseded: the fetch or trust settings changed while a + // download was in flight, so its candidate was discarded (Δ30). + RefreshOutcomeSuperseded = "superseded" + // RefreshOutcomeDisabled: the fetch leg may not issue requests (B10); the + // reason says why. + RefreshOutcomeDisabled = "disabled" +) + +// Refresh reason vocabulary (BundleRefreshStatus.LastReason). Fetch-unavailable +// reasons first, then the disabled reasons (tasks B6, B10). +const ( + RefreshReasonNetworkError = "network_error" + RefreshReasonTimeout = "timeout" + RefreshReasonHTTPStatus = "http_status" + RefreshReasonRateLimited = "rate_limited" + + RefreshReasonFetchOff = "fetch_off" + RefreshReasonNoURL = "no_url" + RefreshReasonInvalidURL = "invalid_url" + RefreshReasonNoTrustedSigner = "no_trusted_signer" + RefreshReasonStateUnavailable = "state_unavailable" +) + +// Re-protection sweep state vocabulary (BundleReprotectStatus.State). +const ( + ReprotectStateIdle = "idle" + ReprotectStateRunning = "running" + ReprotectStateSuspended = "suspended" + ReprotectStateFailed = "failed" +) + +// ReprotectReasonAutoBaselineScanOff is the suspended reason shown while +// security.auto_baseline_scan is off (tasks B17). +const ReprotectReasonAutoBaselineScanOff = "auto_baseline_scan_off" + +// BundleRefreshStatus is the refresh-cycle status merged into BundleInfo at +// read time (B15). It carries the host only, never the URL (B12). +type BundleRefreshStatus struct { + FetchEnabled bool `json:"fetch_enabled"` + // FetchHost is the host of the configured fetch URL ("" when none). + FetchHost string `json:"fetch_host,omitempty"` + IntervalSeconds int `json:"interval_seconds"` + // LastTrigger, LastOutcome and LastReason use the closed vocabularies above. + LastTrigger string `json:"last_trigger,omitempty"` + LastOutcome string `json:"last_outcome,omitempty"` + LastReason string `json:"last_reason,omitempty"` + LastAttemptAt *time.Time `json:"last_attempt_at,omitempty"` + ConsecutiveFailures int `json:"consecutive_failures"` + NextCheckAt *time.Time `json:"next_check_at,omitempty"` +} + +// BundleReprotectStatus is the post-activation re-protection sweep status +// (Spec 101 FR-019, B17), set from the server through SetReprotectStatus. +type BundleReprotectStatus struct { + // State is one of the ReprotectState* constants. + State string `json:"state"` + // Reason qualifies suspended and failed (for example + // ReprotectReasonAutoBaselineScanOff). + Reason string `json:"reason,omitempty"` + LastCompletedAt *time.Time `json:"last_completed_at,omitempty"` + ServersScanned int `json:"servers_scanned"` + ServersWithHits int `json:"servers_with_hits"` +} + +// RefreshOutcomeClass is the coarse class of one refresh cycle handed to the +// refresh observer (SetBundleRefreshObserver). It is scanner's own enum; +// internal/telemetry must not import scanner, so internal/server maps it to +// telemetry.TPABundleRefreshClass (tasks B21). +type RefreshOutcomeClass string + +const ( + // RefreshClassActivated: the cycle activated a bundle. + RefreshClassActivated RefreshOutcomeClass = "activated" + // RefreshClassRejected: the pipeline refused a candidate. + RefreshClassRejected RefreshOutcomeClass = "rejected" + // RefreshClassUnavailable: fetch_unavailable or channel_invalid. + RefreshClassUnavailable RefreshOutcomeClass = "unavailable" + // RefreshClassUnchanged: unchanged, not_modified, superseded or disabled. + RefreshClassUnchanged RefreshOutcomeClass = "unchanged" +) + +// SetReprotectStatus publishes the re-protection sweep status that BundleStatus +// merges into BundleInfo.Reprotect (tasks B15, B17). +// +// STUB (Spec 101 US3 wave 0): no-op until T330. +func SetReprotectStatus(status BundleReprotectStatus) { + _ = status +} diff --git a/internal/security/scanner/tpa_selfcheck.go b/internal/security/scanner/tpa_selfcheck.go new file mode 100644 index 000000000..eca6ff570 --- /dev/null +++ b/internal/security/scanner/tpa_selfcheck.go @@ -0,0 +1,31 @@ +package scanner + +// Activation self-check (Spec 087 FR-008, Spec 101 FR-016a(c); tasks B13, B14). +// The real port of the CI gate lands with T343. + +// selfCheckCorpus is the test seam that replaces the embedded self-check +// corpus bytes (to feed vacuous or unparsable corpora). nil means the embedded +// corpus. +// +//nolint:unused // referenced by the T340 tests and T343 +var selfCheckCorpus func() []byte + +// EnableActivationSelfCheck installs the activation self-check as the pipeline's +// bundleSelfCheck hook. Production wiring (internal/server/server.go) calls it +// once, next to configureTPABundleRuntime; US1 fixtures and clearBundleRuntime +// keep the nil default. +// +// STUB (Spec 101 US3 wave 0): no-op, the hook stays nil, until T343. +func EnableActivationSelfCheck() {} + +// activationSelfCheck scores a compiled candidate over the embedded self-check +// corpus with the same decision as the CI gate and refuses it on a recall or +// false-positive regression. A non-nil error is reported as gate_failure. +// +// STUB (Spec 101 US3 wave 0): accepts everything until T343. +// +//nolint:unused // installed by EnableActivationSelfCheck once T343 lands +func activationSelfCheck(check *BundleCheck) error { + _ = check + return nil +} diff --git a/internal/server/scan_admission_test.go b/internal/server/scan_admission_test.go index 0f7d6566b..da24d38c6 100644 --- a/internal/server/scan_admission_test.go +++ b/internal/server/scan_admission_test.go @@ -55,6 +55,9 @@ type fakeSecurityScanner struct { // startScanTries records EVERY StartScan entry, including the ones that // return startScanErr, so retry-capping can be asserted. startScanTries []string + // cachedScanCalls records every StartScanFromCachedContracts call + // (re-protection sweep, Spec 101 FR-019) as "server@digest". + cachedScanCalls []string } func newFakeSecurityScanner() *fakeSecurityScanner { @@ -189,6 +192,13 @@ func (f *fakeSecurityScanner) HasApprovalBaseline(serverName string) bool { return f.hasBaseline[serverName] } +func (f *fakeSecurityScanner) StartScanFromCachedContracts(_ context.Context, serverName, bundleDigest string) (*scanner.ScanJob, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.cachedScanCalls = append(f.cachedScanCalls, serverName+"@"+bundleDigest) + return &scanner.ScanJob{ServerName: serverName}, nil +} + func (f *fakeSecurityScanner) ApplySecurityConfig(*config.SecurityConfig) {} func (f *fakeSecurityScanner) SetIsolationMode(string) {} func (f *fakeSecurityScanner) DeepScanEnabled() bool { return false } diff --git a/internal/server/scan_bundle_reprotect.go b/internal/server/scan_bundle_reprotect.go new file mode 100644 index 000000000..43beeee2e --- /dev/null +++ b/internal/server/scan_bundle_reprotect.go @@ -0,0 +1,38 @@ +package server + +import ( + "context" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/security/scanner" +) + +// Post-activation re-protection sweep (Spec 101 FR-019, SC-009; tasks B17). +// +// When a new TPA bundle activates, already-approved tools may match a signature +// that did not exist when they were approved. The sweep re-checks them from +// cached data only: leg (i) refreshes the held-scan evidence of pending and +// changed tools, leg (ii) runs a cached informational scan of every +// non-quarantined server with approved tools. It never approves, unquarantines +// or revokes anything (RT-02): findings flow toward human review only. + +// handleBundleActivation is the activation observer. It runs the sweep for one +// activation on a tracked goroutine bound to the informational scan context. +// +// STUB (Spec 101 US3 wave 0): no-op until T369. +// +//nolint:unused // registered as the activation observer by T369 +func (s *Server) handleBundleActivation(act scanner.BundleActivation) { + _ = act +} + +// runBundleReprotectCatchUp re-runs the sweep at startup when the persisted +// marker (storage.BundleReprotectState) differs from the active bundle digest: +// it covers the construction-time activation, a crash mid-sweep and a binary +// upgrade that changes the embedded corpus. +// +// STUB (Spec 101 US3 wave 0): no-op until T369. +// +//nolint:unused // started next to runBaselineSweep by T369 +func (s *Server) runBundleReprotectCatchUp(ctx context.Context) { + _ = ctx +} diff --git a/internal/server/server.go b/internal/server/server.go index 091acd3e1..4bc9b11a3 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -78,6 +78,10 @@ type securityScannerService interface { GetScanStatusByPass(ctx context.Context, serverName string, pass int) (*scanner.ScanJob, error) GetScanReport(ctx context.Context, serverName string) (*scanner.AggregatedReport, error) HasApprovalBaseline(serverName string) bool + // StartScanFromCachedContracts starts the post-activation re-protection scan + // of one server over its already-ingested approved contracts, with no + // upstream connection (Spec 101 FR-019, B17). + StartScanFromCachedContracts(ctx context.Context, serverName, bundleDigest string) (*scanner.ScanJob, error) ApplySecurityConfig(sec *config.SecurityConfig) SetIsolationMode(mode string) DeepScanEnabled() bool @@ -186,6 +190,12 @@ type Server struct { infoRescanSeq uint64 infoRescanDebounce time.Duration + // bundleReprotectWG tracks the post-activation re-protection sweep + // goroutines (see scan_bundle_reprotect.go). Shutdown waits for it before + // runtime.Close. + //nolint:unused // used once the re-protection sweep lands (T369) + bundleReprotectWG sync.WaitGroup + // Spec 024: Shutdown info for lifecycle events shutdownReason string shutdownSignal string diff --git a/internal/storage/manager.go b/internal/storage/manager.go index cd6537709..16901e6e5 100644 --- a/internal/storage/manager.go +++ b/internal/storage/manager.go @@ -942,6 +942,25 @@ func (m *Manager) LoadBaselineSweepState() (*BaselineSweepState, error) { return state, nil } +// TPA bundle re-protection sweep marker + +// SaveBundleReprotectState persists the marker written when a re-protection +// sweep completes with zero failures (Spec 101 FR-019, B17). +// +// STUB (Spec 101 US3 wave 0): writes nothing until T369. +func (m *Manager) SaveBundleReprotectState(state BundleReprotectState) error { + _ = state + return nil +} + +// LoadBundleReprotectState returns the re-protection marker, or the zero value +// when no sweep has ever completed. A read never creates the bucket. +// +// STUB (Spec 101 US3 wave 0): always the zero value until T369. +func (m *Manager) LoadBundleReprotectState() (BundleReprotectState, error) { + return BundleReprotectState{}, nil +} + // Maintenance operations // Backup creates a backup of the database diff --git a/internal/storage/models.go b/internal/storage/models.go index 17a5a6d76..47a290712 100644 --- a/internal/storage/models.go +++ b/internal/storage/models.go @@ -189,6 +189,21 @@ type BaselineSweepState struct { Findings int `json:"findings"` } +// BundleReprotectDoneKey holds the BundleReprotectState marker in MetaBucket. +const BundleReprotectDoneKey = "tpa_bundle_reprotect_done" + +// BundleReprotectState records the last TPA bundle for which the post-activation +// re-protection sweep (Spec 101 FR-019, tasks B17) completed with zero failures. +// Stored in MetaBucket under BundleReprotectDoneKey. Absence (the zero value) +// means "no sweep has completed here". It is installation-wide state and is not +// touched by server purge. +type BundleReprotectState struct { + // FullDigest is the full SHA-256 of the bundle the sweep covered. + FullDigest string `json:"full_digest"` + // CompletedAt is when the sweep finished. + CompletedAt time.Time `json:"completed_at"` +} + // Current schema version const CurrentSchemaVersion = 3 diff --git a/internal/telemetry/tpa_bundle_refresh.go b/internal/telemetry/tpa_bundle_refresh.go new file mode 100644 index 000000000..5b244418b --- /dev/null +++ b/internal/telemetry/tpa_bundle_refresh.go @@ -0,0 +1,38 @@ +package telemetry + +// TPABundleRefreshClass is the coarse class of one TPA bundle refresh cycle +// (Spec 087 / Spec 101 US3 tasks B21). It is a telemetry-local string enum +// because this package must not import internal/security/scanner (see +// tpa_bundle.go); internal/server maps scanner.RefreshOutcomeClass to it. +type TPABundleRefreshClass string + +const ( + // TPABundleRefreshActivated: the cycle activated a bundle. + TPABundleRefreshActivated TPABundleRefreshClass = "activated" + // TPABundleRefreshRejected: the pipeline refused a candidate. + TPABundleRefreshRejected TPABundleRefreshClass = "rejected" + // TPABundleRefreshUnavailable: the fetch was unavailable or the channel was + // malformed. + TPABundleRefreshUnavailable TPABundleRefreshClass = "unavailable" + // TPABundleRefreshUnchanged: nothing changed (unchanged, not modified, + // superseded or disabled). + TPABundleRefreshUnchanged TPABundleRefreshClass = "unchanged" +) + +// RecordTPABundleRefreshOn counts one refresh cycle of the given class on reg if +// reg is non-nil (schema v16): bundle_refresh_cycles always, plus the +// activations, rejections or unavailable counter for the matching class. +// +// STUB (Spec 101 US3 wave 0): no-op until T351. +func RecordTPABundleRefreshOn(reg *CounterRegistry, class TPABundleRefreshClass) { + _, _ = reg, class +} + +// RecordTPABundleReprotectHitOn counts one server whose re-protection scan had +// at least one hard finding (bundle_reprotect_hits) on reg if reg is non-nil +// (schema v16). +// +// STUB (Spec 101 US3 wave 0): no-op until T351. +func RecordTPABundleReprotectHitOn(reg *CounterRegistry) { + _ = reg +} diff --git a/specs/087-tpa-daily-refresh/plan.md b/specs/087-tpa-daily-refresh/plan.md index e68eaf346..93a1add19 100644 --- a/specs/087-tpa-daily-refresh/plan.md +++ b/specs/087-tpa-daily-refresh/plan.md @@ -1,5 +1,7 @@ # Implementation Plan: Daily Offline-First Refresh of the TPA Signature Bundle + mcpproxy Self-Update Awareness +> **Superseded by specs/101-tpa-db/tasks.md US3 (Δ15–Δ31).** The package, config block and file layout below were never built. + **Branch**: `087-tpa-daily-refresh` | **Date**: 2026-07-16 | **Spec**: [spec.md](./spec.md) **Input**: Feature specification from `/specs/087-tpa-daily-refresh/spec.md` diff --git a/specs/087-tpa-daily-refresh/spec.md b/specs/087-tpa-daily-refresh/spec.md index b05009af4..d66b981b7 100644 --- a/specs/087-tpa-daily-refresh/spec.md +++ b/specs/087-tpa-daily-refresh/spec.md @@ -6,6 +6,21 @@ **Implementation note (2026-10-21)**: `plan.md` names a package `internal/security/bundle` (and a nested `security.bundle` config block). Neither exists and neither will be built. Spec 101 US1 shipped the bundle pipeline as `internal/security/scanner/tpa_*.go` plus `internal/security/bundlesig`, with flat `security.tpa_*` config keys. Under Spec 101 D11 this feature is a thin adapter on that pipeline: the daily scheduler and the optional fetch build a source-neutral `Candidate` (`internal/security/scanner/tpa_candidate.go`) and hand it to the same `evaluateCandidate` verifier, which is never forked. The FR-to-task mapping and the amendments to this spec are listed in Spec 101 tasks.md, US3 section. Release awareness (User Story 3, FR-018, FR-019, SC-007) is already covered by Spec 079. **Input**: A once-per-day, non-blocking, offline-FIRST refresh that (a) updates the tpa-db signature bundle — bundled-with-binary default, manual file-drop override, and an OPTIONAL verified/signed network fetch, versioned against the tpa-db `SCHEMA_VERSION`, failing safe to last-known-good; and (b) checks for a new mcpproxy release and surfaces it, respecting the existing update-check/distribution mechanism (no auto-installer). +## Mapping note (2026-10-21, Spec 101 US3) + +This spec is implemented by Spec 101 US3 (`specs/101-tpa-db/tasks.md`, T300 onward). User Story 3, FR-018, FR-019 and SC-007 (mcpproxy release awareness) are covered by Spec 079 (`internal/updatecheck`), unchanged; US3 adds no task for them. + +| Spec 087 | Delivered by | +|---|---| +| FR-001, FR-002, FR-005, FR-006, FR-007 | Spec 101 US1 (shipped) | +| FR-003, FR-004, FR-008 to FR-017 | Spec 101 US3 (T320 to T377) | +| FR-018, FR-019, SC-007, User Story 3 | Spec 079, unchanged | +| FR-020, FR-022 | Spec 101 US1, unchanged | +| FR-021 | Spec 101 US3 (T341, provenance guard) | +| SC-001, SC-008 | Spec 101 US1 and the CI eval gate, unchanged | +| SC-002 to SC-006 | Spec 101 US3 (T321 to T329, T340, T391) | +| SC-009 | Spec 101 US3 (T391) | + ## User Scenarios & Testing *(mandatory)*