From 507390cffa13d909d6c11932e5986a12bf79830b Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:13:51 +0300 Subject: [PATCH 1/6] chore(roadmap): truth-sync statuses against code, clear all --check-github warnings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An evidence-based re-derivation of every non-done roadmap.yaml leaf. Each flip below was verified against the tree, git/gh state and the rendering sites, then independently re-checked by an adversarial pass; claims that did not survive that pass were dropped rather than applied. Statuses that were wrong in the "already shipped" direction: - upgrade-nudge-delta todo -> done (#1085, closes #1081). The note asserted TODO(spec-079/FR-002) markers exist and that the checker calls only /releases/latest; both are false. internal/updatecheck/delta.go now computes the delta and all six surfaces render it. - upgrade-nudge epic in_progress -> done (all five tasks now done). - analytics-dashboard in_progress -> done. Both children shipped; the default-landing half is live (router '/' -> Dashboard, guarded by dashboard-default-landing.spec.ts). Spec 069's one open task is a local Playwright sweep that leaves no artifact and can never tick from code. - discovery-eval-harness todo -> done. Both halves ship: cmd/scan-eval for security recall/FP, and the eval.yml retrieval-d1 gate plus bench/'s own RecallAtK/NDCGAtK + recall@5 parity gate for discovery quality. - hygiene-spec-gardener in_progress -> done (#824, #870, #999, #842). Its recorded blocker is cleared: check-spec-evidence.py is on main. Statuses that were wrong in the "claimed progress it does not have" direction: - windows-tray and windows-tray-window in_review -> todo. No Windows-tray PR is open or merged and native/windows/ is a README placeholder, so "in review" pointed at nothing. The section header is renamed to match. - planning-hygiene todo -> in_progress and action-log-transparency todo -> in_progress; both already have done children. Notes corrected where they would misdirect the next implementer: - token-bench-harness claimed it absorbs discovery-eval-harness by measuring retrieval recall. Recall is already measured, and bench/ already covers token counting, encoding arms (incl. direct_deferred), break-even and reporting. The note now records what is genuinely open: activity-log session replay, measured first-call success/retry rate, and tokens-per-successful-task. - token-bench-public: the MCP-specific third already ships via spec 083 (ToolRet, LiveMCPBench snapshot, lap-score); only agent-loop suites remain. specs/098-tools-preflight/tasks.md: tick T001, T024, T031, each confirmed against the tree (doc.go states both invariants T001 requires; toolslist_snapshot_test.go is labelled T024 and green; README carries all three FR-017 examples). T025/T027/T032/T033 are left unchecked — they are one-time process acts or did not survive verification. Deliberately NOT changed: no epics were added for shipped specs (084-100). roadmap.yaml is the working set by design and ROADMAP.md already renders a generated per-spec progress table, so absence of an epic is not drift. scripts/gen-roadmap.py --check-github: 3 warnings -> 0 errors, 0 warnings. --- ROADMAP.md | 282 ++++++++++++++--------------- roadmap.yaml | 37 ++-- specs/098-tools-preflight/tasks.md | 6 +- 3 files changed, 161 insertions(+), 164 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 422ec88a7..5cc0433f7 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -66,17 +66,16 @@ graph LR classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff; classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; - class sandbox_isolation,scanner_v2,scanner_simplification,schema_deferred done; - class ux_audit,analytics_dashboard,telemetry_identity,telemetry_v7_churn in_progress; - class action_log_transparency,tpa_db,remote_access_tunnel,token_bench,tool_graph,discovery_eval_harness todo; + class sandbox_isolation,scanner_v2,analytics_dashboard,scanner_simplification,schema_deferred,discovery_eval_harness done; + class ux_audit,action_log_transparency,telemetry_identity,telemetry_v7_churn in_progress; + class tpa_db,remote_access_tunnel,token_bench,tool_graph todo; ``` **Independent epics** (14) — no cross-epic prerequisites; each stands alone: -- 🔵 **Upgrade awareness & guided update** — In progress · P0 - 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0 -- 🟡 **Windows native tray app** — In review · P2 -- ⚪ **Planning/docs truth automation** — Todo · P2 +- 🔵 **Planning/docs truth automation** — In progress · P2 +- ⚪ **Windows native tray app** — Todo · P2 - ⚫ **Server marketplace** — Todo · P3 · parked - ⚫ **Audit SIEM integration** — Todo · P3 · parked - ⚫ **Paid-tier MVP (billing / seats / license)** — Todo · P3 · parked @@ -84,6 +83,7 @@ graph LR - ⚫ **SSO (server edition)** — Todo · P3 · parked - ⚪ **MCP protocol upgrade to 2026-07-28 revision** — Todo · P3 - ⚪ **Security gateway Tracks C/D (per-arg least-privilege + signature provenance)** — Todo · P3 +- 🟢 **Upgrade awareness & guided update** — Done · P0 - 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0 - 🟢 **Registries — easier search + add-server** — Done · P1 - 🟢 **Tray↔core decoupling: socket/REST API only, no config-file reads** — Done · P2 @@ -134,42 +134,6 @@ graph LR -
-🔵 Upgrade awareness & guided update — In progress · P0 - -> Corrected CI-filtered telemetry (2026-07-02): ~60% of last-14d active installs run pre-v0.40; latest stable v0.46.0 only 18.7%. Turn the existing internal/updatecheck background poll into a universal, non-intrusive, channel-aware upgrade nudge across every surface. Never blocks/modals; silent offline/CI. 2026-08-29 truth-sync: this epic was marked done, but FR-002's release/age delta was never built — the four shipped tasks are US-slices and the FR belonged to none of them. Spec 079 has only spec.md (no plan.md/tasks.md), so no checkbox surface could catch it; the deferral survived only as TODO(spec-079/FR-002) code comments, which is how an outside contributor found it (#1081). - -Spec: [079-upgrade-nudge](./specs/079-upgrade-nudge/) - -```mermaid -graph LR - upgrade_nudge_status_log["US1 slice: update availability in mcpproxy st…"] - upgrade_nudge_surfacing["US1 remainder: dismissible Web UI banner + up…"] - upgrade_nudge_channel["US2: channel-aware guided update command (bre…"] - upgrade_nudge_quiet["US3: operator control + CI/offline quiet + no…"] - upgrade_nudge_delta["FR-002 remainder: human-readable 'N releases…"] - - upgrade_nudge_status_log --> upgrade_nudge_surfacing - upgrade_nudge_surfacing --> upgrade_nudge_channel - upgrade_nudge_surfacing --> upgrade_nudge_quiet - upgrade_nudge_channel --> upgrade_nudge_delta - - classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; - classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; - class upgrade_nudge_status_log,upgrade_nudge_surfacing,upgrade_nudge_channel,upgrade_nudge_quiet done; - class upgrade_nudge_delta todo; -``` - -| Task | Status | Refs | -| --- | --- | --- | -| US1 slice: update availability in mcpproxy status + deduped startup log | 🟢 Done | #798 | -| US1 remainder: dismissible Web UI banner + update_check config block | 🟢 Done | #805 | -| US2: channel-aware guided update command (brew/dmg/deb/rpm/docker/go-install detection, build-time channel marker) | 🟢 Done | #818 | -| US3: operator control + CI/offline quiet + no prerelease downgrade nudges | 🟢 Done | #911 | -| FR-002 remainder: human-readable "N releases / M weeks behind" delta on status, doctor, startup log, Web UI and the tray | ⚪ Todo | — | - -
-
🔵 Release qualification gate (auto-QA matrix blocks the tag) — In progress · P0 @@ -204,27 +168,34 @@ graph LR
-🔵 Analytics dashboard as default page — In progress · P1 - -> Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done; only the default-landing half remains. +🔵 Action log / transparency — info at a glance — In progress · P1 -Spec: [069-observability-usage-graphs](./specs/069-observability-usage-graphs/) +> Surface the most important activity/security/connection signals at a glance; reduce digging. Vision pillar 'feel control → transparency' — the activity log is a headline feature, polish it and bring it to the tray menu. Builds on the shipped activity-log backend + retention (spec 024, 95% shipped — this epic is the at-a-glance UX on top, not the backend, so 024 is not the progress driver). ```mermaid graph LR - analytics_token_drain_graphs["Per-server / per-tool token-drain graphs"] - analytics_default_landing["Make dashboard the default landing page"] + sessions_web_ui["Sessions in the Web UI: meaningful session na…"] + action_log_glance_view["At-a-glance action log view (top signals, hea…"] + action_log_tray_menu["Activity in the tray menu (recent tool calls…"] + tray_menu_open_telemetry["tray_menu_opened counter: Swift menuWillOpen…"] + action_log_retention_tie_in["Tie activity retention/size into the glance v…"] - analytics_token_drain_graphs --> analytics_default_landing + action_log_glance_view --> action_log_tray_menu + action_log_glance_view --> action_log_retention_tie_in classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; - class analytics_token_drain_graphs,analytics_default_landing done; + classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; + class sessions_web_ui done; + class action_log_glance_view,action_log_tray_menu,tray_menu_open_telemetry,action_log_retention_tie_in todo; ``` | Task | Status | Refs | | --- | --- | --- | -| Per-server / per-tool token-drain graphs | 🟢 Done | — | -| Make dashboard the default landing page | 🟢 Done | #1044 | +| Sessions in the Web UI: meaningful session names in the Activity Log filter + the existing /sessions page linked in the sidebar | 🟢 Done | — | +| At-a-glance action log view (top signals, health) | ⚪ Todo | — | +| Activity in the tray menu (recent tool calls + security events, jump to full log) | ⚪ Todo | — | +| tray_menu_opened counter: Swift menuWillOpen (MCPProxyApp.swift:192) -> lightweight POST /api/v1/telemetry/tray-menu-opened -> registry counter -> heartbeat tray_menu_opened_24h | ⚪ Todo | — | +| Tie activity retention/size into the glance view | ⚪ Todo | — |
@@ -297,59 +268,35 @@ graph LR
-🟡 Windows native tray app — In review · P2 · MCP-43 +🔵 Planning/docs truth automation — In progress · P2 -> No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first. - -```mermaid -graph LR - windows_tray_funnel_qa["Windows first-run QA pass (downloads→actives…"] - windows_tray_window["WebView2 native window + profile submenu
MCP-43"] - - windows_tray_funnel_qa --> windows_tray_window - - classDef in_review fill:#9a6700,stroke:#5c3d00,color:#ffffff; - classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; - class windows_tray_window in_review; - class windows_tray_funnel_qa todo; -``` - -| Task | Status | Refs | -| --- | --- | --- | -| Windows first-run QA pass (downloads→actives 12:1 vs macOS 4:1 — find the funnel break before WebView2 work) | ⚪ Todo | — | -| WebView2 native window + profile submenu | 🟡 In review | `MCP-43` | - -
- -
-⚪ Action log / transparency — info at a glance — Todo · P1 - -> Surface the most important activity/security/connection signals at a glance; reduce digging. Vision pillar 'feel control → transparency' — the activity log is a headline feature, polish it and bring it to the tray menu. Builds on the shipped activity-log backend + retention (spec 024, 95% shipped — this epic is the at-a-glance UX on top, not the backend, so 024 is not the progress driver). +> Automate the consistency checks this very audit had to do by hand: roadmap vs GitHub PR state, tasks.md updates on implementation PRs, volatile CLAUDE.md/README facts, and quickstart contract tests. ```mermaid graph LR - sessions_web_ui["Sessions in the Web UI: meaningful session na…"] - action_log_glance_view["At-a-glance action log view (top signals, hea…"] - action_log_tray_menu["Activity in the tray menu (recent tool calls…"] - tray_menu_open_telemetry["tray_menu_opened counter: Swift menuWillOpen…"] - action_log_retention_tie_in["Tie activity retention/size into the glance v…"] + hygiene_roadmap_github_check["gen-roadmap --check-github: cross-check roadm…"] + hygiene_tasks_reconcile["CI rule: PR touching specs/ implementatio…"] + hygiene_spec_evidence_check["scripts/check-spec-evidence.py: deterministic…"] + hygiene_spec_gardener["Weekly cloud routine: LLM judges only the res…"] + hygiene_docs_facts["Generate volatile CLAUDE.md/README facts (Go…"] + hygiene_quickstart_contract["Run top quickstart.md scenario per spec as co…"] - action_log_glance_view --> action_log_tray_menu - action_log_glance_view --> action_log_retention_tie_in + hygiene_spec_evidence_check --> hygiene_spec_gardener classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; - class sessions_web_ui done; - class action_log_glance_view,action_log_tray_menu,tray_menu_open_telemetry,action_log_retention_tie_in todo; + class hygiene_roadmap_github_check,hygiene_spec_evidence_check,hygiene_spec_gardener done; + class hygiene_tasks_reconcile,hygiene_docs_facts,hygiene_quickstart_contract todo; ``` | Task | Status | Refs | | --- | --- | --- | -| Sessions in the Web UI: meaningful session names in the Activity Log filter + the existing /sessions page linked in the sidebar | 🟢 Done | — | -| At-a-glance action log view (top signals, health) | ⚪ Todo | — | -| Activity in the tray menu (recent tool calls + security events, jump to full log) | ⚪ Todo | — | -| tray_menu_opened counter: Swift menuWillOpen (MCPProxyApp.swift:192) -> lightweight POST /api/v1/telemetry/tray-menu-opened -> registry counter -> heartbeat tray_menu_opened_24h | ⚪ Todo | — | -| Tie activity retention/size into the glance view | ⚪ Todo | — | +| gen-roadmap --check-github: cross-check roadmap.yaml statuses vs gh PR state + dangling spec links | 🟢 Done | #800 | +| CI rule: PR touching specs/ implementation paths must update tasks.md | ⚪ Todo | — | +| scripts/check-spec-evidence.py: deterministic check that every TICKED task cites code that exists | 🟢 Done | — | +| Weekly cloud routine: LLM judges only the residue the evidence-check cannot decide, opens/updates one propose-only PR | 🟢 Done | #824 #870 #999 #842 | +| Generate volatile CLAUDE.md/README facts (Go version, built-in tool list, sample config) from code with --check | ⚪ Todo | — | +| Run top quickstart.md scenario per spec as contract test in test-api-e2e.sh | ⚪ Todo | — |
@@ -409,6 +356,29 @@ graph LR +
+⚪ Windows native tray app — Todo · P2 · MCP-43 + +> No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first. 2026-08-31 audit: reset from in_review to todo — no Windows-tray PR is open or merged, and native/windows/ holds only a README placeholder with no WebView2 code anywhere in the tree, so 'in review' had no PR to point at. + +```mermaid +graph LR + windows_tray_funnel_qa["Windows first-run QA pass (downloads→actives…"] + windows_tray_window["WebView2 native window + profile submenu
MCP-43"] + + windows_tray_funnel_qa --> windows_tray_window + + classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; + class windows_tray_funnel_qa,windows_tray_window todo; +``` + +| Task | Status | Refs | +| --- | --- | --- | +| Windows first-run QA pass (downloads→actives 12:1 vs macOS 4:1 — find the funnel break before WebView2 work) | ⚪ Todo | — | +| WebView2 native window + profile submenu | ⚪ Todo | `MCP-43` | + +
+
⚪ Remote access tunnel (feature-flagged MVP, spec 089) — Todo · P2 @@ -469,41 +439,6 @@ graph LR
-
-⚪ Planning/docs truth automation — Todo · P2 - -> Automate the consistency checks this very audit had to do by hand: roadmap vs GitHub PR state, tasks.md updates on implementation PRs, volatile CLAUDE.md/README facts, and quickstart contract tests. - -```mermaid -graph LR - hygiene_roadmap_github_check["gen-roadmap --check-github: cross-check roadm…"] - hygiene_tasks_reconcile["CI rule: PR touching specs/ implementatio…"] - hygiene_spec_evidence_check["scripts/check-spec-evidence.py: deterministic…"] - hygiene_spec_gardener["Weekly cloud routine: LLM judges only the res…"] - hygiene_docs_facts["Generate volatile CLAUDE.md/README facts (Go…"] - hygiene_quickstart_contract["Run top quickstart.md scenario per spec as co…"] - - hygiene_spec_evidence_check --> hygiene_spec_gardener - - classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; - classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff; - classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; - class hygiene_roadmap_github_check,hygiene_spec_evidence_check done; - class hygiene_spec_gardener in_progress; - class hygiene_tasks_reconcile,hygiene_docs_facts,hygiene_quickstart_contract todo; -``` - -| Task | Status | Refs | -| --- | --- | --- | -| gen-roadmap --check-github: cross-check roadmap.yaml statuses vs gh PR state + dangling spec links | 🟢 Done | #800 | -| CI rule: PR touching specs/ implementation paths must update tasks.md | ⚪ Todo | — | -| scripts/check-spec-evidence.py: deterministic check that every TICKED task cites code that exists | 🟢 Done | — | -| Weekly cloud routine: LLM judges only the residue the evidence-check cannot decide, opens/updates one propose-only PR | 🔵 In progress | — | -| Generate volatile CLAUDE.md/README facts (Go version, built-in tool list, sample config) from code with --check | ⚪ Todo | — | -| Run top quickstart.md scenario per spec as contract test in test-api-e2e.sh | ⚪ Todo | — | - -
-
⚪ MCP protocol upgrade to 2026-07-28 revision — Todo · P3 @@ -522,15 +457,6 @@ Spec: [054-mcp-security-gateway](./specs/054-mcp-security-gateway/)
-
-⚪ Discovery-quality eval harness (Spec 065 second half) — Todo · P3 - -> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness. - -Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) - -
-
⚫ Server marketplace — Todo · parked · P3 · MCP-37 @@ -566,6 +492,40 @@ Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/)
+
+🟢 Upgrade awareness & guided update — Done · P0 + +> Corrected CI-filtered telemetry (2026-07-02): ~60% of last-14d active installs run pre-v0.40; latest stable v0.46.0 only 18.7%. Turn the existing internal/updatecheck background poll into a universal, non-intrusive, channel-aware upgrade nudge across every surface. Never blocks/modals; silent offline/CI. 2026-08-29 truth-sync: this epic was marked done, but FR-002's release/age delta was never built — the four shipped tasks are US-slices and the FR belonged to none of them. Spec 079 has only spec.md (no plan.md/tasks.md), so no checkbox surface could catch it; the deferral survived only as TODO(spec-079/FR-002) code comments, which is how an outside contributor found it (#1081). RESOLVED 2026-08-29 in #1085; all five tasks are now done and the epic is complete. + +Spec: [079-upgrade-nudge](./specs/079-upgrade-nudge/) + +```mermaid +graph LR + upgrade_nudge_status_log["US1 slice: update availability in mcpproxy st…"] + upgrade_nudge_surfacing["US1 remainder: dismissible Web UI banner + up…"] + upgrade_nudge_channel["US2: channel-aware guided update command (bre…"] + upgrade_nudge_quiet["US3: operator control + CI/offline quiet + no…"] + upgrade_nudge_delta["FR-002 remainder: human-readable 'N releases…"] + + upgrade_nudge_status_log --> upgrade_nudge_surfacing + upgrade_nudge_surfacing --> upgrade_nudge_channel + upgrade_nudge_surfacing --> upgrade_nudge_quiet + upgrade_nudge_channel --> upgrade_nudge_delta + + classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; + class upgrade_nudge_status_log,upgrade_nudge_surfacing,upgrade_nudge_channel,upgrade_nudge_quiet,upgrade_nudge_delta done; +``` + +| Task | Status | Refs | +| --- | --- | --- | +| US1 slice: update availability in mcpproxy status + deduped startup log | 🟢 Done | #798 | +| US1 remainder: dismissible Web UI banner + update_check config block | 🟢 Done | #805 | +| US2: channel-aware guided update command (brew/dmg/deb/rpm/docker/go-install detection, build-time channel marker) | 🟢 Done | #818 | +| US3: operator control + CI/offline quiet + no prerelease downgrade nudges | 🟢 Done | #911 | +| FR-002 remainder: human-readable "N releases / M weeks behind" delta on status, doctor, startup log, Web UI and the tray | 🟢 Done | #1085 | + +
+
🟢 Connect step trust: preview, visible backup, one-click undo — Done · P0 @@ -664,6 +624,31 @@ graph LR
+
+🟢 Analytics dashboard as default page — Done · P1 + +> Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done. 2026-08-31 audit: the default-landing half shipped too - frontend/src/router/index.ts routes path '/' to the Dashboard component, guarded by frontend/tests/unit/dashboard-default-landing.spec.ts. Spec 069's one remaining task is a local Playwright verification sweep that leaves no committed artifact, so it can never tick from code; the epic is complete. + +Spec: [069-observability-usage-graphs](./specs/069-observability-usage-graphs/) + +```mermaid +graph LR + analytics_token_drain_graphs["Per-server / per-tool token-drain graphs"] + analytics_default_landing["Make dashboard the default landing page"] + + analytics_token_drain_graphs --> analytics_default_landing + + classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; + class analytics_token_drain_graphs,analytics_default_landing done; +``` + +| Task | Status | Refs | +| --- | --- | --- | +| Per-server / per-tool token-drain graphs | 🟢 Done | — | +| Make dashboard the default landing page | 🟢 Done | #1044 | + +
+
🟢 Registries — easier search + add-server — Done · P1 @@ -755,38 +740,47 @@ graph LR
+
+🟢 Discovery-quality eval harness (Spec 065 second half) — Done · P3 + +> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. 2026-08-31 audit: BOTH HALVES ARE SHIPPED, so this is done rather than pending. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo, and bench/ carries a second in-repo implementation (metrics.go RecallAtK/NDCGAtK, armindex.go's recall@5 parity gate through the production Bleve index). Kept as a stable depends_on target; still do not build a standalone harness. + +Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) + +
+ ## Epics | Epic | Status | Priority | Progress | Spec | PR | | --- | --- | --- | --- | --- | --- | | Web UI + macOS app UX audit | In progress | P0 | — | | | -| Upgrade awareness & guided update | In progress | P0 | — | [079-upgrade-nudge](./specs/079-upgrade-nudge/) | | | Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | | -| Analytics dashboard as default page | In progress | P1 | 25/26 (96%) | [069-observability-usage-graphs](./specs/069-observability-usage-graphs/) | | +| Action log / transparency — info at a glance | In progress | P1 | — | | | | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | -| Windows native tray app `MCP-43` | In review | P2 | — | | | -| Action log / transparency — info at a glance | Todo | P1 | — | | | +| Planning/docs truth automation | In progress | P2 | — | | | | tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | — | [101-tpa-db](./specs/101-tpa-db/) | | | Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | | +| Windows native tray app `MCP-43` | Todo | P2 | — | | | | Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | | | Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | | -| Planning/docs truth automation | Todo | P2 | — | | | | MCP protocol upgrade to 2026-07-28 revision | Todo | P3 | — | [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) | | | Security gateway Tracks C/D (per-arg least-privilege + signature provenance) | Todo | P3 | — | [054-mcp-security-gateway](./specs/054-mcp-security-gateway/) | | -| Discovery-quality eval harness (Spec 065 second half) | Todo | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | | Server marketplace `MCP-37` | Todo (parked) | P3 | — | | | | Audit SIEM integration `MCP-39` | Todo (parked) | P3 | — | | | | Paid-tier MVP (billing / seats / license) `MCP-40` | Todo (parked) | P3 | — | | | | SDK v1 migration | Todo (parked) | P3 | — | | | | SSO (server edition) | Todo (parked) | P3 | — | | | +| Upgrade awareness & guided update | Done | P0 | — | [079-upgrade-nudge](./specs/079-upgrade-nudge/) | | | Connect step trust: preview, visible backup, one-click undo | Done | P0 | — | [078-connect-trust-preview](./specs/078-connect-trust-preview/) | | | Non-Docker sandbox isolation (Landlock) `MCP-34` | Done | P1 | — | | | | Spec 076 deterministic offline tool-scanner `MCP-3574` | Done | P1 | 22/24 (92%) | [076-deterministic-tool-scanner](./specs/076-deterministic-tool-scanner/) | | +| Analytics dashboard as default page | Done | P1 | 25/26 (96%) | [069-observability-usage-graphs](./specs/069-observability-usage-graphs/) | | | Registries — easier search + add-server | Done | P1 | 21/24 (88%) | [070-registry-easy-upstream-add](./specs/070-registry-easy-upstream-add/) | | | Scanner simplification (deterministic default, opt-in deep scan) | Done | P1 | 38/42 (90%) | [077-scanner-simplification](./specs/077-scanner-simplification/) | | | Deferred-schema serialization for the direct tools/list surface (spec 102) | Done | P1 | 89/89 (100%) | [102-schema-deferred](./specs/102-schema-deferred/) | #1063 | | Tray↔core decoupling: socket/REST API only, no config-file reads | Done | P2 | — | | | +| Discovery-quality eval harness (Spec 065 second half) | Done | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | ## Shipped (archived) @@ -899,7 +893,7 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [095-update-failure-ux](./specs/095-update-failure-ux/) | `shipped` | 28/28 (100%) | | [096-batched-call-tools](./specs/096-batched-call-tools/) | `in-flight` | 15/16 (94%) | | [097-stored-scripts](./specs/097-stored-scripts/) | `in-flight` | 13/14 (93%) | -| [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 26/33 (79%) | +| [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 29/33 (88%) | | [099-describe-check-mode](./specs/099-describe-check-mode/) | `in-flight` | 9/10 (90%) | | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | | [101-tpa-db](./specs/101-tpa-db/) | — | — | diff --git a/roadmap.yaml b/roadmap.yaml index 68ae8e008..233ceb527 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -143,14 +143,14 @@ epics: pr: "#780" depends_on: [scanner-v2-eval-gate] - # ── IN REVIEW ─────────────────────────────────────────────────────────── + # ── BLOCKED ON DISCOVERY (was IN REVIEW; nothing is in review as of 2026-08-31) ── - id: windows-tray title: Windows native tray app - status: in_review + status: todo priority: P2 mcp: MCP-43 depends_on: [] - note: "No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first." + note: "No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first. 2026-08-31 audit: reset from in_review to todo — no Windows-tray PR is open or merged, and native/windows/ holds only a README placeholder with no WebView2 code anywhere in the tree, so 'in review' had no PR to point at." tasks: - id: windows-tray-funnel-qa title: "Windows first-run QA pass (downloads→actives 12:1 vs macOS 4:1 — find the funnel break before WebView2 work)" @@ -159,7 +159,7 @@ epics: depends_on: [] - id: windows-tray-window title: WebView2 native window + profile submenu - status: in_review + status: todo mcp: MCP-43 depends_on: [windows-tray-funnel-qa] @@ -220,7 +220,7 @@ epics: - id: action-log-transparency title: Action log / transparency — info at a glance - status: todo + status: in_progress priority: P1 depends_on: [ux-audit] note: "Surface the most important activity/security/connection signals at a glance; reduce digging. Vision pillar 'feel control → transparency' — the activity log is a headline feature, polish it and bring it to the tray menu. Builds on the shipped activity-log backend + retention (spec 024, 95% shipped — this epic is the at-a-glance UX on top, not the backend, so 024 is not the progress driver)." @@ -256,11 +256,11 @@ epics: - id: analytics-dashboard title: Analytics dashboard as default page - status: in_progress + status: done priority: P1 spec: specs/069-observability-usage-graphs depends_on: [ux-audit] - note: "Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done; only the default-landing half remains." + note: "Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done. 2026-08-31 audit: the default-landing half shipped too - frontend/src/router/index.ts routes path '/' to the Dashboard component, guarded by frontend/tests/unit/dashboard-default-landing.spec.ts. Spec 069's one remaining task is a local Playwright verification sweep that leaves no committed artifact, so it can never tick from code; the epic is complete." tasks: - id: analytics-token-drain-graphs title: Per-server / per-tool token-drain graphs @@ -398,11 +398,12 @@ epics: title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries" status: todo depends_on: [] - note: "Absorbs discovery-eval-harness (spec 065 second half) — the same harness measures retrieval recall." + note: "BUILDS ON the existing bench/ harness - do NOT rebuild token counting, encoding arms, break-even or recall. Already shipped there (PRs #747/#748 MCP-42, #851 spec 083): six encoding arms including direct_deferred (spec 102), cl100k_base counting, live response cost, break-even, RecallAtK/NDCGAtK, the v2 report + dashboard, `make bench-discovery` and CI wiring in bench.yml/eval.yml. 2026-08-31 audit correction: the old note claimed this task absorbs discovery-eval-harness because it would measure retrieval recall - recall is ALREADY measured (eval.yml retrieval-d1 + bench/armindex.go), so that is not this task's contribution. GENUINELY OPEN and unique here: (a) activity-log session REPLAY (nothing in bench/ reads the activity log; internal/contracts/activity.go already carries WorkSessionID, ToolName and Response - watch ResponseTruncated, which would undercount tokens); (b) MEASURED first-call success and retry rate (bench/report.go currently uses literature-derived defaults, not observations); (c) tokens-per-SUCCESSFUL-task, which needs a task-completion signal the static harness has no concept of." - id: token-bench-public title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results" status: todo depends_on: [token-bench-harness] + note: "PARTIALLY SHIPPED via spec 083 - the MCP-specific third is already integrated, reproducible and CI-wired: ToolRet (runtime fetch, never committed; seeded subset via -subset/-seed), the 527-tool Apache-2.0 LiveMCPBench snapshot (already load-bearing for spec 102 SC-001 at fleet scale), and the pinned external lap-score linter as an independent verdict. REMAINING: the agent-loop suites, which need a pinned-LLM budget decision. Re-confirm the shortlist against the completed research pass before building - the τ-bench/BFCL naming here predates it." - id: token-bench-blog title: "Publish results + methodology on mcpproxy.app/blog" status: todo @@ -439,11 +440,11 @@ epics: # ── TELEMETRY-DRIVEN REPLAN (2026-07-02) ────────────────────────────────── - id: upgrade-nudge title: Upgrade awareness & guided update - status: in_progress + status: done priority: P0 spec: specs/079-upgrade-nudge depends_on: [] - note: "Corrected CI-filtered telemetry (2026-07-02): ~60% of last-14d active installs run pre-v0.40; latest stable v0.46.0 only 18.7%. Turn the existing internal/updatecheck background poll into a universal, non-intrusive, channel-aware upgrade nudge across every surface. Never blocks/modals; silent offline/CI. 2026-08-29 truth-sync: this epic was marked done, but FR-002's release/age delta was never built — the four shipped tasks are US-slices and the FR belonged to none of them. Spec 079 has only spec.md (no plan.md/tasks.md), so no checkbox surface could catch it; the deferral survived only as TODO(spec-079/FR-002) code comments, which is how an outside contributor found it (#1081)." + note: "Corrected CI-filtered telemetry (2026-07-02): ~60% of last-14d active installs run pre-v0.40; latest stable v0.46.0 only 18.7%. Turn the existing internal/updatecheck background poll into a universal, non-intrusive, channel-aware upgrade nudge across every surface. Never blocks/modals; silent offline/CI. 2026-08-29 truth-sync: this epic was marked done, but FR-002's release/age delta was never built — the four shipped tasks are US-slices and the FR belonged to none of them. Spec 079 has only spec.md (no plan.md/tasks.md), so no checkbox surface could catch it; the deferral survived only as TODO(spec-079/FR-002) code comments, which is how an outside contributor found it (#1081). RESOLVED 2026-08-29 in #1085; all five tasks are now done and the epic is complete." tasks: - id: upgrade-nudge-status-log title: "US1 slice: update availability in mcpproxy status + deduped startup log" @@ -468,8 +469,9 @@ epics: depends_on: [upgrade-nudge-surfacing] - id: upgrade-nudge-delta title: "FR-002 remainder: human-readable \"N releases / M weeks behind\" delta on status, doctor, startup log, Web UI and the tray" - status: todo - note: "Deferred by #798 and never picked up; TODO(spec-079/FR-002) markers at internal/updatecheck/checker.go and cmd/mcpproxy/status_cmd.go. Spec 079 acceptance scenarios 1-3 and FR-003 are unmet — every surface renders latest-version-only. The blocker is data, not rendering: the checker calls only /releases/latest, so it has the latest release's publish date but not the running version's, and no tag sequence to count. Additive VersionInfo fields per FR-021; must stay silent for development builds (FR-017) and never render a negative delta (FR-016). Reported by an outside contributor in #1081." + status: done + pr: "#1085" + note: "SHIPPED in #1085 (2026-08-29), closing #1081. The data blocker is resolved: internal/updatecheck/delta.go (ComputeReleaseDelta/FormatBehindSummary) counts the delta by semver over one ETag-conditional /releases page, with a by-tag lookup for builds older than that page; github.go gained ListReleases + GetReleaseByTagContext. The clause is formatted once in Go and rendered verbatim on all six surfaces: status_cmd.go, doctor_cmd.go, the deduped startup log (checker.go), UpdateBanner.vue, the Go tray and the Swift tray. Fields additive per FR-021 (behind_summary, releases_behind, releases_behind_saturated, weeks_behind); FR-016/FR-017 guarded in resolveDelta; a delta miss never sets CheckError nor touches the FR-018 backoff. The TODO(spec-079/FR-002) markers are gone (2026-08-31 audit: grep -rn \"spec-079\" --include=*.go internal cmd -> no hits)." issue: "#1081" depends_on: [upgrade-nudge-channel] @@ -623,7 +625,7 @@ epics: - id: planning-hygiene title: Planning/docs truth automation - status: todo + status: in_progress priority: P2 depends_on: [] note: "Automate the consistency checks this very audit had to do by hand: roadmap vs GitHub PR state, tasks.md updates on implementation PRs, volatile CLAUDE.md/README facts, and quickstart contract tests." @@ -645,9 +647,10 @@ epics: depends_on: [] - id: hygiene-spec-gardener title: "Weekly cloud routine: LLM judges only the residue the evidence-check cannot decide, opens/updates one propose-only PR" - status: in_progress + status: done priority: P2 - note: "Claude cloud routine 'spec-roadmap gardener' (trig_014GkHno4XSTobu8ViJV7Sno), Mondays 05:07 UTC. Branch claude/spec-gardener (the `claude/` prefix is what the cloud git-proxy allows to push without unrestricted-branch-push). Guardrails: never pushes main; never auto-applies un-ticks (they go in the PR body for a human); evidence required per tick; adversarial refutation pass; 40-tick cap per run. Blocked in practice until check-spec-evidence.py lands on main — the routine no-ops until then." + pr: ["#824", "#870", "#999", "#842"] + note: "SHIPPED and operating: three gardener runs merged (#824 2026-07-13, #870 2026-07-29, #999 2026-08-17), auto-approved by .github/workflows/spec-gardener-auto-approve.yml (#842). The blocker recorded below is cleared - scripts/check-spec-evidence.py is on main. Claude cloud routine 'spec-roadmap gardener' (trig_014GkHno4XSTobu8ViJV7Sno), Mondays 05:07 UTC. Branch claude/spec-gardener (the `claude/` prefix is what the cloud git-proxy allows to push without unrestricted-branch-push). Guardrails: never pushes main; never auto-applies un-ticks (they go in the PR body for a human); evidence required per tick; adversarial refutation pass; 40-tick cap per run." depends_on: [hygiene-spec-evidence-check] - id: hygiene-docs-facts title: "Generate volatile CLAUDE.md/README facts (Go version, built-in tool list, sample config) from code with --check" @@ -722,8 +725,8 @@ epics: note: "Track A→Spec 056, Track B→Spec 059 (both shipped). UNBUILT: Track C per-ARGUMENT allow-listing (per-tool scope exists in mcp_direct_scope.go); Track D provenance + human-readable signature diff (SHA-256 pinning exists via Spec 032). Build ON 032/028, don't re-implement; honor the rug-pull re-quarantine interaction rule vs 032 auto-approve." - id: discovery-eval-harness title: Discovery-quality eval harness (Spec 065 second half) - status: todo + status: done priority: P3 spec: specs/065-evaluation-foundation depends_on: [token-bench-harness] - note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness." + note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. 2026-08-31 audit: BOTH HALVES ARE SHIPPED, so this is done rather than pending. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo, and bench/ carries a second in-repo implementation (metrics.go RecallAtK/NDCGAtK, armindex.go's recall@5 parity gate through the production Bleve index). Kept as a stable depends_on target; still do not build a standalone harness." diff --git a/specs/098-tools-preflight/tasks.md b/specs/098-tools-preflight/tasks.md index 46720e25e..a3c3221ca 100644 --- a/specs/098-tools-preflight/tasks.md +++ b/specs/098-tools-preflight/tasks.md @@ -6,7 +6,7 @@ ## Phase 1: Setup -- [ ] T001 Create `internal/preflight` package skeleton (`reasons.go`, `evaluator.go`, doc comment stating zero-I/O invariant and the error-not-fabricated-reason rule) per plan.md structure +- [x] T001 Create `internal/preflight` package skeleton (`reasons.go`, `evaluator.go`, doc comment stating zero-I/O invariant and the error-not-fabricated-reason rule) per plan.md structure - [x] T002 [P] Capture tools/list golden snapshots for all three routing modes **from merge-base (origin/main)** into `internal/server/testdata/toolslist_goldens/` + snapshot test `internal/server/toolslist_snapshot_test.go` comparing the feature branch byte-for-byte (FR-015; goldens must predate any dispatch refactor) ## Phase 2: Foundational (blocking all stories) @@ -47,7 +47,7 @@ ## Phase 6: User Story 4 — Non-regression (P3) -- [ ] T024 [US4] Golden snapshot test from T002 green on the finished branch (byte-identical tools/list across all three modes) (FR-015) +- [x] T024 [US4] Golden snapshot test from T002 green on the finished branch (byte-identical tools/list across all three modes) (FR-015) - [ ] T025 [US4] E2E: run a `code_execution` script and a stored script (spec 097) against the isolated instance; assert unchanged behavior and that dispatch decisions for sabotaged tools agree with preflight verdicts on all four dispatch paths (no-skew live check) ## Phase 7: Sabotage E2E matrix (acceptance gate) @@ -60,7 +60,7 @@ - [x] T028 [P] `docs/api/rest-api.md`: endpoint reference (envelope, tiers, 400/503 rules, wait semantics) (FR-017) - [x] T029 [P] `docs/cli-management-commands.md`: `tools preflight` reference with exit-code table and cron/CI recipe (FR-017) - [x] T030 [P] NEW `docs/features/tools-preflight.md`: concept, taxonomy + precedence tables, disclosure tiers, transparency/activity story, cron + GitHub Actions + n8n recipes, composition with code_execution/stored scripts (REST-from-harness pattern), Phase-2+ roadmap (FR-017) -- [ ] T031 [P] Usage-examples expansion: README/docs agent-workflow examples — token-saving discovery flow, typical agent actions through mcpproxy, preflight-gated automation example (FR-017) +- [x] T031 [P] Usage-examples expansion: README/docs agent-workflow examples — token-saving discovery flow, typical agent actions through mcpproxy, preflight-gated automation example (FR-017) - [ ] T032 Full gates: `go test -race ./...` (incl. cmd/ CLI tests), server-edition build+test+lint (T021 set), `./scripts/test-api-e2e.sh`, golangci-lint v2 `.github/.golangci.yml`, swagger + generate-types diff-clean - [ ] T033 Cross-model review of the full diff (opencode gpt-5.6-sol), fix→re-review ≤5 rounds; then quickstart walkthrough end-to-end on the isolated instance From dc6890e59e6fc1012f0de5d04b54086881ba64bb Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:24:10 +0300 Subject: [PATCH 2/6] fix(roadmap): correct six claims flagged by cross-model review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All six findings from the opencode gpt-5.6-sol review were independently re-verified against the tree before being applied; none were false positives. (The reviewer's initial claim that spec-gardener-auto-approve.yml is missing WAS a false positive from its glob tool and it self-corrected — the file exists at .github/workflows/spec-gardener-auto-approve.yml.) 1. Un-tick specs/098 T024. T024 asks for a golden snapshot test proving tools/list is BYTE-IDENTICAL across all three modes. Spec 099 (FR-014/015) later converted that test from a no-delta gate into an ENUMERATED-delta gate (toolsListAllowedDelta permits describe_tool and quarantine_security to move), so at HEAD the test no longer demonstrates the property T024 states. It may well have held on 098's own branch, but that is not checkable from the tree, and an unverifiable tick is exactly what this PR exists to remove. Badge 29/33 -> 28/33. 2. discovery-eval-harness note: drop the "SUPERSEDED — folded into token-bench-harness" framing. It contradicted itself once the epic was marked done — token-bench-harness is still unbuilt, so nothing could have been folded into it. Both halves shipped independently. 3. Same note: the eval.yml retrieval-d1 job is REPORT-ONLY on pull requests (continue-on-error, guarding against npx/uvx fetch flake) and blocks only on the nightly schedule. Promoting it to PR-blocking is still open (MCP-742). 4. Same note: the SC-003 recall@5 = 0.68 +/- 0.05 parity gate is asserted in bench/armindex_test.go, not armindex.go — the latter supplies the production index wiring, not the assertion. 5. token-bench-harness note: the literature-derived retry defaults live in bench/session.go (armRetryRates, research D8). bench/report.go only renders and labels them as estimates. The substantive point stands: retry rates are assumed, not observed. 6. token-bench-public note: "CI-wired" overstated the LiveMCPBench snapshot. bench.yml's regular job passes only -corpus-v2; the snapshot is runnable via -livemcptool but not run in CI, and its loader has no retrieval golden set, so it measures token/scale rather than recall. ToolRet does have its own bench.yml job. gen-roadmap.py --check and --check-github both still clean. --- ROADMAP.md | 4 ++-- roadmap.yaml | 6 +++--- specs/098-tools-preflight/tasks.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 5cc0433f7..fcf7861b5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -743,7 +743,7 @@ graph LR
🟢 Discovery-quality eval harness (Spec 065 second half) — Done · P3 -> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. 2026-08-31 audit: BOTH HALVES ARE SHIPPED, so this is done rather than pending. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo, and bench/ carries a second in-repo implementation (metrics.go RecallAtK/NDCGAtK, armindex.go's recall@5 parity gate through the production Bleve index). Kept as a stable depends_on target; still do not build a standalone harness. +> DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness. Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) @@ -893,7 +893,7 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [095-update-failure-ux](./specs/095-update-failure-ux/) | `shipped` | 28/28 (100%) | | [096-batched-call-tools](./specs/096-batched-call-tools/) | `in-flight` | 15/16 (94%) | | [097-stored-scripts](./specs/097-stored-scripts/) | `in-flight` | 13/14 (93%) | -| [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 29/33 (88%) | +| [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 28/33 (85%) | | [099-describe-check-mode](./specs/099-describe-check-mode/) | `in-flight` | 9/10 (90%) | | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | | [101-tpa-db](./specs/101-tpa-db/) | — | — | diff --git a/roadmap.yaml b/roadmap.yaml index 233ceb527..37758b4e2 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -398,12 +398,12 @@ epics: title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries" status: todo depends_on: [] - note: "BUILDS ON the existing bench/ harness - do NOT rebuild token counting, encoding arms, break-even or recall. Already shipped there (PRs #747/#748 MCP-42, #851 spec 083): six encoding arms including direct_deferred (spec 102), cl100k_base counting, live response cost, break-even, RecallAtK/NDCGAtK, the v2 report + dashboard, `make bench-discovery` and CI wiring in bench.yml/eval.yml. 2026-08-31 audit correction: the old note claimed this task absorbs discovery-eval-harness because it would measure retrieval recall - recall is ALREADY measured (eval.yml retrieval-d1 + bench/armindex.go), so that is not this task's contribution. GENUINELY OPEN and unique here: (a) activity-log session REPLAY (nothing in bench/ reads the activity log; internal/contracts/activity.go already carries WorkSessionID, ToolName and Response - watch ResponseTruncated, which would undercount tokens); (b) MEASURED first-call success and retry rate (bench/report.go currently uses literature-derived defaults, not observations); (c) tokens-per-SUCCESSFUL-task, which needs a task-completion signal the static harness has no concept of." + note: "BUILDS ON the existing bench/ harness - do NOT rebuild token counting, encoding arms, break-even or recall. Already shipped there (PRs #747/#748 MCP-42, #851 spec 083): six encoding arms including direct_deferred (spec 102), cl100k_base counting, live response cost, break-even, RecallAtK/NDCGAtK, the v2 report + dashboard, `make bench-discovery` and CI wiring in bench.yml/eval.yml. 2026-08-31 audit correction: the old note claimed this task absorbs discovery-eval-harness because it would measure retrieval recall - recall is ALREADY measured (eval.yml retrieval-d1 + bench/armindex.go), so that is not this task's contribution. GENUINELY OPEN and unique here: (a) activity-log session REPLAY (nothing in bench/ reads the activity log; internal/contracts/activity.go already carries WorkSessionID, ToolName and Response - watch ResponseTruncated, which would undercount tokens); (b) MEASURED first-call success and retry rate — today they are assumed, not observed: bench/session.go defines per-arm literature-derived defaults (armRetryRates, research D8) and bench/report.go only renders and labels them as estimates; (c) tokens-per-SUCCESSFUL-task, which needs a task-completion signal the static harness has no concept of." - id: token-bench-public title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results" status: todo depends_on: [token-bench-harness] - note: "PARTIALLY SHIPPED via spec 083 - the MCP-specific third is already integrated, reproducible and CI-wired: ToolRet (runtime fetch, never committed; seeded subset via -subset/-seed), the 527-tool Apache-2.0 LiveMCPBench snapshot (already load-bearing for spec 102 SC-001 at fleet scale), and the pinned external lap-score linter as an independent verdict. REMAINING: the agent-loop suites, which need a pinned-LLM budget decision. Re-confirm the shortlist against the completed research pass before building - the τ-bench/BFCL naming here predates it." + note: "PARTIALLY SHIPPED via spec 083 - the MCP-specific third is already integrated and reproducible: ToolRet (runtime fetch, never committed; seeded subset via -subset/-seed; run by the toolret-subset job in bench.yml), the 527-tool Apache-2.0 LiveMCPBench snapshot, and the pinned external lap-score linter as an independent verdict. Scope precisely: the LiveMCPBench snapshot is load-bearing for spec 102 SC-001 at fleet scale and is runnable via -livemcptool, but bench.yml does NOT run it (the regular job passes only -corpus-v2), and its loader carries no retrieval golden set — so it measures token/scale, not recall. REMAINING: the agent-loop suites, which need a pinned-LLM budget decision. Re-confirm the shortlist against the completed research pass before building - the τ-bench/BFCL naming here predates it." - id: token-bench-blog title: "Publish results + methodology on mcpproxy.app/blog" status: todo @@ -729,4 +729,4 @@ epics: priority: P3 spec: specs/065-evaluation-foundation depends_on: [token-bench-harness] - note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. 2026-08-31 audit: BOTH HALVES ARE SHIPPED, so this is done rather than pending. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo, and bench/ carries a second in-repo implementation (metrics.go RecallAtK/NDCGAtK, armindex.go's recall@5 parity gate through the production Bleve index). Kept as a stable depends_on target; still do not build a standalone harness." + note: "DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness." diff --git a/specs/098-tools-preflight/tasks.md b/specs/098-tools-preflight/tasks.md index a3c3221ca..c531a37d7 100644 --- a/specs/098-tools-preflight/tasks.md +++ b/specs/098-tools-preflight/tasks.md @@ -47,7 +47,7 @@ ## Phase 6: User Story 4 — Non-regression (P3) -- [x] T024 [US4] Golden snapshot test from T002 green on the finished branch (byte-identical tools/list across all three modes) (FR-015) +- [ ] T024 [US4] Golden snapshot test from T002 green on the finished branch (byte-identical tools/list across all three modes) (FR-015) - [ ] T025 [US4] E2E: run a `code_execution` script and a stored script (spec 097) against the isolated instance; assert unchanged behavior and that dispatch decisions for sabotaged tools agree with preflight verdicts on all four dispatch paths (no-skew live check) ## Phase 7: Sabotage E2E matrix (acceptance gate) From 38bea568b3e417efa01d81cf0137d4cddc417d90 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:28:00 +0300 Subject: [PATCH 3/6] fix(roadmap): tighten two claims from cross-model review round 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both findings verified against the tree before applying. 1. The retrieval-D1 blocking scope was stated too narrowly. eval.yml sets continue-on-error only for github.event_name == 'pull_request', and the workflow triggers on pull_request, workflow_dispatch and schedule — so the job is report-only on PRs and BLOCKING on both nightly and manual dispatch runs, not "only on the nightly schedule". The note now says so, and records that the workflow's own inline comment carries the same omission. 2. discovery-eval-harness is now done but still declared depends_on: [token-bench-harness], which is unbuilt — a done node cannot depend on an open one, and the edge contradicted the corrected note saying both halves shipped independently. Edge removed; the id remains a valid depends_on target for anything that needs it. gen-roadmap.py --check and --check-github both clean. --- ROADMAP.md | 9 ++++----- roadmap.yaml | 4 ++-- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index fcf7861b5..8b2feedcc 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -48,7 +48,6 @@ graph LR tool_graph["Tool co-occurrence graph (experimental,…"] telemetry_identity["Telemetry identity & data quality (mach…"] telemetry_v7_churn["Telemetry v7: honest funnel + churn ins…"] - discovery_eval_harness["Discovery-quality eval harness (Spec 06…"] scanner_v2 --> sandbox_isolation ux_audit --> action_log_transparency @@ -61,17 +60,16 @@ graph LR schema_deferred --> token_bench analytics_dashboard --> tool_graph telemetry_identity --> telemetry_v7_churn - token_bench --> discovery_eval_harness classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff; classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; - class sandbox_isolation,scanner_v2,analytics_dashboard,scanner_simplification,schema_deferred,discovery_eval_harness done; + class sandbox_isolation,scanner_v2,analytics_dashboard,scanner_simplification,schema_deferred done; class ux_audit,action_log_transparency,telemetry_identity,telemetry_v7_churn in_progress; class tpa_db,remote_access_tunnel,token_bench,tool_graph todo; ``` -**Independent epics** (14) — no cross-epic prerequisites; each stands alone: +**Independent epics** (15) — no cross-epic prerequisites; each stands alone: - 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0 - 🔵 **Planning/docs truth automation** — In progress · P2 @@ -87,6 +85,7 @@ graph LR - 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0 - 🟢 **Registries — easier search + add-server** — Done · P1 - 🟢 **Tray↔core decoupling: socket/REST API only, no config-file reads** — Done · P2 +- 🟢 **Discovery-quality eval harness (Spec 065 second half)** — Done · P3 ## Epic details @@ -743,7 +742,7 @@ graph LR
🟢 Discovery-quality eval harness (Spec 065 second half) — Done · P3 -> DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness. +> DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness. Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) diff --git a/roadmap.yaml b/roadmap.yaml index 37758b4e2..59e417ee4 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -728,5 +728,5 @@ epics: status: done priority: P3 spec: specs/065-evaluation-foundation - depends_on: [token-bench-harness] - note: "DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness." + depends_on: [] + note: "DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness." From 0537b04f96d74a499cd64d117a9a33184985da4f Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:34:17 +0300 Subject: [PATCH 4/6] fix(roadmap): scope four over-broad claims from cross-model review round 3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All four verified against GitHub and the tree before applying. 1. windows-tray: "no Windows-tray PR is open or merged" was false. #74 ("feature: windows tray support") merged 2025-10-23 and the cross-platform Go/systray build still ships on Windows. The claim now scopes to what the epic actually tracks — the native WebView2 replacement — for which no PR exists and native/windows/ is a README placeholder. 2. analytics-dashboard: "can never tick from code" conflated two things. A human can run spec 069's T023 Playwright sweep and tick the box; what can never exist is code evidence for it. Reworded so the distinction is explicit. 3. hygiene-spec-gardener: "three gardener runs ... auto-approved by #842" was wrong. Only #999 carries a github-actions APPROVED review. #824 has none and merged ~24 minutes BEFORE the auto-approve workflow landed; #870 records no approving review either. 4. token-bench-public: calling the MCP-specific third "integrated" overstated completion. ToolRet and LiveMCPTool are static retrieval/token corpora and lap-score is a linter — none is a public agent-loop task suite, which is what the task ultimately owes. Also noted for the record: the linux-arm64 Build Binaries failure on the previous commit is a flaky test, not a regression from this docs-only change. TestServiceStartScanDeepOnRunsSourceResolutionAndPass2 (internal/security/scanner) passes locally with -count=3, and two runs of this same branch four minutes apart — 33360429077 and 33360648947, both docs-only — disagreed on it. --- ROADMAP.md | 4 ++-- roadmap.yaml | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 8b2feedcc..5d5b8af86 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -358,7 +358,7 @@ graph LR
⚪ Windows native tray app — Todo · P2 · MCP-43 -> No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first. 2026-08-31 audit: reset from in_review to todo — no Windows-tray PR is open or merged, and native/windows/ holds only a README placeholder with no WebView2 code anywhere in the tree, so 'in review' had no PR to point at. +> No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first. 2026-08-31 audit: reset from in_review to todo. Scoped precisely: Windows tray support DID ship in 2025 via the cross-platform Go/systray build (#74, merged 2025-10-23, cmd/mcpproxy-tray/ + internal/tray under GOOS=windows) — what this epic tracks is the NATIVE WebView2 replacement, and for that no PR is open or merged and native/windows/ holds only a README placeholder with no WebView2 code anywhere in the tree. So 'in review' had no PR to point at. ```mermaid graph LR @@ -626,7 +626,7 @@ graph LR
🟢 Analytics dashboard as default page — Done · P1 -> Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done. 2026-08-31 audit: the default-landing half shipped too - frontend/src/router/index.ts routes path '/' to the Dashboard component, guarded by frontend/tests/unit/dashboard-default-landing.spec.ts. Spec 069's one remaining task is a local Playwright verification sweep that leaves no committed artifact, so it can never tick from code; the epic is complete. +> Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done. 2026-08-31 audit: the default-landing half shipped too - frontend/src/router/index.ts routes path '/' to the Dashboard component, guarded by frontend/tests/unit/dashboard-default-landing.spec.ts. Spec 069's one remaining task (T023) is a local Playwright verification sweep that leaves no committed artifact: a human can run it and tick the box, but no code evidence can ever confirm it, so it cannot gate the epic. The epic is complete. Spec: [069-observability-usage-graphs](./specs/069-observability-usage-graphs/) diff --git a/roadmap.yaml b/roadmap.yaml index 59e417ee4..800fd996d 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -150,7 +150,7 @@ epics: priority: P2 mcp: MCP-43 depends_on: [] - note: "No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first. 2026-08-31 audit: reset from in_review to todo — no Windows-tray PR is open or merged, and native/windows/ holds only a README placeholder with no WebView2 code anywhere in the tree, so 'in review' had no PR to point at." + note: "No spec: link — this epic is the native TRAY app; specs/002-windows-installer is the unrelated INSTALLER spec (35/60) and its badge said nothing about tray progress (wrong link removed 2026-07-10). Option C: WebView2 window reusing shipped Web UI. Most exit criteria already ship; gaps = native window, toasts, profile submenu, Win11 smoke. Telemetry: Windows = ~23% of GitHub downloads but only ~4% of active installs (downloads→actives ~12:1 vs macOS ~4:1) — gate WebView2 work on finding the funnel break first. 2026-08-31 audit: reset from in_review to todo. Scoped precisely: Windows tray support DID ship in 2025 via the cross-platform Go/systray build (#74, merged 2025-10-23, cmd/mcpproxy-tray/ + internal/tray under GOOS=windows) — what this epic tracks is the NATIVE WebView2 replacement, and for that no PR is open or merged and native/windows/ holds only a README placeholder with no WebView2 code anywhere in the tree. So 'in review' had no PR to point at." tasks: - id: windows-tray-funnel-qa title: "Windows first-run QA pass (downloads→actives 12:1 vs macOS 4:1 — find the funnel break before WebView2 work)" @@ -260,7 +260,7 @@ epics: priority: P1 spec: specs/069-observability-usage-graphs depends_on: [ux-audit] - note: "Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done. 2026-08-31 audit: the default-landing half shipped too - frontend/src/router/index.ts routes path '/' to the Dashboard component, guarded by frontend/tests/unit/dashboard-default-landing.spec.ts. Spec 069's one remaining task is a local Playwright verification sweep that leaves no committed artifact, so it can never tick from code; the epic is complete." + note: "Per-server / per-tool token-drain graphs; make the dashboard the default landing page. 2026-07-10 truth-sync: spec 069 is SHIPPED (25/26 — the only open task is a Playwright verification sweep), so the graphs half is done. 2026-08-31 audit: the default-landing half shipped too - frontend/src/router/index.ts routes path '/' to the Dashboard component, guarded by frontend/tests/unit/dashboard-default-landing.spec.ts. Spec 069's one remaining task (T023) is a local Playwright verification sweep that leaves no committed artifact: a human can run it and tick the box, but no code evidence can ever confirm it, so it cannot gate the epic. The epic is complete." tasks: - id: analytics-token-drain-graphs title: Per-server / per-tool token-drain graphs @@ -403,7 +403,7 @@ epics: title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results" status: todo depends_on: [token-bench-harness] - note: "PARTIALLY SHIPPED via spec 083 - the MCP-specific third is already integrated and reproducible: ToolRet (runtime fetch, never committed; seeded subset via -subset/-seed; run by the toolret-subset job in bench.yml), the 527-tool Apache-2.0 LiveMCPBench snapshot, and the pinned external lap-score linter as an independent verdict. Scope precisely: the LiveMCPBench snapshot is load-bearing for spec 102 SC-001 at fleet scale and is runnable via -livemcptool, but bench.yml does NOT run it (the regular job passes only -corpus-v2), and its loader carries no retrieval golden set — so it measures token/scale, not recall. REMAINING: the agent-loop suites, which need a pinned-LLM budget decision. Re-confirm the shortlist against the completed research pass before building - the τ-bench/BFCL naming here predates it." + note: "PARTIALLY SHIPPED via spec 083 - the MCP-specific third is already integrated and reproducible AS RETRIEVAL/TOKEN CORPORA — not as public agent-loop task suites, which is what this task ultimately owes: ToolRet (runtime fetch, never committed; seeded subset via -subset/-seed; run by the toolret-subset job in bench.yml), the 527-tool Apache-2.0 LiveMCPBench snapshot, and the pinned external lap-score linter as an independent verdict. Scope precisely: the LiveMCPBench snapshot is load-bearing for spec 102 SC-001 at fleet scale and is runnable via -livemcptool, but bench.yml does NOT run it (the regular job passes only -corpus-v2), and its loader carries no retrieval golden set — so it measures token/scale, not recall. REMAINING: the agent-loop suites, which need a pinned-LLM budget decision. Re-confirm the shortlist against the completed research pass before building - the τ-bench/BFCL naming here predates it." - id: token-bench-blog title: "Publish results + methodology on mcpproxy.app/blog" status: todo @@ -650,7 +650,7 @@ epics: status: done priority: P2 pr: ["#824", "#870", "#999", "#842"] - note: "SHIPPED and operating: three gardener runs merged (#824 2026-07-13, #870 2026-07-29, #999 2026-08-17), auto-approved by .github/workflows/spec-gardener-auto-approve.yml (#842). The blocker recorded below is cleared - scripts/check-spec-evidence.py is on main. Claude cloud routine 'spec-roadmap gardener' (trig_014GkHno4XSTobu8ViJV7Sno), Mondays 05:07 UTC. Branch claude/spec-gardener (the `claude/` prefix is what the cloud git-proxy allows to push without unrestricted-branch-push). Guardrails: never pushes main; never auto-applies un-ticks (they go in the PR body for a human); evidence required per tick; adversarial refutation pass; 40-tick cap per run." + note: "SHIPPED and operating: three gardener runs merged (#824 2026-07-13, #870 2026-07-29, #999 2026-08-17). Auto-approval is NOT retroactive and did not carry all three — only #999 has a github-actions APPROVED review; #824 carries none and in fact merged ~24 min BEFORE the auto-approve workflow itself (#842, .github/workflows/spec-gardener-auto-approve.yml, merged 2026-07-13 04:40), and #870 records no approving review either. The blocker recorded below is cleared - scripts/check-spec-evidence.py is on main. Claude cloud routine 'spec-roadmap gardener' (trig_014GkHno4XSTobu8ViJV7Sno), Mondays 05:07 UTC. Branch claude/spec-gardener (the `claude/` prefix is what the cloud git-proxy allows to push without unrestricted-branch-push). Guardrails: never pushes main; never auto-applies un-ticks (they go in the PR body for a human); evidence required per tick; adversarial refutation pass; 40-tick cap per run." depends_on: [hygiene-spec-evidence-check] - id: hygiene-docs-facts title: "Generate volatile CLAUDE.md/README facts (Go version, built-in tool list, sample config) from code with --check" From e70e6841eacdaf4f03d7d18a3a86f06aecf1bc14 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:38:47 +0300 Subject: [PATCH 5/6] fix(roadmap): ToolRet is not MCP-specific (cross-model review round 4) Verified: bench/corpusio/toolret.go documents ToolRet as a general tool-retrieval corpus whose documentation fields are "heterogeneous JSON/text, NOT MCP input schemas" (Schema stays nil). Grouping it under "the MCP-specific third" was wrong. The token-bench-public note now says public corpus/linter groundwork is integrated while NO public agent-loop task suite is, and names the ToolRet caveat explicitly so the next implementer does not re-inherit the error. Round 4 found no other issues; the round-3 windows-tray, analytics T023 and gardener corrections were all confirmed accurate against GitHub and the tree. --- roadmap.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roadmap.yaml b/roadmap.yaml index 800fd996d..924233288 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -403,7 +403,7 @@ epics: title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results" status: todo depends_on: [token-bench-harness] - note: "PARTIALLY SHIPPED via spec 083 - the MCP-specific third is already integrated and reproducible AS RETRIEVAL/TOKEN CORPORA — not as public agent-loop task suites, which is what this task ultimately owes: ToolRet (runtime fetch, never committed; seeded subset via -subset/-seed; run by the toolret-subset job in bench.yml), the 527-tool Apache-2.0 LiveMCPBench snapshot, and the pinned external lap-score linter as an independent verdict. Scope precisely: the LiveMCPBench snapshot is load-bearing for spec 102 SC-001 at fleet scale and is runnable via -livemcptool, but bench.yml does NOT run it (the regular job passes only -corpus-v2), and its loader carries no retrieval golden set — so it measures token/scale, not recall. REMAINING: the agent-loop suites, which need a pinned-LLM budget decision. Re-confirm the shortlist against the completed research pass before building - the τ-bench/BFCL naming here predates it." + note: "PARTIALLY SHIPPED via spec 083 - public CORPUS/LINTER groundwork is integrated and reproducible, but NO public agent-loop task suite is, and that is what this task ultimately owes. Do not call any of the following MCP-specific: ToolRet is a general tool-retrieval corpus whose docs are heterogeneous JSON/text and explicitly not MCP input schemas (bench/corpusio/toolret.go). What exists: ToolRet (runtime fetch, never committed; seeded subset via -subset/-seed; run by the toolret-subset job in bench.yml), the 527-tool Apache-2.0 LiveMCPBench snapshot, and the pinned external lap-score linter as an independent verdict. Scope precisely: the LiveMCPBench snapshot is load-bearing for spec 102 SC-001 at fleet scale and is runnable via -livemcptool, but bench.yml does NOT run it (the regular job passes only -corpus-v2), and its loader carries no retrieval golden set — so it measures token/scale, not recall. REMAINING: the agent-loop suites, which need a pinned-LLM budget decision. Re-confirm the shortlist against the completed research pass before building - the τ-bench/BFCL naming here predates it." - id: token-bench-blog title: "Publish results + methodology on mcpproxy.app/blog" status: todo From 492d2d2a7786df0b99425a9b89ceda8d35475007 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:45:59 +0300 Subject: [PATCH 6/6] fix(roadmap): discovery-eval-harness is in_progress, not done (review round 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The final review round caught an error in this PR's own flip, and it is the same failure mode the PR was written to fix: an epic marked done while a governing FR is unmet. Spec 065 FR-009 and SC-005 require CI to FAIL on a discovery regression beyond tolerance. The retrieval-D1 job sets continue-on-error for pull_request (D1 depends on npx/uvx fetches, a known flake source), so on the PR path CI does not fail — and eval.yml's own comment records the promotion to PR-blocking as still planned (MCP-742). The HARNESS shipped; the GATE did not. Reverted to in_progress and added the remaining work as a child task (discovery-eval-pr-blocking, MCP-742) so the epic's status has something in the DAG to point at. The reviewer also read CN-002 (frozen corpus, never live) as violated because D1 boots a live mcpproxy serving seven reference servers. Recorded as a weaker tension to adjudicate rather than a violation: #931 pinned all seven upstreams to freeze-era versions and the job gates on the exact corpus ID set, so the corpus is reproducible in practice. Whether that satisfies CN-002 or a committed snapshot is required is a judgement for the spec owner. gen-roadmap.py --check and --check-github both clean (0 errors, 0 warnings). --- ROADMAP.md | 35 ++++++++++++++++++++++++----------- roadmap.yaml | 11 +++++++++-- 2 files changed, 33 insertions(+), 13 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 5d5b8af86..a3cb75f58 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -73,6 +73,7 @@ graph LR - 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0 - 🔵 **Planning/docs truth automation** — In progress · P2 +- 🔵 **Discovery-quality eval harness (Spec 065 second half)** — In progress · P3 - ⚪ **Windows native tray app** — Todo · P2 - ⚫ **Server marketplace** — Todo · P3 · parked - ⚫ **Audit SIEM integration** — Todo · P3 · parked @@ -85,7 +86,6 @@ graph LR - 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0 - 🟢 **Registries — easier search + add-server** — Done · P1 - 🟢 **Tray↔core decoupling: socket/REST API only, no config-file reads** — Done · P2 -- 🟢 **Discovery-quality eval harness (Spec 065 second half)** — Done · P3 ## Epic details @@ -299,6 +299,28 @@ graph LR
+
+🔵 Discovery-quality eval harness (Spec 065 second half) — In progress · P3 + +> IN PROGRESS — 2026-08-31 audit, corrected on cross-model review: both halves of the HARNESS shipped INDEPENDENTLY (not via token-bench-harness), but spec 065 is NOT fully met, so this is not done. FR-009 and SC-005 require CI to FAIL on a discovery regression beyond tolerance; the retrieval-D1 job is continue-on-error on pull requests, so on the PR path it does not fail — eval.yml itself records the promotion to PR-blocking as still open (MCP-742). A second, weaker tension to adjudicate rather than assume: CN-002 asks that scoring never run against a live drifting corpus, and D1 does boot a live mcpproxy serving 7 reference servers — but #931 pinned all seven upstreams to freeze-era versions and the job gates on the exact corpus ID set, so the corpus is reproducible in practice. Decide whether that satisfies CN-002 or whether a committed snapshot is required. Remaining work is therefore the gating promotion, not the harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness. + +Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) + +```mermaid +graph LR + discovery_eval_pr_blocking["Promote retrieval-D1 from report-only to PR-b…
MCP-742"] + + + classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; + class discovery_eval_pr_blocking todo; +``` + +| Task | Status | Refs | +| --- | --- | --- | +| Promote retrieval-D1 from report-only to PR-blocking (spec 065 FR-009/SC-005), and adjudicate the CN-002 frozen-corpus question | ⚪ Todo | `MCP-742` | + +
+
⚪ tpa-db: versioned TPA signature database for the offline scanner — Todo · P1 @@ -739,15 +761,6 @@ graph LR
-
-🟢 Discovery-quality eval harness (Spec 065 second half) — Done · P3 - -> DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness. - -Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) - -
- ## Epics | Epic | Status | Priority | Progress | Spec | PR | @@ -758,6 +771,7 @@ Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | | Planning/docs truth automation | In progress | P2 | — | | | +| Discovery-quality eval harness (Spec 065 second half) | In progress | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | | tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | — | [101-tpa-db](./specs/101-tpa-db/) | | | Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | | | Windows native tray app `MCP-43` | Todo | P2 | — | | | @@ -779,7 +793,6 @@ Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) | Scanner simplification (deterministic default, opt-in deep scan) | Done | P1 | 38/42 (90%) | [077-scanner-simplification](./specs/077-scanner-simplification/) | | | Deferred-schema serialization for the direct tools/list surface (spec 102) | Done | P1 | 89/89 (100%) | [102-schema-deferred](./specs/102-schema-deferred/) | #1063 | | Tray↔core decoupling: socket/REST API only, no config-file reads | Done | P2 | — | | | -| Discovery-quality eval harness (Spec 065 second half) | Done | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | ## Shipped (archived) diff --git a/roadmap.yaml b/roadmap.yaml index 924233288..71ddc677e 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -725,8 +725,15 @@ epics: note: "Track A→Spec 056, Track B→Spec 059 (both shipped). UNBUILT: Track C per-ARGUMENT allow-listing (per-tool scope exists in mcp_direct_scope.go); Track D provenance + human-readable signature diff (SHA-256 pinning exists via Spec 032). Build ON 032/028, don't re-implement; honor the rug-pull re-quarantine interaction rule vs 032 auto-approve." - id: discovery-eval-harness title: Discovery-quality eval harness (Spec 065 second half) - status: done + status: in_progress priority: P3 spec: specs/065-evaluation-foundation depends_on: [] - note: "DONE — 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness." + tasks: + - id: discovery-eval-pr-blocking + title: "Promote retrieval-D1 from report-only to PR-blocking (spec 065 FR-009/SC-005), and adjudicate the CN-002 frozen-corpus question" + status: todo + mcp: MCP-742 + depends_on: [] + note: "The only thing standing between this epic and done. eval.yml sets continue-on-error for pull_request on the retrieval-D1 job because D1 depends on npx/uvx package fetches (a known flake source); the workflow's own comment plans the promotion after a green soak. Until then CI does not fail on a discovery regression on the PR path, which is exactly what FR-009 and SC-005 require." + note: "IN PROGRESS — 2026-08-31 audit, corrected on cross-model review: both halves of the HARNESS shipped INDEPENDENTLY (not via token-bench-harness), but spec 065 is NOT fully met, so this is not done. FR-009 and SC-005 require CI to FAIL on a discovery regression beyond tolerance; the retrieval-D1 job is continue-on-error on pull requests, so on the PR path it does not fail — eval.yml itself records the promotion to PR-blocking as still open (MCP-742). A second, weaker tension to adjudicate rather than assume: CN-002 asks that scoring never run against a live drifting corpus, and D1 does boot a live mcpproxy serving 7 reference servers — but #931 pinned all seven upstreams to freeze-era versions and the job gates on the exact corpus ID set, so the corpus is reproducible in practice. Decide whether that satisfies CN-002 or whether a committed snapshot is required. Remaining work is therefore the gating promotion, not the harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo — note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness."