Skip to content

Commit fa2a28f

Browse files
Dumbrisclaude
andauthored
chore(roadmap): mcp-2026-upgrade to P1, activity-storage-bounds task, token-bench truth-sync (#1200)
* chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench - mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped 2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532). - action-log-transparency: new task activity-storage-bounds tracking #1173/#1174 (response truncation never wired), #1176 (unbounded per-server tool_calls buckets) and #1175 (omitempty zero erasure). - token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160 are all merged. Clears both --check-github errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 806ffb0 commit fa2a28f

2 files changed

Lines changed: 32 additions & 21 deletions

File tree

‎ROADMAP.md‎

Lines changed: 20 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -74,13 +74,13 @@ graph LR
7474
- 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0
7575
- 🔵 **Planning/docs truth automation** — In progress · P2
7676
- 🔵 **Discovery-quality eval harness (Spec 065 second half)** — In progress · P3
77+
- ⚪ **MCP protocol upgrade to 2026-07-28 revision** — Todo · P1
7778
- ⚪ **Windows native tray app** — Todo · P2
7879
- ⚫ **Server marketplace** — Todo · P3 · parked
7980
- ⚫ **Audit SIEM integration** — Todo · P3 · parked
8081
- ⚫ **Paid-tier MVP (billing / seats / license)** — Todo · P3 · parked
8182
- ⚫ **SDK v1 migration** — Todo · P3 · parked
8283
- ⚫ **SSO (server edition)** — Todo · P3 · parked
83-
- ⚪ **MCP protocol upgrade to 2026-07-28 revision** — Todo · P3
8484
- ⚪ **Security gateway Tracks C/D (per-arg least-privilege + signature provenance)** — Todo · P3
8585
- 🟢 **Upgrade awareness & guided update** — Done · P0
8686
- 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0
@@ -178,13 +178,16 @@ graph LR
178178
action_log_tray_menu["Activity in the tray menu (recent tool calls…"]
179179
tray_menu_open_telemetry["tray_menu_opened counter: Swift menuWillOpen…"]
180180
action_log_retention_tie_in["Tie activity retention/size into the glance v…"]
181+
activity_storage_bounds["Bound every activity-adjacent store: response…"]
181182
182183
action_log_glance_view --> action_log_tray_menu
183184
action_log_glance_view --> action_log_retention_tie_in
184185
185186
classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
187+
classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
186188
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
187189
class sessions_web_ui done;
190+
class activity_storage_bounds in_progress;
188191
class action_log_glance_view,action_log_tray_menu,tray_menu_open_telemetry,action_log_retention_tie_in todo;
189192
```
190193

@@ -195,6 +198,7 @@ graph LR
195198
| Activity in the tray menu (recent tool calls + security events, jump to full log) | ⚪ Todo | — |
196199
| tray_menu_opened counter: Swift menuWillOpen (MCPProxyApp.swift:192) -> lightweight POST /api/v1/telemetry/tray-menu-opened -> registry counter -> heartbeat tray_menu_opened_24h | ⚪ Todo | — |
197200
| Tie activity retention/size into the glance view | ⚪ Todo | — |
201+
| Bound every activity-adjacent store: response truncation on the write path (#1173/#1174), per-server tool_calls buckets (#1176), omitempty zero-erasure (#1175) | 🔵 In progress | #1174 |
198202

199203
</details>
200204

@@ -203,7 +207,7 @@ graph LR
203207

204208
> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product.
205209

206-
Spec: [103-token-bench](./specs/103-token-bench/) · PR: #1141
210+
Spec: [103-token-bench](./specs/103-token-bench/)
207211

208212
```mermaid
209213
graph LR
@@ -215,15 +219,15 @@ graph LR
215219
token_bench_harness --> token_bench_public
216220
token_bench_public --> token_bench_blog
217221
218-
classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
222+
classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
219223
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
220-
class token_bench_harness in_progress;
224+
class token_bench_harness done;
221225
class token_bench_public,token_bench_blog,token_bench_telemetry todo;
222226
```
223227

224228
| Task | Status | Refs |
225229
| --- | --- | --- |
226-
| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | 🔵 In progress | #1141 |
230+
| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | 🟢 Done | #1141 #1147 #1151 #1153 #1160 |
227231
| Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results | ⚪ Todo | — |
228232
| Publish results + methodology on mcpproxy.app/blog | ⚪ Todo | — |
229233
| Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | ⚪ Todo | — |
@@ -381,6 +385,15 @@ graph LR
381385

382386
</details>
383387

388+
<details>
389+
<summary>⚪ MCP protocol upgrade to 2026-07-28 revision — Todo · P1</summary>
390+
391+
> STABLE GATE CLEARED 2026-09-02: mark3labs/mcp-go v1.0.0 (stable) released; go.mod still pins v0.57.0. Raised P3->P1 on the 2026-09-02 issue-prioritization pass — spec 058 has spec.md only, next step is speckit.plan then tasks (tracker #532). Earlier: UNBLOCKED 2026-08-12: the mcp-go gate cleared — v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that — implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/<slug> — Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532.
392+
393+
Spec: [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/)
394+
395+
</details>
396+
384397
<details>
385398
<summary>⚪ Windows native tray app — Todo · P2 · MCP-43</summary>
386399

@@ -464,15 +477,6 @@ graph LR
464477

465478
</details>
466479

467-
<details>
468-
<summary>⚪ MCP protocol upgrade to 2026-07-28 revision — Todo · P3</summary>
469-
470-
> UNBLOCKED 2026-08-12: the mcp-go gate cleared — v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that — implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/<slug> — Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532.
471-
472-
Spec: [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/)
473-
474-
</details>
475-
476480
<details>
477481
<summary>⚪ Security gateway Tracks C/D (per-arg least-privilege + signature provenance) — Todo · P3</summary>
478482

@@ -772,16 +776,16 @@ graph LR
772776
| Web UI + macOS app UX audit | In progress | P0 | — | | |
773777
| Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | |
774778
| Action log / transparency — info at a glance | In progress | P1 | — | | |
775-
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | #1141 |
779+
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | |
776780
| Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | |
777781
| Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | |
778782
| Planning/docs truth automation | In progress | P2 | — | | |
779783
| Discovery-quality eval harness (Spec 065 second half) | In progress | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | |
780784
| tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | — | [101-tpa-db](./specs/101-tpa-db/) | |
785+
| MCP protocol upgrade to 2026-07-28 revision | Todo | P1 | — | [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) | |
781786
| Windows native tray app `MCP-43` | Todo | P2 | — | | |
782787
| Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | |
783788
| Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | |
784-
| MCP protocol upgrade to 2026-07-28 revision | Todo | P3 | — | [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) | |
785789
| Security gateway Tracks C/D (per-arg least-privilege + signature provenance) | Todo | P3 | — | [054-mcp-security-gateway](./specs/054-mcp-security-gateway/) | |
786790
| Server marketplace `MCP-37` | Todo (parked) | P3 | — | | |
787791
| Audit SIEM integration `MCP-39` | Todo (parked) | P3 | — | | |

‎roadmap.yaml‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,13 @@ epics:
253253
status: todo
254254
note: "No spec: link on purpose — 073-activity-size-retention is the SHIPPED retention backend (13/14); this task is the glance-view tie-in on top of it."
255255
depends_on: [action-log-glance-view]
256+
- id: activity-storage-bounds
257+
title: "Bound every activity-adjacent store: response truncation on the write path (#1173/#1174), per-server tool_calls buckets (#1176), omitempty zero-erasure (#1175)"
258+
status: in_progress
259+
priority: P1
260+
pr: "#1174"
261+
note: "User report 2026-09-02 (#1173): config.db reached 940MB; activity_max_response_size existed but was never read, and server_<id>_tool_calls buckets have no size, count or age bound at all (the other ~432MB). #1174 (external contributor, CI green) wires the 64KB cap; #1176 and #1175 are the follow-ons. Retention backend (073) prunes activity records only, not the per-server buckets."
262+
depends_on: []
256263

257264
- id: analytics-dashboard
258265
title: Analytics dashboard as default page
@@ -392,14 +399,14 @@ epics:
392399
status: in_progress
393400
priority: P1
394401
spec: specs/103-token-bench
395-
pr: "#1141"
396402
depends_on: [schema-deferred]
397403
note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product."
398404
tasks:
399405
- id: token-bench-harness
400406
title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"
401-
status: in_progress
402-
pr: "#1141"
407+
status: done
408+
pr: ["#1141", "#1147", "#1151", "#1153", "#1160"]
409+
note: "Shipped 2026-08-31..09-01: #1141 replay MVP, #1147 payload decomposition (US4), #1151 tools/list primitive + live fleet source, #1153 live agent loop + reproducibility + publication, #1160 code-execution response-side saving. Epic stays in_progress: public suites, blog post and heartbeat-v10 counters remain todo."
403410
depends_on: []
404411
note: "BUILDS ON the existing bench/ harness - do NOT rebuild token counting, encoding arms, break-even or recall. Already shipped there (PRs #747/#748 MCP-42, #851 spec 083): six encoding arms including direct_deferred (spec 102), cl100k_base counting, live response cost, break-even, RecallAtK/NDCGAtK, the v2 report + dashboard, `make bench-discovery` and CI wiring in bench.yml/eval.yml. 2026-08-31 audit correction: the old note claimed this task absorbs discovery-eval-harness because it would measure retrieval recall - recall is ALREADY measured (eval.yml retrieval-d1 + bench/armindex.go), so that is not this task's contribution. GENUINELY OPEN and unique here: (a) activity-log session REPLAY (nothing in bench/ reads the activity log; internal/contracts/activity.go already carries WorkSessionID, ToolName and Response - watch ResponseTruncated, which would undercount tokens); (b) MEASURED first-call success and retry rate — today they are assumed, not observed: bench/session.go defines per-arm literature-derived defaults (armRetryRates, research D8) and bench/report.go only renders and labels them as estimates; (c) tokens-per-SUCCESSFUL-task, which needs a task-completion signal the static harness has no concept of."
405412
- id: token-bench-public
@@ -715,10 +722,10 @@ epics:
715722
- id: mcp-2026-upgrade
716723
title: MCP protocol upgrade to 2026-07-28 revision
717724
status: todo
718-
priority: P3
725+
priority: P1
719726
spec: specs/058-mcp-2026-upgrade
720727
depends_on: []
721-
note: "UNBLOCKED 2026-08-12: the mcp-go gate cleared — v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that — implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/<slug> — Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532."
728+
note: "STABLE GATE CLEARED 2026-09-02: mark3labs/mcp-go v1.0.0 (stable) released; go.mod still pins v0.57.0. Raised P3->P1 on the 2026-09-02 issue-prioritization pass — spec 058 has spec.md only, next step is speckit.plan then tasks (tracker #532). Earlier: UNBLOCKED 2026-08-12: the mcp-go gate cleared — v1.0.0-beta.1 (mark3labs/mcp-go#951) ships full 2026-07-28 support with per-request era detection (the pin was v0.55.x, topping out at 2025-11-25). Spec 058 revision MERGED as PR #1033 on 2026-08-27 (kept in this note, not in pr:, because pr: is implementation evidence and a docs(specs) merge is not that — implementation has not started, hence todo): final error-code renumbering (-32020/-32021/-32022), FR-001..006 / FR-014..016 recast as adopt-and-verify, FR-028 legacy-only transport pin as the safe merge state, plus Risks & Watch Items. CROSS-SPEC CONFLICT still open and now the mandatory first task: FR-012 forbids per-connection */list variation; SHIPPED Spec 057 selects toolset by URL path /mcp/p/<slug> — Option A/B decided at plan time (acceptance = the 2 tests failing under beta.1). 028 agent-token scoping is already compatible (header-carried). Tracker: #532."
722729
- id: security-gateway-cd
723730
title: Security gateway Tracks C/D (per-arg least-privilege + signature provenance)
724731
status: todo

0 commit comments

Comments
 (0)