diff --git a/ROADMAP.md b/ROADMAP.md
index 4af85d579..dce01646b 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -43,8 +43,12 @@ graph LR
scanner_simplification["Scanner simplification (deterministic d…"]
tpa_db["tpa-db: versioned TPA signature databas…"]
remote_access_tunnel["Remote access tunnel (feature-flagged M…"]
+ schema_deferred["Deferred-schema serialization for the d…"]
+ token_bench["Token-efficiency benchmark: measured sa…"]
+ tool_graph["Tool co-occurrence graph (experimental,…"]
telemetry_identity["Telemetry identity & data quality (mach…"]
telemetry_v7_churn["Telemetry v7: honest funnel + churn ins…"]
+ discovery_eval_harness["Discovery-quality eval harness (Spec 06…"]
scanner_v2 --> sandbox_isolation
ux_audit --> action_log_transparency
@@ -54,22 +58,24 @@ graph LR
tpa_db --> remote_access_tunnel
ux_audit --> remote_access_tunnel
analytics_dashboard --> remote_access_tunnel
+ schema_deferred --> token_bench
+ analytics_dashboard --> tool_graph
telemetry_identity --> telemetry_v7_churn
+ token_bench --> discovery_eval_harness
classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
classDef in_review fill:#9a6700,stroke:#5c3d00,color:#ffffff;
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
class sandbox_isolation,scanner_v2,scanner_simplification done;
- class analytics_dashboard,telemetry_identity,telemetry_v7_churn in_progress;
+ class analytics_dashboard,schema_deferred,telemetry_identity,telemetry_v7_churn in_progress;
class tpa_db in_review;
- class ux_audit,action_log_transparency,remote_access_tunnel todo;
+ class ux_audit,action_log_transparency,remote_access_tunnel,token_bench,tool_graph,discovery_eval_harness todo;
```
-**Independent epics** (16) — no cross-epic prerequisites; each stands alone:
+**Independent epics** (14) — no cross-epic prerequisites; each stands alone:
- 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0
-- 🔵 **Deferred-schema serialization for the direct tools/list surface (spec 102)** — In progress · P1
- 🔵 **MCP protocol upgrade to 2026-07-28 revision** — In progress · P3
- 🟡 **Windows native tray app** — In review · P2
- ⚪ **Planning/docs truth automation** — Todo · P2
@@ -79,7 +85,6 @@ graph LR
- ⚫ **SDK v1 migration** — Todo · P3 · parked
- ⚫ **SSO (server edition)** — Todo · P3 · parked
- ⚪ **Security gateway Tracks C/D (per-arg least-privilege + signature provenance)** — Todo · P3
-- ⚪ **Discovery-quality eval harness (Spec 065 second half)** — Todo · P3
- 🟢 **Upgrade awareness & guided update** — Done · P0
- 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0
- 🟢 **Registries — easier search + add-server** — Done · P1
@@ -342,6 +347,34 @@ graph LR
+
+⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1
+
+> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix.
+
+```mermaid
+graph LR
+ token_bench_harness["Replay harness: activity-log sessions re-run…"]
+ token_bench_public["Run public suites locally (τ-bench / BFCL / M…"]
+ token_bench_blog["Publish results + methodology on mcpproxy.app…"]
+ token_bench_telemetry["Heartbeat v10: per-tool_response_mode token c…"]
+
+ token_bench_harness --> token_bench_public
+ token_bench_public --> token_bench_blog
+
+ classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
+ class token_bench_harness,token_bench_public,token_bench_blog,token_bench_telemetry todo;
+```
+
+| Task | Status | Refs |
+| --- | --- | --- |
+| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | ⚪ Todo | — |
+| Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results | ⚪ Todo | — |
+| Publish results + methodology on mcpproxy.app/blog | ⚪ Todo | — |
+| Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | ⚪ Todo | — |
+
+
+
⚪ Remote access tunnel (feature-flagged MVP, spec 089) — Todo · P2
@@ -373,6 +406,35 @@ graph LR
+
+⚪ Tool co-occurrence graph (experimental, feature-flagged) — Todo · P2
+
+> Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine.
+
+```mermaid
+graph LR
+ tool_graph_core["Co-occurrence graph from the activity log + r…"]
+ tool_graph_ranking["Session-aware rank boost in retrieve_tools"]
+ tool_graph_mining["Workflow mining: frequent chains → suggested…"]
+ tool_graph_analytics["Usage-chain analytics on the dashboard/stats…"]
+
+ tool_graph_core --> tool_graph_ranking
+ tool_graph_core --> tool_graph_mining
+ tool_graph_core --> tool_graph_analytics
+
+ classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
+ class tool_graph_core,tool_graph_ranking,tool_graph_mining,tool_graph_analytics todo;
+```
+
+| Task | Status | Refs |
+| --- | --- | --- |
+| Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated) | ⚪ Todo | — |
+| Session-aware rank boost in retrieve_tools | ⚪ Todo | — |
+| Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy) | ⚪ Todo | — |
+| Usage-chain analytics on the dashboard/stats page | ⚪ Todo | — |
+
+
+
⚪ Planning/docs truth automation — Todo · P2
@@ -420,7 +482,7 @@ Spec: [054-mcp-security-gateway](./specs/054-mcp-security-gateway/)
⚪ Discovery-quality eval harness (Spec 065 second half) — Todo · P3
-> Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness.
+> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness.
Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/)
@@ -686,7 +748,9 @@ graph LR
| Windows native tray app `MCP-43` | In review | P2 | — | | |
| Web UI + macOS app UX audit | Todo | P0 | — | | |
| Action log / transparency — info at a glance | Todo | P1 | — | | |
+| Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | |
| Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | |
+| Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | |
| Planning/docs truth automation | Todo | P2 | — | | |
| Security gateway Tracks C/D (per-arg least-privilege + signature provenance) | Todo | P3 | — | [054-mcp-security-gateway](./specs/054-mcp-security-gateway/) | |
| Discovery-quality eval harness (Spec 065 second half) | Todo | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | |
diff --git a/roadmap.yaml b/roadmap.yaml
index c5ec2a752..b6a66fb1e 100644
--- a/roadmap.yaml
+++ b/roadmap.yaml
@@ -349,6 +349,55 @@ epics:
depends_on: []
note: "Direct mode enumerates every upstream tool but always ships full inputSchema (~30K tokens for a 100-tool fleet; Spec 083 profiling put ~77% of the payload in schemas agents rarely read). Deferred serialization keeps every tool name, description and annotation and appends the Spec 085 compact signature instead of the schema (~88% smaller listing), with describe_tool on the direct surface to recover it and the shipped pre-dispatch validation turning a wrong guess into one self-healing retry. Not a new routing_mode — a serialization mode of the direct surface, on the same tool_response_mode axis that already governs retrieve_tools. Spec merged in #1035 (issue #971, maintainer-accepted direction); plan in progress."
+ - id: token-bench
+ title: "Token-efficiency benchmark: measured savings, published results"
+ status: todo
+ priority: P1
+ depends_on: [schema-deferred]
+ note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix."
+ tasks:
+ - id: token-bench-harness
+ title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"
+ status: todo
+ depends_on: []
+ note: "Absorbs discovery-eval-harness (spec 065 second half) — the same harness measures retrieval recall."
+ - id: token-bench-public
+ title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results"
+ status: todo
+ depends_on: [token-bench-harness]
+ - id: token-bench-blog
+ title: "Publish results + methodology on mcpproxy.app/blog"
+ status: todo
+ depends_on: [token-bench-public]
+ - id: token-bench-telemetry
+ title: "Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation"
+ status: todo
+ depends_on: []
+
+ - id: tool-graph
+ title: "Tool co-occurrence graph (experimental, feature-flagged)"
+ status: todo
+ priority: P2
+ depends_on: []
+ note: "Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine."
+ tasks:
+ - id: tool-graph-core
+ title: "Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated)"
+ status: todo
+ depends_on: []
+ - id: tool-graph-ranking
+ title: "Session-aware rank boost in retrieve_tools"
+ status: todo
+ depends_on: [tool-graph-core]
+ - id: tool-graph-mining
+ title: "Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy)"
+ status: todo
+ depends_on: [tool-graph-core]
+ - id: tool-graph-analytics
+ title: "Usage-chain analytics on the dashboard/stats page"
+ status: todo
+ depends_on: [tool-graph-core, analytics-default-landing]
+
# ── TELEMETRY-DRIVEN REPLAN (2026-07-02) ──────────────────────────────────
- id: upgrade-nudge
title: Upgrade awareness & guided update
@@ -633,5 +682,5 @@ epics:
status: todo
priority: P3
spec: specs/065-evaluation-foundation
- depends_on: []
- note: "Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."
+ depends_on: [token-bench-harness]
+ note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."