From d4fa1c1375f69db78fa5b412d83804dea1dd8ba2 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Tue, 25 Aug 2026 09:24:25 +0300 Subject: [PATCH] chore(roadmap): add token-bench and tool-graph epics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two new backlog epics: - token-bench (P1): measure real token cost across every routing/savings mode combo (baseline, compact 085, deferred schemas 102, optimistic calling, code_execution 096, stored scripts 097) on replayed sessions and public suites, then publish the numbers on mcpproxy.app/blog. Absorbs the standalone discovery-eval-harness epic, which is now marked SUPERSEDED and points at token-bench-harness. - tool-graph (P2): experimental, flag-gated local co-occurrence graph mined from the activity log — next-tool hints, session-aware ranking, workflow mining and usage-chain analytics. Stacks on #1041 (chore/roadmap-truth-sync); token-bench depends on the schema-deferred epic added there. Merge after #1041. ROADMAP.md regenerated; gen-roadmap.py --check passes. --- ROADMAP.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++----- roadmap.yaml | 53 ++++++++++++++++++++++++++++++++++-- 2 files changed, 121 insertions(+), 8 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 4af85d579..dce01646b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -43,8 +43,12 @@ graph LR scanner_simplification["Scanner simplification (deterministic d…"] tpa_db["tpa-db: versioned TPA signature databas…"] remote_access_tunnel["Remote access tunnel (feature-flagged M…"] + schema_deferred["Deferred-schema serialization for the d…"] + token_bench["Token-efficiency benchmark: measured sa…"] + tool_graph["Tool co-occurrence graph (experimental,…"] telemetry_identity["Telemetry identity & data quality (mach…"] telemetry_v7_churn["Telemetry v7: honest funnel + churn ins…"] + discovery_eval_harness["Discovery-quality eval harness (Spec 06…"] scanner_v2 --> sandbox_isolation ux_audit --> action_log_transparency @@ -54,22 +58,24 @@ graph LR tpa_db --> remote_access_tunnel ux_audit --> remote_access_tunnel analytics_dashboard --> remote_access_tunnel + schema_deferred --> token_bench + analytics_dashboard --> tool_graph telemetry_identity --> telemetry_v7_churn + token_bench --> discovery_eval_harness classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff; classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff; classDef in_review fill:#9a6700,stroke:#5c3d00,color:#ffffff; classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; class sandbox_isolation,scanner_v2,scanner_simplification done; - class analytics_dashboard,telemetry_identity,telemetry_v7_churn in_progress; + class analytics_dashboard,schema_deferred,telemetry_identity,telemetry_v7_churn in_progress; class tpa_db in_review; - class ux_audit,action_log_transparency,remote_access_tunnel todo; + class ux_audit,action_log_transparency,remote_access_tunnel,token_bench,tool_graph,discovery_eval_harness todo; ``` -**Independent epics** (16) — no cross-epic prerequisites; each stands alone: +**Independent epics** (14) — no cross-epic prerequisites; each stands alone: - 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0 -- 🔵 **Deferred-schema serialization for the direct tools/list surface (spec 102)** — In progress · P1 - 🔵 **MCP protocol upgrade to 2026-07-28 revision** — In progress · P3 - 🟡 **Windows native tray app** — In review · P2 - ⚪ **Planning/docs truth automation** — Todo · P2 @@ -79,7 +85,6 @@ graph LR - ⚫ **SDK v1 migration** — Todo · P3 · parked - ⚫ **SSO (server edition)** — Todo · P3 · parked - ⚪ **Security gateway Tracks C/D (per-arg least-privilege + signature provenance)** — Todo · P3 -- ⚪ **Discovery-quality eval harness (Spec 065 second half)** — Todo · P3 - 🟢 **Upgrade awareness & guided update** — Done · P0 - 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0 - 🟢 **Registries — easier search + add-server** — Done · P1 @@ -342,6 +347,34 @@ graph LR +
+⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1 + +> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. + +```mermaid +graph LR + token_bench_harness["Replay harness: activity-log sessions re-run…"] + token_bench_public["Run public suites locally (τ-bench / BFCL / M…"] + token_bench_blog["Publish results + methodology on mcpproxy.app…"] + token_bench_telemetry["Heartbeat v10: per-tool_response_mode token c…"] + + token_bench_harness --> token_bench_public + token_bench_public --> token_bench_blog + + classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; + class token_bench_harness,token_bench_public,token_bench_blog,token_bench_telemetry todo; +``` + +| Task | Status | Refs | +| --- | --- | --- | +| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | ⚪ Todo | — | +| Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results | ⚪ Todo | — | +| Publish results + methodology on mcpproxy.app/blog | ⚪ Todo | — | +| Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | ⚪ Todo | — | + +
+
⚪ Remote access tunnel (feature-flagged MVP, spec 089) — Todo · P2 @@ -373,6 +406,35 @@ graph LR
+
+⚪ Tool co-occurrence graph (experimental, feature-flagged) — Todo · P2 + +> Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine. + +```mermaid +graph LR + tool_graph_core["Co-occurrence graph from the activity log + r…"] + tool_graph_ranking["Session-aware rank boost in retrieve_tools"] + tool_graph_mining["Workflow mining: frequent chains → suggested…"] + tool_graph_analytics["Usage-chain analytics on the dashboard/stats…"] + + tool_graph_core --> tool_graph_ranking + tool_graph_core --> tool_graph_mining + tool_graph_core --> tool_graph_analytics + + classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff; + class tool_graph_core,tool_graph_ranking,tool_graph_mining,tool_graph_analytics todo; +``` + +| Task | Status | Refs | +| --- | --- | --- | +| Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated) | ⚪ Todo | — | +| Session-aware rank boost in retrieve_tools | ⚪ Todo | — | +| Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy) | ⚪ Todo | — | +| Usage-chain analytics on the dashboard/stats page | ⚪ Todo | — | + +
+
⚪ Planning/docs truth automation — Todo · P2 @@ -420,7 +482,7 @@ Spec: [054-mcp-security-gateway](./specs/054-mcp-security-gateway/)
⚪ Discovery-quality eval harness (Spec 065 second half) — Todo · P3 -> Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness. +> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness. Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/) @@ -686,7 +748,9 @@ graph LR | Windows native tray app `MCP-43` | In review | P2 | — | | | | Web UI + macOS app UX audit | Todo | P0 | — | | | | Action log / transparency — info at a glance | Todo | P1 | — | | | +| Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | | | Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | | +| Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | | | Planning/docs truth automation | Todo | P2 | — | | | | Security gateway Tracks C/D (per-arg least-privilege + signature provenance) | Todo | P3 | — | [054-mcp-security-gateway](./specs/054-mcp-security-gateway/) | | | Discovery-quality eval harness (Spec 065 second half) | Todo | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | diff --git a/roadmap.yaml b/roadmap.yaml index c5ec2a752..b6a66fb1e 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -349,6 +349,55 @@ epics: depends_on: [] note: "Direct mode enumerates every upstream tool but always ships full inputSchema (~30K tokens for a 100-tool fleet; Spec 083 profiling put ~77% of the payload in schemas agents rarely read). Deferred serialization keeps every tool name, description and annotation and appends the Spec 085 compact signature instead of the schema (~88% smaller listing), with describe_tool on the direct surface to recover it and the shipped pre-dispatch validation turning a wrong guess into one self-healing retry. Not a new routing_mode — a serialization mode of the direct surface, on the same tool_response_mode axis that already governs retrieve_tools. Spec merged in #1035 (issue #971, maintainer-accepted direction); plan in progress." + - id: token-bench + title: "Token-efficiency benchmark: measured savings, published results" + status: todo + priority: P1 + depends_on: [schema-deferred] + note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix." + tasks: + - id: token-bench-harness + title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries" + status: todo + depends_on: [] + note: "Absorbs discovery-eval-harness (spec 065 second half) — the same harness measures retrieval recall." + - id: token-bench-public + title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results" + status: todo + depends_on: [token-bench-harness] + - id: token-bench-blog + title: "Publish results + methodology on mcpproxy.app/blog" + status: todo + depends_on: [token-bench-public] + - id: token-bench-telemetry + title: "Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation" + status: todo + depends_on: [] + + - id: tool-graph + title: "Tool co-occurrence graph (experimental, feature-flagged)" + status: todo + priority: P2 + depends_on: [] + note: "Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine." + tasks: + - id: tool-graph-core + title: "Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated)" + status: todo + depends_on: [] + - id: tool-graph-ranking + title: "Session-aware rank boost in retrieve_tools" + status: todo + depends_on: [tool-graph-core] + - id: tool-graph-mining + title: "Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy)" + status: todo + depends_on: [tool-graph-core] + - id: tool-graph-analytics + title: "Usage-chain analytics on the dashboard/stats page" + status: todo + depends_on: [tool-graph-core, analytics-default-landing] + # ── TELEMETRY-DRIVEN REPLAN (2026-07-02) ────────────────────────────────── - id: upgrade-nudge title: Upgrade awareness & guided update @@ -633,5 +682,5 @@ epics: status: todo priority: P3 spec: specs/065-evaluation-foundation - depends_on: [] - note: "Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness." + depends_on: [token-bench-harness] + note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."