Skip to content

Commit 126cc05

Browse files
committed
chore(roadmap): add token-bench and tool-graph epics
Two new backlog epics: - token-bench (P1): measure real token cost across every routing/savings mode combo (baseline, compact 085, deferred schemas 102, optimistic calling, code_execution 096, stored scripts 097) on replayed sessions and public suites, then publish the numbers on mcpproxy.app/blog. Absorbs the standalone discovery-eval-harness epic, which is now marked SUPERSEDED and points at token-bench-harness. - tool-graph (P2): experimental, flag-gated local co-occurrence graph mined from the activity log — next-tool hints, session-aware ranking, workflow mining and usage-chain analytics. Stacks on #1041 (chore/roadmap-truth-sync); token-bench depends on the schema-deferred epic added there. Merge after #1041. ROADMAP.md regenerated; gen-roadmap.py --check passes.
1 parent 61f7781 commit 126cc05

2 files changed

Lines changed: 121 additions & 8 deletions

File tree

‎ROADMAP.md‎

Lines changed: 70 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,12 @@ graph LR
4343
scanner_simplification["Scanner simplification (deterministic d…"]
4444
tpa_db["tpa-db: versioned TPA signature databas…"]
4545
remote_access_tunnel["Remote access tunnel (feature-flagged M…"]
46+
schema_deferred["Deferred-schema serialization for the d…"]
47+
token_bench["Token-efficiency benchmark: measured sa…"]
48+
tool_graph["Tool co-occurrence graph (experimental,…"]
4649
telemetry_identity["Telemetry identity & data quality (mach…"]
4750
telemetry_v7_churn["Telemetry v7: honest funnel + churn ins…"]
51+
discovery_eval_harness["Discovery-quality eval harness (Spec 06…"]
4852
4953
scanner_v2 --> sandbox_isolation
5054
ux_audit --> action_log_transparency
@@ -54,22 +58,24 @@ graph LR
5458
tpa_db --> remote_access_tunnel
5559
ux_audit --> remote_access_tunnel
5660
analytics_dashboard --> remote_access_tunnel
61+
schema_deferred --> token_bench
62+
analytics_dashboard --> tool_graph
5763
telemetry_identity --> telemetry_v7_churn
64+
token_bench --> discovery_eval_harness
5865
5966
classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
6067
classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
6168
classDef in_review fill:#9a6700,stroke:#5c3d00,color:#ffffff;
6269
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
6370
class sandbox_isolation,scanner_v2,scanner_simplification done;
64-
class analytics_dashboard,telemetry_identity,telemetry_v7_churn in_progress;
71+
class analytics_dashboard,schema_deferred,telemetry_identity,telemetry_v7_churn in_progress;
6572
class tpa_db in_review;
66-
class ux_audit,action_log_transparency,remote_access_tunnel todo;
73+
class ux_audit,action_log_transparency,remote_access_tunnel,token_bench,tool_graph,discovery_eval_harness todo;
6774
```
6875

69-
**Independent epics** (16) — no cross-epic prerequisites; each stands alone:
76+
**Independent epics** (14) — no cross-epic prerequisites; each stands alone:
7077

7178
- 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0
72-
- 🔵 **Deferred-schema serialization for the direct tools/list surface (spec 102)** — In progress · P1
7379
- 🔵 **MCP protocol upgrade to 2026-07-28 revision** — In progress · P3
7480
- 🟡 **Windows native tray app** — In review · P2
7581
- ⚪ **Planning/docs truth automation** — Todo · P2
@@ -79,7 +85,6 @@ graph LR
7985
- ⚫ **SDK v1 migration** — Todo · P3 · parked
8086
- ⚫ **SSO (server edition)** — Todo · P3 · parked
8187
- ⚪ **Security gateway Tracks C/D (per-arg least-privilege + signature provenance)** — Todo · P3
82-
- ⚪ **Discovery-quality eval harness (Spec 065 second half)** — Todo · P3
8388
- 🟢 **Upgrade awareness & guided update** — Done · P0
8489
- 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0
8590
- 🟢 **Registries — easier search + add-server** — Done · P1
@@ -342,6 +347,34 @@ graph LR
342347

343348
</details>
344349

350+
<details>
351+
<summary>⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1</summary>
352+
353+
> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix.
354+
355+
```mermaid
356+
graph LR
357+
token_bench_harness["Replay harness: activity-log sessions re-run…"]
358+
token_bench_public["Run public suites locally (τ-bench / BFCL / M…"]
359+
token_bench_blog["Publish results + methodology on mcpproxy.app…"]
360+
token_bench_telemetry["Heartbeat v10: per-tool_response_mode token c…"]
361+
362+
token_bench_harness --> token_bench_public
363+
token_bench_public --> token_bench_blog
364+
365+
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
366+
class token_bench_harness,token_bench_public,token_bench_blog,token_bench_telemetry todo;
367+
```
368+
369+
| Task | Status | Refs |
370+
| --- | --- | --- |
371+
| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | ⚪ Todo | — |
372+
| Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results | ⚪ Todo | — |
373+
| Publish results + methodology on mcpproxy.app/blog | ⚪ Todo | — |
374+
| Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | ⚪ Todo | — |
375+
376+
</details>
377+
345378
<details>
346379
<summary>⚪ Remote access tunnel (feature-flagged MVP, spec 089) — Todo · P2</summary>
347380

@@ -373,6 +406,35 @@ graph LR
373406

374407
</details>
375408

409+
<details>
410+
<summary>⚪ Tool co-occurrence graph (experimental, feature-flagged) — Todo · P2</summary>
411+
412+
> Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine.
413+
414+
```mermaid
415+
graph LR
416+
tool_graph_core["Co-occurrence graph from the activity log + r…"]
417+
tool_graph_ranking["Session-aware rank boost in retrieve_tools"]
418+
tool_graph_mining["Workflow mining: frequent chains → suggested…"]
419+
tool_graph_analytics["Usage-chain analytics on the dashboard/stats…"]
420+
421+
tool_graph_core --> tool_graph_ranking
422+
tool_graph_core --> tool_graph_mining
423+
tool_graph_core --> tool_graph_analytics
424+
425+
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
426+
class tool_graph_core,tool_graph_ranking,tool_graph_mining,tool_graph_analytics todo;
427+
```
428+
429+
| Task | Status | Refs |
430+
| --- | --- | --- |
431+
| Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated) | ⚪ Todo | — |
432+
| Session-aware rank boost in retrieve_tools | ⚪ Todo | — |
433+
| Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy) | ⚪ Todo | — |
434+
| Usage-chain analytics on the dashboard/stats page | ⚪ Todo | — |
435+
436+
</details>
437+
376438
<details>
377439
<summary>⚪ Planning/docs truth automation — Todo · P2</summary>
378440

@@ -420,7 +482,7 @@ Spec: [054-mcp-security-gateway](./specs/054-mcp-security-gateway/)
420482
<details>
421483
<summary>⚪ Discovery-quality eval harness (Spec 065 second half) — Todo · P3</summary>
422484

423-
> Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness.
485+
> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness.
424486
425487
Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/)
426488

@@ -686,7 +748,9 @@ graph LR
686748
| Windows native tray app `MCP-43` | In review | P2 | — | | |
687749
| Web UI + macOS app UX audit | Todo | P0 | — | | |
688750
| Action log / transparency — info at a glance | Todo | P1 | — | | |
751+
| Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | |
689752
| Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | |
753+
| Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | |
690754
| Planning/docs truth automation | Todo | P2 | — | | |
691755
| Security gateway Tracks C/D (per-arg least-privilege + signature provenance) | Todo | P3 | — | [054-mcp-security-gateway](./specs/054-mcp-security-gateway/) | |
692756
| Discovery-quality eval harness (Spec 065 second half) | Todo | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | |

‎roadmap.yaml‎

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,55 @@ epics:
348348
depends_on: []
349349
note: "Direct mode enumerates every upstream tool but always ships full inputSchema (~30K tokens for a 100-tool fleet; Spec 083 profiling put ~77% of the payload in schemas agents rarely read). Deferred serialization keeps every tool name, description and annotation and appends the Spec 085 compact signature instead of the schema (~88% smaller listing), with describe_tool on the direct surface to recover it and the shipped pre-dispatch validation turning a wrong guess into one self-healing retry. Not a new routing_mode — a serialization mode of the direct surface, on the same tool_response_mode axis that already governs retrieve_tools. Spec merged in #1035 (issue #971, maintainer-accepted direction); plan in progress."
350350

351+
- id: token-bench
352+
title: "Token-efficiency benchmark: measured savings, published results"
353+
status: todo
354+
priority: P1
355+
depends_on: [schema-deferred]
356+
note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix."
357+
tasks:
358+
- id: token-bench-harness
359+
title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"
360+
status: todo
361+
depends_on: []
362+
note: "Absorbs discovery-eval-harness (spec 065 second half) — the same harness measures retrieval recall."
363+
- id: token-bench-public
364+
title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results"
365+
status: todo
366+
depends_on: [token-bench-harness]
367+
- id: token-bench-blog
368+
title: "Publish results + methodology on mcpproxy.app/blog"
369+
status: todo
370+
depends_on: [token-bench-public]
371+
- id: token-bench-telemetry
372+
title: "Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation"
373+
status: todo
374+
depends_on: []
375+
376+
- id: tool-graph
377+
title: "Tool co-occurrence graph (experimental, feature-flagged)"
378+
status: todo
379+
priority: P2
380+
depends_on: []
381+
note: "Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine."
382+
tasks:
383+
- id: tool-graph-core
384+
title: "Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated)"
385+
status: todo
386+
depends_on: []
387+
- id: tool-graph-ranking
388+
title: "Session-aware rank boost in retrieve_tools"
389+
status: todo
390+
depends_on: [tool-graph-core]
391+
- id: tool-graph-mining
392+
title: "Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy)"
393+
status: todo
394+
depends_on: [tool-graph-core]
395+
- id: tool-graph-analytics
396+
title: "Usage-chain analytics on the dashboard/stats page"
397+
status: todo
398+
depends_on: [tool-graph-core, analytics-default-landing]
399+
351400
# ── TELEMETRY-DRIVEN REPLAN (2026-07-02) ──────────────────────────────────
352401
- id: upgrade-nudge
353402
title: Upgrade awareness & guided update
@@ -632,5 +681,5 @@ epics:
632681
status: todo
633682
priority: P3
634683
spec: specs/065-evaluation-foundation
635-
depends_on: []
636-
note: "Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."
684+
depends_on: [token-bench-harness]
685+
note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."

0 commit comments

Comments
 (0)