Skip to content

Commit d4fa1c1

Browse files
committed
chore(roadmap): add token-bench and tool-graph epics
Two new backlog epics: - token-bench (P1): measure real token cost across every routing/savings mode combo (baseline, compact 085, deferred schemas 102, optimistic calling, code_execution 096, stored scripts 097) on replayed sessions and public suites, then publish the numbers on mcpproxy.app/blog. Absorbs the standalone discovery-eval-harness epic, which is now marked SUPERSEDED and points at token-bench-harness. - tool-graph (P2): experimental, flag-gated local co-occurrence graph mined from the activity log — next-tool hints, session-aware ranking, workflow mining and usage-chain analytics. Stacks on #1041 (chore/roadmap-truth-sync); token-bench depends on the schema-deferred epic added there. Merge after #1041. ROADMAP.md regenerated; gen-roadmap.py --check passes.
1 parent 48892c0 commit d4fa1c1

2 files changed

Lines changed: 121 additions & 8 deletions

File tree

‎ROADMAP.md‎

Lines changed: 70 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,12 @@ graph LR
4343
scanner_simplification["Scanner simplification (deterministic d…"]
4444
tpa_db["tpa-db: versioned TPA signature databas…"]
4545
remote_access_tunnel["Remote access tunnel (feature-flagged M…"]
46+
schema_deferred["Deferred-schema serialization for the d…"]
47+
token_bench["Token-efficiency benchmark: measured sa…"]
48+
tool_graph["Tool co-occurrence graph (experimental,…"]
4649
telemetry_identity["Telemetry identity & data quality (mach…"]
4750
telemetry_v7_churn["Telemetry v7: honest funnel + churn ins…"]
51+
discovery_eval_harness["Discovery-quality eval harness (Spec 06…"]
4852
4953
scanner_v2 --> sandbox_isolation
5054
ux_audit --> action_log_transparency
@@ -54,22 +58,24 @@ graph LR
5458
tpa_db --> remote_access_tunnel
5559
ux_audit --> remote_access_tunnel
5660
analytics_dashboard --> remote_access_tunnel
61+
schema_deferred --> token_bench
62+
analytics_dashboard --> tool_graph
5763
telemetry_identity --> telemetry_v7_churn
64+
token_bench --> discovery_eval_harness
5865
5966
classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
6067
classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
6168
classDef in_review fill:#9a6700,stroke:#5c3d00,color:#ffffff;
6269
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
6370
class sandbox_isolation,scanner_v2,scanner_simplification done;
64-
class analytics_dashboard,telemetry_identity,telemetry_v7_churn in_progress;
71+
class analytics_dashboard,schema_deferred,telemetry_identity,telemetry_v7_churn in_progress;
6572
class tpa_db in_review;
66-
class ux_audit,action_log_transparency,remote_access_tunnel todo;
73+
class ux_audit,action_log_transparency,remote_access_tunnel,token_bench,tool_graph,discovery_eval_harness todo;
6774
```
6875

69-
**Independent epics** (16) — no cross-epic prerequisites; each stands alone:
76+
**Independent epics** (14) — no cross-epic prerequisites; each stands alone:
7077

7178
- 🔵 **Release qualification gate (auto-QA matrix blocks the tag)** — In progress · P0
72-
- 🔵 **Deferred-schema serialization for the direct tools/list surface (spec 102)** — In progress · P1
7379
- 🔵 **MCP protocol upgrade to 2026-07-28 revision** — In progress · P3
7480
- 🟡 **Windows native tray app** — In review · P2
7581
- ⚪ **Planning/docs truth automation** — Todo · P2
@@ -79,7 +85,6 @@ graph LR
7985
- ⚫ **SDK v1 migration** — Todo · P3 · parked
8086
- ⚫ **SSO (server edition)** — Todo · P3 · parked
8187
- ⚪ **Security gateway Tracks C/D (per-arg least-privilege + signature provenance)** — Todo · P3
82-
- ⚪ **Discovery-quality eval harness (Spec 065 second half)** — Todo · P3
8388
- 🟢 **Upgrade awareness & guided update** — Done · P0
8489
- 🟢 **Connect step trust: preview, visible backup, one-click undo** — Done · P0
8590
- 🟢 **Registries — easier search + add-server** — Done · P1
@@ -342,6 +347,34 @@ graph LR
342347

343348
</details>
344349

350+
<details>
351+
<summary>⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1</summary>
352+
353+
> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix.
354+
355+
```mermaid
356+
graph LR
357+
token_bench_harness["Replay harness: activity-log sessions re-run…"]
358+
token_bench_public["Run public suites locally (τ-bench / BFCL / M…"]
359+
token_bench_blog["Publish results + methodology on mcpproxy.app…"]
360+
token_bench_telemetry["Heartbeat v10: per-tool_response_mode token c…"]
361+
362+
token_bench_harness --> token_bench_public
363+
token_bench_public --> token_bench_blog
364+
365+
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
366+
class token_bench_harness,token_bench_public,token_bench_blog,token_bench_telemetry todo;
367+
```
368+
369+
| Task | Status | Refs |
370+
| --- | --- | --- |
371+
| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | ⚪ Todo | — |
372+
| Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results | ⚪ Todo | — |
373+
| Publish results + methodology on mcpproxy.app/blog | ⚪ Todo | — |
374+
| Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | ⚪ Todo | — |
375+
376+
</details>
377+
345378
<details>
346379
<summary>⚪ Remote access tunnel (feature-flagged MVP, spec 089) — Todo · P2</summary>
347380

@@ -373,6 +406,35 @@ graph LR
373406

374407
</details>
375408

409+
<details>
410+
<summary>⚪ Tool co-occurrence graph (experimental, feature-flagged) — Todo · P2</summary>
411+
412+
> Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine.
413+
414+
```mermaid
415+
graph LR
416+
tool_graph_core["Co-occurrence graph from the activity log + r…"]
417+
tool_graph_ranking["Session-aware rank boost in retrieve_tools"]
418+
tool_graph_mining["Workflow mining: frequent chains → suggested…"]
419+
tool_graph_analytics["Usage-chain analytics on the dashboard/stats…"]
420+
421+
tool_graph_core --> tool_graph_ranking
422+
tool_graph_core --> tool_graph_mining
423+
tool_graph_core --> tool_graph_analytics
424+
425+
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
426+
class tool_graph_core,tool_graph_ranking,tool_graph_mining,tool_graph_analytics todo;
427+
```
428+
429+
| Task | Status | Refs |
430+
| --- | --- | --- |
431+
| Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated) | ⚪ Todo | — |
432+
| Session-aware rank boost in retrieve_tools | ⚪ Todo | — |
433+
| Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy) | ⚪ Todo | — |
434+
| Usage-chain analytics on the dashboard/stats page | ⚪ Todo | — |
435+
436+
</details>
437+
376438
<details>
377439
<summary>⚪ Planning/docs truth automation — Todo · P2</summary>
378440

@@ -420,7 +482,7 @@ Spec: [054-mcp-security-gateway](./specs/054-mcp-security-gateway/)
420482
<details>
421483
<summary>⚪ Discovery-quality eval harness (Spec 065 second half) — Todo · P3</summary>
422484

423-
> Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness.
485+
> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness.
424486
425487
Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/)
426488

@@ -686,7 +748,9 @@ graph LR
686748
| Windows native tray app `MCP-43` | In review | P2 | — | | |
687749
| Web UI + macOS app UX audit | Todo | P0 | — | | |
688750
| Action log / transparency — info at a glance | Todo | P1 | — | | |
751+
| Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | |
689752
| Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | |
753+
| Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | |
690754
| Planning/docs truth automation | Todo | P2 | — | | |
691755
| Security gateway Tracks C/D (per-arg least-privilege + signature provenance) | Todo | P3 | — | [054-mcp-security-gateway](./specs/054-mcp-security-gateway/) | |
692756
| Discovery-quality eval harness (Spec 065 second half) | Todo | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | |

‎roadmap.yaml‎

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -349,6 +349,55 @@ epics:
349349
depends_on: []
350350
note: "Direct mode enumerates every upstream tool but always ships full inputSchema (~30K tokens for a 100-tool fleet; Spec 083 profiling put ~77% of the payload in schemas agents rarely read). Deferred serialization keeps every tool name, description and annotation and appends the Spec 085 compact signature instead of the schema (~88% smaller listing), with describe_tool on the direct surface to recover it and the shipped pre-dispatch validation turning a wrong guess into one self-healing retry. Not a new routing_mode — a serialization mode of the direct surface, on the same tool_response_mode axis that already governs retrieve_tools. Spec merged in #1035 (issue #971, maintainer-accepted direction); plan in progress."
351351

352+
- id: token-bench
353+
title: "Token-efficiency benchmark: measured savings, published results"
354+
status: todo
355+
priority: P1
356+
depends_on: [schema-deferred]
357+
note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix."
358+
tasks:
359+
- id: token-bench-harness
360+
title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"
361+
status: todo
362+
depends_on: []
363+
note: "Absorbs discovery-eval-harness (spec 065 second half) — the same harness measures retrieval recall."
364+
- id: token-bench-public
365+
title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results"
366+
status: todo
367+
depends_on: [token-bench-harness]
368+
- id: token-bench-blog
369+
title: "Publish results + methodology on mcpproxy.app/blog"
370+
status: todo
371+
depends_on: [token-bench-public]
372+
- id: token-bench-telemetry
373+
title: "Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation"
374+
status: todo
375+
depends_on: []
376+
377+
- id: tool-graph
378+
title: "Tool co-occurrence graph (experimental, feature-flagged)"
379+
status: todo
380+
priority: P2
381+
depends_on: []
382+
note: "Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine."
383+
tasks:
384+
- id: tool-graph-core
385+
title: "Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated)"
386+
status: todo
387+
depends_on: []
388+
- id: tool-graph-ranking
389+
title: "Session-aware rank boost in retrieve_tools"
390+
status: todo
391+
depends_on: [tool-graph-core]
392+
- id: tool-graph-mining
393+
title: "Workflow mining: frequent chains → suggested stored scripts (spec 097 synergy)"
394+
status: todo
395+
depends_on: [tool-graph-core]
396+
- id: tool-graph-analytics
397+
title: "Usage-chain analytics on the dashboard/stats page"
398+
status: todo
399+
depends_on: [tool-graph-core, analytics-default-landing]
400+
352401
# ── TELEMETRY-DRIVEN REPLAN (2026-07-02) ──────────────────────────────────
353402
- id: upgrade-nudge
354403
title: Upgrade awareness & guided update
@@ -633,5 +682,5 @@ epics:
633682
status: todo
634683
priority: P3
635684
spec: specs/065-evaluation-foundation
636-
depends_on: []
637-
note: "Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."
685+
depends_on: [token-bench-harness]
686+
note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."

0 commit comments

Comments
 (0)