You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d4fa1c1
Browse filesBrowse the repository at this point in the historyBrowse files
chore(roadmap): add token-bench and tool-graph epics
Two new backlog epics:
- token-bench (P1): measure real token cost across every routing/savings
mode combo (baseline, compact 085, deferred schemas 102, optimistic
calling, code_execution 096, stored scripts 097) on replayed sessions
and public suites, then publish the numbers on mcpproxy.app/blog.
Absorbs the standalone discovery-eval-harness epic, which is now marked
SUPERSEDED and points at token-bench-harness.
- tool-graph (P2): experimental, flag-gated local co-occurrence graph
mined from the activity log — next-tool hints, session-aware ranking,
workflow mining and usage-chain analytics.
Stacks on #1041 (chore/roadmap-truth-sync); token-bench depends on the
schema-deferred epic added there. Merge after #1041.
ROADMAP.md regenerated; gen-roadmap.py --check passes.
<summary>⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1</summary>
352
+
353
+
> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix.
<summary>⚪ Tool co-occurrence graph (experimental, feature-flagged) — Todo · P2</summary>
411
+
412
+
> Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine.
413
+
414
+
```mermaid
415
+
graph LR
416
+
tool_graph_core["Co-occurrence graph from the activity log + r…"]
417
+
tool_graph_ranking["Session-aware rank boost in retrieve_tools"]
> SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness.
Copy file name to clipboardExpand all lines: roadmap.yaml
+51-2Lines changed: 51 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -349,6 +349,55 @@ epics:
349
349
depends_on: []
350
350
note: "Direct mode enumerates every upstream tool but always ships full inputSchema (~30K tokens for a 100-tool fleet; Spec 083 profiling put ~77% of the payload in schemas agents rarely read). Deferred serialization keeps every tool name, description and annotation and appends the Spec 085 compact signature instead of the schema (~88% smaller listing), with describe_tool on the direct surface to recover it and the shipped pre-dispatch validation turning a wrong guess into one self-healing retry. Not a new routing_mode — a serialization mode of the direct surface, on the same tool_response_mode axis that already governs retrieve_tools. Spec merged in #1035 (issue #971, maintainer-accepted direction); plan in progress."
351
351
352
+
- id: token-bench
353
+
title: "Token-efficiency benchmark: measured savings, published results"
354
+
status: todo
355
+
priority: P1
356
+
depends_on: [schema-deferred]
357
+
note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix."
358
+
tasks:
359
+
- id: token-bench-harness
360
+
title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"
361
+
status: todo
362
+
depends_on: []
363
+
note: "Absorbs discovery-eval-harness (spec 065 second half) — the same harness measures retrieval recall."
364
+
- id: token-bench-public
365
+
title: "Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results"
366
+
status: todo
367
+
depends_on: [token-bench-harness]
368
+
- id: token-bench-blog
369
+
title: "Publish results + methodology on mcpproxy.app/blog"
370
+
status: todo
371
+
depends_on: [token-bench-public]
372
+
- id: token-bench-telemetry
373
+
title: "Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation"
note: "Local-only co-occurrence graph mined from the activity log: suggests likely-next tools to agents and surfaces usage-chain analytics. Everything sits behind experimental.tool_graph, off by default — nothing leaves the machine."
383
+
tasks:
384
+
- id: tool-graph-core
385
+
title: "Co-occurrence graph from the activity log + related_tools hint in call_tool responses (flag-gated)"
386
+
status: todo
387
+
depends_on: []
388
+
- id: tool-graph-ranking
389
+
title: "Session-aware rank boost in retrieve_tools"
note: "SUPERSEDED — folded into token-bench-harness: the token-efficiency replay harness re-runs real sessions under every mode combo and measures retrieval recall on the same corpus, so a separate discovery-quality harness would duplicate it. Kept here as a pointer (the id is a stable depends_on target); do not build standalone. Security recall/FP half SHIPPED (cmd/scan-eval, backs Spec 076/077 gate). UNBUILT: the discovery-quality (retrieve_tools recall) eval harness."
0 commit comments