From 4ce5761f756e1e90e790f6d1d7f6898bdc32545d Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 13:41:02 +0300 Subject: [PATCH] chore: link spec 103 from the token-bench epic; raise the review cap to 10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two follow-ups from the 2026-08-31 roadmap truth-sync and spec-103 work. 1. roadmap.yaml: the token-bench epic now carries spec: specs/103-token-bench, so ROADMAP.md renders its real progress badge (0/64) instead of a blank. The note records what the spec settled, because three of its 13 cross-model review rounds changed the DESIGN rather than the wording and a future reader of the epic should not have to rediscover them: - a recording carries no prompt, conversation or completion oracle, so replay CANNOT show agent behaviour — US1 (deterministic cost) and US2 (live agent loop) are now separate user stories; - replay needs a FLEET INPUT, because the activity export has no fleet snapshot and a menu is a property of the tool definitions; - bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. Also records that the matrix is 5 distinct behaviours, not a 3x2x2 product. 2. CLAUDE.md escalation trigger 4 was doubly stale: it capped cross-model review at 5 rounds and named Codex as the reviewer. The cap is 10 (maintainer decision 2026-08-31) and the reviewer is the opencode CLI with github-copilot/gpt-5.6-sol (maintainer directive 2026-08-14, Codex not to be used). The rationale for the raise is recorded inline: round 5 on #1136 caught a real defect the first four rounds missed — an epic marked done while its spec's gating FR was unmet, which is the exact failure that PR existed to fix. gen-roadmap.py --check and --check-github both clean. --- CLAUDE.md | 2 +- ROADMAP.md | 6 ++++-- roadmap.yaml | 3 ++- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ad8c7e669..5d47a3dd4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,7 +18,7 @@ Only halt and ask a human IF: 1. You need destructive data operations or to delete core proxy logic that cannot be mocked. 2. A required environment variable is missing from `.env` and cannot be mocked for the task's scope. 3. You are stuck in an error loop for the same `go test` failing after 5 consecutive attempts. -4. **Cross-model (Codex) review round cap — per PR:** when a PR is gated by a cross-model Codex review, run at most **5 fix→re-review rounds on that PR**. If Codex has not returned a clean verdict after the 5th round, STOP and ask the human how to proceed (do not auto-run round 6). The counter is per-PR and resets for each new PR. (Verify each Codex finding is genuine before fixing — Codex can false-positive; a round only counts when you push a fix and re-review.) +4. **Cross-model review round cap — per PR:** when a PR is gated by a cross-model review, run at most **10 fix→re-review rounds on that PR**. If the reviewer has not returned a clean verdict after the 10th round, STOP and ask the human how to proceed (do not auto-run round 11). The counter is per-PR and resets for each new PR. (Verify each finding is genuine before fixing — reviewers do false-positive; a round only counts when you push a fix and re-review.) **The reviewer is the `opencode` CLI (`--model github-copilot/gpt-5.6-sol`), not Codex** — maintainer directive 2026-08-14. The cap was raised from 5 to 10 on 2026-08-31 because round 5 on #1136 caught a real defect the first four missed. ## Project Overview diff --git a/ROADMAP.md b/ROADMAP.md index c4f4d82fa..3a15f6fd3 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -352,7 +352,9 @@ graph LR
⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1 -> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. +> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product. + +Spec: [103-token-bench](./specs/103-token-bench/) ```mermaid graph LR @@ -773,7 +775,7 @@ graph LR | Planning/docs truth automation | In progress | P2 | — | | | | Discovery-quality eval harness (Spec 065 second half) | In progress | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | | tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | — | [101-tpa-db](./specs/101-tpa-db/) | | -| Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | | +| Token-efficiency benchmark: measured savings, published results | Todo | P1 | 0/64 (0%) | [103-token-bench](./specs/103-token-bench/) | | | Windows native tray app `MCP-43` | Todo | P2 | — | | | | Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | | | Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | | diff --git a/roadmap.yaml b/roadmap.yaml index 71ddc677e..182477ccf 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -391,8 +391,9 @@ epics: title: "Token-efficiency benchmark: measured savings, published results" status: todo priority: P1 + spec: specs/103-token-bench depends_on: [schema-deferred] - note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix." + note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product." tasks: - id: token-bench-harness title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"