diff --git a/CLAUDE.md b/CLAUDE.md index ad8c7e669..5d47a3dd4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,7 +18,7 @@ Only halt and ask a human IF: 1. You need destructive data operations or to delete core proxy logic that cannot be mocked. 2. A required environment variable is missing from `.env` and cannot be mocked for the task's scope. 3. You are stuck in an error loop for the same `go test` failing after 5 consecutive attempts. -4. **Cross-model (Codex) review round cap — per PR:** when a PR is gated by a cross-model Codex review, run at most **5 fix→re-review rounds on that PR**. If Codex has not returned a clean verdict after the 5th round, STOP and ask the human how to proceed (do not auto-run round 6). The counter is per-PR and resets for each new PR. (Verify each Codex finding is genuine before fixing — Codex can false-positive; a round only counts when you push a fix and re-review.) +4. **Cross-model review round cap — per PR:** when a PR is gated by a cross-model review, run at most **10 fix→re-review rounds on that PR**. If the reviewer has not returned a clean verdict after the 10th round, STOP and ask the human how to proceed (do not auto-run round 11). The counter is per-PR and resets for each new PR. (Verify each finding is genuine before fixing — reviewers do false-positive; a round only counts when you push a fix and re-review.) **The reviewer is the `opencode` CLI (`--model github-copilot/gpt-5.6-sol`), not Codex** — maintainer directive 2026-08-14. The cap was raised from 5 to 10 on 2026-08-31 because round 5 on #1136 caught a real defect the first four missed. ## Project Overview diff --git a/ROADMAP.md b/ROADMAP.md index c4f4d82fa..3a15f6fd3 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -352,7 +352,9 @@ graph LR
⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1 -> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. +> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product. + +Spec: [103-token-bench](./specs/103-token-bench/) ```mermaid graph LR @@ -773,7 +775,7 @@ graph LR | Planning/docs truth automation | In progress | P2 | — | | | | Discovery-quality eval harness (Spec 065 second half) | In progress | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | | | tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | — | [101-tpa-db](./specs/101-tpa-db/) | | -| Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | | +| Token-efficiency benchmark: measured savings, published results | Todo | P1 | 0/64 (0%) | [103-token-bench](./specs/103-token-bench/) | | | Windows native tray app `MCP-43` | Todo | P2 | — | | | | Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | | | Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | | diff --git a/roadmap.yaml b/roadmap.yaml index 71ddc677e..182477ccf 100644 --- a/roadmap.yaml +++ b/roadmap.yaml @@ -391,8 +391,9 @@ epics: title: "Token-efficiency benchmark: measured savings, published results" status: todo priority: P1 + spec: specs/103-token-bench depends_on: [schema-deferred] - note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix." + note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product." tasks: - id: token-bench-harness title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"