Skip to content

Commit 0750fcd

Browse files
authored
chore: link spec 103 from the token-bench epic; raise the review cap to 10 (#1140)
Two follow-ups from the 2026-08-31 roadmap truth-sync and spec-103 work. 1. roadmap.yaml: the token-bench epic now carries spec: specs/103-token-bench, so ROADMAP.md renders its real progress badge (0/64) instead of a blank. The note records what the spec settled, because three of its 13 cross-model review rounds changed the DESIGN rather than the wording and a future reader of the epic should not have to rediscover them: - a recording carries no prompt, conversation or completion oracle, so replay CANNOT show agent behaviour — US1 (deterministic cost) and US2 (live agent loop) are now separate user stories; - replay needs a FLEET INPUT, because the activity export has no fleet snapshot and a menu is a property of the tool definitions; - bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. Also records that the matrix is 5 distinct behaviours, not a 3x2x2 product. 2. CLAUDE.md escalation trigger 4 was doubly stale: it capped cross-model review at 5 rounds and named Codex as the reviewer. The cap is 10 (maintainer decision 2026-08-31) and the reviewer is the opencode CLI with github-copilot/gpt-5.6-sol (maintainer directive 2026-08-14, Codex not to be used). The rationale for the raise is recorded inline: round 5 on #1136 caught a real defect the first four rounds missed — an epic marked done while its spec's gating FR was unmet, which is the exact failure that PR existed to fix. gen-roadmap.py --check and --check-github both clean.
1 parent 1041e4e commit 0750fcd

3 files changed

Lines changed: 7 additions & 4 deletions

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Only halt and ask a human IF:
1818
1. You need destructive data operations or to delete core proxy logic that cannot be mocked.
1919
2. A required environment variable is missing from `.env` and cannot be mocked for the task's scope.
2020
3. You are stuck in an error loop for the same `go test` failing after 5 consecutive attempts.
21-
4. **Cross-model (Codex) review round cap — per PR:** when a PR is gated by a cross-model Codex review, run at most **5 fix→re-review rounds on that PR**. If Codex has not returned a clean verdict after the 5th round, STOP and ask the human how to proceed (do not auto-run round 6). The counter is per-PR and resets for each new PR. (Verify each Codex finding is genuine before fixing — Codex can false-positive; a round only counts when you push a fix and re-review.)
21+
4. **Cross-model review round cap — per PR:** when a PR is gated by a cross-model review, run at most **10 fix→re-review rounds on that PR**. If the reviewer has not returned a clean verdict after the 10th round, STOP and ask the human how to proceed (do not auto-run round 11). The counter is per-PR and resets for each new PR. (Verify each finding is genuine before fixing — reviewers do false-positive; a round only counts when you push a fix and re-review.) **The reviewer is the `opencode` CLI (`--model github-copilot/gpt-5.6-sol`), not Codex** — maintainer directive 2026-08-14. The cap was raised from 5 to 10 on 2026-08-31 because round 5 on #1136 caught a real defect the first four missed.
2222

2323
## Project Overview
2424

‎ROADMAP.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -352,7 +352,9 @@ graph LR
352352
<details>
353353
<summary>⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1</summary>
354354

355-
> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix.
355+
> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product.
356+
357+
Spec: [103-token-bench](./specs/103-token-bench/)
356358

357359
```mermaid
358360
graph LR
@@ -773,7 +775,7 @@ graph LR
773775
| Planning/docs truth automation | In progress | P2 | — | | |
774776
| Discovery-quality eval harness (Spec 065 second half) | In progress | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | |
775777
| tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | — | [101-tpa-db](./specs/101-tpa-db/) | |
776-
| Token-efficiency benchmark: measured savings, published results | Todo | P1 | — | | |
778+
| Token-efficiency benchmark: measured savings, published results | Todo | P1 | 0/64 (0%) | [103-token-bench](./specs/103-token-bench/) | |
777779
| Windows native tray app `MCP-43` | Todo | P2 | — | | |
778780
| Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | |
779781
| Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | |

‎roadmap.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -391,8 +391,9 @@ epics:
391391
title: "Token-efficiency benchmark: measured savings, published results"
392392
status: todo
393393
priority: P1
394+
spec: specs/103-token-bench
394395
depends_on: [schema-deferred]
395-
note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix."
396+
note: "Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product."
396397
tasks:
397398
- id: token-bench-harness
398399
title: "Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries"

0 commit comments

Comments
 (0)