Repository navigation
docs(specs): 103-token-bench — measured savings, published results - #1137
Conversation
Spec for the token-bench epic, unblocked when spec 102 shipped in #1063. The premise: MCPProxy's "massive token savings" claim currently rests on a static tool-surface measurement. That number is real and CI-gated, but it is not what a user pays. A mode that halves the menu and doubles failed attempts costs more, not less, and nothing in the repo measures that. Two facts make it urgent. Spec 102 projected ~70% and measured 29.7% on the frozen 45-tool corpus, concluding that names/descriptions/annotations rather than schemas dominate the payload — a conclusion drawn from two frozen corpora and never checked against a real fleet. And the session-cost rows in today's report are computed from literature-derived retry defaults, which the dashboard honestly labels as estimates; every session-level savings claim inherits that assumption. Four user stories, P1 first: measure cost per COMPLETED task and first-call success on replayed real sessions; let an outsider reproduce the numbers; confirm or correct the spec-102 payload decomposition; publish. 27 functional requirements, 10 success criteria, no [NEEDS CLARIFICATION] markers. SC-006 is the one that matters most — a mode that costs fewer surface tokens but completes fewer tasks must look WORSE in the headline metric, so the benchmark is required to detect a false saving rather than reward it. A Scope Boundary table names what the existing harness already provides (encoding arms, live-builder-derived token counting, response cost, break-even, retrieval metrics, the provenance-badged report, the frozen corpora) and states that rebuilding any of it is a defect. The genuinely missing pieces are session replay, measured success/retry rates, tokens-per-successful-task, crossing the full three-axis mode matrix, and the payload decomposition. Three decisions were resolved as documented assumptions rather than clarification markers: the pinned model is a plan-stage budget decision and the deterministic half stands alone without it; a task-completion signal is assumed derivable from recorded sessions, with an explicit fallback, and this is the assumption most likely to be wrong; and the public suite is specified as a capability rather than named, because the 2026-08 shortlist needs re-confirmation against the suites' current code before being committed to.
…ound 1 Seven findings from the opencode gpt-5.6-sol review, all verified against the tree and all applied. Two were Critical and changed the spec's premise. 1. Replay cannot answer what the spec asked of it. The activity record carries tool calls, arguments, responses, status and work-session grouping — but no prompt, no conversation, no model state and no completion oracle. Replaying a recording under a different mode therefore cannot reveal how the agent would have BEHAVED: which calls it would attempt, whether the first was right, whether it finished. The original US1 asked for exactly that. Added a binding Replay Boundary section and split US1 into US1 (deterministic counterfactual cost recomputation over real workload shape, no model involved) and US2 (live agent loop under a pinned model, which owns every success and completion figure). FR-004 forbids inferring behaviour from a recording; Out of Scope names it as forbidden rather than merely excluded. 2. Reproducibility and privacy were in conflict and the privacy side was absent. The body-inclusive export deliberately returns full unmasked values including detected secrets — it is the compliance surface, not a browsing one. The spec kept session contents out of published artefacts but said nothing about protecting them during replay, including transmission to model providers. Added FR-006..FR-009 (sensitive by default; never transmitted to third parties; flagged records excluded or reduced; inputs live outside the repo with a documented deletion step) and SC-009. FR-030 now states that figures from private sessions are not independently reproducible and cannot be the sole support for a published claim. Also applied: 3. The headline metric did not penalise worse completion — tokens per completed task can improve while completion falls, if tokens fall faster. FR-018 requires completion rate at equal prominence, FR-019 adds a completion-regression threshold that overrides any token saving, and SC-007 requires demonstrating it with a deliberately degraded mode. 4. SC-001 demanded "every mode combination" while FR-017 requires skipping invalid ones; scoped to every valid combination. 5. Core terms were undefined; added a binding Definitions block that also separates corrective retries from infrastructure retries. 6. Three FRs restated existing harness capabilities; moved to Inherited Constraints (IC-001..IC-004) so they are satisfied rather than rebuilt. 7. The capability axes were not a testable matrix; FR-016 now states them as binary conditions over the routing modes where each is available. 32 functional requirements, 12 success criteria, 4 inherited constraints. The reviewer independently confirmed the spec's current-state claims: nothing in the harness replays activity exports, retry rates are literature-derived defaults, first-call success is not measured, the three axes are not crossed, and the spec-102 figures are accurate.
Deploying mcpproxy-docs with
|
| Latest commit: |
947cb34
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://b02a7d2e.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://103-token-bench.mcpproxy-docs.pages.dev |
Cross-model review round 1 — spec restructuredSeven findings from opencode 1. Replay cannot answer what the spec asked of it
Fix: a binding Replay Boundary section, and US1 split in two:
FR-004 forbids inferring behaviour from a recording; Out of Scope names it as forbidden, not merely excluded. 2. Reproducibility and privacy were in conflict, and the privacy side was absentThe body-inclusive export deliberately returns full unmasked values including detected secrets — it is the compliance/incident-response surface, not a browsing one. The spec kept session contents out of published artefacts but said nothing about protecting them during replay, including transmission to model providers. Fix: FR-006–FR-009 (sensitive by default; never transmitted to third parties; flagged records excluded or reduced; inputs live outside the repo with a documented deletion step) and SC-009. FR-030 now admits figures from private sessions are not independently reproducible and cannot be the sole support for a published claim. Also applied
Now 32 FRs, 12 success criteria, 4 inherited constraints. Current-state claims independently confirmed by the reviewerNothing in the harness replays activity exports; retry rates are literature-derived defaults; first-call success is not measured; the three axes are not crossed; and the spec-102 figures (29.7% / 34.8% / 38.9% ceiling against a 70% projection) are accurate. |
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 33366035721 --repo smart-mcp-proxy/mcpproxy-go
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…en accounting Phase 0 research (5 parallel investigations) and Phase 1 design artifacts. Research changed three of the spec's assumptions, all toward less work. THE MATRIX IS 5 CELLS, NOT 12. The three axes are not a free product: each serialization axis governs exactly one surface. tool_response_mode has a single consumer inside the retrieve_tools handler, and direct_tool_response_mode is read only when building the direct listing, so seven combinations are structurally impossible and become skip-with-reason rows. contracts/mode-matrix.md enumerates all five valid cells and all seven skips with their causes. THE MATRIX CROSSES ON ONE PROXY INSTANCE. All three routing-mode servers are built at startup and all three endpoints stay mounted regardless of config, so a cell is selected by endpoint URL — not by a config change plus restart. This turns the matrix from expensive to nearly free. MCPMARK IS ADOPTED, SHA-PINNED. Both questions the spec left open resolve in its favour: it emits per-task token usage, pass verdicts and turn counts, and it has exactly one MCP factory seam, so pointing it at mcpproxy is a single env-gated branch. The spec's fallback of building an in-repo task set is not needed. Filesystem (30 tasks) + postgres (21) are credential-free and fully local — the FR-027-clean core. Design decisions: - Extension seam is "new measurement mode" (flag + driver + report block), used twice, following the spec-085 flip-gate precedent. No new arm, no new binary. - New bench/replaycorpus/ sibling of corpusio/ (a session trace is not a corpus), plus replay.go, modematrix.go and agentloop.go in package bench. - Report blocks are additive with no version bump. - Token accounting stays split: tiktoken for everything deterministic, provider usage read off the model response for the live loop, never summed — enforced by the harness's existing withhold-rather-than-compute pattern. One production change is required and it is small: request_bytes/response_bytes exist on the storage record, documented as measured pre-truncation, but are absent from the export contract. Adding them turns FR-002 from "exclude every truncated record" into "annotate it accurately". Security design (constitution principle IV) rests on three findings: the export path never masks (masking is wired only into list and detail), has_sensitive_data is written asynchronously after persistence so it is a best-effort reducer rather than a guarantee, and bodies are not needed for the headline at all. Hence bodies-off by default. Five risks carried forward, three of which are pre-existing defects this feature would otherwise inherit silently: the report's tokenizer identity is a document-level singleton that would become false once provider usage lands beside it; provenance is section-level while FR-013 requires measured and estimated figures to coexist in one block; the per-arm retry lookup returns 0.0 for unknown arms, indistinguishable from a measured 0.0; tiktoken fetches its vocabulary over the network unless a cache env var is set, which defeats SC-004 offline; and SC-011 has no enforcement beyond gitignore. Constitution check passes with no violations, so Complexity Tracking is omitted.
Cross-model review round 2 died on tool-permission errors before writing its findings, but its progress output surfaced two real defects, both verified independently against the tree and both fixed here. 1. The matrix framing was wrong. The 3-axis product is 12 combinations that COLLAPSE ONTO 5 distinct behaviours — the extra 7 are configurable and behaviourally redundant, not "structurally impossible". Each is now a skip-with-reason row that also names the valid cell it collapses onto, and the reason codes distinguish an ignored axis (no consumer on that surface) from a genuine override (code_execution forcing full). Verified: effectiveToolResponseMode has one consumer at internal/server/mcp.go:1585, effectiveDirectToolResponseMode one at internal/server/mcp_routing.go:100. 2. "A cell is selected by endpoint URL, not by a config change" was overstated. URL selects the ROUTING-MODE axis only; the two serialization axes still require config. Whether those fields hot-reload or need a fresh instance per value is now recorded as OPEN and must be settled before /speckit.tasks, because it decides whether the matrix crosses on one long-lived proxy or on several — a change to the runner's shape. Note on the review itself: the run auto-rejected its own attempts to read outside the repo (GOMODCACHE) and hit a ripgrep record-size limit, then exited 0 having printed no findings body. Per standing practice that is a FAILED run, not a clean review, so it is not counted as a completed round.
Nine findings from the re-run cross-model review, all verified against the tree before applying. One invalidated a load-bearing design claim. 1. HIGH — the privacy design rested on a false premise. It claimed bodies-off still yields an accurate cost because pre-truncation byte sizes are recorded. But request_bytes/response_bytes are BYTE LENGTHS, and tokenizing requires the text, so they can only support an estimate. Worse, internal retrieve_tools activity emission carries no byte counts at all, so discovery-response cost is unrecoverable from the activity log by any route. The bodies-off default survives, for a better reason: TOOL-SURFACE cost — the term the modes actually change, and US1's headline — is computed from the fleet's tool definitions and the call sequence, not from recorded content, so it is fully measured with bodies off. Response cost is the part that needs text: estimated with bodies off, measured only with bodies on, and the two are never presented interchangeably. 2. HIGH — the proposed SC-011 gate did not work. `git status --porcelain bench/results` cannot see ignored files and would stay silent about an already-tracked report. Replaced with `test -z "$(git ls-files bench/results)"`. 3. HIGH — the report contract contradicted itself: it required a version bump whenever an existing field's meaning changes, then proposed narrowing the document-level tokenizer field's meaning without one. Resolved additively — the existing field keeps its meaning and each new block carries its own accounting_source. 4. The "exactly one consumer" claim for direct_tool_response_mode was false: it has three production call sites (listing, drift detection, reload logging). All three concern the direct surface, so the 12-to-5 collapse holds, but the claim is now stated as one SURFACE, not one call site. 5. Serialization hot-reload was recorded as an open question blocking tasks. It is not open: tool_response_mode reads the live snapshot per call and direct_tool_response_mode rebuilds on config.reloaded. The matrix crosses on one instance with a config apply between serialization cells. Blocker removed. 6. The package placement needed a dependency rule. bench/corpusio imports bench, so a bench/replaycorpus mirroring it could not be imported by bench/replay.go without a second cycle. Now stated explicitly: either replaycorpus imports nothing from bench, or orchestration stays in cmd/bench. 7. Added an explicit compose-don't-reimplement rule for modematrix.go, naming the existing pieces it must reach through structural interfaces. 8. Sensitive-data wording overstated: there is no persisted HasSensitiveData field; it is derived at export from detection metadata added asynchronously after initial persistence. 9. The quickstart contradicted itself on offline operation. Setting the tokenizer cache variable names a cache but does not fill one, so a genuinely offline reproduction needs it pre-populated.
… round 3) Eight findings, all verified before applying. Two corrected an OVER-correction made in round 2: fixing the byte-count error, I replaced one absolute claim with another. Round 3 caught both. 1. "Tool-surface cost carries the headline, so bodies-off works" was too broad. It holds for three of five cells, not all five. For direct_full, direct_deferred and code_exec, what the mode changes IS the static tool-surface payload, computed from the fleet's tool definitions with no recorded content — measurable bodies-off. For retrieve_full and retrieve_compact, what the mode changes IS the retrieve_tools response body, so those two require an explicit bodies-on run and are reported as unavailable otherwise, never as zero. 2. "Discovery-response cost is unrecoverable by any route" was false. The retrieve_tools handler writes the FULL response to the activity log (internal/server/mcp.go:2061-2064), so bodies-on does carry it. It is unavailable only in the bodies-off configuration. 3. SC-002 asks for byte-identical replay reports, but the report carries a wall-clock generated_at stamp, so the documented diff would always fail. The quickstart now excludes that field and the plan records the decision as required before the determinism test is written. 4. The chosen suite records input/output/total/reasoning per task and has NO cache-read field, so FR-014's cache-read axis cannot come from the suite's report. It must be captured in the driver from the provider response or declared out of reach for suite-driven runs. 5-7. Round-2 fixes were applied to the contracts but not to their mirrors in plan.md and research.md, leaving three contradictions: the tokenizer field was still described as needing to be narrowed, the obsolete porcelain SC-011 gate survived in research.md, and the offline remedy still said "set the env var" without the population step. All three now match the contracts. 8. The replay package dependency was left as a choice to settle later while the structure already committed to bench/replay.go. Decided: bench/replaycorpus imports nothing from bench, defines its own domain types, and bench/replay.go imports it. Orchestrating from cmd/bench is rejected — it would put measurement logic outside the package holding every other measurement. Also recorded, because the export carries no fleet snapshot: every replay figure scores a recorded workload against TODAY'S fleet. Internally valid across modes, but not a historical reconstruction, and reports must say so.
… (round 4)
Five findings, all verified. The first is the THIRD consecutive round in which a
fix introduced the next round's defect, so the correction this time is structural
rather than another restatement.
1. HIGH — code_exec was wrongly classed as fully measurable bodies-off. Its menu
is static, but the surface also registers retrieve_tools with a handler
(internal/server/mcp_routing.go:670-676), and those responses are bodies. The
artifacts now split TWO different questions that the previous three drafts kept
conflating:
- MENU cost (what the agent pays to see the tools): measured bodies-off for
ALL FIVE cells.
- COMPLETE WORKLOAD cost (menu plus every response consumed): measured
bodies-off for the two direct cells ONLY. code_exec is partial;
retrieve_full and retrieve_compact need bodies-on.
Stating it as a per-cell table with both columns removes the ambiguity that
produced the last three corrections.
2. The byte-coverage gap was not scoped precisely. Code-execution sub-calls emit
both byte counts as zero (internal/server/mcp_code_execution.go:811-818), so a
truncated sub-call cannot use the byte-length estimate either and must fall to
exclusion accounting rather than contributing zero.
3. The tokenizer contradiction survived in the implementation tree, which still
said live_report.go would "scope Tokenizer to deterministic sections" — the
very narrowing the contract forbids. Replaced with per-block accounting_source.
4. The proposed SC-011 gate was placed in bench.yml, which triggers only on v*
tags and workflow_dispatch and therefore cannot block a PR. Placement is now
called out alongside the assertion: a PR-triggered required workflow, or an
explicit statement that SC-011 is post-merge detection only.
5. "The recording contributes real fleet size" contradicted the adjacent, correct
statement that no fleet snapshot exists. It contributes recorded CALL SHAPE,
evaluated against today's fleet.
…not absolutes (round 5) Five findings, three High, all verified. Two expose real design holes rather than wording problems. 1. HIGH — a fleet input is MANDATORY and was missing from the whole design. A menu cost is a property of the tool definitions the agent was shown, and the activity export carries no fleet snapshot. ProxyToolsForMode returns only the built-in proxy tools, not the upstream fleet, so under `direct` the menu is unavailable without one. The documented `-replay <jsonl>` invocation could therefore compute nothing. Replay now requires pairing with a frozen fleet corpus or a live proxy, and a recording-only run is an error rather than a degraded run. The quickstart commands are corrected accordingly. 2. HIGH — no cell yields an absolute complete-workload cost bodies-off. The previous draft claimed the two direct cells did. Complete workload includes every consumed response and that text is absent, so what bodies-off actually supports is (a) menu cost per cell and (b) the cross-mode DELTA between the two direct cells, whose identical call responses cancel out of the comparison. That delta is the honest bodies-off headline; absolutes are unavailable. 3. HIGH — the loader boundary and bodies-on measurement contradicted each other. ReplayCall carries no response content and no content may leave replaycorpus, yet tokenization was placed downstream in bench/replay.go. Resolved by tokenizing INSIDE the loader and emitting counts, so no text ever crosses the boundary. The two rules can now both hold. 4. Bodies-on does not recover what a truncated retrieve_tools call actually COST the agent: the activity log stores the full pre-truncation response while the agent consumed the truncated text. Tokenizing the stored text would OVERSTATE the agent's cost, so such calls must be excluded or the truncation re-applied. 5. "Provider usage maps 1:1 to input/output/cache-read" is provider-specific and unverified while the provider is unpinned — and is contradicted for the suite's own per-task output, which has no cache-read field. Scoped accordingly.
…ord (round 6) One finding, and it is a genuine follow-through on round 5 rather than a new error: round 5 established that truncated retrieve_tools records must be excluded or have their truncation re-applied, but nothing in the design could IDENTIFY one. wasTruncated is computed in the retrieve_tools handler (internal/server/mcp.go:2019-2035) and only logged. emitActivityInternalToolCall (:810) has no truncation parameter, so the flag never reaches the activity record. A loader would therefore be unable to tell a truncated record from a complete one, and would silently tokenize the stored full response — overstating what the agent actually paid, which is the exact failure class FR-002 exists to prevent. The plan now names a SECOND backend change (propagate the flag onto internal tool-call activity) and records the fallback explicitly: exclude every internal retrieve_tools record from response-cost accounting and report the count. The fallback must be a deliberate decision, never a silent default. Findings per round on this artifact set: 7, 9, 8, 5, 5, 1 — converging.
…the byte-fields gap Completes the round-6 fix: plan.md and contracts/replay-input.md already carry the second backend change, research.md did not.
…kend change (round 7) One finding. The round-6 fix added a second production change (propagating the already-computed wasTruncated flag onto internal tool-call activity) but three earlier statements still said the byte DTO fields were the ONLY production change and that no request-path code is touched. Both were now false. Corrected in all three places, including the Constitution Check row for Principle I: the truncation propagation IS on the retrieve_tools request path, though it passes a value the handler already holds and adds no computation or I/O. The blanket-exclusion fallback is recorded as the alternative that removes the request-path touch entirely, at the cost of coverage — an explicit trade, not a default.
…e line (round 8) One wording finding: the post-design constitution re-check said the design 'otherwise adds no production code' while the paragraph above it documents the export DTO fields and their projection, which are production code. Restated as touching no other REQUEST-PATH code, with the off-request-path footprint named explicitly.
Cross-model review complete — 9 rounds, 38 findings, READY FOR TASKSReviewed with opencode Findings per round: 7 → 9 → 8 → 5 → 5 → 1 → 1 → 1 → 0. The findings that changed the design, not the wordingReplay cannot do what the spec originally asked. A fleet input was missing entirely. A menu cost is a property of the tool definitions, and the export carries no fleet snapshot — Bodies-off yields deltas, not absolutes. Complete workload includes every consumed response; that text is absent. What survives is menu cost per cell plus the cross-mode delta between the two direct cells, whose identical call responses cancel. The privacy boundary contradicted the measurement design. No content may leave Truncated Errors this review caught in my own correctionsThree consecutive rounds found that a fix introduced the next round's defect — each an over-broad claim about what is measurable. The final correction is structural: a per-cell table separating menu cost from complete workload cost, which is the distinction the earlier drafts kept collapsing. Verified current-state claimsNothing in |
Why
MCPProxy's headline promise is "massive token savings". Today that rests on a static tool-surface measurement — how many tokens the tool definitions cost before the agent does any work. That number is real, reproducible and CI-gated. It is also not what a user pays. A mode that halves the menu but doubles failed attempts costs more, not less, and nothing in the repo measures that.
Two facts make it urgent:
estimated. Every session-level savings claim inherits that assumption.Scope boundary — the main risk this spec manages
bench/is already a 74-file harness (PRs #747/#748/#851). The spec opens with a table naming what it already provides — encoding arms including spec 102's deferred rendering, token counting derived from the live tool builders, response cost, break-even, recall/nDCG, the provenance-badged v2 report, the frozen corpora — and states that rebuilding any of it is a defect.Described in capability terms rather than file names, since a speckit
spec.mdmust stay free of implementation detail.What is genuinely missing, and is the scope here:
bench/reads the activity logShape
4 user stories (P1: measure cost per completed task on replayed real sessions), 27 FRs, 10 success criteria, zero
[NEEDS CLARIFICATION]markers.The criterion that matters most is SC-006: a mode that costs fewer surface tokens but completes fewer tasks must look worse in the headline metric. The benchmark is required to detect a false saving rather than reward one — which surface-only measurement structurally cannot do.
Also specified because it is the most dangerous silent failure: recorded responses truncated at capture time under-count tokens in the project's favour, so detection is mandatory and silent inclusion is forbidden (FR-002, SC-007).
Assumptions resolved instead of clarification markers
Next
/speckit.plan. Cross-model spec review (opencodegpt-5.6-sol) is running against the tree to check for duplication withbench/.