Skip to content

docs(specs): 103-token-bench — measured savings, published results - #1137

Merged
Dumbris merged 12 commits into
mainfrom
103-token-bench
Aug 31, 2026
Merged

Dumbris merged 12 commits into
mainfrom
103-token-bench

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member

Why

MCPProxy's headline promise is "massive token savings". Today that rests on a static tool-surface measurement — how many tokens the tool definitions cost before the agent does any work. That number is real, reproducible and CI-gated. It is also not what a user pays. A mode that halves the menu but doubles failed attempts costs more, not less, and nothing in the repo measures that.

Two facts make it urgent:

  1. Spec 102 projected ~70% and measured 29.7% on the frozen 45-tool corpus (34.8% at 527 tools). SC-001 was restated per corpus shape rather than the design changed, and its gate concluded that names, descriptions and annotations — not schemas — dominate the payload, with 38.9% the arithmetic ceiling on that corpus. That conclusion comes from two frozen corpora and has never been checked against a real fleet.
  2. Retry rates are assumed, not observed. Session-cost rows are computed from literature-derived per-arm defaults; the dashboard honestly labels them estimated. Every session-level savings claim inherits that assumption.

Scope boundary — the main risk this spec manages

bench/ is already a 74-file harness (PRs #747/#748/#851). The spec opens with a table naming what it already provides — encoding arms including spec 102's deferred rendering, token counting derived from the live tool builders, response cost, break-even, recall/nDCG, the provenance-badged v2 report, the frozen corpora — and states that rebuilding any of it is a defect.

Described in capability terms rather than file names, since a speckit spec.md must stay free of implementation detail.

What is genuinely missing, and is the scope here:

  1. Session replay — nothing in bench/ reads the activity log
  2. Measured first-call success and retry rate, replacing the assumed defaults
  3. Tokens per completed task as the headline, which needs a completion signal the static harness has no concept of
  4. Crossing the full three-axis mode matrix (routing mode × discovery-surface serialization × direct-surface serialization), plus batching, stored scripts and validate-before-dispatch
  5. Payload decomposition that confirms or corrects the spec-102 conclusion on real fleets

Shape

4 user stories (P1: measure cost per completed task on replayed real sessions), 27 FRs, 10 success criteria, zero [NEEDS CLARIFICATION] markers.

The criterion that matters most is SC-006: a mode that costs fewer surface tokens but completes fewer tasks must look worse in the headline metric. The benchmark is required to detect a false saving rather than reward one — which surface-only measurement structurally cannot do.

Also specified because it is the most dangerous silent failure: recorded responses truncated at capture time under-count tokens in the project's favour, so detection is mandatory and silent inclusion is forbidden (FR-002, SC-007).

Assumptions resolved instead of clarification markers

  • Pinned model / spend for agent-loop runs → plan-stage budget decision; the deterministic half is specified to stand alone without it.
  • A task-completion signal is derivable from recorded sessions → assumed, with an explicit fallback. This is the assumption most likely to be wrong and the checklist names it as the first thing plan stage should validate.
  • Public suite left as a capability, not named — the 2026-08 shortlist needs re-confirmation against the suites' current code (specifically whether they emit per-task token counts and can target a single proxy endpoint; FR-020/FR-022 are unsatisfiable if neither holds).

Next

/speckit.plan. Cross-model spec review (opencode gpt-5.6-sol) is running against the tree to check for duplication with bench/.

Spec for the token-bench epic, unblocked when spec 102 shipped in #1063.

The premise: MCPProxy's "massive token savings" claim currently rests on a
static tool-surface measurement. That number is real and CI-gated, but it is
not what a user pays. A mode that halves the menu and doubles failed attempts
costs more, not less, and nothing in the repo measures that.

Two facts make it urgent. Spec 102 projected ~70% and measured 29.7% on the
frozen 45-tool corpus, concluding that names/descriptions/annotations rather
than schemas dominate the payload — a conclusion drawn from two frozen corpora
and never checked against a real fleet. And the session-cost rows in today's
report are computed from literature-derived retry defaults, which the dashboard
honestly labels as estimates; every session-level savings claim inherits that
assumption.

Four user stories, P1 first: measure cost per COMPLETED task and first-call
success on replayed real sessions; let an outsider reproduce the numbers;
confirm or correct the spec-102 payload decomposition; publish.

27 functional requirements, 10 success criteria, no [NEEDS CLARIFICATION]
markers. SC-006 is the one that matters most — a mode that costs fewer surface
tokens but completes fewer tasks must look WORSE in the headline metric, so the
benchmark is required to detect a false saving rather than reward it.

A Scope Boundary table names what the existing harness already provides
(encoding arms, live-builder-derived token counting, response cost, break-even,
retrieval metrics, the provenance-badged report, the frozen corpora) and states
that rebuilding any of it is a defect. The genuinely missing pieces are session
replay, measured success/retry rates, tokens-per-successful-task, crossing the
full three-axis mode matrix, and the payload decomposition.

Three decisions were resolved as documented assumptions rather than
clarification markers: the pinned model is a plan-stage budget decision and the
deterministic half stands alone without it; a task-completion signal is assumed
derivable from recorded sessions, with an explicit fallback, and this is the
assumption most likely to be wrong; and the public suite is specified as a
capability rather than named, because the 2026-08 shortlist needs
re-confirmation against the suites' current code before being committed to.
…ound 1

Seven findings from the opencode gpt-5.6-sol review, all verified against the
tree and all applied. Two were Critical and changed the spec's premise.

1. Replay cannot answer what the spec asked of it. The activity record carries
   tool calls, arguments, responses, status and work-session grouping — but no
   prompt, no conversation, no model state and no completion oracle. Replaying a
   recording under a different mode therefore cannot reveal how the agent would
   have BEHAVED: which calls it would attempt, whether the first was right,
   whether it finished. The original US1 asked for exactly that.

   Added a binding Replay Boundary section and split US1 into US1 (deterministic
   counterfactual cost recomputation over real workload shape, no model
   involved) and US2 (live agent loop under a pinned model, which owns every
   success and completion figure). FR-004 forbids inferring behaviour from a
   recording; Out of Scope names it as forbidden rather than merely excluded.

2. Reproducibility and privacy were in conflict and the privacy side was absent.
   The body-inclusive export deliberately returns full unmasked values including
   detected secrets — it is the compliance surface, not a browsing one. The spec
   kept session contents out of published artefacts but said nothing about
   protecting them during replay, including transmission to model providers.

   Added FR-006..FR-009 (sensitive by default; never transmitted to third
   parties; flagged records excluded or reduced; inputs live outside the repo
   with a documented deletion step) and SC-009. FR-030 now states that figures
   from private sessions are not independently reproducible and cannot be the
   sole support for a published claim.

Also applied:

3. The headline metric did not penalise worse completion — tokens per completed
   task can improve while completion falls, if tokens fall faster. FR-018
   requires completion rate at equal prominence, FR-019 adds a
   completion-regression threshold that overrides any token saving, and SC-007
   requires demonstrating it with a deliberately degraded mode.
4. SC-001 demanded "every mode combination" while FR-017 requires skipping
   invalid ones; scoped to every valid combination.
5. Core terms were undefined; added a binding Definitions block that also
   separates corrective retries from infrastructure retries.
6. Three FRs restated existing harness capabilities; moved to Inherited
   Constraints (IC-001..IC-004) so they are satisfied rather than rebuilt.
7. The capability axes were not a testable matrix; FR-016 now states them as
   binary conditions over the routing modes where each is available.

32 functional requirements, 12 success criteria, 4 inherited constraints.

The reviewer independently confirmed the spec's current-state claims: nothing in
the harness replays activity exports, retry rates are literature-derived
defaults, first-call success is not measured, the three axes are not crossed,
and the spec-102 figures are accurate.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 947cb34
Status: ✅  Deploy successful!
Preview URL: https://b02a7d2e.mcpproxy-docs.pages.dev
Branch Preview URL: https://103-token-bench.mcpproxy-docs.pages.dev

View logs

@Dumbris

Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

Cross-model review round 1 — spec restructured

Seven findings from opencode gpt-5.6-sol, all verified against the tree and all applied. Two were Critical and changed the spec's premise, so the PR description above is superseded on one key point.

1. Replay cannot answer what the spec asked of it

ActivityRecord carries tool calls, arguments, responses, status and work-session grouping — but no prompt, no conversation, no model state and no completion oracle. Replaying a recording under a different mode therefore cannot reveal how the agent would have behaved: which calls it would attempt, whether the first was right, whether it finished. The original US1 asked for exactly that and was unsatisfiable — as was the roadmap task's original framing ("activity-log sessions re-run under each mode combo").

Fix: a binding Replay Boundary section, and US1 split in two:

  • US1 — deterministic counterfactual cost recomputation over real workload shape. No model, byte-reproducible.
  • US2 — live agent loop under a pinned model, which now owns every success/completion figure.

FR-004 forbids inferring behaviour from a recording; Out of Scope names it as forbidden, not merely excluded.

2. Reproducibility and privacy were in conflict, and the privacy side was absent

The body-inclusive export deliberately returns full unmasked values including detected secrets — it is the compliance/incident-response surface, not a browsing one. The spec kept session contents out of published artefacts but said nothing about protecting them during replay, including transmission to model providers.

Fix: FR-006–FR-009 (sensitive by default; never transmitted to third parties; flagged records excluded or reduced; inputs live outside the repo with a documented deletion step) and SC-009. FR-030 now admits figures from private sessions are not independently reproducible and cannot be the sole support for a published claim.

Also applied

  1. The headline metric did not penalise worse completion. Tokens-per-completed-task can improve while completion falls, if tokens fall faster — the old SC-006 was not generally true. FR-018 requires completion rate at equal prominence, FR-019 adds a completion-regression threshold that overrides any token saving, SC-007 requires demonstrating it with a deliberately degraded mode.
  2. SC-001 demanded "every mode combination" while FR-017 requires skipping invalid ones — scoped to every valid combination.
  3. Core terms were undefined — added a binding Definitions block, which also separates corrective retries from infrastructure retries.
  4. Three FRs restated existing harness capabilities — moved to Inherited Constraints (IC-001–IC-004) so they are satisfied, not rebuilt.
  5. The capability axes were not a testable matrix — FR-016 states them as binary conditions over the routing modes where each is available.

Now 32 FRs, 12 success criteria, 4 inherited constraints.

Current-state claims independently confirmed by the reviewer

Nothing in the harness replays activity exports; retry rates are literature-derived defaults; first-call success is not measured; the three axes are not crossed; and the spec-102 figures (29.7% / 34.8% / 38.9% ceiling against a 70% projection) are accurate.

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: 103-token-bench

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (26 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (21 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 33366035721 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Dumbris added 10 commits August 31, 2026 09:16
…en accounting

Phase 0 research (5 parallel investigations) and Phase 1 design artifacts.
Research changed three of the spec's assumptions, all toward less work.

THE MATRIX IS 5 CELLS, NOT 12. The three axes are not a free product: each
serialization axis governs exactly one surface. tool_response_mode has a single
consumer inside the retrieve_tools handler, and direct_tool_response_mode is
read only when building the direct listing, so seven combinations are
structurally impossible and become skip-with-reason rows. contracts/mode-matrix.md
enumerates all five valid cells and all seven skips with their causes.

THE MATRIX CROSSES ON ONE PROXY INSTANCE. All three routing-mode servers are
built at startup and all three endpoints stay mounted regardless of config, so a
cell is selected by endpoint URL — not by a config change plus restart. This
turns the matrix from expensive to nearly free.

MCPMARK IS ADOPTED, SHA-PINNED. Both questions the spec left open resolve in its
favour: it emits per-task token usage, pass verdicts and turn counts, and it has
exactly one MCP factory seam, so pointing it at mcpproxy is a single env-gated
branch. The spec's fallback of building an in-repo task set is not needed.
Filesystem (30 tasks) + postgres (21) are credential-free and fully local — the
FR-027-clean core.

Design decisions:
- Extension seam is "new measurement mode" (flag + driver + report block), used
  twice, following the spec-085 flip-gate precedent. No new arm, no new binary.
- New bench/replaycorpus/ sibling of corpusio/ (a session trace is not a corpus),
  plus replay.go, modematrix.go and agentloop.go in package bench.
- Report blocks are additive with no version bump.
- Token accounting stays split: tiktoken for everything deterministic, provider
  usage read off the model response for the live loop, never summed — enforced by
  the harness's existing withhold-rather-than-compute pattern.

One production change is required and it is small: request_bytes/response_bytes
exist on the storage record, documented as measured pre-truncation, but are
absent from the export contract. Adding them turns FR-002 from "exclude every
truncated record" into "annotate it accurately".

Security design (constitution principle IV) rests on three findings: the export
path never masks (masking is wired only into list and detail), has_sensitive_data
is written asynchronously after persistence so it is a best-effort reducer rather
than a guarantee, and bodies are not needed for the headline at all. Hence
bodies-off by default.

Five risks carried forward, three of which are pre-existing defects this feature
would otherwise inherit silently: the report's tokenizer identity is a
document-level singleton that would become false once provider usage lands beside
it; provenance is section-level while FR-013 requires measured and estimated
figures to coexist in one block; the per-arm retry lookup returns 0.0 for unknown
arms, indistinguishable from a measured 0.0; tiktoken fetches its vocabulary over
the network unless a cache env var is set, which defeats SC-004 offline; and
SC-011 has no enforcement beyond gitignore.

Constitution check passes with no violations, so Complexity Tracking is omitted.
Cross-model review round 2 died on tool-permission errors before writing its
findings, but its progress output surfaced two real defects, both verified
independently against the tree and both fixed here.

1. The matrix framing was wrong. The 3-axis product is 12 combinations that
   COLLAPSE ONTO 5 distinct behaviours — the extra 7 are configurable and
   behaviourally redundant, not "structurally impossible". Each is now a
   skip-with-reason row that also names the valid cell it collapses onto, and
   the reason codes distinguish an ignored axis (no consumer on that surface)
   from a genuine override (code_execution forcing full). Verified:
   effectiveToolResponseMode has one consumer at internal/server/mcp.go:1585,
   effectiveDirectToolResponseMode one at internal/server/mcp_routing.go:100.

2. "A cell is selected by endpoint URL, not by a config change" was overstated.
   URL selects the ROUTING-MODE axis only; the two serialization axes still
   require config. Whether those fields hot-reload or need a fresh instance per
   value is now recorded as OPEN and must be settled before /speckit.tasks,
   because it decides whether the matrix crosses on one long-lived proxy or on
   several — a change to the runner's shape.

Note on the review itself: the run auto-rejected its own attempts to read
outside the repo (GOMODCACHE) and hit a ripgrep record-size limit, then exited 0
having printed no findings body. Per standing practice that is a FAILED run, not
a clean review, so it is not counted as a completed round.
Nine findings from the re-run cross-model review, all verified against the tree
before applying. One invalidated a load-bearing design claim.

1. HIGH — the privacy design rested on a false premise. It claimed bodies-off
   still yields an accurate cost because pre-truncation byte sizes are recorded.
   But request_bytes/response_bytes are BYTE LENGTHS, and tokenizing requires the
   text, so they can only support an estimate. Worse, internal retrieve_tools
   activity emission carries no byte counts at all, so discovery-response cost is
   unrecoverable from the activity log by any route.

   The bodies-off default survives, for a better reason: TOOL-SURFACE cost — the
   term the modes actually change, and US1's headline — is computed from the
   fleet's tool definitions and the call sequence, not from recorded content, so
   it is fully measured with bodies off. Response cost is the part that needs
   text: estimated with bodies off, measured only with bodies on, and the two are
   never presented interchangeably.

2. HIGH — the proposed SC-011 gate did not work. `git status --porcelain
   bench/results` cannot see ignored files and would stay silent about an
   already-tracked report. Replaced with `test -z "$(git ls-files bench/results)"`.

3. HIGH — the report contract contradicted itself: it required a version bump
   whenever an existing field's meaning changes, then proposed narrowing the
   document-level tokenizer field's meaning without one. Resolved additively —
   the existing field keeps its meaning and each new block carries its own
   accounting_source.

4. The "exactly one consumer" claim for direct_tool_response_mode was false: it
   has three production call sites (listing, drift detection, reload logging).
   All three concern the direct surface, so the 12-to-5 collapse holds, but the
   claim is now stated as one SURFACE, not one call site.

5. Serialization hot-reload was recorded as an open question blocking tasks. It
   is not open: tool_response_mode reads the live snapshot per call and
   direct_tool_response_mode rebuilds on config.reloaded. The matrix crosses on
   one instance with a config apply between serialization cells. Blocker removed.

6. The package placement needed a dependency rule. bench/corpusio imports bench,
   so a bench/replaycorpus mirroring it could not be imported by bench/replay.go
   without a second cycle. Now stated explicitly: either replaycorpus imports
   nothing from bench, or orchestration stays in cmd/bench.

7. Added an explicit compose-don't-reimplement rule for modematrix.go, naming the
   existing pieces it must reach through structural interfaces.

8. Sensitive-data wording overstated: there is no persisted HasSensitiveData
   field; it is derived at export from detection metadata added asynchronously
   after initial persistence.

9. The quickstart contradicted itself on offline operation. Setting the tokenizer
   cache variable names a cache but does not fill one, so a genuinely offline
   reproduction needs it pre-populated.
… round 3)

Eight findings, all verified before applying. Two corrected an OVER-correction
made in round 2: fixing the byte-count error, I replaced one absolute claim with
another. Round 3 caught both.

1. "Tool-surface cost carries the headline, so bodies-off works" was too broad.
   It holds for three of five cells, not all five. For direct_full,
   direct_deferred and code_exec, what the mode changes IS the static
   tool-surface payload, computed from the fleet's tool definitions with no
   recorded content — measurable bodies-off. For retrieve_full and
   retrieve_compact, what the mode changes IS the retrieve_tools response body,
   so those two require an explicit bodies-on run and are reported as
   unavailable otherwise, never as zero.

2. "Discovery-response cost is unrecoverable by any route" was false. The
   retrieve_tools handler writes the FULL response to the activity log
   (internal/server/mcp.go:2061-2064), so bodies-on does carry it. It is
   unavailable only in the bodies-off configuration.

3. SC-002 asks for byte-identical replay reports, but the report carries a
   wall-clock generated_at stamp, so the documented diff would always fail. The
   quickstart now excludes that field and the plan records the decision as
   required before the determinism test is written.

4. The chosen suite records input/output/total/reasoning per task and has NO
   cache-read field, so FR-014's cache-read axis cannot come from the suite's
   report. It must be captured in the driver from the provider response or
   declared out of reach for suite-driven runs.

5-7. Round-2 fixes were applied to the contracts but not to their mirrors in
   plan.md and research.md, leaving three contradictions: the tokenizer field
   was still described as needing to be narrowed, the obsolete porcelain SC-011
   gate survived in research.md, and the offline remedy still said "set the env
   var" without the population step. All three now match the contracts.

8. The replay package dependency was left as a choice to settle later while the
   structure already committed to bench/replay.go. Decided:
   bench/replaycorpus imports nothing from bench, defines its own domain types,
   and bench/replay.go imports it. Orchestrating from cmd/bench is rejected —
   it would put measurement logic outside the package holding every other
   measurement.

Also recorded, because the export carries no fleet snapshot: every replay figure
scores a recorded workload against TODAY'S fleet. Internally valid across modes,
but not a historical reconstruction, and reports must say so.
… (round 4)

Five findings, all verified. The first is the THIRD consecutive round in which a
fix introduced the next round's defect, so the correction this time is structural
rather than another restatement.

1. HIGH — code_exec was wrongly classed as fully measurable bodies-off. Its menu
   is static, but the surface also registers retrieve_tools with a handler
   (internal/server/mcp_routing.go:670-676), and those responses are bodies. The
   artifacts now split TWO different questions that the previous three drafts kept
   conflating:
     - MENU cost (what the agent pays to see the tools): measured bodies-off for
       ALL FIVE cells.
     - COMPLETE WORKLOAD cost (menu plus every response consumed): measured
       bodies-off for the two direct cells ONLY. code_exec is partial;
       retrieve_full and retrieve_compact need bodies-on.
   Stating it as a per-cell table with both columns removes the ambiguity that
   produced the last three corrections.

2. The byte-coverage gap was not scoped precisely. Code-execution sub-calls emit
   both byte counts as zero (internal/server/mcp_code_execution.go:811-818), so a
   truncated sub-call cannot use the byte-length estimate either and must fall to
   exclusion accounting rather than contributing zero.

3. The tokenizer contradiction survived in the implementation tree, which still
   said live_report.go would "scope Tokenizer to deterministic sections" — the
   very narrowing the contract forbids. Replaced with per-block accounting_source.

4. The proposed SC-011 gate was placed in bench.yml, which triggers only on v*
   tags and workflow_dispatch and therefore cannot block a PR. Placement is now
   called out alongside the assertion: a PR-triggered required workflow, or an
   explicit statement that SC-011 is post-merge detection only.

5. "The recording contributes real fleet size" contradicted the adjacent, correct
   statement that no fleet snapshot exists. It contributes recorded CALL SHAPE,
   evaluated against today's fleet.
…not absolutes (round 5)

Five findings, three High, all verified. Two expose real design holes rather
than wording problems.

1. HIGH — a fleet input is MANDATORY and was missing from the whole design. A
   menu cost is a property of the tool definitions the agent was shown, and the
   activity export carries no fleet snapshot. ProxyToolsForMode returns only the
   built-in proxy tools, not the upstream fleet, so under `direct` the menu is
   unavailable without one. The documented `-replay <jsonl>` invocation could
   therefore compute nothing. Replay now requires pairing with a frozen fleet
   corpus or a live proxy, and a recording-only run is an error rather than a
   degraded run. The quickstart commands are corrected accordingly.

2. HIGH — no cell yields an absolute complete-workload cost bodies-off. The
   previous draft claimed the two direct cells did. Complete workload includes
   every consumed response and that text is absent, so what bodies-off actually
   supports is (a) menu cost per cell and (b) the cross-mode DELTA between the
   two direct cells, whose identical call responses cancel out of the comparison.
   That delta is the honest bodies-off headline; absolutes are unavailable.

3. HIGH — the loader boundary and bodies-on measurement contradicted each other.
   ReplayCall carries no response content and no content may leave replaycorpus,
   yet tokenization was placed downstream in bench/replay.go. Resolved by
   tokenizing INSIDE the loader and emitting counts, so no text ever crosses the
   boundary. The two rules can now both hold.

4. Bodies-on does not recover what a truncated retrieve_tools call actually COST
   the agent: the activity log stores the full pre-truncation response while the
   agent consumed the truncated text. Tokenizing the stored text would OVERSTATE
   the agent's cost, so such calls must be excluded or the truncation re-applied.

5. "Provider usage maps 1:1 to input/output/cache-read" is provider-specific and
   unverified while the provider is unpinned — and is contradicted for the suite's
   own per-task output, which has no cache-read field. Scoped accordingly.
…ord (round 6)

One finding, and it is a genuine follow-through on round 5 rather than a new
error: round 5 established that truncated retrieve_tools records must be excluded
or have their truncation re-applied, but nothing in the design could IDENTIFY
one.

wasTruncated is computed in the retrieve_tools handler
(internal/server/mcp.go:2019-2035) and only logged. emitActivityInternalToolCall
(:810) has no truncation parameter, so the flag never reaches the activity
record. A loader would therefore be unable to tell a truncated record from a
complete one, and would silently tokenize the stored full response — overstating
what the agent actually paid, which is the exact failure class FR-002 exists to
prevent.

The plan now names a SECOND backend change (propagate the flag onto internal
tool-call activity) and records the fallback explicitly: exclude every internal
retrieve_tools record from response-cost accounting and report the count. The
fallback must be a deliberate decision, never a silent default.

Findings per round on this artifact set: 7, 9, 8, 5, 5, 1 — converging.
…the byte-fields gap

Completes the round-6 fix: plan.md and contracts/replay-input.md already carry
the second backend change, research.md did not.
…kend change (round 7)

One finding. The round-6 fix added a second production change (propagating the
already-computed wasTruncated flag onto internal tool-call activity) but three
earlier statements still said the byte DTO fields were the ONLY production change
and that no request-path code is touched. Both were now false.

Corrected in all three places, including the Constitution Check row for Principle
I: the truncation propagation IS on the retrieve_tools request path, though it
passes a value the handler already holds and adds no computation or I/O. The
blanket-exclusion fallback is recorded as the alternative that removes the
request-path touch entirely, at the cost of coverage — an explicit trade, not a
default.
…e line (round 8)

One wording finding: the post-design constitution re-check said the design
'otherwise adds no production code' while the paragraph above it documents the
export DTO fields and their projection, which are production code. Restated as
touching no other REQUEST-PATH code, with the off-request-path footprint named
explicitly.
@Dumbris

Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

Cross-model review complete — 9 rounds, 38 findings, READY FOR TASKS

Reviewed with opencode gpt-5.6-sol against the tree. Every finding was independently verified before being applied; one reviewer claim (a glob miss) was rejected with evidence.

Findings per round: 7 → 9 → 8 → 5 → 5 → 1 → 1 → 1 → 0.

The findings that changed the design, not the wording

Replay cannot do what the spec originally asked. ActivityRecord carries tool calls, arguments and responses — but no prompt, conversation, model state or completion oracle. So replaying a recording under a different mode cannot reveal how the agent would behave. US1 was split into deterministic cost recomputation (no model) and US2, a live agent loop that owns every success/completion figure. FR-004 now forbids inferring behaviour from a recording.

A fleet input was missing entirely. A menu cost is a property of the tool definitions, and the export carries no fleet snapshot — ProxyToolsForMode returns only built-in proxy tools, not the upstream fleet. The documented -replay <jsonl> invocation could compute nothing under direct. Replay now requires a frozen corpus or a live proxy.

Bodies-off yields deltas, not absolutes. Complete workload includes every consumed response; that text is absent. What survives is menu cost per cell plus the cross-mode delta between the two direct cells, whose identical call responses cancel.

The privacy boundary contradicted the measurement design. No content may leave replaycorpus, yet tokenization sat downstream. Resolved by tokenizing inside the loader and emitting counts only.

Truncated retrieve_tools records cannot be identified. wasTruncated is computed in the handler and only logged — the internal activity emitter has no truncation parameter. A loader would silently tokenize the stored full response and overstate what the agent paid. This forced a second backend change, with an explicit blanket-exclusion fallback.

Errors this review caught in my own corrections

Three consecutive rounds found that a fix introduced the next round's defect — each an over-broad claim about what is measurable. The final correction is structural: a per-cell table separating menu cost from complete workload cost, which is the distinction the earlier drafts kept collapsing.

Verified current-state claims

Nothing in bench/ replays activity exports; retry rates are literature-derived defaults; first-call success is not measured; the three config axes are not crossed; spec 102's figures (29.7% / 34.8% / 38.9% ceiling vs a 70% projection) are accurate; the export path never masks; has_sensitive_data is derived from asynchronously-added metadata; code-execution sub-calls emit zero byte counts; bench.yml runs only on v* tags so an SC-011 gate there could never block a PR.

@Dumbris
Dumbris merged commit 286885a into main Aug 31, 2026
55 of 56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants