Skip to content

feat(bench): spec 103 MVP — replay cost recomputation over real recorded workloads - #1141

Merged
Dumbris merged 5 commits into
mainfrom
103-token-bench-impl
Aug 31, 2026
Merged

Dumbris merged 5 commits into
mainfrom
103-token-bench-impl

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member

Implements the spec 103 MVP: T001–T033 of 64. US1 is independently shippable and needs zero model spend.

What it does

bench -replay <activity.jsonl> -corpus-v2 <fleet> recomputes what a real recorded workload would have cost under each of the 5 mode cells.

On the frozen 45-tool corpus it reports a 1296-token / 29.5% direct_full → direct_deferred delta — independently matching the DirectDeferredArm's own documented measurement, which is a useful corroboration that the two paths agree.

What it deliberately refuses

This is the part the spec was corrected five times to get right:

  • A fleet input is mandatory. -replay alone exits 1: a menu is a property of the tool definitions and the export carries no fleet snapshot, so a recording alone computes nothing. An error, not a degraded run.
  • No absolute complete-workload cost for any cell. It includes every consumed response, and that's a body. Every cell carries a withheld flag plus a reason — rendering zero would read as "measured, and it cost nothing."
  • Only the direct_full ↔ direct_deferred delta is emitted, the one comparison whose identical call responses cancel.
  • Every figure carries the FR-004 counterfactual label, enforced at emission rather than left to rendering.

Two production changes, both small

  1. request_bytes/response_bytes now reach the export. Copied unconditionally rather than inside the includeBodies guard — they're sizes, not content, and bodies-off is exactly when they're the only cost signal left. omitempty is load-bearing: absent means UNKNOWN and routes to exclusion accounting; a present zero would read as a costless call.

  2. The truncation flag now reaches the record. wasTruncated was computed and only logged. It now travels the same two hops the non-internal path always used. Without it, a truncated retrieve_tools record is indistinguishable from a complete one, and anything recomputing cost tokenizes text the agent never paid for — overstating what mcpproxy cost, the one direction of error this benchmark exists to prevent.

Defects the adversarial verification pass caught

The withheld reason was posture-blind. Bodies-on, the two direct cells emitted a response figure while the same row printed "that text is absent from a bodies-off replay" — a caveat stating a false premise beside the number contradicting it.

The ExclusionReport was only partially surfaced. Only TotalDropped, folded into one row. The per-reason breakdown, costs withheld on records that did contribute, flagged records and orphaned sub-calls never reached the report. Bodies-on, a single withheld response silently collapsed a cell's response total to nil with nothing saying why — the silent accounting SC-008 forbids.

Added ReplayLoaderAccounting with the three buckets kept separate: "12 exclusions" spanning dropped, withheld and flagged is uninterpretable, and an uninterpretable count satisfies the letter of "everything is counted" while destroying its purpose.

Also fixed a silent no-op: bench -live -replay x ran live mode and discarded -replay with no warning. Now a hard error.

Structure

bench/replaycorpus imports nothing from bench — bench/corpusio imports bench, so mirroring it would have made bench/replay.go uncompilable through a cycle. A go/parser guard test enforces this in the package where the mistake would be made, since violating it doesn't break that package's own build.

Privacy is enforced in code, not just documented: bodies-off is the zero value, bodies-on must be named and warns that the export path doesn't mask, and LoadFile refuses any path inside the working tree before opening it. Tokenization happens inside the loader and emits counts only.

Verification

Both new tests were mutation-tested: reverting the posture-awareness fails the first, nil-ing the loader accounting fails the second, both pass once reverted.

go build ./...                                    clean
go test ./bench/... -count=1                      green
go test -race ./bench/... -count=1                green
golangci-lint run --config .github/.golangci.yml  0 issues
git ls-files bench/results                        empty (SC-011)

Report validates against report-v2.schema.json — test confirmed PASS, not skipped. oas/swagger.yaml regenerated for the two new fields; note make swagger-verify isn't wired into any workflow, so CI would not have caught that drift.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 52eedb2
Status: ✅  Deploy successful!
Preview URL: https://7b5bf50a.mcpproxy-docs.pages.dev
Branch Preview URL: https://103-token-bench-impl.mcpproxy-docs.pages.dev

View logs

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: 103-token-bench-impl

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (26 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (21 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 33392141086 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@codecov-commenter

Copy link
Copy Markdown

…ag, report blocks, mode matrix, replay loader

Spec 103 (token-bench) tasks T001-T018 and T022-T026. Five units were built in
parallel on disjoint file sets, then integrated here.

PRODUCTION CHANGES (two, both small, both required before US1 can be honest):

1. request_bytes/response_bytes now reach the activity export
   (internal/contracts/activity.go, internal/httpapi/activity.go). They exist on
   the storage record as pre-truncation measurements but never reached the
   export contract. Copied UNCONDITIONALLY rather than inside the includeBodies
   guard: they are sizes, not content, and bodies-off — the default — is exactly
   the case where they are the only cost signal left. omitempty is load-bearing:
   an absent key means UNKNOWN and routes the record to exclusion accounting,
   whereas a present zero would read as a costless call and understate the
   workload.

2. The truncation flag now reaches the record. wasTruncated was computed in the
   retrieve_tools handler and only logged; it now travels the same two hops the
   non-internal path has always used — Runtime.EmitActivityInternalToolCallTruncated
   carries "response_truncated" in the event payload, and
   ActivityService.handleInternalToolCall reads it back onto
   ActivityRecord.ResponseTruncated. Both new methods delegate from the old
   signatures, so every existing call site still compiles.

   This matters because the activity log deliberately stores the FULL
   pre-truncation retrieve_tools response while the agent consumed only the cut
   text. Without the flag a truncated record is indistinguishable from a complete
   one, and anything recomputing cost tokenizes text the agent never paid for —
   overstating what mcpproxy cost, which is the one direction of error this
   benchmark exists to prevent.

BENCH ADDITIONS:

- bench/reportv2.go: the three spec-103 blocks (replay, agent_loop,
  payload_decomposition) as additive optional fields, each with its own
  accounting_source and each row with its own provenance. The document-level
  Tokenizer field is deliberately NOT narrowed — that would be a meaning change
  requiring a report_version bump. Blocks declared in the reviewed JSON schema.
- bench/modematrix.go: the 5 valid cells, with the other 7 combinations of the
  3-axis product rendered as skip rows naming the cell each collapses onto, plus
  the FR-016 capability conditions. Reuses the existing Skipped/SkipReason shape
  rather than inventing a second one.
- bench/replaycorpus/: the activity-JSONL loader. It imports NOTHING from bench —
  bench/corpusio imports bench, so mirroring that would have made bench/replay.go
  uncompilable through an import cycle. A go/parser guard test enforces the rule
  in the package where the mistake would be made, because violating it does not
  break this package's own build.

  Privacy is enforced in code, not just documented: BodyPolicy's zero value is
  bodies-off, bodies-on must be named and prints a warning that the export path
  does not mask, and LoadFile refuses any path inside the repository working tree
  before opening it. Tokenization happens inside the package and emits counts
  only, so no body text can cross the boundary.

CI: tiktoken cache populated (not merely named) in bench.yml, and a PR-triggered
workflow asserting no generated report is tracked via git ls-files — bench.yml
runs only on v* tags and could never have blocked a PR.

oas/swagger.yaml + oas/docs.go regenerated for the two new fields. Note
make swagger-verify is not wired into any workflow, so CI would not have caught
the drift.

Verified: go build ./... clean; go vet clean; bench, replaycorpus, contracts,
httpapi, runtime and the internal/server truncation tests all pass. The
truncation test was red before the runtime wiring and green after, so it is a
real assertion rather than a vacuous one.
…orded workload

Completes the spec 103 MVP: tasks T019-T021 and T027-T033, on top of the
foundations in the previous commit. 33/64 tasks done; US1 is independently
shippable and needs zero model spend.

WHAT REPLAY PRODUCES, and what it deliberately refuses to:

- A fleet input is MANDATORY. `-replay <jsonl>` alone exits 1 with a message
  explaining that a menu is a property of the tool definitions and the activity
  export carries no fleet snapshot, so a recording on its own computes nothing.
  This is an error, not a degraded run.
- Menu cost per cell, measured from the fleet input.
- NO absolute complete-workload cost for ANY cell — it includes every consumed
  response, and that is a body. Every cell carries the withheld flag plus a
  reason; rendering zero would read as "measured, and it cost nothing".
- The direct_full -> direct_deferred cross-mode delta, the one comparison whose
  identical call responses cancel. On the frozen 45-tool corpus this measures
  1296 tokens / 29.5%, independently matching the DirectDeferredArm's own
  documented figure.
- Every figure carries the FR-004 counterfactual label, enforced at emission
  rather than left to rendering.
- generated_at is pinned so two runs are byte-identical (SC-002).

TWO DEFECTS FOUND BY THE ADVERSARIAL VERIFICATION PASS, both fixed here:

1. The withheld reason was posture-blind. On a bodies-on run the two direct
   cells emitted a response figure while the SAME row printed "that text is
   absent from a bodies-off replay" — a caveat stating a false premise next to
   the number contradicting it. The reason now tracks the posture: bodies-on it
   explains that the figure prices the RECORDED traffic rather than the workload
   a mode would incur in general, which is why the absolute cost stays withheld
   for a different and true reason.

2. The ExclusionReport was only partially surfaced — just TotalDropped, folded
   into one row. The per-reason breakdown, the costs WITHHELD on records that
   did contribute, the flagged records and the orphaned sub-calls never reached
   the report. Bodies-on, a single withheld response silently collapsed a cell's
   response total to nil with nothing anywhere saying why, which is exactly the
   silent accounting SC-008 forbids. Added ReplayLoaderAccounting, declared in
   the JSON schema, with the three buckets kept SEPARATE: "12 exclusions"
   spanning dropped, withheld and flagged is uninterpretable, and an
   uninterpretable count satisfies the letter of "everything is counted" while
   destroying its purpose.

Also fixed a silent no-op: `bench -live -replay x` ran live mode and discarded
-replay with no output and no warning. Now a hard error — a benchmark that
quietly measures something other than what was asked is worse than one that
refuses.

Both new tests were mutation-tested to prove they are not vacuous: reverting the
posture-awareness fails the first, nil-ing the loader accounting fails the
second, and both pass again once reverted.

Gates: go build ./... clean; go test ./bench/... and -race both green;
golangci-lint v2 with .github/.golangci.yml reports 0 issues; the report
validates against report-v2.schema.json (test confirmed PASS, not skipped);
git ls-files bench/results is empty.
Found by cross-model review of the implementation.

A sub-call whose parent fell outside the exported window was counted in
OrphanedSubCalls and THEN dropped as ReasonUnattributed — the same record
reported under two counters with two different meanings, while the field's own
doc comment claimed orphans are "KEPT at top level rather than dropped". That
claim was false for the common case: sandbox sub-calls inherit their session
from the parent, so losing the parent loses the session too, and the record fell
straight through to the unattributed drop.

Both are records with no unit of work, but they tell the operator different
things, and only one is actionable. An unattributed record never had a session.
An orphan had a parent that the EXPORT WINDOW cut off — a wider re-export brings
it back. Folding the second into the first hides the one exclusion reason
someone can actually do something about.

Now: ReasonOrphanedSubCall is its own drop reason, OrphanedSubCalls spans both
outcomes so the attribution loss stays visible either way, and the doc states
both cases instead of asserting the one that is often untrue.

Two tests, mutation-verified: routing the drop back to ReasonUnattributed fails
the first, and the second pins that an orphan which CAN stand alone is kept
rather than dropped, since dropping it would understate the workload.

go test ./bench/... green; golangci-lint v2 0 issues.
…omething

Found by cross-model review round 2, which cleared all three correctness
priorities (loader cost branches, replay aggregation, the four-hop truncation
path) and found this instead.

TestUsabilityFlagsAreComputedOnceAtLoad compared the flags before and after a
downstream read and asserted they were equal. It never asserted what they
should BE. Delete load-time classification entirely and before == after ==
Flags{}, so "nothing changed" still holds and the test reports green on a loader
that classifies nothing at all — a green light on exactly the failure it exists
to catch.

Now asserts Truncated is already set before testing stability. Mutation-verified:
removing the load-time assignment in flags.go fails it with
"load-time classification must already have set Truncated, got {Truncated:false ...}",
where before it passed.

Review verdict on the production code: no correctness issues in cost handling
(estimated/unavailable responses make the aggregate nil rather than adding zero;
a truncated internal response cannot reach the tokenizer; sensitivity blocks
both request and response tokenization), replay aggregation (AllCalls() used for
both call count and response totals; the delta is strictly direct_full minus
direct_deferred over the direct_full denominator), or the truncation path (the
handler's real result reaches ActivityRecord through both delegating overloads,
with the integration test proving both the true and false outcomes against real
storage).
Rebased onto main, which now carries the spec link from #1140, so ROADMAP.md
regenerates with the epic's real progress badge (33/64) instead of a blank.

Flipped token-bench and token-bench-harness from todo to in_progress with the
implementation PR linked. US1 — the deterministic replay half — is shipped and
independently useful; the harness task stays in_progress because its measured
success and retry rates belong to US2, which is gated on a pinned model.

gen-roadmap.py --check and --check-github both clean.
@Dumbris
Dumbris force-pushed the 103-token-bench-impl branch from 92eb84c to 52eedb2 Compare August 31, 2026 12:30
@Dumbris
Dumbris merged commit 3d076c3 into main Aug 31, 2026
50 checks passed
Dumbris added a commit that referenced this pull request Sep 2, 2026
… token-bench truth-sync (#1200)

* chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench

- mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped
  2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532).
- action-log-transparency: new task activity-storage-bounds tracking
  #1173/#1174 (response truncation never wired), #1176 (unbounded
  per-server tool_calls buckets) and #1175 (omitempty zero erasure).
- token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160
  are all merged. Clears both --check-github errors.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants