Skip to content

feat(bench): MCP tools/list primitive and a live fleet source for replay - #1151

Merged
Dumbris merged 3 commits into
mainfrom
103-fleet-and-decomp
Sep 1, 2026
Merged

Dumbris merged 3 commits into
mainfrom
103-fleet-and-decomp

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member

Adds bench/mcptools.go — a tools/list fetch mapping a live proxy's catalog onto []bench.Tool with real schemas — and wires it as a second fleet source, so -replay <jsonl> -proxy <url> works alongside -corpus-v2.

The fleet input stays mandatory; this adds a way to supply one, not a way to omit it. Supplying both is an error — two fleets is ambiguous, and silently preferring one would make the report's fleet_id a claim about definitions it wasn't priced over.

I had the premise backwards, and the research phase caught it

I told the team REST /api/v1/tools serves stub schemas. It doesn't. MCP-3167 was already fixed in internal/contracts/converters.go, and a live probe found 42/45 real schemas — the 3 empties being genuinely parameter-less tools. Neither withholding branch in live_report.go fires today; a live run emits authoritative_headline=true.

The hazard is real but sits on the other side. With direct_tool_response_mode: "deferred" (spec 102), /mcp/all serves {"type":"object"} for every upstream tool while REST keeps serving real ones:

Surface Under deferred mode
/mcp/all 4805 → 2939 tokens (−38.8%), 45/46 stubbed
REST /api/v1/tools unchanged, 42/45 real

A live fleet read from that surface under-counts the direct menu and flips the sign of the direct delta. That's why the stub guard is the primitive rather than a detail of it.

Also verified: a live tools/list capture reproduces bench's in-process ProxyModeToolDefs derivation exactly, to the token (5783 and 3781).

Defects from the adversarial pass

  • The guard now runs in ResolveFleet's live branch, not only inside FetchToolsList. A check living in one FleetSource implementation protects only that implementation.
  • An all-empty-properties population is refused with an honest cause. The shape is genuinely ambiguous — a stubbed surface and an honestly parameter-less tool emit identical bytes — so the error says so rather than blaming the supervisor stub. Refusal is still right: admitting a stubbed fleet inflates the headline; refusing an honest one costs one clear message.
  • isStubSchema treats an undecodable schema as a stub. A schema of true, [] or 0 previously passed as REAL and was priced at ~1 token.
  • FleetShape.SchemalessTools — the guard only refuses an all-stub fleet, so a 3-of-45 partial regression flowed through leaving no trace. Now in the report and the JSON schema.
  • MCP client closed when Start fails; the "__" mis-split's double-charging consequence documented.

One finding rejected, with the reasoning left in the code

The review called the guard's absence on the frozen-corpus path an asymmetry defect. It isn't. corpus_v1 is 45 tools with zero schemas by design and is the default -corpus value; CountToolWithSchema is documented to count schema-less tools identically so the two mix. Guarding there would reject the project's own default corpus.

The asymmetry tracks a real difference: a live surface can be misconfigured out from under the operator; a committed corpus is a reviewed artifact whose shape is visible in the diff.

Three vacuous tests fixed, each mutation-verified

  • The stub-fleet test injected the sentinel error from a fake source — it passed with guardToolSchemas deleted entirely. Now feeds real deferred-mode definitions through the real guard.
  • The API key was never asserted to reach the wire; if false on the key-appending branch left the suite green.
  • The repeated-cursor and nil-transport guards were untested; disabling either left the suite green.

Verification

go build ./...                                    clean
go test ./bench/... -count=1                      green
go test -race ./bench/ -count=1                   green
golangci-lint run --config .github/.golangci.yml  0 issues

…50-T052)

Spec 103 US4. Attributes a corpus's rendered payload to name, description,
schema and structural overhead, and recomputes the achievable ceiling FOR THAT
CORPUS rather than carrying one forward — the error spec 102 made when it
projected ~70% from an assumption about one corpus shape.

TOKENS ARE NOT ADDITIVE ACROSS CONCATENATION, so the shares cannot be measured
by counting components separately: BPE merges across boundaries, and
Count(name) + Count(description) + Count(schema) does not equal Count of the
concatenation. Independent counts would produce shares that look plausible,
never sum, and misattribute every boundary. The payload is therefore tokenized
ONCE and each token attributed by byte offset, reusing the span partition
AttributeTokens already implements for response cost. T050 asserts the shares
partition the payload exactly.

RUNNING IT AGAINST THE REAL CORPORA EXPOSED TWO DEFECTS IN MY OWN VERDICT LOGIC,
both fixed here, and both of the same kind — a confident statement about a
corpus that does not support it:

1. The 527-tool snapshot carries ZERO schemas. The first version reported
   "deferring schemas cannot exceed 0.0% however it is implemented", which reads
   as a decisive confirmation of spec 102 and is really a corpus that does not
   contain the thing under test. Now INAPPLICABLE, with ToolsWithSchema on the
   struct so the distinction cannot collapse again.

2. The verdict claimed to CONFIRM or CORRECT spec 102 from corpus evidence.
   Spec 102 measured the production WIRE payload — annotations, mcp.Tool
   marshalling, the "[server] " prefix; this decomposition covers the corpus
   rendering, which has none of those. Adjudicating across that gap is answering
   with evidence about something else, and the code already said so in its
   Limitation field while the verdict ignored it. Verdicts are now scoped to the
   corpus and explicitly decline to settle spec 102's figure.

THE MEASUREMENT ITSELF IS INTERESTING. On the 45-tool corpus_v2 the payload is
59.4% schema, 36.5% description, 2.4% name, 1.7% structural — schemas are the
DOMINANT term in the corpus rendering, which is the opposite of what spec 102
concluded about the wire payload. That does not overturn spec 102; it localises
the disagreement to the wire framing and annotations, which is a far more useful
place to look than "schemas are cheap".

T053 (emitting the block into the report) is deliberately NOT ticked — the
decomposition computes, but nothing wires it into ReportV2 yet.

go test ./bench/... green; golangci-lint v2 0 issues.
…s null, not zero

Completes T053. Spec 103 US4 is done: 37/64.

The block type already existed with ShareAnnotationsPct as a plain float64, and
the JSON schema marked it required. Both were wrong for the only source that can
populate this block today: the frozen corpora carry no annotations field, so the
share is NOT MEASURABLE, and emitting 0 would claim a measurement that was never
taken — the silent-zero failure the rest of this report works to avoid.

So the field is now a pointer that marshals to null, and the schema types it
["number","null"] with a comment explaining that null and zero differ here. A
future source that CAN measure annotations — the production wire payload, which
is what spec 102 measured — populates it without a schema change.

PayloadDecompositionBlockFor refuses fewer than two fleet shapes. FR-024 asks
for at least two so the attribution can be seen to MOVE with fleet size, and one
shape is exactly the evidence base spec 102 projected from; making that
unemittable is the point of the requirement.

Two existing reportv2_test fixtures were updated to pointer values rather than
having the field dropped: they model a future annotation-capable source, and
keeping real values there preserves what they were testing.

Gates: go test ./bench/... green, -race green, golangci-lint v2 0 issues, all
three schema-validation tests PASS (confirmed not skipped).
Spec 103. Adds bench/mcptools.go — a tools/list fetch that maps a live proxy's
catalog onto []bench.Tool with real schemas — and wires it as a second fleet
source so `-replay <jsonl> -proxy <url>` works alongside -corpus-v2. The fleet
input stays MANDATORY; this adds a way to supply one, not a way to omit it.
Supplying both is an error: two fleets is ambiguous, and silently preferring one
would make the report's fleet_id a claim about definitions it was not priced over.

THE PREMISE I STARTED FROM WAS INVERTED, and the research phase caught it before
any code was written. I had said REST /api/v1/tools serves stub schemas. It does
not: MCP-3167 was already fixed in internal/contracts/converters.go, and a live
probe found 42/45 real schemas, the 3 empties being genuinely parameter-less
tools. Neither withholding branch in live_report.go fires today — a live run
emits authoritative_headline=true.

The hazard is real but sits on the OTHER side. With
direct_tool_response_mode:"deferred" (spec 102), /mcp/all serves {"type":"object"}
for every upstream tool while REST keeps serving real ones. Measured: /mcp/all
drops 4805 -> 2939 tokens, -38.8%. A live fleet read from that surface
under-counts the direct menu and FLIPS THE SIGN of the direct delta. Hence the
stub guard, which is the point of the primitive rather than a detail of it.

Also verified: a live tools/list capture reproduces bench's in-process
ProxyModeToolDefs derivation exactly, to the token (5783 and 3781).

DEFECTS FROM THE ADVERSARIAL PASS, and one finding rejected:

- The guard now runs in ResolveFleet's live branch, not only inside
  FetchToolsList. A check living in one FleetSource implementation protects only
  that implementation; ResolveFleet is where every live fleet passes exactly once.
- An all-empty-properties population is REFUSED with an honest cause. The shape
  is genuinely ambiguous — a stubbed surface and an honestly parameter-less tool
  emit identical bytes — so the error says so instead of blaming the supervisor
  stub, and points at -corpus-v2 for a fleet that really is parameter-less.
  Refusal is still right: admitting a stubbed fleet inflates the headline, while
  refusing an honest one costs one clear message.
- isStubSchema now treats an undecodable schema as a stub. A schema of `true`,
  `[]` or `0` previously passed as REAL and was priced at ~1 token.
- FleetShape carries SchemalessTools. The guard only refuses an ALL-stub fleet, so
  a 3-of-45 partial regression flowed through leaving no trace; the count is now
  in the report and the JSON schema.
- The MCP client is closed when Start fails, symmetric with the Initialize path.
- The "__" mis-split's cost consequence is documented: a server name containing
  "__" leaves a stale prefix that the arms then re-prepend, double-charging it.

REJECTED, with the reasoning recorded in the code so it is not re-fixed: the
review called the guard's absence on the frozen-corpus path an asymmetry defect.
It is not. corpus_v1 is 45 tools with ZERO schemas by design and is the default
-corpus value, and CountToolWithSchema is documented to count schema-less tools
identically so the two mix. Guarding there would reject the project's own default
corpus. The asymmetry tracks a real difference: a live surface can be
misconfigured out from under the operator, a committed corpus is a reviewed
artifact whose shape is visible in the diff.

THREE VACUOUS TESTS FIXED, each mutation-verified after the change:
- the stub-fleet test injected the sentinel error from a fake source, so it
  passed with guardToolSchemas deleted entirely. It now feeds real deferred-mode
  tool definitions through the real guard.
- the API key was never asserted to reach the wire; `if false` on the
  key-appending branch left the suite green.
- the repeated-cursor and nil-transport guards were untested; disabling either
  left the suite green.

Gates: go build ./... clean; go test ./bench/... and -race green; golangci-lint
v2 0 issues.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 52fb0ea
Status: ✅  Deploy successful!
Preview URL: https://a8c1c372.mcpproxy-docs.pages.dev
Branch Preview URL: https://103-fleet-and-decomp.mcpproxy-docs.pages.dev

View logs

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 75.87209% with 83 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
bench/mcptools.go 83.64% 19 Missing and 7 partials ⚠️
bench/payloaddecomp.go 82.64% 12 Missing and 9 partials ⚠️
bench/mcpcaller.go 0.00% 19 Missing ⚠️
bench/cmd/bench/main.go 0.00% 12 Missing ⚠️
bench/replay.go 87.50% 2 Missing and 2 partials ⚠️
bench/live_report.go 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: 103-fleet-and-decomp

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (26 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (21 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 33420945393 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@Dumbris
Dumbris merged commit 2161262 into main Sep 1, 2026
47 of 48 checks passed
Dumbris added a commit that referenced this pull request Sep 2, 2026
… token-bench truth-sync (#1200)

* chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench

- mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped
  2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532).
- action-log-transparency: new task activity-storage-bounds tracking
  #1173/#1174 (response truncation never wired), #1176 (unbounded
  per-server tool_calls buckets) and #1175 (omitempty zero erasure).
- token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160
  are all merged. Clears both --check-github errors.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants