Repository navigation
feat(bench): MCP tools/list primitive and a live fleet source for replay - #1151
Merged
Merged
Conversation
…50-T052) Spec 103 US4. Attributes a corpus's rendered payload to name, description, schema and structural overhead, and recomputes the achievable ceiling FOR THAT CORPUS rather than carrying one forward — the error spec 102 made when it projected ~70% from an assumption about one corpus shape. TOKENS ARE NOT ADDITIVE ACROSS CONCATENATION, so the shares cannot be measured by counting components separately: BPE merges across boundaries, and Count(name) + Count(description) + Count(schema) does not equal Count of the concatenation. Independent counts would produce shares that look plausible, never sum, and misattribute every boundary. The payload is therefore tokenized ONCE and each token attributed by byte offset, reusing the span partition AttributeTokens already implements for response cost. T050 asserts the shares partition the payload exactly. RUNNING IT AGAINST THE REAL CORPORA EXPOSED TWO DEFECTS IN MY OWN VERDICT LOGIC, both fixed here, and both of the same kind — a confident statement about a corpus that does not support it: 1. The 527-tool snapshot carries ZERO schemas. The first version reported "deferring schemas cannot exceed 0.0% however it is implemented", which reads as a decisive confirmation of spec 102 and is really a corpus that does not contain the thing under test. Now INAPPLICABLE, with ToolsWithSchema on the struct so the distinction cannot collapse again. 2. The verdict claimed to CONFIRM or CORRECT spec 102 from corpus evidence. Spec 102 measured the production WIRE payload — annotations, mcp.Tool marshalling, the "[server] " prefix; this decomposition covers the corpus rendering, which has none of those. Adjudicating across that gap is answering with evidence about something else, and the code already said so in its Limitation field while the verdict ignored it. Verdicts are now scoped to the corpus and explicitly decline to settle spec 102's figure. THE MEASUREMENT ITSELF IS INTERESTING. On the 45-tool corpus_v2 the payload is 59.4% schema, 36.5% description, 2.4% name, 1.7% structural — schemas are the DOMINANT term in the corpus rendering, which is the opposite of what spec 102 concluded about the wire payload. That does not overturn spec 102; it localises the disagreement to the wire framing and annotations, which is a far more useful place to look than "schemas are cheap". T053 (emitting the block into the report) is deliberately NOT ticked — the decomposition computes, but nothing wires it into ReportV2 yet. go test ./bench/... green; golangci-lint v2 0 issues.
…s null, not zero Completes T053. Spec 103 US4 is done: 37/64. The block type already existed with ShareAnnotationsPct as a plain float64, and the JSON schema marked it required. Both were wrong for the only source that can populate this block today: the frozen corpora carry no annotations field, so the share is NOT MEASURABLE, and emitting 0 would claim a measurement that was never taken — the silent-zero failure the rest of this report works to avoid. So the field is now a pointer that marshals to null, and the schema types it ["number","null"] with a comment explaining that null and zero differ here. A future source that CAN measure annotations — the production wire payload, which is what spec 102 measured — populates it without a schema change. PayloadDecompositionBlockFor refuses fewer than two fleet shapes. FR-024 asks for at least two so the attribution can be seen to MOVE with fleet size, and one shape is exactly the evidence base spec 102 projected from; making that unemittable is the point of the requirement. Two existing reportv2_test fixtures were updated to pointer values rather than having the field dropped: they model a future annotation-capable source, and keeping real values there preserves what they were testing. Gates: go test ./bench/... green, -race green, golangci-lint v2 0 issues, all three schema-validation tests PASS (confirmed not skipped).
Spec 103. Adds bench/mcptools.go — a tools/list fetch that maps a live proxy's
catalog onto []bench.Tool with real schemas — and wires it as a second fleet
source so `-replay <jsonl> -proxy <url>` works alongside -corpus-v2. The fleet
input stays MANDATORY; this adds a way to supply one, not a way to omit it.
Supplying both is an error: two fleets is ambiguous, and silently preferring one
would make the report's fleet_id a claim about definitions it was not priced over.
THE PREMISE I STARTED FROM WAS INVERTED, and the research phase caught it before
any code was written. I had said REST /api/v1/tools serves stub schemas. It does
not: MCP-3167 was already fixed in internal/contracts/converters.go, and a live
probe found 42/45 real schemas, the 3 empties being genuinely parameter-less
tools. Neither withholding branch in live_report.go fires today — a live run
emits authoritative_headline=true.
The hazard is real but sits on the OTHER side. With
direct_tool_response_mode:"deferred" (spec 102), /mcp/all serves {"type":"object"}
for every upstream tool while REST keeps serving real ones. Measured: /mcp/all
drops 4805 -> 2939 tokens, -38.8%. A live fleet read from that surface
under-counts the direct menu and FLIPS THE SIGN of the direct delta. Hence the
stub guard, which is the point of the primitive rather than a detail of it.
Also verified: a live tools/list capture reproduces bench's in-process
ProxyModeToolDefs derivation exactly, to the token (5783 and 3781).
DEFECTS FROM THE ADVERSARIAL PASS, and one finding rejected:
- The guard now runs in ResolveFleet's live branch, not only inside
FetchToolsList. A check living in one FleetSource implementation protects only
that implementation; ResolveFleet is where every live fleet passes exactly once.
- An all-empty-properties population is REFUSED with an honest cause. The shape
is genuinely ambiguous — a stubbed surface and an honestly parameter-less tool
emit identical bytes — so the error says so instead of blaming the supervisor
stub, and points at -corpus-v2 for a fleet that really is parameter-less.
Refusal is still right: admitting a stubbed fleet inflates the headline, while
refusing an honest one costs one clear message.
- isStubSchema now treats an undecodable schema as a stub. A schema of `true`,
`[]` or `0` previously passed as REAL and was priced at ~1 token.
- FleetShape carries SchemalessTools. The guard only refuses an ALL-stub fleet, so
a 3-of-45 partial regression flowed through leaving no trace; the count is now
in the report and the JSON schema.
- The MCP client is closed when Start fails, symmetric with the Initialize path.
- The "__" mis-split's cost consequence is documented: a server name containing
"__" leaves a stale prefix that the arms then re-prepend, double-charging it.
REJECTED, with the reasoning recorded in the code so it is not re-fixed: the
review called the guard's absence on the frozen-corpus path an asymmetry defect.
It is not. corpus_v1 is 45 tools with ZERO schemas by design and is the default
-corpus value, and CountToolWithSchema is documented to count schema-less tools
identically so the two mix. Guarding there would reject the project's own default
corpus. The asymmetry tracks a real difference: a live surface can be
misconfigured out from under the operator, a committed corpus is a reviewed
artifact whose shape is visible in the diff.
THREE VACUOUS TESTS FIXED, each mutation-verified after the change:
- the stub-fleet test injected the sentinel error from a fake source, so it
passed with guardToolSchemas deleted entirely. It now feeds real deferred-mode
tool definitions through the real guard.
- the API key was never asserted to reach the wire; `if false` on the
key-appending branch left the suite green.
- the repeated-cursor and nil-transport guards were untested; disabling either
left the suite green.
Gates: go build ./... clean; go test ./bench/... and -race green; golangci-lint
v2 0 issues.
Deploying mcpproxy-docs with
|
| Latest commit: |
52fb0ea
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://a8c1c372.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://103-fleet-and-decomp.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 33420945393 --repo smart-mcp-proxy/mcpproxy-go
|
Dumbris
added a commit
that referenced
this pull request
Sep 2, 2026
… token-bench truth-sync (#1200) * chore(roadmap): raise mcp-2026-upgrade to P1 (mcp-go v1.0.0 stable), add activity-storage-bounds, sync token-bench - mcp-2026-upgrade: P3 -> P1. mark3labs/mcp-go v1.0.0 stable shipped 2026-09-02; go.mod still pins v0.57.0. Next step is speckit.plan (#532). - action-log-transparency: new task activity-storage-bounds tracking #1173/#1174 (response truncation never wired), #1176 (unbounded per-server tool_calls buckets) and #1175 (omitempty zero erasure). - token-bench-harness: in_progress -> done; #1141/#1147/#1151/#1153/#1160 are all merged. Clears both --check-github errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(roadmap): keep merged token-bench PRs on the done task, not the in_progress epic Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
bench/mcptools.go— atools/listfetch mapping a live proxy's catalog onto[]bench.Toolwith real schemas — and wires it as a second fleet source, so-replay <jsonl> -proxy <url>works alongside-corpus-v2.The fleet input stays mandatory; this adds a way to supply one, not a way to omit it. Supplying both is an error — two fleets is ambiguous, and silently preferring one would make the report's
fleet_ida claim about definitions it wasn't priced over.I had the premise backwards, and the research phase caught it
I told the team REST
/api/v1/toolsserves stub schemas. It doesn't. MCP-3167 was already fixed ininternal/contracts/converters.go, and a live probe found 42/45 real schemas — the 3 empties being genuinely parameter-less tools. Neither withholding branch inlive_report.gofires today; a live run emitsauthoritative_headline=true.The hazard is real but sits on the other side. With
direct_tool_response_mode: "deferred"(spec 102),/mcp/allserves{"type":"object"}for every upstream tool while REST keeps serving real ones:/mcp/all/api/v1/toolsA live fleet read from that surface under-counts the direct menu and flips the sign of the direct delta. That's why the stub guard is the primitive rather than a detail of it.
Also verified: a live
tools/listcapture reproduces bench's in-processProxyModeToolDefsderivation exactly, to the token (5783 and 3781).Defects from the adversarial pass
ResolveFleet's live branch, not only insideFetchToolsList. A check living in oneFleetSourceimplementation protects only that implementation.isStubSchematreats an undecodable schema as a stub. A schema oftrue,[]or0previously passed as REAL and was priced at ~1 token.FleetShape.SchemalessTools— the guard only refuses an all-stub fleet, so a 3-of-45 partial regression flowed through leaving no trace. Now in the report and the JSON schema.Startfails; the"__"mis-split's double-charging consequence documented.One finding rejected, with the reasoning left in the code
The review called the guard's absence on the frozen-corpus path an asymmetry defect. It isn't.
corpus_v1is 45 tools with zero schemas by design and is the default-corpusvalue;CountToolWithSchemais documented to count schema-less tools identically so the two mix. Guarding there would reject the project's own default corpus.The asymmetry tracks a real difference: a live surface can be misconfigured out from under the operator; a committed corpus is a reviewed artifact whose shape is visible in the diff.
Three vacuous tests fixed, each mutation-verified
guardToolSchemasdeleted entirely. Now feeds real deferred-mode definitions through the real guard.if falseon the key-appending branch left the suite green.Verification