Repository navigation
chore: link spec 103 from the token-bench epic; raise the review cap to 10 - #1140
Merged
Merged
Conversation
…to 10
Two follow-ups from the 2026-08-31 roadmap truth-sync and spec-103 work.
1. roadmap.yaml: the token-bench epic now carries spec: specs/103-token-bench,
so ROADMAP.md renders its real progress badge (0/64) instead of a blank. The
note records what the spec settled, because three of its 13 cross-model review
rounds changed the DESIGN rather than the wording and a future reader of the
epic should not have to rediscover them:
- a recording carries no prompt, conversation or completion oracle, so
replay CANNOT show agent behaviour — US1 (deterministic cost) and US2
(live agent loop) are now separate user stories;
- replay needs a FLEET INPUT, because the activity export has no fleet
snapshot and a menu is a property of the tool definitions;
- bodies-off yields menu costs plus one cross-mode delta, never an absolute
workload cost.
Also records that the matrix is 5 distinct behaviours, not a 3x2x2 product.
2. CLAUDE.md escalation trigger 4 was doubly stale: it capped cross-model review
at 5 rounds and named Codex as the reviewer. The cap is 10 (maintainer
decision 2026-08-31) and the reviewer is the opencode CLI with
github-copilot/gpt-5.6-sol (maintainer directive 2026-08-14, Codex not to be
used). The rationale for the raise is recorded inline: round 5 on #1136 caught
a real defect the first four rounds missed — an epic marked done while its
spec's gating FR was unmet, which is the exact failure that PR existed to fix.
gen-roadmap.py --check and --check-github both clean.
Deploying mcpproxy-docs with
|
| Latest commit: |
4ce5761
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://d364ac14.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://roadmap-spec-link-and-cap.mcpproxy-docs.pages.dev |
Dumbris
enabled auto-merge (squash)
August 31, 2026 10:44
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Dumbris
added a commit
that referenced
this pull request
Aug 31, 2026
Rebased onto main, which now carries the spec link from #1140, so ROADMAP.md regenerates with the epic's real progress badge (33/64) instead of a blank. Flipped token-bench and token-bench-harness from todo to in_progress with the implementation PR linked. US1 — the deterministic replay half — is shipped and independently useful; the harness task stays in_progress because its measured success and retry rates belong to US2, which is gated on a pinned model. gen-roadmap.py --check and --check-github both clean.
Dumbris
added a commit
that referenced
this pull request
Aug 31, 2026
…ded workloads (#1141) * feat(bench): spec 103 foundations — export byte fields, truncation flag, report blocks, mode matrix, replay loader Spec 103 (token-bench) tasks T001-T018 and T022-T026. Five units were built in parallel on disjoint file sets, then integrated here. PRODUCTION CHANGES (two, both small, both required before US1 can be honest): 1. request_bytes/response_bytes now reach the activity export (internal/contracts/activity.go, internal/httpapi/activity.go). They exist on the storage record as pre-truncation measurements but never reached the export contract. Copied UNCONDITIONALLY rather than inside the includeBodies guard: they are sizes, not content, and bodies-off — the default — is exactly the case where they are the only cost signal left. omitempty is load-bearing: an absent key means UNKNOWN and routes the record to exclusion accounting, whereas a present zero would read as a costless call and understate the workload. 2. The truncation flag now reaches the record. wasTruncated was computed in the retrieve_tools handler and only logged; it now travels the same two hops the non-internal path has always used — Runtime.EmitActivityInternalToolCallTruncated carries "response_truncated" in the event payload, and ActivityService.handleInternalToolCall reads it back onto ActivityRecord.ResponseTruncated. Both new methods delegate from the old signatures, so every existing call site still compiles. This matters because the activity log deliberately stores the FULL pre-truncation retrieve_tools response while the agent consumed only the cut text. Without the flag a truncated record is indistinguishable from a complete one, and anything recomputing cost tokenizes text the agent never paid for — overstating what mcpproxy cost, which is the one direction of error this benchmark exists to prevent. BENCH ADDITIONS: - bench/reportv2.go: the three spec-103 blocks (replay, agent_loop, payload_decomposition) as additive optional fields, each with its own accounting_source and each row with its own provenance. The document-level Tokenizer field is deliberately NOT narrowed — that would be a meaning change requiring a report_version bump. Blocks declared in the reviewed JSON schema. - bench/modematrix.go: the 5 valid cells, with the other 7 combinations of the 3-axis product rendered as skip rows naming the cell each collapses onto, plus the FR-016 capability conditions. Reuses the existing Skipped/SkipReason shape rather than inventing a second one. - bench/replaycorpus/: the activity-JSONL loader. It imports NOTHING from bench — bench/corpusio imports bench, so mirroring that would have made bench/replay.go uncompilable through an import cycle. A go/parser guard test enforces the rule in the package where the mistake would be made, because violating it does not break this package's own build. Privacy is enforced in code, not just documented: BodyPolicy's zero value is bodies-off, bodies-on must be named and prints a warning that the export path does not mask, and LoadFile refuses any path inside the repository working tree before opening it. Tokenization happens inside the package and emits counts only, so no body text can cross the boundary. CI: tiktoken cache populated (not merely named) in bench.yml, and a PR-triggered workflow asserting no generated report is tracked via git ls-files — bench.yml runs only on v* tags and could never have blocked a PR. oas/swagger.yaml + oas/docs.go regenerated for the two new fields. Note make swagger-verify is not wired into any workflow, so CI would not have caught the drift. Verified: go build ./... clean; go vet clean; bench, replaycorpus, contracts, httpapi, runtime and the internal/server truncation tests all pass. The truncation test was red before the runtime wiring and green after, so it is a real assertion rather than a vacuous one. * feat(bench): spec 103 US1 — replay cost recomputation over a real recorded workload Completes the spec 103 MVP: tasks T019-T021 and T027-T033, on top of the foundations in the previous commit. 33/64 tasks done; US1 is independently shippable and needs zero model spend. WHAT REPLAY PRODUCES, and what it deliberately refuses to: - A fleet input is MANDATORY. `-replay <jsonl>` alone exits 1 with a message explaining that a menu is a property of the tool definitions and the activity export carries no fleet snapshot, so a recording on its own computes nothing. This is an error, not a degraded run. - Menu cost per cell, measured from the fleet input. - NO absolute complete-workload cost for ANY cell — it includes every consumed response, and that is a body. Every cell carries the withheld flag plus a reason; rendering zero would read as "measured, and it cost nothing". - The direct_full -> direct_deferred cross-mode delta, the one comparison whose identical call responses cancel. On the frozen 45-tool corpus this measures 1296 tokens / 29.5%, independently matching the DirectDeferredArm's own documented figure. - Every figure carries the FR-004 counterfactual label, enforced at emission rather than left to rendering. - generated_at is pinned so two runs are byte-identical (SC-002). TWO DEFECTS FOUND BY THE ADVERSARIAL VERIFICATION PASS, both fixed here: 1. The withheld reason was posture-blind. On a bodies-on run the two direct cells emitted a response figure while the SAME row printed "that text is absent from a bodies-off replay" — a caveat stating a false premise next to the number contradicting it. The reason now tracks the posture: bodies-on it explains that the figure prices the RECORDED traffic rather than the workload a mode would incur in general, which is why the absolute cost stays withheld for a different and true reason. 2. The ExclusionReport was only partially surfaced — just TotalDropped, folded into one row. The per-reason breakdown, the costs WITHHELD on records that did contribute, the flagged records and the orphaned sub-calls never reached the report. Bodies-on, a single withheld response silently collapsed a cell's response total to nil with nothing anywhere saying why, which is exactly the silent accounting SC-008 forbids. Added ReplayLoaderAccounting, declared in the JSON schema, with the three buckets kept SEPARATE: "12 exclusions" spanning dropped, withheld and flagged is uninterpretable, and an uninterpretable count satisfies the letter of "everything is counted" while destroying its purpose. Also fixed a silent no-op: `bench -live -replay x` ran live mode and discarded -replay with no output and no warning. Now a hard error — a benchmark that quietly measures something other than what was asked is worse than one that refuses. Both new tests were mutation-tested to prove they are not vacuous: reverting the posture-awareness fails the first, nil-ing the loader accounting fails the second, and both pass again once reverted. Gates: go build ./... clean; go test ./bench/... and -race both green; golangci-lint v2 with .github/.golangci.yml reports 0 issues; the report validates against report-v2.schema.json (test confirmed PASS, not skipped); git ls-files bench/results is empty. * fix(bench): an orphaned sub-call is not merely "unattributed" Found by cross-model review of the implementation. A sub-call whose parent fell outside the exported window was counted in OrphanedSubCalls and THEN dropped as ReasonUnattributed — the same record reported under two counters with two different meanings, while the field's own doc comment claimed orphans are "KEPT at top level rather than dropped". That claim was false for the common case: sandbox sub-calls inherit their session from the parent, so losing the parent loses the session too, and the record fell straight through to the unattributed drop. Both are records with no unit of work, but they tell the operator different things, and only one is actionable. An unattributed record never had a session. An orphan had a parent that the EXPORT WINDOW cut off — a wider re-export brings it back. Folding the second into the first hides the one exclusion reason someone can actually do something about. Now: ReasonOrphanedSubCall is its own drop reason, OrphanedSubCalls spans both outcomes so the attribution loss stays visible either way, and the doc states both cases instead of asserting the one that is often untrue. Two tests, mutation-verified: routing the drop back to ReasonUnattributed fails the first, and the second pins that an orphan which CAN stand alone is kept rather than dropped, since dropping it would understate the workload. go test ./bench/... green; golangci-lint v2 0 issues. * test(bench): make the flags-computed-once assertion actually assert something Found by cross-model review round 2, which cleared all three correctness priorities (loader cost branches, replay aggregation, the four-hop truncation path) and found this instead. TestUsabilityFlagsAreComputedOnceAtLoad compared the flags before and after a downstream read and asserted they were equal. It never asserted what they should BE. Delete load-time classification entirely and before == after == Flags{}, so "nothing changed" still holds and the test reports green on a loader that classifies nothing at all — a green light on exactly the failure it exists to catch. Now asserts Truncated is already set before testing stability. Mutation-verified: removing the load-time assignment in flags.go fails it with "load-time classification must already have set Truncated, got {Truncated:false ...}", where before it passed. Review verdict on the production code: no correctness issues in cost handling (estimated/unavailable responses make the aggregate nil rather than adding zero; a truncated internal response cannot reach the tokenizer; sensitivity blocks both request and response tokenization), replay aggregation (AllCalls() used for both call count and response totals; the delta is strictly direct_full minus direct_deferred over the direct_full denominator), or the truncation path (the handler's real result reaches ActivityRecord through both delegating overloads, with the integration test proving both the true and false outcomes against real storage). * chore(roadmap): token-bench is in_progress — US1 shipped in #1141 Rebased onto main, which now carries the spec link from #1140, so ROADMAP.md regenerates with the epic's real progress badge (33/64) instead of a blank. Flipped token-bench and token-bench-harness from todo to in_progress with the implementation PR linked. US1 — the deterministic replay half — is shipped and independently useful; the harness task stays in_progress because its measured success and retry rates belong to US2, which is gated on a pinned model. gen-roadmap.py --check and --check-github both clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two follow-ups from the 2026-08-31 roadmap truth-sync (#1136) and spec-103 work (#1137, #1139).
1.
roadmap.yaml— link spec 103The
token-benchepic now carriesspec: specs/103-token-bench, soROADMAP.mdrenders its real progress badge (0/64) instead of a blank cell.The note records what the spec settled, because three of its 13 cross-model review rounds changed the design rather than the wording — a future reader of the epic shouldn't have to rediscover them:
Also records that the mode matrix is 5 distinct behaviours, not a 3×2×2 product.
2.
CLAUDE.md— escalation trigger 4 was doubly staleIt capped cross-model review at 5 rounds and named Codex as the reviewer. Neither is current:
opencodeCLI withgithub-copilot/gpt-5.6-sol(maintainer directive 2026-08-14; Codex is not to be used).The rationale for the raise is recorded inline: round 5 on #1136 caught a real defect the first four missed — an epic marked
donewhile its spec's gating FR was unmet, which is the exact failure that PR existed to fix. Stopping at 5 would have shipped it.Verification