Skip to content

chore: link spec 103 from the token-bench epic; raise the review cap to 10 - #1140

Merged
Dumbris merged 1 commit into
mainfrom
roadmap-spec-link-and-cap
Aug 31, 2026
Merged

Dumbris merged 1 commit into
mainfrom
roadmap-spec-link-and-cap

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member

Two follow-ups from the 2026-08-31 roadmap truth-sync (#1136) and spec-103 work (#1137, #1139).

1. roadmap.yaml — link spec 103

The token-bench epic now carries spec: specs/103-token-bench, so ROADMAP.md renders its real progress badge (0/64) instead of a blank cell.

The note records what the spec settled, because three of its 13 cross-model review rounds changed the design rather than the wording — a future reader of the epic shouldn't have to rediscover them:

  • A recording carries no prompt, conversation or completion oracle, so replay cannot show agent behaviour. US1 (deterministic cost) and US2 (live agent loop) are now separate user stories.
  • Replay needs a fleet input — the activity export has no fleet snapshot, and a menu is a property of the tool definitions.
  • Bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost.

Also records that the mode matrix is 5 distinct behaviours, not a 3×2×2 product.

2. CLAUDE.md — escalation trigger 4 was doubly stale

It capped cross-model review at 5 rounds and named Codex as the reviewer. Neither is current:

  • The cap is 10 (maintainer decision 2026-08-31).
  • The reviewer is the opencode CLI with github-copilot/gpt-5.6-sol (maintainer directive 2026-08-14; Codex is not to be used).

The rationale for the raise is recorded inline: round 5 on #1136 caught a real defect the first four missed — an epic marked done while its spec's gating FR was unmet, which is the exact failure that PR existed to fix. Stopping at 5 would have shipped it.

Verification

python3 scripts/gen-roadmap.py --check          # up to date
python3 scripts/gen-roadmap.py --check-github   # 0 errors, 0 warnings

…to 10

Two follow-ups from the 2026-08-31 roadmap truth-sync and spec-103 work.

1. roadmap.yaml: the token-bench epic now carries spec: specs/103-token-bench,
   so ROADMAP.md renders its real progress badge (0/64) instead of a blank. The
   note records what the spec settled, because three of its 13 cross-model review
   rounds changed the DESIGN rather than the wording and a future reader of the
   epic should not have to rediscover them:
     - a recording carries no prompt, conversation or completion oracle, so
       replay CANNOT show agent behaviour — US1 (deterministic cost) and US2
       (live agent loop) are now separate user stories;
     - replay needs a FLEET INPUT, because the activity export has no fleet
       snapshot and a menu is a property of the tool definitions;
     - bodies-off yields menu costs plus one cross-mode delta, never an absolute
       workload cost.
   Also records that the matrix is 5 distinct behaviours, not a 3x2x2 product.

2. CLAUDE.md escalation trigger 4 was doubly stale: it capped cross-model review
   at 5 rounds and named Codex as the reviewer. The cap is 10 (maintainer
   decision 2026-08-31) and the reviewer is the opencode CLI with
   github-copilot/gpt-5.6-sol (maintainer directive 2026-08-14, Codex not to be
   used). The rationale for the raise is recorded inline: round 5 on #1136 caught
   a real defect the first four rounds missed — an epic marked done while its
   spec's gating FR was unmet, which is the exact failure that PR existed to fix.

gen-roadmap.py --check and --check-github both clean.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4ce5761
Status: ✅  Deploy successful!
Preview URL: https://d364ac14.mcpproxy-docs.pages.dev
Branch Preview URL: https://roadmap-spec-link-and-cap.mcpproxy-docs.pages.dev

View logs

@Dumbris
Dumbris enabled auto-merge (squash) August 31, 2026 10:44
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Dumbris
Dumbris merged commit 0750fcd into main Aug 31, 2026
47 checks passed
Dumbris added a commit that referenced this pull request Aug 31, 2026
Rebased onto main, which now carries the spec link from #1140, so ROADMAP.md
regenerates with the epic's real progress badge (33/64) instead of a blank.

Flipped token-bench and token-bench-harness from todo to in_progress with the
implementation PR linked. US1 — the deterministic replay half — is shipped and
independently useful; the harness task stays in_progress because its measured
success and retry rates belong to US2, which is gated on a pinned model.

gen-roadmap.py --check and --check-github both clean.
Dumbris added a commit that referenced this pull request Aug 31, 2026
…ded workloads (#1141)

* feat(bench): spec 103 foundations — export byte fields, truncation flag, report blocks, mode matrix, replay loader

Spec 103 (token-bench) tasks T001-T018 and T022-T026. Five units were built in
parallel on disjoint file sets, then integrated here.

PRODUCTION CHANGES (two, both small, both required before US1 can be honest):

1. request_bytes/response_bytes now reach the activity export
   (internal/contracts/activity.go, internal/httpapi/activity.go). They exist on
   the storage record as pre-truncation measurements but never reached the
   export contract. Copied UNCONDITIONALLY rather than inside the includeBodies
   guard: they are sizes, not content, and bodies-off — the default — is exactly
   the case where they are the only cost signal left. omitempty is load-bearing:
   an absent key means UNKNOWN and routes the record to exclusion accounting,
   whereas a present zero would read as a costless call and understate the
   workload.

2. The truncation flag now reaches the record. wasTruncated was computed in the
   retrieve_tools handler and only logged; it now travels the same two hops the
   non-internal path has always used — Runtime.EmitActivityInternalToolCallTruncated
   carries "response_truncated" in the event payload, and
   ActivityService.handleInternalToolCall reads it back onto
   ActivityRecord.ResponseTruncated. Both new methods delegate from the old
   signatures, so every existing call site still compiles.

   This matters because the activity log deliberately stores the FULL
   pre-truncation retrieve_tools response while the agent consumed only the cut
   text. Without the flag a truncated record is indistinguishable from a complete
   one, and anything recomputing cost tokenizes text the agent never paid for —
   overstating what mcpproxy cost, which is the one direction of error this
   benchmark exists to prevent.

BENCH ADDITIONS:

- bench/reportv2.go: the three spec-103 blocks (replay, agent_loop,
  payload_decomposition) as additive optional fields, each with its own
  accounting_source and each row with its own provenance. The document-level
  Tokenizer field is deliberately NOT narrowed — that would be a meaning change
  requiring a report_version bump. Blocks declared in the reviewed JSON schema.
- bench/modematrix.go: the 5 valid cells, with the other 7 combinations of the
  3-axis product rendered as skip rows naming the cell each collapses onto, plus
  the FR-016 capability conditions. Reuses the existing Skipped/SkipReason shape
  rather than inventing a second one.
- bench/replaycorpus/: the activity-JSONL loader. It imports NOTHING from bench —
  bench/corpusio imports bench, so mirroring that would have made bench/replay.go
  uncompilable through an import cycle. A go/parser guard test enforces the rule
  in the package where the mistake would be made, because violating it does not
  break this package's own build.

  Privacy is enforced in code, not just documented: BodyPolicy's zero value is
  bodies-off, bodies-on must be named and prints a warning that the export path
  does not mask, and LoadFile refuses any path inside the repository working tree
  before opening it. Tokenization happens inside the package and emits counts
  only, so no body text can cross the boundary.

CI: tiktoken cache populated (not merely named) in bench.yml, and a PR-triggered
workflow asserting no generated report is tracked via git ls-files — bench.yml
runs only on v* tags and could never have blocked a PR.

oas/swagger.yaml + oas/docs.go regenerated for the two new fields. Note
make swagger-verify is not wired into any workflow, so CI would not have caught
the drift.

Verified: go build ./... clean; go vet clean; bench, replaycorpus, contracts,
httpapi, runtime and the internal/server truncation tests all pass. The
truncation test was red before the runtime wiring and green after, so it is a
real assertion rather than a vacuous one.

* feat(bench): spec 103 US1 — replay cost recomputation over a real recorded workload

Completes the spec 103 MVP: tasks T019-T021 and T027-T033, on top of the
foundations in the previous commit. 33/64 tasks done; US1 is independently
shippable and needs zero model spend.

WHAT REPLAY PRODUCES, and what it deliberately refuses to:

- A fleet input is MANDATORY. `-replay <jsonl>` alone exits 1 with a message
  explaining that a menu is a property of the tool definitions and the activity
  export carries no fleet snapshot, so a recording on its own computes nothing.
  This is an error, not a degraded run.
- Menu cost per cell, measured from the fleet input.
- NO absolute complete-workload cost for ANY cell — it includes every consumed
  response, and that is a body. Every cell carries the withheld flag plus a
  reason; rendering zero would read as "measured, and it cost nothing".
- The direct_full -> direct_deferred cross-mode delta, the one comparison whose
  identical call responses cancel. On the frozen 45-tool corpus this measures
  1296 tokens / 29.5%, independently matching the DirectDeferredArm's own
  documented figure.
- Every figure carries the FR-004 counterfactual label, enforced at emission
  rather than left to rendering.
- generated_at is pinned so two runs are byte-identical (SC-002).

TWO DEFECTS FOUND BY THE ADVERSARIAL VERIFICATION PASS, both fixed here:

1. The withheld reason was posture-blind. On a bodies-on run the two direct
   cells emitted a response figure while the SAME row printed "that text is
   absent from a bodies-off replay" — a caveat stating a false premise next to
   the number contradicting it. The reason now tracks the posture: bodies-on it
   explains that the figure prices the RECORDED traffic rather than the workload
   a mode would incur in general, which is why the absolute cost stays withheld
   for a different and true reason.

2. The ExclusionReport was only partially surfaced — just TotalDropped, folded
   into one row. The per-reason breakdown, the costs WITHHELD on records that
   did contribute, the flagged records and the orphaned sub-calls never reached
   the report. Bodies-on, a single withheld response silently collapsed a cell's
   response total to nil with nothing anywhere saying why, which is exactly the
   silent accounting SC-008 forbids. Added ReplayLoaderAccounting, declared in
   the JSON schema, with the three buckets kept SEPARATE: "12 exclusions"
   spanning dropped, withheld and flagged is uninterpretable, and an
   uninterpretable count satisfies the letter of "everything is counted" while
   destroying its purpose.

Also fixed a silent no-op: `bench -live -replay x` ran live mode and discarded
-replay with no output and no warning. Now a hard error — a benchmark that
quietly measures something other than what was asked is worse than one that
refuses.

Both new tests were mutation-tested to prove they are not vacuous: reverting the
posture-awareness fails the first, nil-ing the loader accounting fails the
second, and both pass again once reverted.

Gates: go build ./... clean; go test ./bench/... and -race both green;
golangci-lint v2 with .github/.golangci.yml reports 0 issues; the report
validates against report-v2.schema.json (test confirmed PASS, not skipped);
git ls-files bench/results is empty.

* fix(bench): an orphaned sub-call is not merely "unattributed"

Found by cross-model review of the implementation.

A sub-call whose parent fell outside the exported window was counted in
OrphanedSubCalls and THEN dropped as ReasonUnattributed — the same record
reported under two counters with two different meanings, while the field's own
doc comment claimed orphans are "KEPT at top level rather than dropped". That
claim was false for the common case: sandbox sub-calls inherit their session
from the parent, so losing the parent loses the session too, and the record fell
straight through to the unattributed drop.

Both are records with no unit of work, but they tell the operator different
things, and only one is actionable. An unattributed record never had a session.
An orphan had a parent that the EXPORT WINDOW cut off — a wider re-export brings
it back. Folding the second into the first hides the one exclusion reason
someone can actually do something about.

Now: ReasonOrphanedSubCall is its own drop reason, OrphanedSubCalls spans both
outcomes so the attribution loss stays visible either way, and the doc states
both cases instead of asserting the one that is often untrue.

Two tests, mutation-verified: routing the drop back to ReasonUnattributed fails
the first, and the second pins that an orphan which CAN stand alone is kept
rather than dropped, since dropping it would understate the workload.

go test ./bench/... green; golangci-lint v2 0 issues.

* test(bench): make the flags-computed-once assertion actually assert something

Found by cross-model review round 2, which cleared all three correctness
priorities (loader cost branches, replay aggregation, the four-hop truncation
path) and found this instead.

TestUsabilityFlagsAreComputedOnceAtLoad compared the flags before and after a
downstream read and asserted they were equal. It never asserted what they
should BE. Delete load-time classification entirely and before == after ==
Flags{}, so "nothing changed" still holds and the test reports green on a loader
that classifies nothing at all — a green light on exactly the failure it exists
to catch.

Now asserts Truncated is already set before testing stability. Mutation-verified:
removing the load-time assignment in flags.go fails it with
"load-time classification must already have set Truncated, got {Truncated:false ...}",
where before it passed.

Review verdict on the production code: no correctness issues in cost handling
(estimated/unavailable responses make the aggregate nil rather than adding zero;
a truncated internal response cannot reach the tokenizer; sensitivity blocks
both request and response tokenization), replay aggregation (AllCalls() used for
both call count and response totals; the delta is strictly direct_full minus
direct_deferred over the direct_full denominator), or the truncation path (the
handler's real result reaches ActivityRecord through both delegating overloads,
with the integration test proving both the true and false outcomes against real
storage).

* chore(roadmap): token-bench is in_progress — US1 shipped in #1141

Rebased onto main, which now carries the spec link from #1140, so ROADMAP.md
regenerates with the epic's real progress badge (33/64) instead of a blank.

Flipped token-bench and token-bench-harness from todo to in_progress with the
implementation PR linked. US1 — the deterministic replay half — is shipped and
independently useful; the harness task stays in_progress because its measured
success and retry rates belong to US2, which is gated on a pinned model.

gen-roadmap.py --check and --check-github both clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants