diff --git a/CLAUDE.md b/CLAUDE.md index cebe443ed..ad8c7e669 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -158,7 +158,6 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/ - **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md). ## Recent Changes +- 103-token-bench: Go 1.25 (`bench/`) + existing only — tiktoken-go v0.1.8 (cl100k_base), the mcp-go transport already used by `bench/mcpcaller.go`. MCPMark is an external SHA-pinned tool invoked out of process, not a module dependency. **No new dependencies.** - 102-schema-deferred: Go 1.25.5 module toolchain (`go.mod`), backend-only + existing only — mark3labs/mcp-go v0.57.0 (tool surface), santhosh-tekuri/jsonschema/v6 (Spec-085 validator reuse), zap, stdlib. **No new dependencies.** - 101-tpa-db: Go 1.25.5 module toolchain (`go.mod`), backend-only + stdlib only for the new work — `crypto/ed25519`, `crypto/sha256`, `syscall` (no ed25519 usage in the tree before this); existing bbolt (state), zap, Cobra. **No new dependencies.** -- 098-tools-preflight: Added Go 1.24 module toolchain (repo builds with local Go 1.25) + existing only — chi (httpapi), bbolt (storage), Bleve (index), zap (logging), Cobra (CLI), swaggo/swag v2 (contract regen). **No new dependencies.** -- 097-stored-scripts: Added Go 1.25 (os.Root/Root.ReadFile available — R1) + stdlib only (os.Root). **No new dependencies.** diff --git a/specs/103-token-bench/checklists/requirements.md b/specs/103-token-bench/checklists/requirements.md new file mode 100644 index 000000000..b3d988149 --- /dev/null +++ b/specs/103-token-bench/checklists/requirements.md @@ -0,0 +1,119 @@ +# Specification Quality Checklist: Token-efficiency benchmark — measured savings, published results + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-31 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +Validation ran over three iterations; the following were fixed rather than waived. + +**Iteration 1 — implementation detail leaked into a spec that must stay declarative.** +The draft named concrete files, flags, CLI commands, struct fields and PR numbers +(`bench/session.go`, `armRetryRates`, `ResponseTruncated`, `-corpus-v2`, `#747`). All were +removed from requirements and success criteria. This was the hardest constraint to hold, +because the feature's defining risk — duplicating a harness that already exists — is +naturally expressed in file names. It is instead expressed as the **Scope Boundary** table +in capability terms, which a non-implementer can still act on. + +**Iteration 2 — success criteria were not all verifiable without implementation knowledge.** +Rewritten so each states an outcome a reader can check: SC-003 (an outsider reproduces the +numbers), SC-006 (the benchmark detects a false saving), SC-007 (truncated input is never +counted silently). + +**Iteration 3 — scope was bounded but the *anti*-scope was not.** +Added Out of Scope, because the likeliest failure of this feature is not under-delivery but +sprawl into optimisation work the measurement merely suggests. + +**No [NEEDS CLARIFICATION] markers were used.** Three candidates were resolved by informed +default and recorded in Assumptions instead: +- *Which model to pin for agent-loop runs* — deferred to plan stage as a budget decision, + and the deterministic half is specified to stand alone without it (FR-015 constrains how + results are reported, not which model produces them). +- *Whether a completion signal can be derived from recorded sessions* — assumed derivable, + with an explicit fallback (report sessions lacking the signal rather than assuming + success). This is the assumption most likely to be wrong and should be validated first at + plan stage. +- *Which public suite to adopt* — deliberately left as a capability requirement (FR-020: + "at least one full agent-loop suite") rather than naming one, because the shortlist was + researched in 2026-08 and needs re-confirmation against the suites' current code before + being committed to. + +**Carried into planning**: the research pass flagged two items as unverified — whether the +candidate suites emit per-task token counts, and whether they can be pointed at a single +proxy endpoint. FR-020 and FR-022 are unsatisfiable if neither holds, so plan stage must +read the suites' code before committing to one. + +## Cross-model review round 1 (opencode gpt-5.6-sol, 2026-08-31) + +Seven findings, all verified against the tree and all applied. Two were **Critical and +changed the spec's premise**: + +1. **Replay cannot answer what the spec asked it to.** The activity record carries tool + calls, arguments, responses, status and work-session grouping — but no prompt, no + conversation, no model state and no completion oracle (`internal/storage/activity_models.go`). + So replaying a recording under a different mode cannot reveal how the agent would have + *behaved*: which calls it would attempt, whether the first was right, whether it finished. + The original US1 asked for exactly that and was unsatisfiable. + **Fix**: added a binding **Replay Boundary** section and split the old US1 into US1 + (deterministic counterfactual cost recomputation over real workload shape, no model) and + US2 (live agent loop under a pinned model, which owns every success/completion figure). + FR-004 now forbids inferring behaviour from a recording, and Out of Scope names it. + +2. **Reproducibility and privacy were in direct conflict, and the privacy side was absent.** + The body-inclusive export deliberately returns full unmasked values including detected + secrets — it is the compliance surface, not a browsing one (`internal/httpapi/activity.go`). + The spec said session contents must stay out of published artefacts but said nothing about + protecting them *during* replay, including transmission to model providers. + **Fix**: new FR-006..FR-009 (sensitive by default, never transmitted to third parties, + flagged records excluded or reduced, inputs live outside the repo with a documented + deletion step) plus SC-009. FR-030 now admits that figures from private sessions are not + independently reproducible and cannot be the sole support for a published claim. + +Also applied: + +3. **The headline metric did not actually penalise worse completion.** Tokens-per-completed-task + can *improve* while completion falls, if tokens fall faster — so the old SC-006 was not + generally true. **Fix**: FR-018 requires completion rate at equal prominence, FR-019 adds a + completion-regression threshold that overrides any token saving, and SC-007 now requires + demonstrating this with a deliberately degraded mode. +4. **SC-001 demanded results for "every mode combination"** while FR-017 requires skipping + invalid ones. **Fix**: scoped to every *valid* combination exercised by the live task set. +5. **Core terms were undefined** ("first attempt", "accepted", "retry", "unit of work", + "completion", token accounting source). **Fix**: a binding Definitions block, which also + separates corrective retries from infrastructure retries. +6. **Three FRs restated existing capabilities.** **Fix**: moved to an Inherited Constraints + block (IC-001..IC-004) so they are satisfied, not rebuilt. +7. **The capability axes were not a testable matrix.** **Fix**: FR-016 states them as binary + conditions over the routing modes where each is available, with the report enumerating + applicable rows. + +The reviewer independently confirmed the spec's current-state claims: nothing in the harness +replays activity exports, retry rates are literature-derived defaults, first-call success is +not measured, the three axes are not crossed, and the spec-102 figures (29.7% / 34.8% / +38.9% ceiling vs a 70% projection) are accurate. diff --git a/specs/103-token-bench/contracts/mode-matrix.md b/specs/103-token-bench/contracts/mode-matrix.md new file mode 100644 index 000000000..f8fe584de --- /dev/null +++ b/specs/103-token-bench/contracts/mode-matrix.md @@ -0,0 +1,89 @@ +# Contract: the mode matrix + +**5 distinct behaviours + 1 baseline. The 3-axis product's other 7 combinations are +configurable but redundant, each a skip-with-reason +row.** This file is the source of truth for FR-015, FR-016 and FR-017. + +## Valid cells + +| id | Endpoint | `tool_response_mode` | `direct_tool_response_mode` | Requires | +|----|----------|----------------------|------------------------------|----------| +| `retrieve_full` | retrieve_tools surface | `full` | n/a | — (default) | +| `retrieve_compact` | retrieve_tools surface | `compact` | n/a | spec 085 | +| `direct_full` | direct surface | n/a | `full` | — | +| `direct_deferred` | direct surface | n/a | `deferred` | spec 102 | +| `code_exec` | code-execution surface | forced `full` | n/a | `enable_code_execution: true` | +| `baseline` | none — all upstream tools loaded directly | n/a | n/a | FR-020 denominator | + +`baseline` is not an mcpproxy mode. It is the same agent doing the same tasks with every +upstream tool inline, and it is what every percentage is measured against. + +## The other 7 combinations of the 3-axis product (FR-017) + +The naive product is 3 routing modes x 2 discovery serializations x 2 direct serializations = +**12 combinations, which collapse onto 5 distinct behaviours.** + +They are **not impossible to configure** — they are *configurable and behaviourally redundant*, +because on each surface one or both serialization axes have no consumer. Calling them +"impossible" would be wrong; rendering them as zeros would be worse. Each is a +**skip-with-reason row naming the cell it collapses onto.** + +| Configured combination | Collapses onto | Reason code | +|---|---|---| +| `retrieve_tools` x full x **deferred** | `retrieve_full` | `axis-ignored` | +| `retrieve_tools` x compact x **deferred** | `retrieve_compact` | `axis-ignored` | +| `direct` x **compact** x full | `direct_full` | `axis-ignored` | +| `direct` x **compact** x deferred | `direct_deferred` | `axis-ignored` | +| `code_execution` x **compact** x full | `code_exec` | `forced-full` | +| `code_execution` x full x **deferred** | `code_exec` | `axis-ignored` | +| `code_execution` x **compact** x **deferred** | `code_exec` | `forced-full` + `axis-ignored` | + +### Why each axis is ignored where it is + +- **`tool_response_mode` governs exactly one surface**: resolved by + `effectiveToolResponseMode`, whose single production call is in the `retrieve_tools` handler + (`internal/server/mcp.go:1584-1585`). Neither the direct nor the code-execution surface reads + it — the direct surface has no `retrieve_tools` tool at all. +- **`direct_tool_response_mode` governs exactly one surface**: resolved by + `effectiveDirectToolResponseMode`. Note it has THREE production call sites, not one — + building the listing (`mcp_routing.go:100`), detecting serialization drift (`:1040-1045`) + and logging a reload (`:1071-1076`) — but all three concern the direct surface only, so the + collapse holds. The claim is one *surface*, not one *call site*. +- **`code_execution` forces full**: that surface overwrites the response mode with `full` and + blanks the detail parameter; `detail` is not in its schema. That is a genuine *override*, + distinct from an ignored axis, hence its own reason code. + +### Separately: a degenerate configuration + +`code_execution` with `enable_code_execution: false` is not part of the product above. It is a +configuration in which the surface can discover tools and call none of them. Skip it with +reason `degenerate`. + +## Rules + +1. **The routing-mode axis is selected by endpoint URL — no config change, no restart.** All + three routing-mode servers are built at startup and all three endpoints stay mounted + regardless of config (`internal/server/server.go:2514-2536`). +2. **The two serialization axes DO require config**, and each affects only its own surface. + A cell is therefore (URL, serialization-config) — **not URL alone**. + **Both hot-reload, so the matrix still crosses on ONE long-lived instance**, with a config + apply between serialization cells: `tool_response_mode` is read from the live snapshot on + every call (`internal/server/profile_resolver.go:49-64`, documented as taking effect "on + the very next call, without reconstructing the server"), and `direct_tool_response_mode` is + rebuilt on the `config.reloaded` event (`internal/server/server.go:569-597`). +3. **An ignored axis is recorded as not-applicable**, never as a default value — a + default would imply a measurement that was never taken. +4. **A skipped row carries a reason code AND the cell it collapses onto, and never renders + as a zero.** Reuse the existing + skipped-row shape (`Skipped` / `SkipReason` plus the skipped-result constructor) rather + than inventing a second one. +5. **Capability toggles** (batching, stored scripts, validate-before-dispatch) are binary + conditions applied to the cells where each is available, and the report enumerates which + cells each applies to. They are not additional axes of a product. +6. **Cell ids are stable** across runs and releases, so results remain comparable (FR-028). + +## Fleet shaping + +Every cell must be measured against the SAME fleet in a given comparison, and the fleet must +be large enough for proxy modes to differ from baseline. A single upstream server's toolset is +too small to show the asymptote; load the full fleet even when exercising one service's tasks. diff --git a/specs/103-token-bench/contracts/replay-input.md b/specs/103-token-bench/contracts/replay-input.md new file mode 100644 index 000000000..c14512266 --- /dev/null +++ b/specs/103-token-bench/contracts/replay-input.md @@ -0,0 +1,110 @@ +# Contract: replay input + +## Source + +`GET /api/v1/activity/export?format=json` — equivalently `mcpproxy activity export --format +json`. JSONL, one activity record per line. + +**CSV is not a valid input.** It drops `work_session_id`, arguments, response and every byte +field, so it can neither group units of work nor account tokens. + +## Required fields + +| Field | Use | +|---|---| +| `work_session_id` | Groups records into one unit of work (spec 082). Records lacking it are unattributed and reported as such. | +| `request_id`, `parent_id` | Joins code-execution sub-calls to the call that issued them. | +| `tool_name`, `server_name` | Resolves the call against a mode's tool surface. | +| `status` | success / error / blocked / rejected. | +| `response_truncated` | Marks a record whose stored content was cut. | +| `request_bytes`, `response_bytes` | **Byte lengths measured pre-truncation — not token counts.** Basis for an explicitly-estimated response cost only. | +| `has_sensitive_data` | Best-effort exclusion signal — see limitation below. | + +### Backend change this contract requires + +`request_bytes` and `response_bytes` exist on the storage record and are documented as +measured pre-truncation, but are **absent from the export contract**. They must be added to +the export DTO and copied in the export projection. + +Without them, FR-002 can only *exclude* truncated records. With them, a truncated record can +carry an explicitly-estimated response cost instead of being dropped. + +**They do NOT make response cost measurable.** Tokenizing requires the text; a byte length +yields an estimate at best. See the cost split below. + +**Known gaps in byte coverage** — records where even the byte-length estimate is unavailable, +so they must fall to exclusion accounting rather than silently contributing zero: + +- **Internal `retrieve_tools` emission carries no byte counts.** With bodies off its response + cost cannot even be estimated. It is *not* unrecoverable in general — the handler writes the + FULL response to the activity log (`internal/server/mcp.go:2061-2064`), so a bodies-on export + carries it. +- **Code-execution sub-calls record both byte counts as zero** + (`internal/server/mcp_code_execution.go:811-818`). A truncated sub-call therefore cannot use + the byte-length estimate either, and must be counted as excluded. +- **A truncated `retrieve_tools` record does not tell you what the agent PAID, and today you + cannot even identify one.** The activity log stores the FULL pre-truncation response while + the agent consumed truncated text, so tokenizing the stored text OVERSTATES the agent's cost. + Worse, the truncation is not recorded: `wasTruncated` is computed in the handler + (`internal/server/mcp.go:2019-2035`) and only logged — `emitActivityInternalToolCall` + (`:810`) has no truncation parameter, so the flag never reaches the activity record. + + **This makes a SECOND backend change necessary** for the exclusion rule to be implementable: + propagate the truncation flag onto internal tool-call activity. Without it the loader cannot + distinguish a truncated `retrieve_tools` record from a complete one and would silently + overstate. + + **Fallback if that change is not wanted**: exclude ALL internal `retrieve_tools` records from + response-cost accounting and report the exclusion count. Blunt, but honest — and it must be + an explicit decision, not a silent default. + +## Privacy contract + +1. **Replay requires a FLEET INPUT as well as the recording.** A menu cost is a property of the + tool definitions the agent was shown, and the export carries no fleet snapshot. Built-in + proxy tools alone are not the menu — under `direct` the menu is the whole upstream fleet. + So `-replay ` on its own can compute nothing; it must be paired with either a frozen + fleet corpus (the existing committed corpora already serve this) or a live proxy to read + the current fleet from. **A recording-only invocation is an error, not a degraded run.** + +2. **Default is bodies-off. What that yields, per cell:** + + | Mode cell | Menu cost | Absolute complete-workload cost | Cross-mode delta | + |---|---|---|---| + | `direct_full` | measured (needs fleet input) | **NO** — response text absent | **measured** vs `direct_deferred` | + | `direct_deferred` | measured (needs fleet input) | **NO** — response text absent | **measured** vs `direct_full` | + | `code_exec` | measured (needs fleet input) | **NO** | not comparable bodies-off | + | `retrieve_full` | measured (needs fleet input) | **NO** | **NO** — the mode changes the response body | + | `retrieve_compact` | measured (needs fleet input) | **NO** | **NO** — same | + + **No cell has a measured ABSOLUTE complete-workload cost bodies-off**, because complete + workload includes every consumed response and that text is absent. What bodies-off does give + is (a) menu cost per cell, and (b) the cross-mode DELTA between the two direct cells, whose + identical call responses cancel out of the comparison. That delta is the honest bodies-off + headline; an absolute figure is not available. + + **What the recording contributes** is the *recorded call shape* — call sequence, tool mix and + call counts — evaluated against the supplied fleet input. It does not contribute fleet size. + + **Consequence to state in every replay report**: a replay scores a recorded workload against + the SUPPLIED fleet, not the fleet as it stood when the session was recorded. Internally valid + across modes; not a historical reconstruction. + +3. **Bodies-on is a separate, explicit opt-in** and prints a warning. The export path does + **not** mask: masking is wired into the list and detail handlers only, so a bodies-on + export is raw and unmasked by design — it is the compliance surface. +4. **`has_sensitive_data` is not a guarantee.** There is no persisted sensitivity field: the + flag is derived at export from detection metadata added asynchronously after initial + persistence, so a freshly exported record may be sensitive but not yet flagged. + Exclude-by-flag is a best-effort reducer and must be documented as such wherever relied on. +5. **Nothing crosses the loader boundary but counts.** Sessions and calls surrender sizes, + statuses and derived measurements. No content reaches a report, a dashboard, or any + third-party service including model providers. +6. **Inputs live outside the repository**, are never committed, and the documented procedure + tells an operator how to delete them. + +## Loader output + +Grouped `ReplaySession` values carrying usability flags computed once at load: +`truncated`, `bodies_missing`, `sensitive`, `unreplayable`. Every exclusion is counted and +reported (FR-003, SC-008). Silence is never success. diff --git a/specs/103-token-bench/contracts/report-v2-additions.md b/specs/103-token-bench/contracts/report-v2-additions.md new file mode 100644 index 000000000..1af0b4557 --- /dev/null +++ b/specs/103-token-bench/contracts/report-v2-additions.md @@ -0,0 +1,94 @@ +# Contract: report v2 additions + +## Versioning + +**No `report_version` bump.** New blocks are additive optional top-level fields with +`omitempty`, plus matching optional properties declared in the v2 JSON schema. A version bump +is reserved for changing an existing field's meaning or shape; this feature changes none. + +Precedent: the latency block's nested fields were added beside the existing flat fields the +same way. + +> The schema has no `additionalProperties: false`, so an undeclared block would validate +> silently. Do not rely on that — the schema file is the reviewed contract. Declare every new +> block. + +## New blocks + +### `replay` (deterministic, US1) + +Per-cell recomputed cost for a real recorded workload, plus the exclusion accounting: +sessions supplied, sessions used, and a count per exclusion reason. Carries the fleet shape. + +### `agent_loop` (live, US2) + +Per-cell tokens-per-completed-task, completion rate, first-attempt success rate, corrective +and infrastructure retry counts, run count and spread. **Carries its own accounting-source +field** (provider + pinned model). + +### `payload_decomposition` + +Per fleet shape: attribution across names, descriptions, annotations and schemas; the +recomputed achievable ceiling; and an explicit `confirmed` / `corrected` verdict on spec 102. + +## Two required changes to existing structures + +### 1. Scope the tokenizer identity + +The report envelope carries **one** tokenizer identity for the whole document. Once +provider-reported usage enters the same envelope, that field silently claims the entire report +was tokenizer-counted. + +**Do not redefine the existing field.** This contract states that a version bump is required +whenever an existing field's meaning changes, and narrowing the document-level tokenizer to +"the deterministic sections only" would be exactly such a change — the two rules contradict. + +Resolve it additively instead: leave the existing field's meaning intact (it names the +deterministic estimator, which remains true of every section that has one), and give each new +block its own explicit `accounting_source` naming either that estimator or the provider plus +pinned model. A reader then never has to infer scope from a document-level field. + +If a future change really must narrow the existing field, bump the version — that is what the +rule is for. + +### 2. Add per-row provenance + +Provenance today is **section-level**: a map from section key to +`measured` / `computed` / `estimated`, rendered as a badge on each section heading. + +FR-013 lets measured and estimated figures coexist *inside* one section, which the section +badge cannot express. Every new row type therefore carries its own `provenance` field, +constrained to the same three-value enum. + +This matters concretely: the per-arm retry-rate lookup returns `0.0` for unknown arms, which +is indistinguishable from a measured `0.0`. Without per-row provenance, one table can mix a +measured rate and a defaulted one under a single `estimated` badge. + +## Aggregation rules + +1. **Never sum across accounting sources.** A cross-source aggregate is **withheld with a + stated reason**, never computed — reusing the harness's existing withhold-rather-than-compute + pattern for non-authoritative headlines. +2. **A model-dependent figure needs `runs >= 4`** plus a spread, or it is not a headline. +3. **Every percentage carries its fleet shape.** +4. **Partial runs are marked and excluded from headlines.** + +## Enforcement gaps to close + +- **`SC-011` has no gate today.** "Reports are never committed" is gitignore plus convention; + no CI job or test fails on a committed report. + **`git status --porcelain bench/results` is NOT a valid gate** — porcelain does not list + ignored files, and it would also stay silent about a report that is already tracked. The + correct assertion is that no file under the results directory is tracked at all: + `test -z "$(git ls-files bench/results)"`. + **Placement matters as much as the assertion**: `bench.yml` triggers only on `v*` tags and + `workflow_dispatch`, so a gate there cannot block a pull request. Put it in a PR-triggered + required workflow, or state plainly that SC-011 remains post-merge detection only. +- **`SC-004` is not currently achievable offline.** The tokenizer fetches its vocabulary over + the network unless a cache directory env var is set, and nothing in the repo or CI sets it. + Setting the variable is necessary but **not sufficient** — it names a cache without filling + one. CI must also populate or restore that cache, and the reproduction procedure must tell + an outsider to warm it once with network access. +- **`SC-002` (byte-identical reruns) collides with the report's wall-clock `generated_at` + stamp.** Replay must pin or exclude that field, or every determinism check fails on it + alone. diff --git a/specs/103-token-bench/data-model.md b/specs/103-token-bench/data-model.md new file mode 100644 index 000000000..a9a6518ff --- /dev/null +++ b/specs/103-token-bench/data-model.md @@ -0,0 +1,189 @@ +# Phase 1 Data Model: token-bench (spec 103) + +Entities are described by what they hold and what must be true of them. Field names are +indicative; the binding shapes are in [contracts/](./contracts/). + +--- + +## ReplaySession + +One unit of real recorded work, reassembled from exported activity JSONL. + +| Field | Meaning | +|---|---| +| `work_session_id` | The spec-082 grouping key: one client, one project, across reconnects. This is the unit of work for US1. | +| `calls` | Ordered `ReplayCall` list. | +| `usability` | See below — computed once at load, never re-derived downstream. | +| `call_count`, `span` | Derived counts for reporting. | + +**Usability flags** (each a reason a session may be partly or wholly unusable): + +- `truncated` — one or more calls had their stored response truncated at capture. +- `bodies_missing` — the export was taken without bodies (the default), so content is absent. +- `sensitive` — one or more calls carry the sensitive-data flag. +- `unreplayable` — the tools referenced no longer resolve against the configured fleet. + +**Validation rules** + +- A session with `truncated` calls MUST NOT contribute a silent token total. It either + contributes an annotated figure derived from pre-truncation byte sizes, or is excluded and + counted in the exclusion report (FR-002, FR-003, SC-008). +- `sensitive` is a **best-effort** reducer. There is no persisted sensitivity field; the flag + is derived at export from detection metadata that is added asynchronously after initial + persistence, so its absence does not prove a record is clean. This limitation MUST be stated + wherever the flag is relied upon (Principle IV). +- A session MUST NOT carry its call contents into any report structure. Only counts, sizes and + derived measurements propagate (FR-006). +- **Therefore tokenization of response bodies happens INSIDE the loader**, not downstream: a + bodies-on run tokenizes within `replaycorpus` and emits token counts, so no text ever + crosses the boundary. A design that hands bodies to the scoring layer to tokenize would + violate the invariant above — the two cannot both hold. + +**State transitions**: `loaded → grouped → classified → {scored | excluded}`. Exclusion is +terminal and always counted. + +--- + +## ReplayCall + +One recorded tool call inside a session. + +| Field | Meaning | +|---|---| +| `tool_name`, `server_name` | Identity, used to resolve the call against a mode's tool surface. | +| `request_bytes`, `response_bytes` | **Byte lengths, measured pre-truncation — NOT token counts.** They support an explicitly-estimated response cost when bodies are absent; they cannot produce a measured one, because tokenizing requires the text. | +| `response_truncated` | Whether the stored content was cut. | +| `status` | success / error / blocked / rejected. | +| `parent_id` | Links a code-execution sub-call to the call that issued it. | +| `has_sensitive_data` | Derived at export from detection metadata, which is added asynchronously after initial persistence. Best-effort (see above). | + +**Validation rules** + +- **Tool-surface cost does not depend on recorded content** — it is computed from the fleet's + tool definitions under each mode. It is fully `measured` with bodies off, and it is what the + direct and code-execution cells' modes change. +- **NO cell yields an absolute complete-workload cost with bodies off**, because complete + workload includes every consumed response and that text is absent. Bodies-off yields menu + cost per cell (given a fleet input) and the cross-mode DELTA between the two direct cells, + whose identical call responses cancel. Absolute figures MUST be reported as unavailable, + never as zero. +- **Menu cost requires a FLEET INPUT** — a frozen fleet corpus or a live proxy. The export + carries no fleet snapshot, so a recording-only run computes nothing. +- **Response cost generally requires the text.** With bodies off it may only be an explicit + `estimated` figure derived from byte length; with bodies on it becomes `measured`. The two + MUST NOT be presented interchangeably. +- **The export carries no fleet snapshot**, so every replay figure is against the currently + configured fleet. Reports MUST state this: it is internally valid across modes but is not a + historical reconstruction. +- A call whose `parent_id` resolves MUST be attributed to its parent's session, so + code-execution sub-calls are neither double-counted nor orphaned. + +--- + +## ModeCell + +One point in the matrix. **There are exactly 5 distinct behaviours plus a baseline**; see +[contracts/mode-matrix.md](./contracts/mode-matrix.md). + +| Field | Meaning | +|---|---| +| `id` | Stable cell identifier used as a report key. | +| `endpoint` | Which mounted MCP endpoint serves this cell. Selects the routing-mode axis with no config change; the serialization axes still need config. | +| `discovery_serialization` | full / compact / not-applicable. | +| `direct_serialization` | full / deferred / not-applicable. | +| `capabilities` | Binary conditions applicable to this cell (batching, stored scripts, validate-before-dispatch). | +| `skipped`, `skip_reason`, `collapses_onto` | Populated for the 7 redundant combinations of the 3-axis product; each names the valid cell it collapses onto. | + +**Validation rules** + +- An ignored axis MUST be recorded as not-applicable, never as a default value — a default + would imply a measurement that was never taken (FR-017). +- A skipped row MUST carry a reason code AND the cell it collapses onto, and MUST NOT be + rendered as a zero. These combinations are configurable and redundant, NOT impossible. +- Cell identity MUST be stable across runs so results are comparable over time (FR-028). + +--- + +## RunRecord + +The raw outcome of one unit of work under one mode cell. This is the traceable unit behind +every headline figure (FR-029). + +| Field | Meaning | +|---|---| +| `cell_id`, `unit_id` | What was run, and under which cell. | +| `tokens` | Input / output / cache-read, kept separate (FR-014). | +| `accounting_source` | `deterministic-tokenizer` or `provider-usage`. **Load-bearing** — records from different sources must never be summed. | +| `attempts` | Ordered attempt outcomes for first-attempt-success derivation. | +| `retries_corrective`, `retries_infrastructure` | Counted separately (FR-011). | +| `completion` | The suite's pass verdict, or `no-signal`. Never inferred. | +| `partial` | True if the run did not finish; such records are excluded from headlines (FR-032). | + +**Validation rules** + +- `accounting_source` is mandatory. A record without it cannot enter any aggregate. +- `completion: no-signal` excludes the record from completion-dependent figures rather than + counting it as either success or failure. +- For provider-sourced records, the number of provider responses summed MUST be recorded, so a + partially-captured unit is distinguishable from a genuinely cheap one. + +--- + +## Measurement + +An aggregate over `RunRecord`s, and the only thing a report renders. + +| Field | Meaning | +|---|---| +| `value`, `unit` | The figure. | +| `provenance` | measured / computed / estimated — **per row**, not only per section. | +| `accounting_source` | Inherited from its records; an aggregate spanning sources is forbidden. | +| `runs`, `spread` | Required for any model-dependent figure (FR-021). | +| `fleet_shape` | Tool count + definition-size distribution the figure holds for (FR-016 / IC-004). | +| `withheld`, `withheld_reason` | Set instead of a value when the figure cannot be computed honestly. | + +**Validation rules** + +- A cross-source aggregate MUST be **withheld with a stated reason**, never computed. This + reuses the harness's existing withhold-rather-than-compute pattern. +- A model-dependent figure with `runs < 4` MUST NOT be marked as a headline (FR-021). +- A figure without `fleet_shape` MUST NOT be published as a percentage. +- `provenance` at row level is required because FR-013 permits measured and estimated figures + to coexist inside one section — the existing section-level badge cannot express that. + +--- + +## PayloadDecomposition + +Attribution of tool-definition cost for one fleet shape (FR-024, FR-025). + +| Field | Meaning | +|---|---| +| `fleet_shape` | Which corpus/fleet this describes. | +| `share_names`, `share_descriptions`, `share_annotations`, `share_schemas` | The four attributions. | +| `achievable_ceiling` | Recomputed **per corpus** — never carried forward as a constant. | +| `spec102_verdict` | Explicit `confirmed` or `corrected`, with the delta. | + +**Validation rule**: the ceiling MUST be recomputed for each fleet shape. Reusing a previously +published ceiling is precisely the error spec 102 made when it projected from one corpus. + +--- + +## Relationships + +```text +ReplaySession 1──* ReplayCall +ReplaySession *──* ModeCell → RunRecord (US1: deterministic, one record per pair) +TaskSet *──* ModeCell → RunRecord (US2: live loop, k runs per pair) +RunRecord *──1 Measurement (aggregated, never across accounting sources) +FleetShape 1──1 PayloadDecomposition +``` + +## Invariants that must hold across the whole model + +1. **No recorded content leaves the loader.** Sessions and calls surrender counts and sizes; + nothing downstream can reach a body. +2. **Accounting sources never mix.** Enforced at aggregation by withholding. +3. **Silence is never success.** Truncated, partial, no-signal and skipped are each distinct + from zero and each are reported. +4. **Every published percentage carries its fleet shape.** diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md new file mode 100644 index 000000000..ad10a0d0b --- /dev/null +++ b/specs/103-token-bench/plan.md @@ -0,0 +1,249 @@ +# Implementation Plan: Token-efficiency benchmark — measured savings, published results + +**Branch**: `103-token-bench` | **Date**: 2026-08-31 | **Spec**: [spec.md](./spec.md) +**Input**: Feature specification from `/specs/103-token-bench/spec.md` + +## Summary + +Add two measurement capabilities to the existing `bench/` harness, and one small backend +contract fix that makes the first of them honest. + +1. **US1 — deterministic replay** (`-replay` PLUS a fleet input): load exported activity JSONL, + group it into units of work, and recompute menu cost per mode cell and the direct-cell delta + for that real workload shape. **A fleet input is mandatory** — a frozen fleet corpus or a + live proxy — because a menu is a property of the tool definitions and the export carries no + fleet snapshot. No model. Reproducible **modulo the report's `generated_at` stamp**, which + must be excluded or pinned before any byte-identical comparison (see Risks). +2. **US2 — live agent loop**: run MCPMark against mcpproxy under each mode, capturing + provider-reported token usage and per-task pass verdicts, to produce tokens per + *completed* task, first-attempt success and retry rates. + +The research pass changed three things materially versus the spec's assumptions, all in the +direction of less work: + +- **The mode matrix is 5 distinct behaviours, not 12.** The three axes are not a free product: + each serialization axis has exactly one consumer, so the other 7 combinations are + configurable but **behaviourally redundant** — each collapses onto one of the 5. They are + reported as skip-with-reason rows naming the cell they collapse onto, never as zeros and + never as "impossible". +- **The routing-mode axis costs nothing to cross.** All three routing-mode servers are built + at startup and permanently mounted, so that axis is selected by endpoint URL with no config + change and no restart. The two serialization axes still require config, but **both + hot-reload**, so the whole matrix still crosses on one long-lived instance with a config + apply between serialization cells. +- **MCPMark needs one `elif` to point at mcpproxy**, and already emits per-task token usage, + pass verdicts, turn counts and pass^k. The spec's fallback plan (build an in-repo task set) + is not needed. + +## Technical Context + +**Language/Version**: Go 1.25 (`bench/`, `internal/`); Python 3 for the pinned external suite +**Primary Dependencies**: existing only — `tiktoken-go` (pinned `v0.1.8`, `cl100k_base`), +`mark3labs/mcp-go` transport via `bench/mcpcaller.go`, chi/bbolt/zap unchanged. +**No new Go module dependencies.** MCPMark is an external, SHA-pinned tool invoked out of +process, not vendored and not a module dependency. +**Storage**: N/A for the harness. Replay inputs are operator-supplied JSONL files that live +outside the repository and are never committed. +**Testing**: `go test ./bench/...` (unit + invariant + schema-conformance), plus the existing +`reportv2_e2e_test.go` JSON-schema validation path. +**Target Platform**: developer machine + CI (ubuntu-latest) for the deterministic half; the +live half is operator-run and never CI-gated, because it costs model spend. +**Project Type**: single Go project; `bench/` is an existing in-repo tool tree. +**Performance Goals**: N/A — this is a measurement tool. The relevant budget is model spend, +not latency. +**Constraints**: reports are never committed; the deterministic half must be byte-reproducible +and must run offline; no recorded session content may reach a report or a third party. +**Scale/Scope**: 5 matrix cells + 1 baseline; MCPMark's 51 credential-free tasks +(filesystem 30 + postgres 21) as the clean core, 127 total if credentialed services are added. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design.* + +| Principle | Assessment | +|---|---| +| **I. Performance at Scale** | Two production-touching changes, both trivial. (a) Two integers added to an export DTO — off the request path. (b) Propagating an already-computed `wasTruncated` boolean into the internal tool-call activity emit, which IS on the `retrieve_tools` request path: it passes a value the handler already has, adds no computation and no I/O. If even that is unwanted, the blanket-exclusion fallback removes it entirely at the cost of coverage. | +| **II. Actor-Based Concurrency** | No new concurrency in production code. The harness is a batch tool. | +| **III. Configuration-Driven** | PASS: the matrix is crossed by endpoint URL plus existing config fields, adding no benchmark-only configuration to the product. | +| **IV. Security by Default** | **The binding gate for this feature.** Replay inputs are raw user traffic; the export path deliberately does not mask. Design must default to bodies-off and must never transmit recorded content off-box. See Security Design below. | +| **V. TDD** | PASS. Every unit below is specified test-first; the deterministic half is fully unit-testable with no proxy, following the `flipgate.go` precedent of parameterising the driver over a function type. | +| **VI. Documentation Hygiene** | PASS. `bench/README.md` is the feature's user-facing doc and must gain the replay and live-loop sections; `CLAUDE.md` needs no change (no new commands at the top level, no architecture change). | + +**No violations. Complexity Tracking is therefore empty and omitted.** + +### Security Design (Principle IV) + +Three findings drive this and each becomes a design rule: + +1. **The export path never masks.** `maskActivityPayloads` is wired into the list and detail + handlers only, not into export. Bodies-on export is raw and unmasked. + → **Rule**: replay defaults to bodies-off. Bodies-on is a separate, explicitly-opted-in + mode with a printed warning. +2. **`has_sensitive_data` is set asynchronously after the record is persisted.** A freshly + exported record can be sensitive but not yet flagged. + → **Rule**: exclude-by-flag is a best-effort reducer and MUST NOT be described as a + guarantee, in code comments or in the published methodology. +3. **Bodies-off yields menu costs and one cross-mode delta — never an absolute workload cost.** + Complete workload includes every consumed response, and that text is absent bodies-off, so + NO cell has a measured absolute figure. What it does give is menu cost per cell, plus the + delta between the two direct cells, whose identical call responses cancel in the comparison. + The recorded byte sizes are byte *lengths*, not token counts, and support only an estimate. + → **Rule**: bodies-off is the default and reports menu costs plus the direct-cell delta; + absolute workload figures are reported unavailable, never as zero. + → **Rule**: response tokenization happens inside the loader, so no body text ever reaches + the scoring layer (otherwise the no-content-leaves-the-loader invariant cannot hold). + +## Phase 0: Research + +Complete. See [research.md](./research.md) for the full decisions, alternatives and evidence. +Five topics were resolved; the load-bearing outcomes: + +1. **Extension seams** — three exist. Spec 103 uses the "new measurement mode" seam twice + (flag on `cmd/bench` + driver in package `bench` + report block), following the spec-085 + flip-gate precedent. **No new arm is needed**, and no new binary. +2. **The matrix is 5 distinct behaviours**; the other 7 combinations of the 3-axis product + are configurable but redundant and collapse onto them. The routing-mode axis is selected by + endpoint URL with no restart; the serialization axes still need config. +3. **Two backend changes are needed, both small.** + (a) `request_bytes` / `response_bytes` exist on the storage record, measured pre-truncation, + but are absent from the export contract. Adding them lets a truncated record carry an + explicitly-estimated response cost instead of being dropped. They are byte lengths, **not** + token counts, so they do not make response cost measurable. + (b) **Truncation is not recorded for internal tool calls.** `wasTruncated` is computed in the + `retrieve_tools` handler and only logged; `emitActivityInternalToolCall` has no truncation + parameter, so the flag never reaches the activity record. Without propagating it the + exclusion rule for truncated `retrieve_tools` records is unimplementable and the loader would + silently overstate agent cost. The fallback — excluding every internal `retrieve_tools` + record from response-cost accounting — must be an explicit decision, not a silent default. +4. **MCPMark is adopted**, SHA-pinned, with one `elif` in its single MCP factory. Its per-task + `meta.json` feeds FR-010/011/012/018 directly. +5. **Token accounting stays split**: tiktoken for everything deterministic; provider `usage` + read off the model response for the live loop; the two never summed, enforced by the + harness's existing withhold-rather-than-compute pattern. + +### Risks carried into Phase 1 + +- **`ReportV2.Tokenizer` is a report-level singleton.** The moment provider-usage figures + enter the same envelope, a reader could take that field to describe them too. Resolve it + **additively** — leave the existing field's meaning intact and give every new block its own + `accounting_source` — rather than by narrowing the existing field, which would itself require + a version bump. See contracts/report-v2-additions.md. +- **Provenance is section-level, not per-figure.** FR-013 requires measured and estimated + figures to coexist inside one block, so new row types need their own provenance field. +- **tiktoken fetches its vocabulary over the network** unless `TIKTOKEN_CACHE_DIR` is set; + nothing in the repo or CI sets it. Setting the variable only *names* a cache — a first run + still downloads — so an offline reproduction additionally needs the cache **populated** + (warmed once with network, or restored/vendored in CI). Without that, SC-004 is not + achievable on a restricted network. +- **The report carries a wall-clock `generated_at` stamp.** SC-002 asks for byte-identical + replay reports; two runs will differ on that field alone unless replay pins or excludes it. + Decide which before writing the determinism test. +- **`RetryRateForArm` returns 0.0 for unknown arms**, indistinguishable from a measured 0.0. + Mixing a measured rate with a defaulted one under a single section badge is a live hazard + for FR-013. +- **SC-011 has no enforcement.** "Reports are never committed" is gitignore plus convention; + no CI job or test fails on a committed report. + +## Project Structure + +### Documentation (this feature) + +```text +specs/103-token-bench/ +├── plan.md # This file +├── spec.md # Feature specification +├── research.md # Phase 0 decisions + evidence +├── data-model.md # Phase 1 entities +├── quickstart.md # Phase 1 operator walkthrough +├── contracts/ +│ ├── report-v2-additions.md # additive report blocks + per-row provenance +│ ├── mode-matrix.md # the 5 valid cells + 7 skip reasons +│ └── replay-input.md # activity JSONL contract consumed by replay +├── checklists/ +│ └── requirements.md +└── tasks.md # /speckit.tasks output — NOT created here +``` + +### Source Code (repository root) + +```text +bench/ +├── replaycorpus/ # NEW — activity JSONL loader, grouping, usability flags +│ ├── load.go +│ ├── group.go # work_session_id grouping; parent_id ↔ request_id join +│ └── flags.go # truncated / bodies-missing / sensitive / unreplayable +├── replay.go # NEW — deterministic cost recomputation over the matrix +├── modematrix.go # NEW — the 5 valid cells, skip reasons, cell→URL mapping +├── agentloop.go # NEW — live-loop driver + MCPMark meta.json ingestion +├── reportv2.go # EXTEND — additive optional blocks; per-row provenance +├── live_report.go # EXTEND — per-block accounting_source (do NOT narrow Tokenizer) +├── session.go # EXTEND — measured rates supersede armRetryRates +└── cmd/bench/main.go # EXTEND — -replay and -agent-loop branches + +internal/contracts/activity.go # EXTEND — add request_bytes / response_bytes +internal/httpapi/activity.go # EXTEND — copy them in the export projection +internal/server/mcp.go # EXTEND — propagate wasTruncated to internal tool-call + # activity (or accept blanket exclusion) + +specs/083-discovery-profiler/contracts/report-v2.schema.json # EXTEND — declare new blocks +.github/workflows/bench.yml # EXTEND — TIKTOKEN_CACHE_DIR set AND cache populated + # SC-011 gate — bench.yml runs only on v* tags and + # workflow_dispatch, so a gate there cannot block a PR +``` + +**Structure Decision**: The feature lives almost entirely inside the existing `bench/` tree, +using its established seams. `bench/replaycorpus/` is a new sibling of `bench/corpusio/` +rather than an addition to it, because `corpusio` is scoped by its own doc comment to public +tool-retrieval corpora and produces `Corpus`/`GoldenSet` values — a session trace is neither. +`replay.go`, `modematrix.go` and `agentloop.go` sit in package `bench` (not a subpackage) +because they need `Tokenizer`, `ProxyToolsForMode` and the `EncodingArm` interface, and +`bench` cannot import `bench/arms` (a real cycle — arms import `bench`); they cross that +boundary with the same structural interface `RunArms` already uses. + +**Dependency rule for `bench/replaycorpus/` — decided, not deferred**: the existing +`bench/corpusio/` imports `bench`, so a `replaycorpus` that mirrored it could not be imported +by `bench/replay.go` without a second cycle. **Decision: `bench/replaycorpus` imports nothing +from `bench`.** It defines its own replay domain types, and `bench/replay.go` imports it. The +alternative (orchestrating from `cmd/bench`) is rejected because it would put measurement +logic outside the package that holds every other measurement. + +**`modematrix.go` composes, it does not reimplement.** The pieces it needs already exist — +mode constants and proxy catalogs (`bench/tokens.go`), full rendering +(`bench/arms/baseline.go`), compact measurement (`bench/flipgate.go`), deferred-direct +rendering (`bench/arms/directdeferred.go`) and MCP transport (`bench/mcpcaller.go`). It must +reach them through passed structural interfaces, never by re-deriving a serialization. + +The production-code changes are two, both deliberately minimal: the additive DTO fields plus +their export projection, and propagation of the already-computed truncation flag onto internal +tool-call activity. The first is justified because without it a truncated record must be +dropped rather than estimated; the second because without it a truncated `retrieve_tools` +record cannot be identified at all, and the loader would silently overstate agent cost. + +If the second is rejected, the blanket-exclusion fallback stands in — at the cost of excluding +every internal `retrieve_tools` record from response-cost accounting. That trade must be made +explicitly and reported, not defaulted into. + +## Phase 1: Design & Contracts + +Artifacts produced: [data-model.md](./data-model.md), [contracts/](./contracts/), +[quickstart.md](./quickstart.md). + +**Post-design constitution re-check: PASS.** The design touches the request path in exactly one +place — passing an already-computed boolean into an existing activity emit — and touches no +other request-path code. Its remaining production footprint is the additive export DTO fields +and their projection, both off the request path. It introduces no new abstraction beyond one loader package that mirrors an +existing sibling, and no module dependency. Principle IV is satisfied by the bodies-off default +rather than by a masking layer, which is the simpler of the two available designs. + +## Open decisions for the operator (not blockers) + +These need a human, and none blocks starting US1: + +1. **Pinned model and spend ceiling** for the live loop. US1 delivers alone without it. +2. **Whether cache-read tokens count toward "tokens per completed task."** FR-014 requires + tracking them separately; FR-018 does not say which composite the headline uses. This is a + definition decision, not a discovery. +3. **Whether to publish the measured tiktoken→Claude divergence.** The repo's caveat says up + to ~60%; Anthropic's guidance says ~15–20% on typical text. Neither is sourced in-repo and + they differ threefold. `count_tokens` settles it with no inference spend, and doing so turns + FR-022's tolerance into a measured number. diff --git a/specs/103-token-bench/quickstart.md b/specs/103-token-bench/quickstart.md new file mode 100644 index 000000000..0ddb07e90 --- /dev/null +++ b/specs/103-token-bench/quickstart.md @@ -0,0 +1,165 @@ +# Quickstart: token-bench (spec 103) + +Two independent halves. **Start with the deterministic one** — it needs no model spend and no +credentials, and it delivers value alone. It runs offline *once the tokenizer vocabulary is +cached*; see "Offline note" below, which is a real prerequisite rather than a footnote. + +--- + +## Half 1 — Deterministic replay (US1) + +### 1. Export a real workload + +From a machine that has been using mcpproxy for real work: + +```bash +mcpproxy activity export --format json > ~/replay-corpus.jsonl +``` + +**Bodies stay off.** The headline — per-mode tool-surface cost — is computed from the fleet's +tool definitions and the call sequence, not from recorded content, so it is fully measured +with bodies off. The export path does not mask, so a bodies-on export is raw and unmasked by +design; take one only if you specifically need measured *response* cost, and delete it after. + +> The file is real user traffic. Keep it outside the repository, never commit it, and delete +> it when finished. It is deliberately gitignored nowhere, because it should never be inside +> the repo in the first place. + +### 2. Recompute cost across the matrix + +```bash +# A fleet input is MANDATORY — a menu is a property of the tool definitions, and the +# activity export carries no fleet snapshot. Use a frozen corpus: +go run ./bench/cmd/bench \ + -replay ~/replay-corpus.jsonl \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \ + -out bench/results +# ...or point at a live proxy to read today's fleet instead. +``` + +Produces the `replay` block in `bench/results/report.json` plus the dashboard section: per +mode cell, what this workload would have cost, with the exclusion accounting beside it. + +### 3. Know which figures you just got + +- **Menu cost: `measured` for all five cells** — from the FLEET INPUT you supplied, not from + the recording. +- **Absolute complete-workload cost: unavailable for every cell.** It includes every consumed + response, and that text is absent bodies-off. +- **Cross-mode delta: `measured` for `direct_full` vs `direct_deferred` only** — their call + responses are identical, so they cancel in the comparison. This is the honest bodies-off + headline. +- **`retrieve_full` / `retrieve_compact` deltas: unavailable bodies-off**, because the mode + changes the `retrieve_tools` response body itself. Same for `code_exec`, whose surface also + serves `retrieve_tools`. +- **Response cost generally: `estimated` or absent.** Tokenizing needs the text; a byte length + supports only an estimate. +- **The fleet is today's fleet.** The export carries no fleet snapshot, so this scores a + recorded workload against the currently configured servers — internally valid across modes, + but not a historical reconstruction. + +### 4. Read the exclusion report before the headline + +The exclusion counts tell you whether the headline is trustworthy. A corpus that is 80% +truncated produces a real number over a small slice, and the report says so rather than +pretending otherwise. + +### 5. Verify determinism + +```bash +CORPUS=specs/083-discovery-profiler/datasets/corpus_v2.tools.json +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-a +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-b +# generated_at is a wall-clock stamp and differs between runs by design. +diff <(jq 'del(.generated_at)' /tmp/run-a/report.json) \ + <(jq 'del(.generated_at)' /tmp/run-b/report.json) \ + && echo "byte-identical modulo generated_at (SC-002)" +``` + +### Offline note + +The tokenizer downloads its vocabulary on first use unless a cache directory is set: + +```bash +export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" +# warms it; needs network once +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json -out /tmp/warm +``` + +Setting the variable only names a cache; it does not fill one. A first run still fetches the +vocabulary, so a genuinely offline reproduction needs the cache pre-populated (warm it once +with network access, or vendor/restore a known-good cache in CI). Without that, a reproducer +on a restricted network fails at step one — which is exactly what SC-004 requires to work. + +--- + +## Half 2 — Live agent loop (US2) + +**Costs model spend.** Do not start until a model and a ceiling are pinned. + +### 1. Stand up the full fleet on one proxy + +Configure mcpproxy with **all** suite services at once — filesystem, postgres, and the +credentialed ones if used — even while running a single service's tasks. A single server's +toolset is too small a fleet for proxy modes to differ from baseline, and the full fleet is +also the honest baseline: the same agent, the same tasks, all tools loaded directly. + +Enable code execution if the `code_exec` cell is in scope; without it that cell is degenerate +and is skipped with a reason. + +Leave the instance running. **The whole matrix crosses on one instance** — cells are selected +by endpoint URL, and only the two serialization axes need config. + +### 2. Point the suite at mcpproxy + +MCPMark is pinned by commit SHA. Patch its single MCP factory with an env-gated branch that +returns an HTTP MCP server at mcpproxy's URL with the API key header. That is the only change +the suite needs — `list_tools()` and `call_tool()` are the only calls its agent loop makes +against the server object. + +### 3. Start with the credential-free core + +Filesystem (30 tasks) and postgres (21) need no third-party credentials and are fully local. +That is the clean starting point. GitHub and Notion add real-fleet realism but **perform real +writes** and must be run so they cannot damage real data. + +### 4. Run each cell at k >= 4 + +A single agentic run is noise. Every model-dependent figure is an average over at least four +runs with its spread, or it is not a headline. + +### 5. Ingest the results + +The suite's per-task metadata carries the pass verdict, token usage split, and turn count; its +trajectory files are what make corrective-versus-infrastructure retry classification possible. +These feed tokens-per-completed-task, completion rate, first-attempt success and retry rates. + +--- + +## Reading the report honestly + +- **Check the accounting source on every figure.** Deterministic figures come from the local + tokenizer; live figures come from provider-reported usage. They are never summed — a + cross-source aggregate is withheld with a stated reason instead. +- **Check provenance per row**, not just the section badge. A section can legitimately hold a + measured figure beside an estimated one. +- **A withheld figure is a result**, not a bug. It means the honest number could not be + computed from what was measured. +- **Completion rate sits beside cost.** A mode that costs less and completes less is not a + saving, and the report marks it as a regression regardless of its token figure. +- **Every percentage carries its fleet shape.** A percentage without one is not publishable. + +## Before publishing + +1. Recompute the achievable ceiling **for each fleet shape** — never carry a previous + ceiling forward. That is the error spec 102 made. +2. Measure the tokenizer's divergence from the pinned model using the token-counting endpoint + (no inference spend) rather than quoting either of the two contradictory figures currently + in circulation. +3. Confirm no report is tracked: + ```bash + test -z "$(git ls-files bench/results)" && echo "clean (SC-011)" + ``` + Use `git ls-files`, not `git status --porcelain` — porcelain does not list ignored files and + would stay silent about an already-tracked report. diff --git a/specs/103-token-bench/research.md b/specs/103-token-bench/research.md new file mode 100644 index 000000000..563ca4caa --- /dev/null +++ b/specs/103-token-bench/research.md @@ -0,0 +1,279 @@ +# Phase 0 Research: token-bench (spec 103) + +Five topics, resolved 2026-08-31 against the tree at `b7d803e76` and against the suites' +current published code. Every decision below carries the evidence that settled it. + +--- + +## 1. How to attach to the existing `bench/` harness + +**Decision**: Use the "new measurement mode" seam twice — a flag on `cmd/bench`, a driver file +in package `bench`, and an additive report block. Add no new arm, no new binary, no new +package tree beyond one loader. + +**Rationale**: `bench/` has three distinct extension seams and only one of them fits: + +| Seam | For | Fits spec 103? | +|---|---|---| +| New **arm** (`bench/arms/`) | one deterministic way of rendering *tool definitions* into context text | **No** — spec 103 measures modes, not renderings. The renderings already exist. | +| New **corpus** (`bench/corpusio/` + an `OfflineSection`) | one more frozen tool universe to score | **No** — replay crosses the mode matrix; it is not one more corpus | +| New **measurement mode** (flag + driver + report block) | a new kind of measurement | **Yes, twice** | + +The precedent to copy is spec 085's flip gates: a `-flip-gates` flag, a driver +(`bench/flipgate.go`) parameterised over a function type so the arithmetic is unit-testable +with no proxy running, production transport isolated in `bench/mcpcaller.go`, and the result +hung off the live report. `-live` is the same shape at larger scale. + +**Alternatives considered**: adding replay as an arm (rejected — an arm renders tool +definitions and must satisfy the arm-interface contract including determinism over a corpus, +which a session trace is not); adding it as a corpus (rejected — a corpus is one tool universe +scored by many arms, whereas replay is one workload scored across the mode matrix). + +**Report extension cost**: additive optional fields with `omitempty`, plus matching optional +properties in `specs/083-discovery-profiler/contracts/report-v2.schema.json`, plus one +provenance key per block. **No `report_version` bump** — that is reserved for changing an +existing field's meaning or shape, and spec 103 changes none. Precedent: `LatencyV2`'s +`rest_search`/`mcp_discovery` were added beside the flat fields the same way. + +> The schema has no `additionalProperties: false`, so an undeclared block would validate +> silently. Do not rely on that — the schema file is the reviewed contract and new blocks must +> be declared in it. + +**Finding that becomes work**: SC-011 ("no generated report is committed") is enforced only by +`bench/.gitignore` and convention. There is no CI job or test that fails on a committed +report. The gate must assert that nothing under the results directory is **tracked** — +`test -z "$(git ls-files bench/results)"`. A `git status --porcelain` check does NOT work: +porcelain omits ignored files and would stay silent about an already-committed report. + +--- + +## 2. What replay can consume, and the privacy posture it forces + +**Decision**: Consume JSONL from `GET /api/v1/activity/export?format=json` (equivalently +`mcpproxy activity export --format json`) in a new `bench/replaycorpus/` package. **Default to +bodies-off.** Add `request_bytes` / `response_bytes` to the export contract. + +**Rationale**: + +- **CSV is unusable** — it drops `work_session_id`, arguments, response and all byte fields. +- **Nothing in `bench/` references activity today**, so the loader is purely additive. +- **The export path never masks.** `maskActivityPayloads` is wired into the list and detail + handlers only. Bodies-on export is raw and unmasked, by design — it is the + compliance/incident-response surface. This is why bodies-off is the default and bodies-on is + an explicit opt-in with a warning. +- **`has_sensitive_data` is set asynchronously *after* the record is persisted.** A freshly + exported record can be sensitive but not yet flagged. Exclude-by-flag is therefore a + best-effort reducer and must never be documented as a guarantee. + +**The one worthwhile backend change**: `request_bytes` and `response_bytes` exist on the +storage record, documented as *measured pre-truncation*, but are absent from the export +contract entirely. Adding them (two DTO fields, two lines in the export projection) lets a +truncated record carry an explicitly-estimated response cost instead of being dropped. + +**A second backend change turns out to be necessary too.** Truncation is not recorded for +internal tool calls: `wasTruncated` is computed in the `retrieve_tools` handler +(`internal/server/mcp.go:2019-2035`) and only logged, and `emitActivityInternalToolCall` +(`:810`) has no truncation parameter. So a truncated `retrieve_tools` record is +indistinguishable from a complete one, and tokenizing its stored full response would overstate +what the agent paid — the exact failure class FR-002 exists to prevent. Either propagate the +flag onto internal tool-call activity, or exclude every internal `retrieve_tools` record from +response-cost accounting as an explicit, reported decision. + +**Correction applied after review, then corrected again**: an early draft claimed these give +an *accurate* cost basis without bodies. They do not — they are byte lengths, and tokenizing +needs the text. A second draft over-corrected, claiming tool-surface cost carries the headline +for all five cells. The precise position: + +- **A fleet input is mandatory.** A menu is a property of the tool definitions, and the export + carries no fleet snapshot, so replay must be paired with a frozen fleet corpus or a live + proxy. A recording-only invocation computes nothing. +- **No cell yields an absolute complete-workload cost bodies-off.** Complete workload includes + every consumed response, and that text is absent. Bodies-off gives menu cost per cell plus + the cross-mode DELTA between the two direct cells, whose identical call responses cancel. +- **`code_exec` and both `retrieve_tools` cells have no bodies-off delta either**: the + code-execution surface also registers `retrieve_tools` + (`internal/server/mcp_routing.go:670-676`), and the `retrieve_tools` cells' mode changes the + response body itself. +- **Tokenization of bodies must happen inside the loader**, so no text crosses the boundary + that the privacy invariant draws. +- **`retrieve_tools` response cost is recoverable with bodies on** — the handler writes the + FULL response to the activity log (`internal/server/mcp.go:2061-2064`). It is unavailable + only in the bodies-off configuration, not unrecoverable in general. +- **The export carries no fleet snapshot**, so a replay scores a recorded workload against + today's fleet. Internally valid across modes; not a historical reconstruction. + +**Alternatives considered**: reading BBolt directly (rejected — bypasses the contract, couples +the harness to storage internals, and defeats the privacy gating entirely); building a new +capture mechanism (rejected — explicitly out of scope in the spec, and unnecessary). + +--- + +## 3. The mode matrix: 5 valid cells, not 12 + +**Decision**: A matrix cell is **one MCP endpoint plus the one serialization axis that governs +that endpoint's surface** — not a 3×2×2 config product. + +| # | Routing mode / endpoint | `tool_response_mode` | `direct_tool_response_mode` | Notes | +|---|---|---|---|---| +| 1 | `retrieve_tools` — `/mcp/call` | full | n/a | today's default | +| 2 | `retrieve_tools` — `/mcp/call` | compact | n/a | spec 085 | +| 3 | `direct` — `/mcp/all` | n/a | full | pre-102 direct | +| 4 | `direct` — `/mcp/all` | n/a | deferred | spec 102 | +| 5 | `code_execution` — `/mcp/code` | forced full | n/a | needs `enable_code_execution: true` | + +Plus the existing `baseline` arm as the FR-020 denominator. + +**The 7 skipped combinations and their FR-017 reasons**: + +- `code_execution × compact` (×2 across the direct axis) — **forced**: the code-execution + surface overwrites the response mode with `full` and blanks the detail parameter; `detail` + is not even in that surface's schema. +- `direct × {full, compact}` on the *discovery* axis (×2) — **inapplicable**: + `tool_response_mode` has exactly one consumer, inside the `retrieve_tools` handler. The + direct surface has no `retrieve_tools` tool at all. +- `retrieve_tools × deferred`, `code_execution × deferred` (×2) — **inapplicable**: + `direct_tool_response_mode` is read only when building the direct listing. +- `code_execution` with `enable_code_execution: false` — **degenerate**: the surface can + discover tools and call none of them. + +**Crossing the matrix is cheap.** All three routing-mode servers are built at startup and all +three endpoints are permanently mounted regardless of config — the source comments this +explicitly — so the routing-mode axis is selected **by URL**, with no restart. The two +serialization axes do need config, but **both hot-reload**: `tool_response_mode` is read from +the live snapshot on every call (documented as taking effect "on the very next call, without +reconstructing the server") and `direct_tool_response_mode` is rebuilt on the `config.reloaded` +event. So the whole matrix crosses on ONE long-lived instance, with a config apply between +serialization cells. + +**Reuse the existing skip shape** rather than inventing one: `ArmResult.Skipped` / +`SkipReason` and the `SkippedArmResult` constructor already implement "a skipped row with a +reason, never a zero", which is exactly what FR-017 asks for. + +**This directly simplifies the spec**: FR-015/FR-016/FR-017 describe a matrix that is 5 rows +plus 7 documented skips, not a 12-cell product. + +--- + +## 4. Public agent-loop suite + +**Decision**: Adopt **MCPMark** (`eval-sys/mcpmark`, Apache-2.0), SHA-pinned. Adopt +**MCP-Atlas** only as a *corpus donor* for fleet-shape work, not as an agent loop. + +**Both previously-unverified questions resolve in MCPMark's favour**: + +1. **Per-task token counts**: yes. Its per-task `meta.json` carries + `execution_result.success`, `token_usage{input, output, total, reasoning}`, `turn_count` + and `agent_execution_time`; `messages.json` holds the full trajectory, which is what makes + corrective-vs-infrastructure retry classification possible. Its aggregator already emits + `pass@1{avg,std}`, `pass@k`, `pass^k` and per-run cost. +2. **Can it point at a single endpoint?** Yes — the load-bearing question, and the answer is + that MCPMark has exactly **one** MCP factory seam. Pointing it at mcpproxy is a single + env-gated `elif` branch returning an HTTP MCP server at mcpproxy's URL. Its transport is + the official Python SDK streamable-HTTP — the same transport `bench/mcpcaller.go` already + drives. `list_tools()` and `call_tool()` are the only two calls its agent loop makes. + +**Consequence**: the spec's fallback (build an in-repo task set) is **not needed**. + +**Fleet shaping matters for honesty.** Stock MCPMark connects to one server per run, so the +per-task fleet is a single server's toolset — far too small for mcpproxy to show its +asymptote. Configure mcpproxy with *all* MCPMark services simultaneously while running one +service's tasks. This is a pure mcpproxy-config change and it is also the honest FR-020 +baseline: the same agent, the same tasks, all tools loaded directly. + +**Credential-free core**: filesystem (30 tasks) + postgres (21) need zero third-party +credentials and are fully local — the FR-027-clean starting point. GitHub (23) and Notion (28) +add real-fleet realism but perform real writes and need credentials. + +**Alternatives considered**: MCP-Atlas as the loop (rejected — its value is its 36-server / +220-tool *shape*, ideal for FR-024's fleet-shape decomposition, but it is not an agent-loop +harness); τ²-bench (rejected — needs an MCP bridge that does not exist); BFCL / ToolBench / +API-Bank (rejected — not MCP); LiveMCPBench as a loop (rejected — stale since 2025-08, though +its committed 527-tool snapshot remains useful as a corpus). + +--- + +## 5. Token accounting for a live loop + +**Decision**: Keep tiktoken (`cl100k_base`) as the *sole* source for everything deterministic. +Read provider-reported `usage` **directly off the model response** in the agent driver for the +live loop. **Never sum the two.** + +**Rationale**: The harness already solved "two token shapes must not be mixed into one +number", for schemas: when either side lacks schema counts, the savings ratio is **withheld +with a stated reason** rather than computed. The break-even computation carries the same rule +in its own doc comment and errors rather than fabricating a verdict. Spec 103's never-summed +constraint is that same rule on a new axis, so it is enforceable with machinery that already +exists — the extension is a new section and a source label, not new aggregation logic. + +For the provider this project would pin, `usage` maps 1:1 onto FR-014's required split +(input / output / cache-read) with no derivation, so reading response fields is simpler than +proxying. **This is provider-specific and unverified until the provider is pinned** — the +mapping must be re-checked against whichever provider is chosen, and it does NOT hold for the +suite's own per-task output (see the gap below). + +**Gap to close at task time**: the suite's own per-task output records input / output / total / +reasoning — it has **no cache-read field**. So FR-014's cache-read axis cannot come from the +suite's report; it must be captured in the driver from the provider response, or explicitly +declared out of reach for suite-driven runs. The single MCP-factory patch alone does not +establish FR-014 coverage. An LLM proxy is kept +only as the fallback for a closed-box suite whose loop cannot be edited — which MCPMark, being +patchable at one seam, is not. + +**Alternatives considered**: an LLM proxy as the primary path (rejected — an extra hop and a +second failure mode on a run that costs real money, and it sees nothing the response object +does not already carry; a silently dropped record would understate cost *in the project's +favour*, the same failure class FR-002 forbids for truncation); feeding provider-derived retry +rates into the existing session estimator (rejected — that formula's other terms are tiktoken +counts, so it would produce exactly the summed hybrid the spec forbids; a measured session +cost must be a new row type sourced wholly from provider usage); reusing +`internal/server/tokens/savings.go` (rejected — a production/Web-UI path whose "query result" +is an arbitrary prefix of the tool list rather than a ranking, and whose "all tools" side is a +hand-built JSON shape rather than the live builders; reusing it would reintroduce exactly the +drift IC-002 forbids). + +**There is no existing code anywhere in the repo that reads a provider usage field.** Every +token count in the tree is produced by the local tiktoken wrapper. Cache-token handling is +entirely greenfield — no `cache_read` concept exists anywhere. + +### Two hazards this creates + +- **`ReportV2.Tokenizer` is a report-level singleton** naming one estimator. Once provider + usage enters the same envelope a reader could take that field to describe it too. Resolve + **additively**: leave the existing field's meaning intact and give each new block its own + accounting-source field (provider + pinned model). Narrowing the existing field would itself + be a meaning change and would require a version bump. +- **`RetryRateForArm` returns 0.0 for unknown arms**, indistinguishable from a measured 0.0. + With FR-013 replacing assumed rates only where measurements exist, one table could mix a + measured rate and a defaulted one under a single section-level `estimated` badge. Session + cost rows need a **per-row** provenance field. + +--- + +## Cross-cutting: reproducibility hazards found + +1. **tiktoken downloads its vocabulary at runtime** from a remote host unless + `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` is set. Nothing in the repo or in + `bench.yml` sets it, so the documented "offline estimator" property holds only after a warm + cache. **An outside reproducer on a restricted network fails at step one** — which defeats + SC-004. Fix: setting the variable is necessary but NOT sufficient — it names a cache without + filling one. CI must also **populate or restore** the cache (a cache-restore step, or a + vendored copy), and the reproduction procedure must tell an outsider to warm it once. +2. **The tokenizer caveat is unsourced and internally contradicted.** The repo states up to + ~60% underestimation versus Claude; Anthropic's own guidance says ~15–20% on typical text + (more on code). The two differ threefold and neither is sourced in-repo. Publishing + FR-022's numeric tolerance on top of either is a credibility risk. Fix: measure it with + `count_tokens` against the frozen corpora for the pinned model — no inference spend, and it + doubles as a genuine `measured` figure for the per-fleet-shape ceiling. +3. **The two savings numbers have never been compared.** The Web-UI/status figure and the + bench headline are independently implemented over the same fleet. The spec's own + Assumptions say a contradiction between measurements is a finding to investigate. Running + both against one live proxy is cheap and should happen early. + +## Deferred to the operator + +- Pinned model and spend ceiling for the live loop (US1 is unblocked without it). +- Whether cache-read tokens count toward the headline composite (a definition decision). +- Whether the `playwright_webarena` task subset (21 of 127) needs a self-hosted instance. +- The exact corrective-vs-infrastructure retry boundary. Whatever rule is chosen **must be + applied identically to the baseline arm and the proxy arms**, or the comparison is biased + toward whichever arm carries richer error signal. diff --git a/specs/103-token-bench/spec.md b/specs/103-token-bench/spec.md new file mode 100644 index 000000000..fff4f5f02 --- /dev/null +++ b/specs/103-token-bench/spec.md @@ -0,0 +1,497 @@ +# Feature Specification: Token-efficiency benchmark — measured savings, published results + +**Feature Branch**: `103-token-bench` +**Created**: 2026-08-31 +**Status**: Draft +**Input**: User description: "token-bench: measure real token cost AND task-success rate across every mcpproxy routing/serialization mode combination, on replayed real agent sessions and public agent-loop suites, so every savings number mcpproxy quotes becomes a reproducible measurement" + +## Why This Exists + +MCPProxy's headline promise is "massive token savings". Today that promise rests on a +measurement of the **static tool surface** — how many tokens the tool definitions cost an +agent before it does any work. That number is real, reproducible and already gated in CI. + +It is also not the number a user cares about. A user cares what a *task* costs. A mode that +halves the tool surface but doubles the number of failed attempts costs more, not less. +Nothing in the repo measures that today. + +Two concrete facts make this urgent: + +1. **Spec 102 projected ~70% reduction and measured 29.7%** on the frozen 45-tool corpus + (34.8% on a 527-tool snapshot). Its success criterion was restated per corpus shape + rather than the design being changed, and its own gate concluded that names, + descriptions and annotations — not schemas — dominate the payload, with 38.9% the + arithmetic ceiling on that corpus even if both schema and signature were deleted. + That conclusion is drawn from two frozen corpora and has never been checked against a + real fleet. +2. **Retry rates are assumed, not observed.** The session-cost rows in today's report are + computed from per-arm literature-derived defaults, and the dashboard honestly labels + them `estimated`. Every session-level savings claim inherits that assumption. + +This feature replaces assumptions with measurements, and publishes the methodology so the +numbers can be checked by someone who does not trust us. + +## Scope Boundary *(read before planning)* + +An extensive benchmark harness already exists and is CI-wired. **This feature extends it. +Re-implementing any of the following is out of scope and would be a defect:** + +| Already exists | Do not rebuild | +|---|---| +| Six tool-definition encoding arms, incl. the spec-102 deferred rendering | arm framework, arm contracts | +| Token counting derived from the *live* tool builders (cannot drift from production) | tokenizer, catalog extraction | +| Live response-cost measurement, break-even analysis | response cost, break-even | +| Retrieval quality metrics (recall@k, nDCG) and a parity gate through the production index | retrieval scoring | +| Versioned report + self-contained dashboard with `measured`/`computed`/`estimated` provenance badges on every headline number | report schema, dashboard | +| Frozen tool corpora at 45-tool and 527-tool scale, plus an external tool-retrieval corpus and an independent third-party linter | corpus loaders | + +The provenance-badge convention is load-bearing for this feature: everything it adds must +declare whether it is measured, computed or estimated, and the point of the work is to move +specific numbers from `estimated` to `measured`. + +## Replay Boundary *(read before planning — this bounds US1)* + +A recorded session is a trace of what an agent **did** under the mode it was running. It +carries the tool calls, their arguments, the responses, status, timing and the grouping that +ties calls into one unit of user work. It does **not** carry the user's prompt, the +conversation, the model's state, or any oracle for whether the user's goal was met. + +Therefore replay CAN answer: what this real workload, at this real fleet size, would have +cost under a different serialization or routing mode. That is arithmetic over recorded +traffic and is fully deterministic. + +Replay CANNOT answer: how the agent would have *behaved* differently — which calls it would +have attempted, whether its first attempt would have been right, whether it would have +finished. Those are new decisions by a model that is not present in the recording. + +Any requirement about success, retries or completion therefore belongs to a live agent loop +(US2), not to replay (US1). A design that infers agent behaviour from a recording is wrong, +and this boundary is the single most important thing for planning to respect. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 — Recompute real workload cost under every mode (Priority: P1) + +A maintainer takes a set of previously recorded real sessions — the actual sequence of tool +calls a real agent made, at real fleet size — and asks what that same workload would have +cost under every other mode combination. The answer is arithmetic over recorded traffic: no +model is invoked and no agent decision is predicted. + +**Why this priority**: It is the cheapest way to replace synthetic corpora with real workload +shapes, it needs no model spend, and it is fully deterministic. It is also strictly bounded: +see the Replay Boundary below for what it cannot answer. + +**Independent Test**: Supply exported real sessions, run the recomputation across the mode +matrix, and confirm a per-mode cost for that workload, each figure marked `measured` or +`computed` and never `estimated`. + +**Acceptance Scenarios**: + +1. **Given** a set of exported real sessions, **When** a maintainer runs the recomputation, + **Then** the report shows per-mode token cost for that workload, broken into tool-surface + cost and response cost, on real fleet sizes rather than a frozen corpus. +2. **Given** a recorded session whose responses were truncated at capture, **When** it is + processed, **Then** it is excluded from token totals or explicitly annotated as + under-counting, and the report states how many sessions were affected — never silently + included. +3. **Given** two runs over the same sessions and modes, **When** their reports are compared, + **Then** the figures are byte-identical, since no model is involved. +4. **Given** any recomputed figure, **When** a reader inspects it, **Then** it is labelled as + a counterfactual over recorded traffic, not as observed agent behaviour. + +--- + +### User Story 2 — Measure whether a mode helps the agent succeed (Priority: P1) + +A maintainer wants the number that actually matters: tokens per *completed* task. This +requires a live agent loop under a pinned model, because it depends on decisions the agent +makes differently under each mode — which calls it attempts, how often the first attempt is +right, and whether it finishes. + +**Why this priority**: This is the feature's core claim. Without it every savings figure +remains a statement about menu size rather than about work done. It is separated from US1 +because it has a fundamentally different cost, determinism and data requirement. + +**Independent Test**: Run a fixed task set through a live agent under at least two mode +combinations and confirm the report gives, per mode, tokens per completed task, task +completion rate, first-attempt success rate and retry count, each `measured`. + +**Acceptance Scenarios**: + +1. **Given** a fixed task set, **When** it is run under each mode combination, **Then** the + report gives per mode: task completion rate, tokens per completed task, first-attempt + success rate and retry count, each `measured`. +2. **Given** a mode that lowers token use but completes fewer tasks, **When** the report is + read, **Then** the completion drop is displayed with equal prominence to the token + saving, and the mode is not presented as a saving. +3. **Given** any model-dependent figure, **When** it is published, **Then** it is an average + over at least four runs with its spread, never a single run. + +--- + +### User Story 3 — A skeptical reader reproduces the published numbers (Priority: P2) + +Someone who does not trust the project reads a published claim, follows the documented +procedure, and arrives at the same numbers within a stated tolerance — including the +comparison against not using MCPProxy at all. + +**Why this priority**: An unreproducible benchmark is marketing. This is what separates +this work from the thin-methodology comparisons already circulating in this space. It +depends on US1 and US2 producing numbers worth reproducing. + +**Independent Test**: A person with no prior context follows the published procedure on +their own machine and reproduces the deterministic figures exactly and the model-dependent +figures within the stated tolerance. + +**Acceptance Scenarios**: + +1. **Given** the published methodology, **When** an outside reader follows it, **Then** every + input needed is either included in the repository or fetched by a documented pinned + procedure, and no step depends on data only the maintainers hold. +2. **Given** a published comparison, **When** a reader inspects the baseline, **Then** the + baseline is the same agent doing the same tasks with all tools loaded directly, not a + different or weaker configuration. +3. **Given** any published percentage, **When** a reader looks it up, **Then** it is + accompanied by the size and shape of the tool set it was measured on, because the + figure moves with fleet size. +4. **Given** a headline figure, **When** a reader traces it, **Then** they can reach the raw + per-run records it was computed from. + +--- + +### User Story 4 — The 29.7% shortfall gets an answer (Priority: P2) + +A maintainer wants to know whether the spec-102 conclusion — that names, descriptions and +annotations dominate the payload, capping achievable savings well below the projection — +holds on real fleets, or was an artefact of two frozen corpora. + +**Why this priority**: This determines whether further serialization work is worth doing at +all, and it is the first question anyone will ask about the published numbers. It reuses +the existing tool-surface measurement machinery rather than adding its own. + +**Independent Test**: Produce a payload decomposition across corpora of different sizes and +shapes and compare it against the spec-102 conclusion, yielding an explicit confirmation +or correction. + +**Acceptance Scenarios**: + +1. **Given** tool sets of varying size and shape, **When** the decomposition runs, **Then** + the report attributes payload share to names, descriptions, annotations and schemas + separately, and shows how each share moves with fleet size. +2. **Given** that decomposition, **When** it is compared with the spec-102 conclusion, + **Then** the result explicitly states whether that conclusion is confirmed or corrected, + and the arithmetic ceiling is recomputed per corpus rather than assumed. + +--- + +### User Story 5 — Results reach the people deciding whether to adopt (Priority: P3) + +A developer evaluating MCPProxy reads a public write-up with real numbers, an honest +account of where savings do *not* materialise, and enough method to judge it. + +**Why this priority**: The measurement has no external value until it is published, but it +is strictly downstream of having trustworthy numbers. + +**Independent Test**: The published write-up states the measured figures, the corpus shapes +they hold for, the known limitations, and the reproduction procedure. + +**Acceptance Scenarios**: + +1. **Given** the write-up, **When** a reader finishes it, **Then** they know which mode to + choose for their own fleet size and why. +2. **Given** a case where a mode does not help, **When** the reader looks for it, **Then** + the write-up says so plainly rather than omitting it. + +--- + +### Edge Cases + +- **A recorded session cannot be replayed** because the upstream servers it used are gone + or have changed: the session must be reported as unreplayable rather than counted as a + failure of the mode under test, since that would blame the mode for missing infrastructure. +- **Recorded responses were truncated at capture time**, so replaying them under-counts + tokens. Detection is mandatory; silent inclusion is the single most dangerous failure + mode here, because it biases every number in the project's favour. +- **Response bodies are unavailable** because the export withheld them: the harness must + say what it cannot measure rather than substituting a guess. +- **A mode combination is not valid** (an axis that does not apply to the selected routing + mode): the matrix must skip it explicitly and show it as skipped, not as zero. +- **A task suite performs real writes** against live third-party services: destructive + operations must be handled deliberately, and a benchmark run must never be capable of + damaging a real user's data. +- **A run is interrupted partway** through an expensive matrix: partial results must be + identifiable as partial and must never be published as a complete comparison. +- **A model-dependent figure comes from a single run**: single-run agentic numbers are + noise; the report must refuse to present one as a headline. +- **Sessions contain sensitive data** — real recorded traffic may include secrets or + personal data. Replay inputs must be handled so that publishing results never publishes + session contents. + +## Requirements *(mandatory)* + +### Definitions *(binding — these terms are used with exactly these meanings)* + +- **Unit of work**: one task from a fixed task set (US2), or one recorded work-session + grouping (US1). Never an individual tool call. +- **Attempt**: one tool call issued by the agent toward a given intent. +- **First-attempt success**: the first attempt for an intent returned a non-error result AND + was not followed by a corrective retry for the same intent. A schema-valid call that the + agent immediately re-issues differently is NOT a success. +- **Retry**: a subsequent attempt for the same intent after a failed or corrected one. + Transport-level and infrastructure retries are counted and reported SEPARATELY, because + they measure the network, not the mode. +- **Task completion**: the task suite's own pass verdict. Where a suite has no verdict, the + unit of work is reported as having no completion signal and is excluded from + completion-dependent figures rather than assumed complete. +- **Token accounting**: provider-reported usage where the run is model-backed; the existing + deterministic tokenizer where it is not. Which one produced a figure MUST be stated; the + two MUST NOT be summed into one number. +- **Fleet shape**: the tool count plus the distribution of definition sizes for the tool set + a figure was measured on. + +### Inherited constraints *(already provided by the existing harness — satisfy, do not rebuild)* + +- **IC-001**: Every figure carries a `measured` / `computed` / `estimated` provenance badge. +- **IC-002**: Static tool-surface cost is measured from the live tool builders for every + routing mode and MUST NOT be re-derived. +- **IC-003**: Generated reports are never committed; only code, fixtures, thresholds and + methodology are versioned. +- **IC-004**: Any quoted percentage carries the fleet shape it was measured on. + +### Functional Requirements + +**Replay of real sessions (deterministic; see Replay Boundary)** + +- **FR-001**: The harness MUST accept previously recorded real agent sessions as input via + the existing export path, without requiring a new capture mechanism. +- **FR-002**: The harness MUST detect recorded entries whose stored content was truncated at + capture time and MUST either exclude them from token totals or annotate them as + under-counting; it MUST NOT include them silently. +- **FR-003**: The harness MUST report how many supplied sessions were unusable and why, + distinguishing truncation, missing bodies, and unreplayable upstreams. +- **FR-004**: Replay output MUST be labelled as a counterfactual cost recomputation over + recorded traffic. It MUST NOT be presented as observed agent behaviour, and the harness + MUST NOT infer success, retry or completion figures from a recording. +- **FR-005**: Replay MUST report cost split into tool-surface cost and response cost, so the + two can be reasoned about separately at real fleet shapes. + +**Handling of recorded session data** + +Recorded sessions are real user traffic. The export path that includes bodies deliberately +returns full unmasked values — it is the compliance/incident-response surface — so replay +inputs can contain secrets and personal data that the browsing surfaces mask. + +- **FR-006**: Replay inputs MUST be treated as sensitive by default. Neither they nor any + excerpt MUST appear in a report, dashboard or published artefact. +- **FR-007**: The harness MUST NOT transmit recorded session content to any third-party + service, including model providers. Replay is local arithmetic; a design requiring such + transmission is outside this feature. +- **FR-008**: Records flagged as containing sensitive data MUST be excluded, or reduced to + the non-sensitive measurements needed for token accounting, before use; the count of + affected records MUST be reported. +- **FR-009**: Replay inputs MUST live outside the repository, MUST NOT be committed, and the + documented procedure MUST tell an operator how to delete them when finished. + +**Measured success, retries and completion (live agent loop)** + +- **FR-010**: The harness MUST measure first-attempt success rate per mode combination, as + defined above. +- **FR-011**: The harness MUST measure retry counts per mode combination, reporting + corrective retries separately from infrastructure retries. +- **FR-012**: The harness MUST record a task-completion verdict per unit of work, taken from + the task suite rather than inferred. +- **FR-013**: Measured success and retry figures MUST replace the assumed per-arm defaults + for any mode where a measurement exists; any figure still derived from an assumption MUST + remain badged `estimated` and MUST NOT be presented without that distinction. +- **FR-014**: Token accounting MUST separate input, output and cached-read consumption. + +**The mode matrix** + +- **FR-015**: The matrix MUST cross the three configuration axes that determine what an agent + sees: routing mode, discovery-surface serialization, and direct-surface serialization. +- **FR-016**: Batching, stored scripts and validate-before-dispatch MUST each be covered as a + binary condition applied to the routing modes where they are available, and the report MUST + enumerate which rows each applies to. +- **FR-017**: Combinations that are not meaningful MUST be reported as skipped with a reason, + never as a zero or a missing row. +- **FR-018**: The report MUST express headline cost as tokens per completed task, with + completion rate displayed alongside it at equal prominence. +- **FR-019**: A mode whose completion rate falls below the baseline's by more than a stated + threshold MUST be marked as a regression regardless of its token cost, and MUST NOT be + described as a saving. + +**Honest comparison** + +- **FR-020**: The baseline arm MUST be the same agent performing the same tasks with all + tools loaded directly. A comparison against a different or weaker configuration MUST NOT + be published. +- **FR-021**: Any model-dependent figure MUST be reported as an average over at least four + runs together with a measure of consistency across runs; a single run MUST NOT be reported + as a headline result. +- **FR-022**: The reproduction tolerance for model-dependent figures MUST be stated + numerically in the published methodology. +- **FR-023**: The report MUST include a cost-versus-outcome view so a reader can see which + modes are worth their savings rather than only which are cheapest. + +**Payload decomposition** + +- **FR-024**: The harness MUST attribute tool-definition payload share to names, + descriptions, annotations and schemas separately, across at least two fleet shapes. +- **FR-025**: The result MUST explicitly confirm or correct the spec-102 conclusion about + what dominates the payload, and MUST recompute the achievable ceiling per corpus rather + than carrying a fixed figure forward. + +**Public suites** + +- **FR-026**: The harness MUST support at least one public task suite that exercises a full + agent loop and emits a per-task pass verdict. +- **FR-027**: Any suite performing real writes against third-party services MUST be run in a + way that cannot damage real user data. +- **FR-028**: Suite versions and configuration MUST be pinned so a later run is comparable to + an earlier one. + +**Reproducibility and publication** + +- **FR-029**: Raw per-run records MUST be retained and referenced by the report so a headline + figure can be traced to its inputs, without embedding session contents (FR-006). +- **FR-030**: Every input MUST be either included in the repository or obtainable by a + documented, pinned procedure. Figures that depend on private recorded sessions MUST be + marked as not independently reproducible, and MUST NOT be the sole support for a published + claim. +- **FR-031**: The published write-up MUST state measured figures, the fleet shapes they hold + for, known limitations, and the reproduction procedure, and MUST state plainly where + savings do not materialise. +- **FR-032**: A partial or interrupted run MUST be identifiable as partial and MUST NOT be + publishable as a complete comparison. + +### Key Entities + +- **Replayed session**: A previously recorded unit of real agent work — its tool calls, + arguments and responses — with usability flags (truncated, missing bodies, unreplayable, + sensitive). +- **Mode combination**: One point in the matrix — routing mode plus serialization settings + and capability toggles determining what the agent sees and how it calls. +- **Run record**: The raw outcome of one unit of work under one mode: token consumption by + kind, attempts, first-attempt outcome, retries by class, completion verdict, errors. +- **Measurement**: An aggregate over run records, carrying provenance and, where + model-dependent, run count and spread. +- **Payload decomposition**: Attribution of tool-definition cost to names, descriptions, + annotations and schemas for a given fleet shape. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: For every VALID mode combination exercised by the live task set, a maintainer + can obtain tokens per completed task, completion rate and first-attempt success rate, each + badged `measured`. +- **SC-002**: For every valid mode combination, a maintainer can obtain the counterfactual + cost of a real recorded workload, reproducible byte-for-byte across runs. +- **SC-003**: No session-level cost figure is derived from an assumed retry rate unless it is + explicitly badged `estimated`. +- **SC-004**: A person outside the project, following only the published procedure, + reproduces the deterministic figures exactly and the model-dependent figures within the + stated numeric tolerance. +- **SC-005**: Every published percentage states the fleet shape it was measured on; a + percentage without that context does not appear. +- **SC-006**: The spec-102 payload conclusion is explicitly confirmed or corrected, with the + achievable ceiling recomputed for each fleet shape measured. +- **SC-007**: A mode that lowers token cost while completing fewer tasks is flagged as a + regression by the completion threshold and is not reported as a saving — demonstrated by + running a deliberately degraded mode. +- **SC-008**: Recorded sessions with truncated content are never counted silently: the report + states how many were affected and how they were handled. +- **SC-009**: No recorded session content, and no excerpt of it, appears in any report, + dashboard or published artefact, and no such content is transmitted to a third-party + service. +- **SC-010**: At least one full agent-loop public task suite runs against MCPProxy and + produces comparable results across at least two mode combinations. +- **SC-011**: No generated report is committed to the repository. +- **SC-012**: A reader of the published write-up can state which mode suits their fleet shape + and can name at least one case where MCPProxy's savings do not materialise. + +## Assumptions + +- **Existing harness is the foundation.** Everything listed under Scope Boundary is treated + as available and correct; this feature adds to it rather than replacing it. If a + measurement here contradicts an existing one, that is a finding to investigate, not a + reason to fork the harness. +- **Recorded sessions supply real workload SHAPE, not agent behaviour.** They drive the + cost recomputation (US1) because real fleets beat synthetic corpora. They cannot drive the + success/completion headline, because a recording contains no prompt, conversation, model + state or completion oracle — see Replay Boundary. The live task set (US2) owns that. +- **Task completion comes from the task suite's own verdict**, never inferred from a call + trace. Units of work with no verdict are reported as lacking a completion signal and + excluded from completion-dependent figures rather than assumed complete. +- **Model-dependent measurement costs money and needs a decision.** Any full agent-loop run + requires a pinned model and a spending decision before it can be built. US1 is designed to + deliver value alone without it, so the feature is not blocked on that decision. +- **Private recorded sessions cannot be published**, so figures derived from them are not + independently reproducible by an outsider. Public suites and frozen corpora therefore carry + the reproducible claims; recorded-session figures corroborate them at real fleet shape. +- **Reporting conventions carry over.** The provenance-badge discipline, the + reports-are-never-committed rule, and the requirement to quote corpus size alongside any + percentage all continue to apply. +- **Telemetry cross-validation is downstream.** Confirming these findings against the real + user population is a later, separate step and is not required for this feature to be + complete. +- **Publication happens outside this repository**, so the write-up is a deliverable of this + feature but lands elsewhere; it extends the existing published comparison work rather + than duplicating it. + +## Out of Scope + +- Changing any routing or serialization behaviour. This feature measures; it does not + optimise. Findings may motivate later work. +- Real-user population validation through telemetry counters. +- Retrieval-quality evaluation, which already exists and is separately gated. +- Suites that are not MCP-native, and any suite that cannot be pointed at a single proxy + endpoint. +- Inferring agent behaviour, success or completion from recorded traces. This is not merely + out of scope: it is forbidden by FR-004, because it would fabricate the feature's headline. +- Any new activity capture mechanism. If the existing export proves insufficient for cost + recomputation, that is a finding to report, not scope to absorb here. + +## Dependencies + +- The recorded-session export path must expose enough per-call detail — including response + content when explicitly requested — for token accounting. It is already known NOT to carry + prompts, conversation or completion verdicts, which is why US2 exists separately. +- A task suite with per-task pass verdicts is required for every completion-dependent + figure; without one, US2 cannot be satisfied by any amount of recorded data. +- The mode configuration axes must be settable per run so the matrix can be crossed. +- A pinned model and a spending decision are prerequisites for the agent-loop portions + (US2's public-suite half and any measured retry rate that requires live inference). + +## Commit Message Conventions *(mandatory)* + +When committing changes for this feature, follow these guidelines: + +### Issue References +- ✅ **Use**: `Related #[issue-number]` - Links the commit to the issue without auto-closing +- ❌ **Do NOT use**: `Fixes #[issue-number]`, `Closes #[issue-number]`, `Resolves #[issue-number]` - These auto-close issues on merge + +**Rationale**: Issues should only be closed manually after verification and testing in production, not automatically on merge. + +### Co-Authorship +- ❌ **Do NOT include**: `Co-Authored-By: Claude ` +- ❌ **Do NOT include**: "🤖 Generated with [Claude Code](https://claude.com/claude-code)" + +**Rationale**: Commit authorship should reflect the human contributors, not the AI tools used. + +### Example Commit Message +``` +feat: [brief description of change] + +Related #[issue-number] + +[Detailed description of what was changed and why] + +## Changes +- [Bulleted list of key changes] +- [Each change on a new line] + +## Testing +- [Test results summary] +- [Key test scenarios covered] +```