From 61dead4d1bea3c0eadd0184a07f4897cc8e94369 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:50:31 +0300 Subject: [PATCH 01/12] =?UTF-8?q?docs(specs):=20103-token-bench=20?= =?UTF-8?q?=E2=80=94=20measured=20savings,=20published=20results?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec for the token-bench epic, unblocked when spec 102 shipped in #1063. The premise: MCPProxy's "massive token savings" claim currently rests on a static tool-surface measurement. That number is real and CI-gated, but it is not what a user pays. A mode that halves the menu and doubles failed attempts costs more, not less, and nothing in the repo measures that. Two facts make it urgent. Spec 102 projected ~70% and measured 29.7% on the frozen 45-tool corpus, concluding that names/descriptions/annotations rather than schemas dominate the payload — a conclusion drawn from two frozen corpora and never checked against a real fleet. And the session-cost rows in today's report are computed from literature-derived retry defaults, which the dashboard honestly labels as estimates; every session-level savings claim inherits that assumption. Four user stories, P1 first: measure cost per COMPLETED task and first-call success on replayed real sessions; let an outsider reproduce the numbers; confirm or correct the spec-102 payload decomposition; publish. 27 functional requirements, 10 success criteria, no [NEEDS CLARIFICATION] markers. SC-006 is the one that matters most — a mode that costs fewer surface tokens but completes fewer tasks must look WORSE in the headline metric, so the benchmark is required to detect a false saving rather than reward it. A Scope Boundary table names what the existing harness already provides (encoding arms, live-builder-derived token counting, response cost, break-even, retrieval metrics, the provenance-badged report, the frozen corpora) and states that rebuilding any of it is a defect. The genuinely missing pieces are session replay, measured success/retry rates, tokens-per-successful-task, crossing the full three-axis mode matrix, and the payload decomposition. Three decisions were resolved as documented assumptions rather than clarification markers: the pinned model is a plan-stage budget decision and the deterministic half stands alone without it; a task-completion signal is assumed derivable from recorded sessions, with an explicit fallback, and this is the assumption most likely to be wrong; and the public suite is specified as a capability rather than named, because the 2026-08 shortlist needs re-confirmation against the suites' current code before being committed to. --- .../checklists/requirements.md | 70 ++++ specs/103-token-bench/spec.md | 385 ++++++++++++++++++ 2 files changed, 455 insertions(+) create mode 100644 specs/103-token-bench/checklists/requirements.md create mode 100644 specs/103-token-bench/spec.md diff --git a/specs/103-token-bench/checklists/requirements.md b/specs/103-token-bench/checklists/requirements.md new file mode 100644 index 000000000..ee0dd6452 --- /dev/null +++ b/specs/103-token-bench/checklists/requirements.md @@ -0,0 +1,70 @@ +# Specification Quality Checklist: Token-efficiency benchmark — measured savings, published results + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-08-31 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +Validation ran over three iterations; the following were fixed rather than waived. + +**Iteration 1 — implementation detail leaked into a spec that must stay declarative.** +The draft named concrete files, flags, CLI commands, struct fields and PR numbers +(`bench/session.go`, `armRetryRates`, `ResponseTruncated`, `-corpus-v2`, `#747`). All were +removed from requirements and success criteria. This was the hardest constraint to hold, +because the feature's defining risk — duplicating a harness that already exists — is +naturally expressed in file names. It is instead expressed as the **Scope Boundary** table +in capability terms, which a non-implementer can still act on. + +**Iteration 2 — success criteria were not all verifiable without implementation knowledge.** +Rewritten so each states an outcome a reader can check: SC-003 (an outsider reproduces the +numbers), SC-006 (the benchmark detects a false saving), SC-007 (truncated input is never +counted silently). + +**Iteration 3 — scope was bounded but the *anti*-scope was not.** +Added Out of Scope, because the likeliest failure of this feature is not under-delivery but +sprawl into optimisation work the measurement merely suggests. + +**No [NEEDS CLARIFICATION] markers were used.** Three candidates were resolved by informed +default and recorded in Assumptions instead: +- *Which model to pin for agent-loop runs* — deferred to plan stage as a budget decision, + and the deterministic half is specified to stand alone without it (FR-015 constrains how + results are reported, not which model produces them). +- *Whether a completion signal can be derived from recorded sessions* — assumed derivable, + with an explicit fallback (report sessions lacking the signal rather than assuming + success). This is the assumption most likely to be wrong and should be validated first at + plan stage. +- *Which public suite to adopt* — deliberately left as a capability requirement (FR-020: + "at least one full agent-loop suite") rather than naming one, because the shortlist was + researched in 2026-08 and needs re-confirmation against the suites' current code before + being committed to. + +**Carried into planning**: the research pass flagged two items as unverified — whether the +candidate suites emit per-task token counts, and whether they can be pointed at a single +proxy endpoint. FR-020 and FR-022 are unsatisfiable if neither holds, so plan stage must +read the suites' code before committing to one. diff --git a/specs/103-token-bench/spec.md b/specs/103-token-bench/spec.md new file mode 100644 index 000000000..bbcb45928 --- /dev/null +++ b/specs/103-token-bench/spec.md @@ -0,0 +1,385 @@ +# Feature Specification: Token-efficiency benchmark — measured savings, published results + +**Feature Branch**: `103-token-bench` +**Created**: 2026-08-31 +**Status**: Draft +**Input**: User description: "token-bench: measure real token cost AND task-success rate across every mcpproxy routing/serialization mode combination, on replayed real agent sessions and public agent-loop suites, so every savings number mcpproxy quotes becomes a reproducible measurement" + +## Why This Exists + +MCPProxy's headline promise is "massive token savings". Today that promise rests on a +measurement of the **static tool surface** — how many tokens the tool definitions cost an +agent before it does any work. That number is real, reproducible and already gated in CI. + +It is also not the number a user cares about. A user cares what a *task* costs. A mode that +halves the tool surface but doubles the number of failed attempts costs more, not less. +Nothing in the repo measures that today. + +Two concrete facts make this urgent: + +1. **Spec 102 projected ~70% reduction and measured 29.7%** on the frozen 45-tool corpus + (34.8% on a 527-tool snapshot). Its success criterion was restated per corpus shape + rather than the design being changed, and its own gate concluded that names, + descriptions and annotations — not schemas — dominate the payload, with 38.9% the + arithmetic ceiling on that corpus even if both schema and signature were deleted. + That conclusion is drawn from two frozen corpora and has never been checked against a + real fleet. +2. **Retry rates are assumed, not observed.** The session-cost rows in today's report are + computed from per-arm literature-derived defaults, and the dashboard honestly labels + them `estimated`. Every session-level savings claim inherits that assumption. + +This feature replaces assumptions with measurements, and publishes the methodology so the +numbers can be checked by someone who does not trust us. + +## Scope Boundary *(read before planning)* + +An extensive benchmark harness already exists and is CI-wired. **This feature extends it. +Re-implementing any of the following is out of scope and would be a defect:** + +| Already exists | Do not rebuild | +|---|---| +| Six tool-definition encoding arms, incl. the spec-102 deferred rendering | arm framework, arm contracts | +| Token counting derived from the *live* tool builders (cannot drift from production) | tokenizer, catalog extraction | +| Live response-cost measurement, break-even analysis | response cost, break-even | +| Retrieval quality metrics (recall@k, nDCG) and a parity gate through the production index | retrieval scoring | +| Versioned report + self-contained dashboard with `measured`/`computed`/`estimated` provenance badges on every headline number | report schema, dashboard | +| Frozen tool corpora at 45-tool and 527-tool scale, plus an external tool-retrieval corpus and an independent third-party linter | corpus loaders | + +The provenance-badge convention is load-bearing for this feature: everything it adds must +declare whether it is measured, computed or estimated, and the point of the work is to move +specific numbers from `estimated` to `measured`. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 — A maintainer measures what a task actually costs (Priority: P1) + +A maintainer wants to know, for a given mode combination, how many tokens it takes to +finish a real unit of work — not how many tokens the menu costs. They point the harness at +a set of previously recorded real sessions, run every mode combination over them, and get a +per-mode cost-per-completed-task figure alongside how often the agent got the call right +the first time. + +**Why this priority**: This is the feature's core claim and the only part that cannot be +approximated from what exists. Without it, every session-level number stays an estimate. + +**Independent Test**: Record or supply a set of real sessions, run the harness across the +mode matrix, and confirm the report contains a per-mode cost-per-completed-task and a +first-call success rate, each marked `measured` rather than `estimated`. + +**Acceptance Scenarios**: + +1. **Given** a set of exported real sessions, **When** a maintainer runs the replay across + all mode combinations, **Then** the report shows, per mode, tokens per completed task, + first-call success rate and retry count, each carrying a `measured` provenance badge. +2. **Given** a recorded session whose stored responses were truncated when captured, + **When** the replay processes it, **Then** that session is excluded from token totals or + explicitly annotated as under-counting, and the report states how many sessions were + affected — it is never silently included. +3. **Given** two runs of the same replay over the same session set and the same modes, + **When** their reports are compared, **Then** the deterministic figures are identical and + any model-dependent figure is reported as a mean across runs with its spread. +4. **Given** a mode that reduces tool-surface tokens but lowers task completion, + **When** the report is generated, **Then** its cost-per-completed-task rises, making the + regression visible rather than hidden behind a favourable surface number. + +--- + +### User Story 2 — A skeptical reader reproduces the published numbers (Priority: P2) + +Someone who does not trust the project reads a published claim, follows the documented +procedure, and arrives at the same numbers within a stated tolerance — including the +comparison against not using MCPProxy at all. + +**Why this priority**: An unreproducible benchmark is marketing. This is what separates +this work from the thin-methodology comparisons already circulating in this space. It +depends on US1 producing numbers worth reproducing. + +**Independent Test**: A person with no prior context follows the published procedure on +their own machine and reproduces the deterministic figures exactly and the model-dependent +figures within the stated tolerance. + +**Acceptance Scenarios**: + +1. **Given** the published methodology, **When** an outside reader follows it, **Then** every + input needed is either included in the repository or fetched by a documented pinned + procedure, and no step depends on data only the maintainers hold. +2. **Given** a published comparison, **When** a reader inspects the baseline, **Then** the + baseline is the same agent doing the same tasks with all tools loaded directly, not a + different or weaker configuration. +3. **Given** any published percentage, **When** a reader looks it up, **Then** it is + accompanied by the size and shape of the tool set it was measured on, because the + figure moves with fleet size. +4. **Given** a headline figure, **When** a reader traces it, **Then** they can reach the raw + per-run records it was computed from. + +--- + +### User Story 3 — The 29.7% shortfall gets an answer (Priority: P2) + +A maintainer wants to know whether the spec-102 conclusion — that names, descriptions and +annotations dominate the payload, capping achievable savings well below the projection — +holds on real fleets, or was an artefact of two frozen corpora. + +**Why this priority**: This determines whether further serialization work is worth doing at +all, and it is the first question anyone will ask about the published numbers. It reuses +US1's machinery rather than adding its own. + +**Independent Test**: Produce a payload decomposition across corpora of different sizes and +shapes and compare it against the spec-102 conclusion, yielding an explicit confirmation +or correction. + +**Acceptance Scenarios**: + +1. **Given** tool sets of varying size and shape, **When** the decomposition runs, **Then** + the report attributes payload share to names, descriptions, annotations and schemas + separately, and shows how each share moves with fleet size. +2. **Given** that decomposition, **When** it is compared with the spec-102 conclusion, + **Then** the result explicitly states whether that conclusion is confirmed or corrected, + and the arithmetic ceiling is recomputed per corpus rather than assumed. + +--- + +### User Story 4 — Results reach the people deciding whether to adopt (Priority: P3) + +A developer evaluating MCPProxy reads a public write-up with real numbers, an honest +account of where savings do *not* materialise, and enough method to judge it. + +**Why this priority**: The measurement has no external value until it is published, but it +is strictly downstream of having trustworthy numbers. + +**Independent Test**: The published write-up states the measured figures, the corpus shapes +they hold for, the known limitations, and the reproduction procedure. + +**Acceptance Scenarios**: + +1. **Given** the write-up, **When** a reader finishes it, **Then** they know which mode to + choose for their own fleet size and why. +2. **Given** a case where a mode does not help, **When** the reader looks for it, **Then** + the write-up says so plainly rather than omitting it. + +--- + +### Edge Cases + +- **A recorded session cannot be replayed** because the upstream servers it used are gone + or have changed: the session must be reported as unreplayable rather than counted as a + failure of the mode under test, since that would blame the mode for missing infrastructure. +- **Recorded responses were truncated at capture time**, so replaying them under-counts + tokens. Detection is mandatory; silent inclusion is the single most dangerous failure + mode here, because it biases every number in the project's favour. +- **Response bodies are unavailable** because the export withheld them: the harness must + say what it cannot measure rather than substituting a guess. +- **A mode combination is not valid** (an axis that does not apply to the selected routing + mode): the matrix must skip it explicitly and show it as skipped, not as zero. +- **A task suite performs real writes** against live third-party services: destructive + operations must be handled deliberately, and a benchmark run must never be capable of + damaging a real user's data. +- **A run is interrupted partway** through an expensive matrix: partial results must be + identifiable as partial and must never be published as a complete comparison. +- **A model-dependent figure comes from a single run**: single-run agentic numbers are + noise; the report must refuse to present one as a headline. +- **Sessions contain sensitive data** — real recorded traffic may include secrets or + personal data. Replay inputs must be handled so that publishing results never publishes + session contents. + +## Requirements *(mandatory)* + +### Functional Requirements + +**Replay of real sessions** + +- **FR-001**: The harness MUST accept previously recorded real agent sessions as input via + the existing export path, without requiring a new capture mechanism. +- **FR-002**: The harness MUST detect recorded entries whose stored content was truncated at + capture time and MUST either exclude them from token totals or annotate them as + under-counting; it MUST NOT include them silently. +- **FR-003**: The harness MUST report how many supplied sessions were unusable and why, + distinguishing truncation, missing bodies, and unreplayable upstreams. +- **FR-004**: The harness MUST treat recorded session content as sensitive: results and + published artefacts MUST NOT contain session contents. + +**Measured success and retries** + +- **FR-005**: The harness MUST measure first-call success rate — how often the agent's first + attempt at a tool call is accepted — per mode combination. +- **FR-006**: The harness MUST measure retry counts per mode combination, replacing the + current assumed per-arm defaults for any mode where a measurement exists. +- **FR-007**: Any figure still derived from an assumption MUST remain marked as an estimate + and MUST NOT be presented alongside measured figures without that distinction. +- **FR-008**: The harness MUST report a task-completion signal per replayed unit of work, so + that cost can be expressed per *completed* task rather than per attempt. + +**The mode matrix** + +- **FR-009**: The harness MUST cross all three configuration axes that govern what an agent + sees: the routing mode, the serialization mode of the discovery surface, and the + serialization mode of the direct surface. +- **FR-010**: The matrix MUST additionally cover the batching, stored-script and + validate-before-dispatch capabilities, since each trades one kind of cost for another. +- **FR-011**: Combinations that are not meaningful MUST be reported as skipped with a reason, + never as a zero or a missing row. +- **FR-012**: The report MUST express headline cost as tokens per completed task, with the + static tool-surface cost retained as a separate, clearly-labelled figure. +- **FR-013**: Token accounting MUST separate input, output and cached-read consumption. + +**Honest comparison** + +- **FR-014**: The baseline arm MUST be the same agent performing the same tasks with all + tools loaded directly. A comparison against a different or weaker configuration MUST NOT + be published. +- **FR-015**: Any model-dependent figure MUST be reported as an average over at least four + runs together with a measure of consistency across runs; a single run MUST NOT be + reported as a headline result. +- **FR-016**: Every published percentage MUST carry the size and shape of the tool set it was + measured on. +- **FR-017**: The report MUST include a cost-versus-outcome view so a reader can see which + modes are worth their savings rather than only which are cheapest. + +**Payload decomposition** + +- **FR-018**: The harness MUST attribute tool-definition payload share to names, + descriptions, annotations and schemas separately, across at least two corpus shapes. +- **FR-019**: The result MUST explicitly confirm or correct the spec-102 conclusion about + what dominates the payload, and MUST recompute the achievable ceiling per corpus rather + than carrying a fixed figure forward. + +**Public suites** + +- **FR-020**: The harness MUST support running at least one public task suite that exercises + a full agent loop, in addition to the retrieval and token corpora that already exist. +- **FR-021**: Any suite performing real writes against third-party services MUST be run in a + way that cannot damage real user data. +- **FR-022**: Suite versions and configuration MUST be pinned so a later run is comparable to + an earlier one. + +**Reproducibility and publication** + +- **FR-023**: Generated reports MUST NOT be committed to the repository; only code, fixtures, + thresholds and methodology are versioned. +- **FR-024**: Raw per-run records MUST be retained and referenced by the report so a headline + figure can be traced to its inputs. +- **FR-025**: Every input MUST be either included in the repository or obtainable by a + documented, pinned procedure. +- **FR-026**: The published write-up MUST state measured figures, the corpus shapes they hold + for, known limitations, and the reproduction procedure, and MUST state plainly where + savings do not materialise. +- **FR-027**: A partial or interrupted run MUST be identifiable as partial and MUST NOT be + publishable as a complete comparison. + +### Key Entities + +- **Replayed session**: A previously recorded unit of real agent work, with its sequence of + tool calls and enough context to re-run it. Carries usability flags (truncated, missing + bodies, unreplayable). +- **Mode combination**: One point in the configuration matrix — a routing mode plus the + serialization settings and capability toggles that determine what the agent sees and how + it calls. +- **Run record**: The raw outcome of executing one session under one mode combination: + token consumption split by kind, attempts, first-call outcome, completion, and any error. +- **Measurement**: An aggregate over run records, always carrying its provenance + (`measured`, `computed`, or `estimated`) and, where model-dependent, its run count and + spread. +- **Payload decomposition**: The attribution of tool-definition cost to names, descriptions, + annotations and schemas for a given corpus. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: For every mode combination in the matrix, a maintainer can obtain a cost per + completed task and a first-call success rate, both marked as measured rather than + estimated. +- **SC-002**: No session-level cost figure in the report is derived from an assumed retry + rate unless it is explicitly labelled as an estimate. +- **SC-003**: A person outside the project, following only the published procedure, + reproduces the deterministic figures exactly and the model-dependent figures within the + stated tolerance. +- **SC-004**: Every published percentage states the tool-set size and shape it was measured + on; a percentage without that context does not appear. +- **SC-005**: The spec-102 payload conclusion is explicitly confirmed or corrected, with the + achievable ceiling recomputed for each corpus measured. +- **SC-006**: A deliberately degraded mode — one that costs fewer surface tokens but + completes fewer tasks — is visibly worse in the report's headline metric, demonstrating + that the benchmark detects a false saving rather than rewarding it. +- **SC-007**: Recorded sessions with truncated content are never counted silently: the + report states how many were affected and how they were handled. +- **SC-008**: At least one full agent-loop public task suite runs against MCPProxy and + produces comparable results across at least two mode combinations. +- **SC-009**: No generated report is committed to the repository. +- **SC-010**: A reader of the published write-up can state which mode suits their fleet size + and can name at least one case where MCPProxy's savings do not materialise. + +## Assumptions + +- **Existing harness is the foundation.** Everything listed under Scope Boundary is treated + as available and correct; this feature adds to it rather than replacing it. If a + measurement here contradicts an existing one, that is a finding to investigate, not a + reason to fork the harness. +- **Recorded sessions are the primary corpus.** Real recorded work is more representative + than synthetic tasks, so it drives the headline. Public suites provide external validity + and comparability, not the headline. +- **A task-completion signal can be derived from recorded sessions.** Where it cannot be + derived reliably, the affected sessions are reported as lacking a completion signal + rather than being assumed complete. +- **Model-dependent measurement costs money and needs a decision.** Any full agent-loop run + requires a pinned model and a spending decision before it can be built; the deterministic + parts of this feature are designed to stand alone without it. +- **Reporting conventions carry over.** The provenance-badge discipline, the + reports-are-never-committed rule, and the requirement to quote corpus size alongside any + percentage all continue to apply. +- **Telemetry cross-validation is downstream.** Confirming these findings against the real + user population is a later, separate step and is not required for this feature to be + complete. +- **Publication happens outside this repository**, so the write-up is a deliverable of this + feature but lands elsewhere; it extends the existing published comparison work rather + than duplicating it. + +## Out of Scope + +- Changing any routing or serialization behaviour. This feature measures; it does not + optimise. Findings may motivate later work. +- Real-user population validation through telemetry counters. +- Retrieval-quality evaluation, which already exists and is separately gated. +- Suites that are not MCP-native, and any suite that cannot be pointed at a single proxy + endpoint. + +## Dependencies + +- The recorded-session export path must expose enough per-call detail — including response + content when explicitly requested — for token accounting. +- The mode configuration axes must be settable per run so the matrix can be crossed. +- A pinned model and a spending decision are prerequisites for the agent-loop portions + (US2's public-suite half and any measured retry rate that requires live inference). + +## Commit Message Conventions *(mandatory)* + +When committing changes for this feature, follow these guidelines: + +### Issue References +- ✅ **Use**: `Related #[issue-number]` - Links the commit to the issue without auto-closing +- ❌ **Do NOT use**: `Fixes #[issue-number]`, `Closes #[issue-number]`, `Resolves #[issue-number]` - These auto-close issues on merge + +**Rationale**: Issues should only be closed manually after verification and testing in production, not automatically on merge. + +### Co-Authorship +- ❌ **Do NOT include**: `Co-Authored-By: Claude ` +- ❌ **Do NOT include**: "🤖 Generated with [Claude Code](https://claude.com/claude-code)" + +**Rationale**: Commit authorship should reflect the human contributors, not the AI tools used. + +### Example Commit Message +``` +feat: [brief description of change] + +Related #[issue-number] + +[Detailed description of what was changed and why] + +## Changes +- [Bulleted list of key changes] +- [Each change on a new line] + +## Testing +- [Test results summary] +- [Key test scenarios covered] +``` From 367ea8fe40fbc6112aa46f31cd92003558c57861 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 08:57:00 +0300 Subject: [PATCH 02/12] =?UTF-8?q?docs(specs):=20103-token-bench=20?= =?UTF-8?q?=E2=80=94=20restructure=20after=20cross-model=20review=20round?= =?UTF-8?q?=201?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven findings from the opencode gpt-5.6-sol review, all verified against the tree and all applied. Two were Critical and changed the spec's premise. 1. Replay cannot answer what the spec asked of it. The activity record carries tool calls, arguments, responses, status and work-session grouping — but no prompt, no conversation, no model state and no completion oracle. Replaying a recording under a different mode therefore cannot reveal how the agent would have BEHAVED: which calls it would attempt, whether the first was right, whether it finished. The original US1 asked for exactly that. Added a binding Replay Boundary section and split US1 into US1 (deterministic counterfactual cost recomputation over real workload shape, no model involved) and US2 (live agent loop under a pinned model, which owns every success and completion figure). FR-004 forbids inferring behaviour from a recording; Out of Scope names it as forbidden rather than merely excluded. 2. Reproducibility and privacy were in conflict and the privacy side was absent. The body-inclusive export deliberately returns full unmasked values including detected secrets — it is the compliance surface, not a browsing one. The spec kept session contents out of published artefacts but said nothing about protecting them during replay, including transmission to model providers. Added FR-006..FR-009 (sensitive by default; never transmitted to third parties; flagged records excluded or reduced; inputs live outside the repo with a documented deletion step) and SC-009. FR-030 now states that figures from private sessions are not independently reproducible and cannot be the sole support for a published claim. Also applied: 3. The headline metric did not penalise worse completion — tokens per completed task can improve while completion falls, if tokens fall faster. FR-018 requires completion rate at equal prominence, FR-019 adds a completion-regression threshold that overrides any token saving, and SC-007 requires demonstrating it with a deliberately degraded mode. 4. SC-001 demanded "every mode combination" while FR-017 requires skipping invalid ones; scoped to every valid combination. 5. Core terms were undefined; added a binding Definitions block that also separates corrective retries from infrastructure retries. 6. Three FRs restated existing harness capabilities; moved to Inherited Constraints (IC-001..IC-004) so they are satisfied rather than rebuilt. 7. The capability axes were not a testable matrix; FR-016 now states them as binary conditions over the routing modes where each is available. 32 functional requirements, 12 success criteria, 4 inherited constraints. The reviewer independently confirmed the spec's current-state claims: nothing in the harness replays activity exports, retry rates are literature-derived defaults, first-call success is not measured, the three axes are not crossed, and the spec-102 figures are accurate. --- .../checklists/requirements.md | 49 +++ specs/103-token-bench/spec.md | 342 ++++++++++++------ 2 files changed, 276 insertions(+), 115 deletions(-) diff --git a/specs/103-token-bench/checklists/requirements.md b/specs/103-token-bench/checklists/requirements.md index ee0dd6452..b3d988149 100644 --- a/specs/103-token-bench/checklists/requirements.md +++ b/specs/103-token-bench/checklists/requirements.md @@ -68,3 +68,52 @@ default and recorded in Assumptions instead: candidate suites emit per-task token counts, and whether they can be pointed at a single proxy endpoint. FR-020 and FR-022 are unsatisfiable if neither holds, so plan stage must read the suites' code before committing to one. + +## Cross-model review round 1 (opencode gpt-5.6-sol, 2026-08-31) + +Seven findings, all verified against the tree and all applied. Two were **Critical and +changed the spec's premise**: + +1. **Replay cannot answer what the spec asked it to.** The activity record carries tool + calls, arguments, responses, status and work-session grouping — but no prompt, no + conversation, no model state and no completion oracle (`internal/storage/activity_models.go`). + So replaying a recording under a different mode cannot reveal how the agent would have + *behaved*: which calls it would attempt, whether the first was right, whether it finished. + The original US1 asked for exactly that and was unsatisfiable. + **Fix**: added a binding **Replay Boundary** section and split the old US1 into US1 + (deterministic counterfactual cost recomputation over real workload shape, no model) and + US2 (live agent loop under a pinned model, which owns every success/completion figure). + FR-004 now forbids inferring behaviour from a recording, and Out of Scope names it. + +2. **Reproducibility and privacy were in direct conflict, and the privacy side was absent.** + The body-inclusive export deliberately returns full unmasked values including detected + secrets — it is the compliance surface, not a browsing one (`internal/httpapi/activity.go`). + The spec said session contents must stay out of published artefacts but said nothing about + protecting them *during* replay, including transmission to model providers. + **Fix**: new FR-006..FR-009 (sensitive by default, never transmitted to third parties, + flagged records excluded or reduced, inputs live outside the repo with a documented + deletion step) plus SC-009. FR-030 now admits that figures from private sessions are not + independently reproducible and cannot be the sole support for a published claim. + +Also applied: + +3. **The headline metric did not actually penalise worse completion.** Tokens-per-completed-task + can *improve* while completion falls, if tokens fall faster — so the old SC-006 was not + generally true. **Fix**: FR-018 requires completion rate at equal prominence, FR-019 adds a + completion-regression threshold that overrides any token saving, and SC-007 now requires + demonstrating this with a deliberately degraded mode. +4. **SC-001 demanded results for "every mode combination"** while FR-017 requires skipping + invalid ones. **Fix**: scoped to every *valid* combination exercised by the live task set. +5. **Core terms were undefined** ("first attempt", "accepted", "retry", "unit of work", + "completion", token accounting source). **Fix**: a binding Definitions block, which also + separates corrective retries from infrastructure retries. +6. **Three FRs restated existing capabilities.** **Fix**: moved to an Inherited Constraints + block (IC-001..IC-004) so they are satisfied, not rebuilt. +7. **The capability axes were not a testable matrix.** **Fix**: FR-016 states them as binary + conditions over the routing modes where each is available, with the report enumerating + applicable rows. + +The reviewer independently confirmed the spec's current-state claims: nothing in the harness +replays activity exports, retry rates are literature-derived defaults, first-call success is +not measured, the three axes are not crossed, and the spec-102 figures (29.7% / 34.8% / +38.9% ceiling vs a 70% projection) are accurate. diff --git a/specs/103-token-bench/spec.md b/specs/103-token-bench/spec.md index bbcb45928..fff4f5f02 100644 --- a/specs/103-token-bench/spec.md +++ b/specs/103-token-bench/spec.md @@ -49,42 +49,87 @@ The provenance-badge convention is load-bearing for this feature: everything it declare whether it is measured, computed or estimated, and the point of the work is to move specific numbers from `estimated` to `measured`. +## Replay Boundary *(read before planning — this bounds US1)* + +A recorded session is a trace of what an agent **did** under the mode it was running. It +carries the tool calls, their arguments, the responses, status, timing and the grouping that +ties calls into one unit of user work. It does **not** carry the user's prompt, the +conversation, the model's state, or any oracle for whether the user's goal was met. + +Therefore replay CAN answer: what this real workload, at this real fleet size, would have +cost under a different serialization or routing mode. That is arithmetic over recorded +traffic and is fully deterministic. + +Replay CANNOT answer: how the agent would have *behaved* differently — which calls it would +have attempted, whether its first attempt would have been right, whether it would have +finished. Those are new decisions by a model that is not present in the recording. + +Any requirement about success, retries or completion therefore belongs to a live agent loop +(US2), not to replay (US1). A design that infers agent behaviour from a recording is wrong, +and this boundary is the single most important thing for planning to respect. + ## User Scenarios & Testing *(mandatory)* -### User Story 1 — A maintainer measures what a task actually costs (Priority: P1) +### User Story 1 — Recompute real workload cost under every mode (Priority: P1) -A maintainer wants to know, for a given mode combination, how many tokens it takes to -finish a real unit of work — not how many tokens the menu costs. They point the harness at -a set of previously recorded real sessions, run every mode combination over them, and get a -per-mode cost-per-completed-task figure alongside how often the agent got the call right -the first time. +A maintainer takes a set of previously recorded real sessions — the actual sequence of tool +calls a real agent made, at real fleet size — and asks what that same workload would have +cost under every other mode combination. The answer is arithmetic over recorded traffic: no +model is invoked and no agent decision is predicted. -**Why this priority**: This is the feature's core claim and the only part that cannot be -approximated from what exists. Without it, every session-level number stays an estimate. +**Why this priority**: It is the cheapest way to replace synthetic corpora with real workload +shapes, it needs no model spend, and it is fully deterministic. It is also strictly bounded: +see the Replay Boundary below for what it cannot answer. -**Independent Test**: Record or supply a set of real sessions, run the harness across the -mode matrix, and confirm the report contains a per-mode cost-per-completed-task and a -first-call success rate, each marked `measured` rather than `estimated`. +**Independent Test**: Supply exported real sessions, run the recomputation across the mode +matrix, and confirm a per-mode cost for that workload, each figure marked `measured` or +`computed` and never `estimated`. **Acceptance Scenarios**: -1. **Given** a set of exported real sessions, **When** a maintainer runs the replay across - all mode combinations, **Then** the report shows, per mode, tokens per completed task, - first-call success rate and retry count, each carrying a `measured` provenance badge. -2. **Given** a recorded session whose stored responses were truncated when captured, - **When** the replay processes it, **Then** that session is excluded from token totals or - explicitly annotated as under-counting, and the report states how many sessions were - affected — it is never silently included. -3. **Given** two runs of the same replay over the same session set and the same modes, - **When** their reports are compared, **Then** the deterministic figures are identical and - any model-dependent figure is reported as a mean across runs with its spread. -4. **Given** a mode that reduces tool-surface tokens but lowers task completion, - **When** the report is generated, **Then** its cost-per-completed-task rises, making the - regression visible rather than hidden behind a favourable surface number. +1. **Given** a set of exported real sessions, **When** a maintainer runs the recomputation, + **Then** the report shows per-mode token cost for that workload, broken into tool-surface + cost and response cost, on real fleet sizes rather than a frozen corpus. +2. **Given** a recorded session whose responses were truncated at capture, **When** it is + processed, **Then** it is excluded from token totals or explicitly annotated as + under-counting, and the report states how many sessions were affected — never silently + included. +3. **Given** two runs over the same sessions and modes, **When** their reports are compared, + **Then** the figures are byte-identical, since no model is involved. +4. **Given** any recomputed figure, **When** a reader inspects it, **Then** it is labelled as + a counterfactual over recorded traffic, not as observed agent behaviour. --- -### User Story 2 — A skeptical reader reproduces the published numbers (Priority: P2) +### User Story 2 — Measure whether a mode helps the agent succeed (Priority: P1) + +A maintainer wants the number that actually matters: tokens per *completed* task. This +requires a live agent loop under a pinned model, because it depends on decisions the agent +makes differently under each mode — which calls it attempts, how often the first attempt is +right, and whether it finishes. + +**Why this priority**: This is the feature's core claim. Without it every savings figure +remains a statement about menu size rather than about work done. It is separated from US1 +because it has a fundamentally different cost, determinism and data requirement. + +**Independent Test**: Run a fixed task set through a live agent under at least two mode +combinations and confirm the report gives, per mode, tokens per completed task, task +completion rate, first-attempt success rate and retry count, each `measured`. + +**Acceptance Scenarios**: + +1. **Given** a fixed task set, **When** it is run under each mode combination, **Then** the + report gives per mode: task completion rate, tokens per completed task, first-attempt + success rate and retry count, each `measured`. +2. **Given** a mode that lowers token use but completes fewer tasks, **When** the report is + read, **Then** the completion drop is displayed with equal prominence to the token + saving, and the mode is not presented as a saving. +3. **Given** any model-dependent figure, **When** it is published, **Then** it is an average + over at least four runs with its spread, never a single run. + +--- + +### User Story 3 — A skeptical reader reproduces the published numbers (Priority: P2) Someone who does not trust the project reads a published claim, follows the documented procedure, and arrives at the same numbers within a stated tolerance — including the @@ -92,7 +137,7 @@ comparison against not using MCPProxy at all. **Why this priority**: An unreproducible benchmark is marketing. This is what separates this work from the thin-methodology comparisons already circulating in this space. It -depends on US1 producing numbers worth reproducing. +depends on US1 and US2 producing numbers worth reproducing. **Independent Test**: A person with no prior context follows the published procedure on their own machine and reproduces the deterministic figures exactly and the model-dependent @@ -114,7 +159,7 @@ figures within the stated tolerance. --- -### User Story 3 — The 29.7% shortfall gets an answer (Priority: P2) +### User Story 4 — The 29.7% shortfall gets an answer (Priority: P2) A maintainer wants to know whether the spec-102 conclusion — that names, descriptions and annotations dominate the payload, capping achievable savings well below the projection — @@ -122,7 +167,7 @@ holds on real fleets, or was an artefact of two frozen corpora. **Why this priority**: This determines whether further serialization work is worth doing at all, and it is the first question anyone will ask about the published numbers. It reuses -US1's machinery rather than adding its own. +the existing tool-surface measurement machinery rather than adding its own. **Independent Test**: Produce a payload decomposition across corpora of different sizes and shapes and compare it against the spec-102 conclusion, yielding an explicit confirmation @@ -139,7 +184,7 @@ or correction. --- -### User Story 4 — Results reach the people deciding whether to adopt (Priority: P3) +### User Story 5 — Results reach the people deciding whether to adopt (Priority: P3) A developer evaluating MCPProxy reads a public write-up with real numbers, an honest account of where savings do *not* materialise, and enough method to judge it. @@ -184,9 +229,38 @@ they hold for, the known limitations, and the reproduction procedure. ## Requirements *(mandatory)* +### Definitions *(binding — these terms are used with exactly these meanings)* + +- **Unit of work**: one task from a fixed task set (US2), or one recorded work-session + grouping (US1). Never an individual tool call. +- **Attempt**: one tool call issued by the agent toward a given intent. +- **First-attempt success**: the first attempt for an intent returned a non-error result AND + was not followed by a corrective retry for the same intent. A schema-valid call that the + agent immediately re-issues differently is NOT a success. +- **Retry**: a subsequent attempt for the same intent after a failed or corrected one. + Transport-level and infrastructure retries are counted and reported SEPARATELY, because + they measure the network, not the mode. +- **Task completion**: the task suite's own pass verdict. Where a suite has no verdict, the + unit of work is reported as having no completion signal and is excluded from + completion-dependent figures rather than assumed complete. +- **Token accounting**: provider-reported usage where the run is model-backed; the existing + deterministic tokenizer where it is not. Which one produced a figure MUST be stated; the + two MUST NOT be summed into one number. +- **Fleet shape**: the tool count plus the distribution of definition sizes for the tool set + a figure was measured on. + +### Inherited constraints *(already provided by the existing harness — satisfy, do not rebuild)* + +- **IC-001**: Every figure carries a `measured` / `computed` / `estimated` provenance badge. +- **IC-002**: Static tool-surface cost is measured from the live tool builders for every + routing mode and MUST NOT be re-derived. +- **IC-003**: Generated reports are never committed; only code, fixtures, thresholds and + methodology are versioned. +- **IC-004**: Any quoted percentage carries the fleet shape it was measured on. + ### Functional Requirements -**Replay of real sessions** +**Replay of real sessions (deterministic; see Replay Boundary)** - **FR-001**: The harness MUST accept previously recorded real agent sessions as input via the existing export path, without requiring a new capture mechanism. @@ -195,118 +269,145 @@ they hold for, the known limitations, and the reproduction procedure. under-counting; it MUST NOT include them silently. - **FR-003**: The harness MUST report how many supplied sessions were unusable and why, distinguishing truncation, missing bodies, and unreplayable upstreams. -- **FR-004**: The harness MUST treat recorded session content as sensitive: results and - published artefacts MUST NOT contain session contents. - -**Measured success and retries** - -- **FR-005**: The harness MUST measure first-call success rate — how often the agent's first - attempt at a tool call is accepted — per mode combination. -- **FR-006**: The harness MUST measure retry counts per mode combination, replacing the - current assumed per-arm defaults for any mode where a measurement exists. -- **FR-007**: Any figure still derived from an assumption MUST remain marked as an estimate - and MUST NOT be presented alongside measured figures without that distinction. -- **FR-008**: The harness MUST report a task-completion signal per replayed unit of work, so - that cost can be expressed per *completed* task rather than per attempt. +- **FR-004**: Replay output MUST be labelled as a counterfactual cost recomputation over + recorded traffic. It MUST NOT be presented as observed agent behaviour, and the harness + MUST NOT infer success, retry or completion figures from a recording. +- **FR-005**: Replay MUST report cost split into tool-surface cost and response cost, so the + two can be reasoned about separately at real fleet shapes. + +**Handling of recorded session data** + +Recorded sessions are real user traffic. The export path that includes bodies deliberately +returns full unmasked values — it is the compliance/incident-response surface — so replay +inputs can contain secrets and personal data that the browsing surfaces mask. + +- **FR-006**: Replay inputs MUST be treated as sensitive by default. Neither they nor any + excerpt MUST appear in a report, dashboard or published artefact. +- **FR-007**: The harness MUST NOT transmit recorded session content to any third-party + service, including model providers. Replay is local arithmetic; a design requiring such + transmission is outside this feature. +- **FR-008**: Records flagged as containing sensitive data MUST be excluded, or reduced to + the non-sensitive measurements needed for token accounting, before use; the count of + affected records MUST be reported. +- **FR-009**: Replay inputs MUST live outside the repository, MUST NOT be committed, and the + documented procedure MUST tell an operator how to delete them when finished. + +**Measured success, retries and completion (live agent loop)** + +- **FR-010**: The harness MUST measure first-attempt success rate per mode combination, as + defined above. +- **FR-011**: The harness MUST measure retry counts per mode combination, reporting + corrective retries separately from infrastructure retries. +- **FR-012**: The harness MUST record a task-completion verdict per unit of work, taken from + the task suite rather than inferred. +- **FR-013**: Measured success and retry figures MUST replace the assumed per-arm defaults + for any mode where a measurement exists; any figure still derived from an assumption MUST + remain badged `estimated` and MUST NOT be presented without that distinction. +- **FR-014**: Token accounting MUST separate input, output and cached-read consumption. **The mode matrix** -- **FR-009**: The harness MUST cross all three configuration axes that govern what an agent - sees: the routing mode, the serialization mode of the discovery surface, and the - serialization mode of the direct surface. -- **FR-010**: The matrix MUST additionally cover the batching, stored-script and - validate-before-dispatch capabilities, since each trades one kind of cost for another. -- **FR-011**: Combinations that are not meaningful MUST be reported as skipped with a reason, +- **FR-015**: The matrix MUST cross the three configuration axes that determine what an agent + sees: routing mode, discovery-surface serialization, and direct-surface serialization. +- **FR-016**: Batching, stored scripts and validate-before-dispatch MUST each be covered as a + binary condition applied to the routing modes where they are available, and the report MUST + enumerate which rows each applies to. +- **FR-017**: Combinations that are not meaningful MUST be reported as skipped with a reason, never as a zero or a missing row. -- **FR-012**: The report MUST express headline cost as tokens per completed task, with the - static tool-surface cost retained as a separate, clearly-labelled figure. -- **FR-013**: Token accounting MUST separate input, output and cached-read consumption. +- **FR-018**: The report MUST express headline cost as tokens per completed task, with + completion rate displayed alongside it at equal prominence. +- **FR-019**: A mode whose completion rate falls below the baseline's by more than a stated + threshold MUST be marked as a regression regardless of its token cost, and MUST NOT be + described as a saving. **Honest comparison** -- **FR-014**: The baseline arm MUST be the same agent performing the same tasks with all +- **FR-020**: The baseline arm MUST be the same agent performing the same tasks with all tools loaded directly. A comparison against a different or weaker configuration MUST NOT be published. -- **FR-015**: Any model-dependent figure MUST be reported as an average over at least four - runs together with a measure of consistency across runs; a single run MUST NOT be - reported as a headline result. -- **FR-016**: Every published percentage MUST carry the size and shape of the tool set it was - measured on. -- **FR-017**: The report MUST include a cost-versus-outcome view so a reader can see which +- **FR-021**: Any model-dependent figure MUST be reported as an average over at least four + runs together with a measure of consistency across runs; a single run MUST NOT be reported + as a headline result. +- **FR-022**: The reproduction tolerance for model-dependent figures MUST be stated + numerically in the published methodology. +- **FR-023**: The report MUST include a cost-versus-outcome view so a reader can see which modes are worth their savings rather than only which are cheapest. **Payload decomposition** -- **FR-018**: The harness MUST attribute tool-definition payload share to names, - descriptions, annotations and schemas separately, across at least two corpus shapes. -- **FR-019**: The result MUST explicitly confirm or correct the spec-102 conclusion about +- **FR-024**: The harness MUST attribute tool-definition payload share to names, + descriptions, annotations and schemas separately, across at least two fleet shapes. +- **FR-025**: The result MUST explicitly confirm or correct the spec-102 conclusion about what dominates the payload, and MUST recompute the achievable ceiling per corpus rather than carrying a fixed figure forward. **Public suites** -- **FR-020**: The harness MUST support running at least one public task suite that exercises - a full agent loop, in addition to the retrieval and token corpora that already exist. -- **FR-021**: Any suite performing real writes against third-party services MUST be run in a +- **FR-026**: The harness MUST support at least one public task suite that exercises a full + agent loop and emits a per-task pass verdict. +- **FR-027**: Any suite performing real writes against third-party services MUST be run in a way that cannot damage real user data. -- **FR-022**: Suite versions and configuration MUST be pinned so a later run is comparable to +- **FR-028**: Suite versions and configuration MUST be pinned so a later run is comparable to an earlier one. **Reproducibility and publication** -- **FR-023**: Generated reports MUST NOT be committed to the repository; only code, fixtures, - thresholds and methodology are versioned. -- **FR-024**: Raw per-run records MUST be retained and referenced by the report so a headline - figure can be traced to its inputs. -- **FR-025**: Every input MUST be either included in the repository or obtainable by a - documented, pinned procedure. -- **FR-026**: The published write-up MUST state measured figures, the corpus shapes they hold +- **FR-029**: Raw per-run records MUST be retained and referenced by the report so a headline + figure can be traced to its inputs, without embedding session contents (FR-006). +- **FR-030**: Every input MUST be either included in the repository or obtainable by a + documented, pinned procedure. Figures that depend on private recorded sessions MUST be + marked as not independently reproducible, and MUST NOT be the sole support for a published + claim. +- **FR-031**: The published write-up MUST state measured figures, the fleet shapes they hold for, known limitations, and the reproduction procedure, and MUST state plainly where savings do not materialise. -- **FR-027**: A partial or interrupted run MUST be identifiable as partial and MUST NOT be +- **FR-032**: A partial or interrupted run MUST be identifiable as partial and MUST NOT be publishable as a complete comparison. ### Key Entities -- **Replayed session**: A previously recorded unit of real agent work, with its sequence of - tool calls and enough context to re-run it. Carries usability flags (truncated, missing - bodies, unreplayable). -- **Mode combination**: One point in the configuration matrix — a routing mode plus the - serialization settings and capability toggles that determine what the agent sees and how - it calls. -- **Run record**: The raw outcome of executing one session under one mode combination: - token consumption split by kind, attempts, first-call outcome, completion, and any error. -- **Measurement**: An aggregate over run records, always carrying its provenance - (`measured`, `computed`, or `estimated`) and, where model-dependent, its run count and - spread. -- **Payload decomposition**: The attribution of tool-definition cost to names, descriptions, - annotations and schemas for a given corpus. +- **Replayed session**: A previously recorded unit of real agent work — its tool calls, + arguments and responses — with usability flags (truncated, missing bodies, unreplayable, + sensitive). +- **Mode combination**: One point in the matrix — routing mode plus serialization settings + and capability toggles determining what the agent sees and how it calls. +- **Run record**: The raw outcome of one unit of work under one mode: token consumption by + kind, attempts, first-attempt outcome, retries by class, completion verdict, errors. +- **Measurement**: An aggregate over run records, carrying provenance and, where + model-dependent, run count and spread. +- **Payload decomposition**: Attribution of tool-definition cost to names, descriptions, + annotations and schemas for a given fleet shape. ## Success Criteria *(mandatory)* ### Measurable Outcomes -- **SC-001**: For every mode combination in the matrix, a maintainer can obtain a cost per - completed task and a first-call success rate, both marked as measured rather than - estimated. -- **SC-002**: No session-level cost figure in the report is derived from an assumed retry - rate unless it is explicitly labelled as an estimate. -- **SC-003**: A person outside the project, following only the published procedure, +- **SC-001**: For every VALID mode combination exercised by the live task set, a maintainer + can obtain tokens per completed task, completion rate and first-attempt success rate, each + badged `measured`. +- **SC-002**: For every valid mode combination, a maintainer can obtain the counterfactual + cost of a real recorded workload, reproducible byte-for-byte across runs. +- **SC-003**: No session-level cost figure is derived from an assumed retry rate unless it is + explicitly badged `estimated`. +- **SC-004**: A person outside the project, following only the published procedure, reproduces the deterministic figures exactly and the model-dependent figures within the - stated tolerance. -- **SC-004**: Every published percentage states the tool-set size and shape it was measured - on; a percentage without that context does not appear. -- **SC-005**: The spec-102 payload conclusion is explicitly confirmed or corrected, with the - achievable ceiling recomputed for each corpus measured. -- **SC-006**: A deliberately degraded mode — one that costs fewer surface tokens but - completes fewer tasks — is visibly worse in the report's headline metric, demonstrating - that the benchmark detects a false saving rather than rewarding it. -- **SC-007**: Recorded sessions with truncated content are never counted silently: the - report states how many were affected and how they were handled. -- **SC-008**: At least one full agent-loop public task suite runs against MCPProxy and + stated numeric tolerance. +- **SC-005**: Every published percentage states the fleet shape it was measured on; a + percentage without that context does not appear. +- **SC-006**: The spec-102 payload conclusion is explicitly confirmed or corrected, with the + achievable ceiling recomputed for each fleet shape measured. +- **SC-007**: A mode that lowers token cost while completing fewer tasks is flagged as a + regression by the completion threshold and is not reported as a saving — demonstrated by + running a deliberately degraded mode. +- **SC-008**: Recorded sessions with truncated content are never counted silently: the report + states how many were affected and how they were handled. +- **SC-009**: No recorded session content, and no excerpt of it, appears in any report, + dashboard or published artefact, and no such content is transmitted to a third-party + service. +- **SC-010**: At least one full agent-loop public task suite runs against MCPProxy and produces comparable results across at least two mode combinations. -- **SC-009**: No generated report is committed to the repository. -- **SC-010**: A reader of the published write-up can state which mode suits their fleet size +- **SC-011**: No generated report is committed to the repository. +- **SC-012**: A reader of the published write-up can state which mode suits their fleet shape and can name at least one case where MCPProxy's savings do not materialise. ## Assumptions @@ -315,15 +416,19 @@ they hold for, the known limitations, and the reproduction procedure. as available and correct; this feature adds to it rather than replacing it. If a measurement here contradicts an existing one, that is a finding to investigate, not a reason to fork the harness. -- **Recorded sessions are the primary corpus.** Real recorded work is more representative - than synthetic tasks, so it drives the headline. Public suites provide external validity - and comparability, not the headline. -- **A task-completion signal can be derived from recorded sessions.** Where it cannot be - derived reliably, the affected sessions are reported as lacking a completion signal - rather than being assumed complete. +- **Recorded sessions supply real workload SHAPE, not agent behaviour.** They drive the + cost recomputation (US1) because real fleets beat synthetic corpora. They cannot drive the + success/completion headline, because a recording contains no prompt, conversation, model + state or completion oracle — see Replay Boundary. The live task set (US2) owns that. +- **Task completion comes from the task suite's own verdict**, never inferred from a call + trace. Units of work with no verdict are reported as lacking a completion signal and + excluded from completion-dependent figures rather than assumed complete. - **Model-dependent measurement costs money and needs a decision.** Any full agent-loop run - requires a pinned model and a spending decision before it can be built; the deterministic - parts of this feature are designed to stand alone without it. + requires a pinned model and a spending decision before it can be built. US1 is designed to + deliver value alone without it, so the feature is not blocked on that decision. +- **Private recorded sessions cannot be published**, so figures derived from them are not + independently reproducible by an outsider. Public suites and frozen corpora therefore carry + the reproducible claims; recorded-session figures corroborate them at real fleet shape. - **Reporting conventions carry over.** The provenance-badge discipline, the reports-are-never-committed rule, and the requirement to quote corpus size alongside any percentage all continue to apply. @@ -342,11 +447,18 @@ they hold for, the known limitations, and the reproduction procedure. - Retrieval-quality evaluation, which already exists and is separately gated. - Suites that are not MCP-native, and any suite that cannot be pointed at a single proxy endpoint. +- Inferring agent behaviour, success or completion from recorded traces. This is not merely + out of scope: it is forbidden by FR-004, because it would fabricate the feature's headline. +- Any new activity capture mechanism. If the existing export proves insufficient for cost + recomputation, that is a finding to report, not scope to absorb here. ## Dependencies - The recorded-session export path must expose enough per-call detail — including response - content when explicitly requested — for token accounting. + content when explicitly requested — for token accounting. It is already known NOT to carry + prompts, conversation or completion verdicts, which is why US2 exists separately. +- A task suite with per-task pass verdicts is required for every completion-dependent + figure; without one, US2 cannot be satisfied by any amount of recorded data. - The mode configuration axes must be settable per run so the matrix can be crossed. - A pinned model and a spending decision are prerequisites for the agent-loop portions (US2's public-suite half and any measured retry rate that requires live inference). From c84ea65132dd93812eb7b1492245b3f48a8ade6c Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:16:03 +0300 Subject: [PATCH 03/12] =?UTF-8?q?docs(specs):=20103-token-bench=20plan=20?= =?UTF-8?q?=E2=80=94=205-cell=20matrix,=20MCPMark,=20split=20token=20accou?= =?UTF-8?q?nting?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0 research (5 parallel investigations) and Phase 1 design artifacts. Research changed three of the spec's assumptions, all toward less work. THE MATRIX IS 5 CELLS, NOT 12. The three axes are not a free product: each serialization axis governs exactly one surface. tool_response_mode has a single consumer inside the retrieve_tools handler, and direct_tool_response_mode is read only when building the direct listing, so seven combinations are structurally impossible and become skip-with-reason rows. contracts/mode-matrix.md enumerates all five valid cells and all seven skips with their causes. THE MATRIX CROSSES ON ONE PROXY INSTANCE. All three routing-mode servers are built at startup and all three endpoints stay mounted regardless of config, so a cell is selected by endpoint URL — not by a config change plus restart. This turns the matrix from expensive to nearly free. MCPMARK IS ADOPTED, SHA-PINNED. Both questions the spec left open resolve in its favour: it emits per-task token usage, pass verdicts and turn counts, and it has exactly one MCP factory seam, so pointing it at mcpproxy is a single env-gated branch. The spec's fallback of building an in-repo task set is not needed. Filesystem (30 tasks) + postgres (21) are credential-free and fully local — the FR-027-clean core. Design decisions: - Extension seam is "new measurement mode" (flag + driver + report block), used twice, following the spec-085 flip-gate precedent. No new arm, no new binary. - New bench/replaycorpus/ sibling of corpusio/ (a session trace is not a corpus), plus replay.go, modematrix.go and agentloop.go in package bench. - Report blocks are additive with no version bump. - Token accounting stays split: tiktoken for everything deterministic, provider usage read off the model response for the live loop, never summed — enforced by the harness's existing withhold-rather-than-compute pattern. One production change is required and it is small: request_bytes/response_bytes exist on the storage record, documented as measured pre-truncation, but are absent from the export contract. Adding them turns FR-002 from "exclude every truncated record" into "annotate it accurately". Security design (constitution principle IV) rests on three findings: the export path never masks (masking is wired only into list and detail), has_sensitive_data is written asynchronously after persistence so it is a best-effort reducer rather than a guarantee, and bodies are not needed for the headline at all. Hence bodies-off by default. Five risks carried forward, three of which are pre-existing defects this feature would otherwise inherit silently: the report's tokenizer identity is a document-level singleton that would become false once provider usage lands beside it; provenance is section-level while FR-013 requires measured and estimated figures to coexist in one block; the per-arm retry lookup returns 0.0 for unknown arms, indistinguishable from a measured 0.0; tiktoken fetches its vocabulary over the network unless a cache env var is set, which defeats SC-004 offline; and SC-011 has no enforcement beyond gitignore. Constitution check passes with no violations, so Complexity Tracking is omitted. --- CLAUDE.md | 3 +- .../103-token-bench/contracts/mode-matrix.md | 52 ++++ .../103-token-bench/contracts/replay-input.md | 52 ++++ .../contracts/report-v2-additions.md | 74 ++++++ specs/103-token-bench/data-model.md | 169 +++++++++++++ specs/103-token-bench/plan.md | 197 +++++++++++++++ specs/103-token-bench/quickstart.md | 128 ++++++++++ specs/103-token-bench/research.md | 231 ++++++++++++++++++ 8 files changed, 904 insertions(+), 2 deletions(-) create mode 100644 specs/103-token-bench/contracts/mode-matrix.md create mode 100644 specs/103-token-bench/contracts/replay-input.md create mode 100644 specs/103-token-bench/contracts/report-v2-additions.md create mode 100644 specs/103-token-bench/data-model.md create mode 100644 specs/103-token-bench/plan.md create mode 100644 specs/103-token-bench/quickstart.md create mode 100644 specs/103-token-bench/research.md diff --git a/CLAUDE.md b/CLAUDE.md index cebe443ed..ad8c7e669 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -158,7 +158,6 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/ - **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md). ## Recent Changes +- 103-token-bench: Go 1.25 (`bench/`) + existing only — tiktoken-go v0.1.8 (cl100k_base), the mcp-go transport already used by `bench/mcpcaller.go`. MCPMark is an external SHA-pinned tool invoked out of process, not a module dependency. **No new dependencies.** - 102-schema-deferred: Go 1.25.5 module toolchain (`go.mod`), backend-only + existing only — mark3labs/mcp-go v0.57.0 (tool surface), santhosh-tekuri/jsonschema/v6 (Spec-085 validator reuse), zap, stdlib. **No new dependencies.** - 101-tpa-db: Go 1.25.5 module toolchain (`go.mod`), backend-only + stdlib only for the new work — `crypto/ed25519`, `crypto/sha256`, `syscall` (no ed25519 usage in the tree before this); existing bbolt (state), zap, Cobra. **No new dependencies.** -- 098-tools-preflight: Added Go 1.24 module toolchain (repo builds with local Go 1.25) + existing only — chi (httpapi), bbolt (storage), Bleve (index), zap (logging), Cobra (CLI), swaggo/swag v2 (contract regen). **No new dependencies.** -- 097-stored-scripts: Added Go 1.25 (os.Root/Root.ReadFile available — R1) + stdlib only (os.Root). **No new dependencies.** diff --git a/specs/103-token-bench/contracts/mode-matrix.md b/specs/103-token-bench/contracts/mode-matrix.md new file mode 100644 index 000000000..022a646f6 --- /dev/null +++ b/specs/103-token-bench/contracts/mode-matrix.md @@ -0,0 +1,52 @@ +# Contract: the mode matrix + +**5 valid cells + 1 baseline. 7 structurally impossible combinations, each a skip-with-reason +row.** This file is the source of truth for FR-015, FR-016 and FR-017. + +## Valid cells + +| id | Endpoint | `tool_response_mode` | `direct_tool_response_mode` | Requires | +|----|----------|----------------------|------------------------------|----------| +| `retrieve_full` | retrieve_tools surface | `full` | n/a | — (default) | +| `retrieve_compact` | retrieve_tools surface | `compact` | n/a | spec 085 | +| `direct_full` | direct surface | n/a | `full` | — | +| `direct_deferred` | direct surface | n/a | `deferred` | spec 102 | +| `code_exec` | code-execution surface | forced `full` | n/a | `enable_code_execution: true` | +| `baseline` | none — all upstream tools loaded directly | n/a | n/a | FR-020 denominator | + +`baseline` is not an mcpproxy mode. It is the same agent doing the same tasks with every +upstream tool inline, and it is what every percentage is measured against. + +## Skipped combinations (FR-017) + +| Combination | Reason code | Why | +|---|---|---| +| `code_execution` × `compact` | `forced` | The code-execution surface overwrites the response mode with `full` and blanks the detail parameter; `detail` is not in that surface's schema. | +| `code_execution` × `compact` × direct axis | `forced` | Same cause, second axis. | +| `direct` × `full` (discovery axis) | `inapplicable` | `tool_response_mode` has exactly one consumer, inside the retrieve_tools handler. The direct surface has no `retrieve_tools` tool. | +| `direct` × `compact` (discovery axis) | `inapplicable` | Same cause. | +| `retrieve_tools` × `deferred` | `inapplicable` | `direct_tool_response_mode` is read only when building the direct listing. | +| `code_execution` × `deferred` | `inapplicable` | Same cause. | +| `code_execution` with `enable_code_execution: false` | `degenerate` | The surface can discover tools and call none of them. | + +## Rules + +1. **A cell is selected by endpoint URL, not by a config change.** All three routing-mode + servers are built at startup and all three endpoints stay mounted regardless of config. + The whole matrix therefore crosses on ONE long-lived proxy instance. +2. **Only the two serialization axes need config**, and each affects only its own surface. +3. **An inapplicable axis is recorded as not-applicable**, never as a default value — a + default would imply a measurement that was never taken. +4. **A skipped cell carries a reason code and never renders as a zero.** Reuse the existing + skipped-row shape (`Skipped` / `SkipReason` plus the skipped-result constructor) rather + than inventing a second one. +5. **Capability toggles** (batching, stored scripts, validate-before-dispatch) are binary + conditions applied to the cells where each is available, and the report enumerates which + cells each applies to. They are not additional axes of a product. +6. **Cell ids are stable** across runs and releases, so results remain comparable (FR-028). + +## Fleet shaping + +Every cell must be measured against the SAME fleet in a given comparison, and the fleet must +be large enough for proxy modes to differ from baseline. A single upstream server's toolset is +too small to show the asymptote; load the full fleet even when exercising one service's tasks. diff --git a/specs/103-token-bench/contracts/replay-input.md b/specs/103-token-bench/contracts/replay-input.md new file mode 100644 index 000000000..ca37f1a1c --- /dev/null +++ b/specs/103-token-bench/contracts/replay-input.md @@ -0,0 +1,52 @@ +# Contract: replay input + +## Source + +`GET /api/v1/activity/export?format=json` — equivalently `mcpproxy activity export --format +json`. JSONL, one activity record per line. + +**CSV is not a valid input.** It drops `work_session_id`, arguments, response and every byte +field, so it can neither group units of work nor account tokens. + +## Required fields + +| Field | Use | +|---|---| +| `work_session_id` | Groups records into one unit of work (spec 082). Records lacking it are unattributed and reported as such. | +| `request_id`, `parent_id` | Joins code-execution sub-calls to the call that issued them. | +| `tool_name`, `server_name` | Resolves the call against a mode's tool surface. | +| `status` | success / error / blocked / rejected. | +| `response_truncated` | Marks a record whose stored content was cut. | +| `request_bytes`, `response_bytes` | **Measured pre-truncation.** The accurate cost basis. | +| `has_sensitive_data` | Best-effort exclusion signal — see limitation below. | + +### Backend change this contract requires + +`request_bytes` and `response_bytes` exist on the storage record and are documented as +measured pre-truncation, but are **absent from the export contract**. They must be added to +the export DTO and copied in the export projection. + +Without them, FR-002 can only *exclude* truncated records. With them, it can *annotate them +accurately*, which materially increases how much real traffic the headline can use. This is +the only production-code change the feature requires. + +## Privacy contract + +1. **Default is bodies-off.** The headline is computable from byte sizes alone. +2. **Bodies-on is a separate, explicit opt-in** and prints a warning. The export path does + **not** mask: masking is wired into the list and detail handlers only, so a bodies-on + export is raw and unmasked by design — it is the compliance surface. +3. **`has_sensitive_data` is not a guarantee.** It is written asynchronously *after* the + record is persisted, so a freshly exported record may be sensitive but not yet flagged. + Exclude-by-flag is a best-effort reducer and must be documented as such wherever relied on. +4. **Nothing crosses the loader boundary but counts.** Sessions and calls surrender sizes, + statuses and derived measurements. No content reaches a report, a dashboard, or any + third-party service including model providers. +5. **Inputs live outside the repository**, are never committed, and the documented procedure + tells an operator how to delete them. + +## Loader output + +Grouped `ReplaySession` values carrying usability flags computed once at load: +`truncated`, `bodies_missing`, `sensitive`, `unreplayable`. Every exclusion is counted and +reported (FR-003, SC-008). Silence is never success. diff --git a/specs/103-token-bench/contracts/report-v2-additions.md b/specs/103-token-bench/contracts/report-v2-additions.md new file mode 100644 index 000000000..facaa1632 --- /dev/null +++ b/specs/103-token-bench/contracts/report-v2-additions.md @@ -0,0 +1,74 @@ +# Contract: report v2 additions + +## Versioning + +**No `report_version` bump.** New blocks are additive optional top-level fields with +`omitempty`, plus matching optional properties declared in the v2 JSON schema. A version bump +is reserved for changing an existing field's meaning or shape; this feature changes none. + +Precedent: the latency block's nested fields were added beside the existing flat fields the +same way. + +> The schema has no `additionalProperties: false`, so an undeclared block would validate +> silently. Do not rely on that — the schema file is the reviewed contract. Declare every new +> block. + +## New blocks + +### `replay` (deterministic, US1) + +Per-cell recomputed cost for a real recorded workload, plus the exclusion accounting: +sessions supplied, sessions used, and a count per exclusion reason. Carries the fleet shape. + +### `agent_loop` (live, US2) + +Per-cell tokens-per-completed-task, completion rate, first-attempt success rate, corrective +and infrastructure retry counts, run count and spread. **Carries its own accounting-source +field** (provider + pinned model). + +### `payload_decomposition` + +Per fleet shape: attribution across names, descriptions, annotations and schemas; the +recomputed achievable ceiling; and an explicit `confirmed` / `corrected` verdict on spec 102. + +## Two required changes to existing structures + +### 1. Scope the tokenizer identity + +The report envelope carries **one** tokenizer identity for the whole document. Once +provider-reported usage enters the same envelope, that field silently claims the entire report +was tokenizer-counted. + +It must be **scoped to the deterministic sections**, with the live block naming its own +accounting source. Supplementing is not enough — the existing field becomes false otherwise. + +### 2. Add per-row provenance + +Provenance today is **section-level**: a map from section key to +`measured` / `computed` / `estimated`, rendered as a badge on each section heading. + +FR-013 lets measured and estimated figures coexist *inside* one section, which the section +badge cannot express. Every new row type therefore carries its own `provenance` field, +constrained to the same three-value enum. + +This matters concretely: the per-arm retry-rate lookup returns `0.0` for unknown arms, which +is indistinguishable from a measured `0.0`. Without per-row provenance, one table can mix a +measured rate and a defaulted one under a single `estimated` badge. + +## Aggregation rules + +1. **Never sum across accounting sources.** A cross-source aggregate is **withheld with a + stated reason**, never computed — reusing the harness's existing withhold-rather-than-compute + pattern for non-authoritative headlines. +2. **A model-dependent figure needs `runs >= 4`** plus a spread, or it is not a headline. +3. **Every percentage carries its fleet shape.** +4. **Partial runs are marked and excluded from headlines.** + +## Enforcement gaps to close + +- **`SC-011` has no gate today.** "Reports are never committed" is gitignore plus convention; + no CI job or test fails on a committed report. Add a CI step asserting the results directory + is clean, mirroring the existing OAS verification script. +- **`SC-004` is not currently achievable offline.** The tokenizer fetches its vocabulary over + the network unless a cache directory env var is set, and nothing in the repo or CI sets it. + Set it in CI and document it in the reproduction procedure. diff --git a/specs/103-token-bench/data-model.md b/specs/103-token-bench/data-model.md new file mode 100644 index 000000000..9a1080fec --- /dev/null +++ b/specs/103-token-bench/data-model.md @@ -0,0 +1,169 @@ +# Phase 1 Data Model: token-bench (spec 103) + +Entities are described by what they hold and what must be true of them. Field names are +indicative; the binding shapes are in [contracts/](./contracts/). + +--- + +## ReplaySession + +One unit of real recorded work, reassembled from exported activity JSONL. + +| Field | Meaning | +|---|---| +| `work_session_id` | The spec-082 grouping key: one client, one project, across reconnects. This is the unit of work for US1. | +| `calls` | Ordered `ReplayCall` list. | +| `usability` | See below — computed once at load, never re-derived downstream. | +| `call_count`, `span` | Derived counts for reporting. | + +**Usability flags** (each a reason a session may be partly or wholly unusable): + +- `truncated` — one or more calls had their stored response truncated at capture. +- `bodies_missing` — the export was taken without bodies (the default), so content is absent. +- `sensitive` — one or more calls carry the sensitive-data flag. +- `unreplayable` — the tools referenced no longer resolve against the configured fleet. + +**Validation rules** + +- A session with `truncated` calls MUST NOT contribute a silent token total. It either + contributes an annotated figure derived from pre-truncation byte sizes, or is excluded and + counted in the exclusion report (FR-002, FR-003, SC-008). +- `sensitive` is a **best-effort** reducer. The flag is written asynchronously after the + record is persisted, so its absence does not prove a record is clean. This limitation MUST + be stated wherever the flag is relied upon (Principle IV). +- A session MUST NOT carry its call contents into any report structure. Only counts, sizes and + derived measurements propagate (FR-006). + +**State transitions**: `loaded → grouped → classified → {scored | excluded}`. Exclusion is +terminal and always counted. + +--- + +## ReplayCall + +One recorded tool call inside a session. + +| Field | Meaning | +|---|---| +| `tool_name`, `server_name` | Identity, used to resolve the call against a mode's tool surface. | +| `request_bytes`, `response_bytes` | **Measured pre-truncation.** The reason the backend contract change matters: these permit an accurate response-cost figure without reading a body. | +| `response_truncated` | Whether the stored content was cut. | +| `status` | success / error / blocked / rejected. | +| `parent_id` | Links a code-execution sub-call to the call that issued it. | +| `has_sensitive_data` | Best-effort flag (see above). | + +**Validation rules** + +- When bodies are absent (the default), token cost MUST be derived from the byte sizes, and + the resulting figure labelled as byte-derived rather than content-derived. +- A call whose `parent_id` resolves MUST be attributed to its parent's session, so + code-execution sub-calls are neither double-counted nor orphaned. + +--- + +## ModeCell + +One point in the matrix. **There are exactly 5 valid cells plus a baseline**; see +[contracts/mode-matrix.md](./contracts/mode-matrix.md). + +| Field | Meaning | +|---|---| +| `id` | Stable cell identifier used as a report key. | +| `endpoint` | Which mounted MCP endpoint this cell is served from — this is how a cell is selected, not a config change. | +| `discovery_serialization` | full / compact / not-applicable. | +| `direct_serialization` | full / deferred / not-applicable. | +| `capabilities` | Binary conditions applicable to this cell (batching, stored scripts, validate-before-dispatch). | +| `skipped`, `skip_reason` | Populated for the 7 structurally impossible combinations. | + +**Validation rules** + +- An inapplicable axis MUST be recorded as not-applicable, never as a default value — a + default would imply a measurement that was never taken (FR-017). +- A skipped cell MUST carry a reason and MUST NOT be rendered as a zero. +- Cell identity MUST be stable across runs so results are comparable over time (FR-028). + +--- + +## RunRecord + +The raw outcome of one unit of work under one mode cell. This is the traceable unit behind +every headline figure (FR-029). + +| Field | Meaning | +|---|---| +| `cell_id`, `unit_id` | What was run, and under which cell. | +| `tokens` | Input / output / cache-read, kept separate (FR-014). | +| `accounting_source` | `deterministic-tokenizer` or `provider-usage`. **Load-bearing** — records from different sources must never be summed. | +| `attempts` | Ordered attempt outcomes for first-attempt-success derivation. | +| `retries_corrective`, `retries_infrastructure` | Counted separately (FR-011). | +| `completion` | The suite's pass verdict, or `no-signal`. Never inferred. | +| `partial` | True if the run did not finish; such records are excluded from headlines (FR-032). | + +**Validation rules** + +- `accounting_source` is mandatory. A record without it cannot enter any aggregate. +- `completion: no-signal` excludes the record from completion-dependent figures rather than + counting it as either success or failure. +- For provider-sourced records, the number of provider responses summed MUST be recorded, so a + partially-captured unit is distinguishable from a genuinely cheap one. + +--- + +## Measurement + +An aggregate over `RunRecord`s, and the only thing a report renders. + +| Field | Meaning | +|---|---| +| `value`, `unit` | The figure. | +| `provenance` | measured / computed / estimated — **per row**, not only per section. | +| `accounting_source` | Inherited from its records; an aggregate spanning sources is forbidden. | +| `runs`, `spread` | Required for any model-dependent figure (FR-021). | +| `fleet_shape` | Tool count + definition-size distribution the figure holds for (FR-016 / IC-004). | +| `withheld`, `withheld_reason` | Set instead of a value when the figure cannot be computed honestly. | + +**Validation rules** + +- A cross-source aggregate MUST be **withheld with a stated reason**, never computed. This + reuses the harness's existing withhold-rather-than-compute pattern. +- A model-dependent figure with `runs < 4` MUST NOT be marked as a headline (FR-021). +- A figure without `fleet_shape` MUST NOT be published as a percentage. +- `provenance` at row level is required because FR-013 permits measured and estimated figures + to coexist inside one section — the existing section-level badge cannot express that. + +--- + +## PayloadDecomposition + +Attribution of tool-definition cost for one fleet shape (FR-024, FR-025). + +| Field | Meaning | +|---|---| +| `fleet_shape` | Which corpus/fleet this describes. | +| `share_names`, `share_descriptions`, `share_annotations`, `share_schemas` | The four attributions. | +| `achievable_ceiling` | Recomputed **per corpus** — never carried forward as a constant. | +| `spec102_verdict` | Explicit `confirmed` or `corrected`, with the delta. | + +**Validation rule**: the ceiling MUST be recomputed for each fleet shape. Reusing a previously +published ceiling is precisely the error spec 102 made when it projected from one corpus. + +--- + +## Relationships + +```text +ReplaySession 1──* ReplayCall +ReplaySession *──* ModeCell → RunRecord (US1: deterministic, one record per pair) +TaskSet *──* ModeCell → RunRecord (US2: live loop, k runs per pair) +RunRecord *──1 Measurement (aggregated, never across accounting sources) +FleetShape 1──1 PayloadDecomposition +``` + +## Invariants that must hold across the whole model + +1. **No recorded content leaves the loader.** Sessions and calls surrender counts and sizes; + nothing downstream can reach a body. +2. **Accounting sources never mix.** Enforced at aggregation by withholding. +3. **Silence is never success.** Truncated, partial, no-signal and skipped are each distinct + from zero and each are reported. +4. **Every published percentage carries its fleet shape.** diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md new file mode 100644 index 000000000..e8dcc6c4c --- /dev/null +++ b/specs/103-token-bench/plan.md @@ -0,0 +1,197 @@ +# Implementation Plan: Token-efficiency benchmark — measured savings, published results + +**Branch**: `103-token-bench` | **Date**: 2026-08-31 | **Spec**: [spec.md](./spec.md) +**Input**: Feature specification from `/specs/103-token-bench/spec.md` + +## Summary + +Add two measurement capabilities to the existing `bench/` harness, and one small backend +contract fix that makes the first of them honest. + +1. **US1 — deterministic replay** (`-replay`): load exported activity JSONL, group it into + units of work, and recompute what that real workload would have cost under every valid + mode. No model, byte-reproducible. +2. **US2 — live agent loop**: run MCPMark against mcpproxy under each mode, capturing + provider-reported token usage and per-task pass verdicts, to produce tokens per + *completed* task, first-attempt success and retry rates. + +The research pass changed three things materially versus the spec's assumptions, all in the +direction of less work: + +- **The mode matrix is 5 cells, not 12.** The three axes are not a free product; two of them + each govern exactly one surface. Seven combinations are structurally impossible and get + skip-with-reason rows. +- **The matrix crosses on ONE long-lived proxy instance.** All three routing-mode servers are + built at startup and permanently mounted, so a cell is selected by URL, not by a config + change plus restart. Only the two serialization axes need config, and only for their own + surface. +- **MCPMark needs one `elif` to point at mcpproxy**, and already emits per-task token usage, + pass verdicts, turn counts and pass^k. The spec's fallback plan (build an in-repo task set) + is not needed. + +## Technical Context + +**Language/Version**: Go 1.25 (`bench/`, `internal/`); Python 3 for the pinned external suite +**Primary Dependencies**: existing only — `tiktoken-go` (pinned `v0.1.8`, `cl100k_base`), +`mark3labs/mcp-go` transport via `bench/mcpcaller.go`, chi/bbolt/zap unchanged. +**No new Go module dependencies.** MCPMark is an external, SHA-pinned tool invoked out of +process, not vendored and not a module dependency. +**Storage**: N/A for the harness. Replay inputs are operator-supplied JSONL files that live +outside the repository and are never committed. +**Testing**: `go test ./bench/...` (unit + invariant + schema-conformance), plus the existing +`reportv2_e2e_test.go` JSON-schema validation path. +**Target Platform**: developer machine + CI (ubuntu-latest) for the deterministic half; the +live half is operator-run and never CI-gated, because it costs model spend. +**Project Type**: single Go project; `bench/` is an existing in-repo tool tree. +**Performance Goals**: N/A — this is a measurement tool. The relevant budget is model spend, +not latency. +**Constraints**: reports are never committed; the deterministic half must be byte-reproducible +and must run offline; no recorded session content may reach a report or a third party. +**Scale/Scope**: 5 matrix cells + 1 baseline; MCPMark's 51 credential-free tasks +(filesystem 30 + postgres 21) as the clean core, 127 total if credentialed services are added. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-checked after Phase 1 design.* + +| Principle | Assessment | +|---|---| +| **I. Performance at Scale** | N/A to production paths. The harness adds no code to a request path. The one production-touching change (Phase 0 item 3 below) adds two integers to an export DTO. | +| **II. Actor-Based Concurrency** | No new concurrency in production code. The harness is a batch tool. | +| **III. Configuration-Driven** | PASS, and strengthened: the matrix is crossed by URL + existing config fields, adding no benchmark-only configuration to the product. | +| **IV. Security by Default** | **The binding gate for this feature.** Replay inputs are raw user traffic; the export path deliberately does not mask. Design must default to bodies-off and must never transmit recorded content off-box. See Security Design below. | +| **V. TDD** | PASS. Every unit below is specified test-first; the deterministic half is fully unit-testable with no proxy, following the `flipgate.go` precedent of parameterising the driver over a function type. | +| **VI. Documentation Hygiene** | PASS. `bench/README.md` is the feature's user-facing doc and must gain the replay and live-loop sections; `CLAUDE.md` needs no change (no new commands at the top level, no architecture change). | + +**No violations. Complexity Tracking is therefore empty and omitted.** + +### Security Design (Principle IV) + +Three findings drive this and each becomes a design rule: + +1. **The export path never masks.** `maskActivityPayloads` is wired into the list and detail + handlers only, not into export. Bodies-on export is raw and unmasked. + → **Rule**: replay defaults to bodies-off. Bodies-on is a separate, explicitly-opted-in + mode with a printed warning. +2. **`has_sensitive_data` is set asynchronously after the record is persisted.** A freshly + exported record can be sensitive but not yet flagged. + → **Rule**: exclude-by-flag is a best-effort reducer and MUST NOT be described as a + guarantee, in code comments or in the published methodology. +3. **Bodies are not needed for the headline.** Byte sizes recorded pre-truncation give an + accurate response-cost figure without reading content (see Phase 0 item 3). + → **Rule**: the default configuration produces the headline without ever loading a body. + +## Phase 0: Research + +Complete. See [research.md](./research.md) for the full decisions, alternatives and evidence. +Five topics were resolved; the load-bearing outcomes: + +1. **Extension seams** — three exist. Spec 103 uses the "new measurement mode" seam twice + (flag on `cmd/bench` + driver in package `bench` + report block), following the spec-085 + flip-gate precedent. **No new arm is needed**, and no new binary. +2. **The valid matrix is 5 cells**, with 7 documented skip reasons; cells are selected by + endpoint URL on one running proxy. +3. **One backend change is required**: `request_bytes` / `response_bytes` exist on the storage + record, explicitly measured pre-truncation, but are absent from the export contract. Adding + them turns FR-002 from "exclude truncated records" into "annotate them accurately", which + materially improves the headline's coverage. +4. **MCPMark is adopted**, SHA-pinned, with one `elif` in its single MCP factory. Its per-task + `meta.json` feeds FR-010/011/012/018 directly. +5. **Token accounting stays split**: tiktoken for everything deterministic; provider `usage` + read off the model response for the live loop; the two never summed, enforced by the + harness's existing withhold-rather-than-compute pattern. + +### Risks carried into Phase 1 + +- **`ReportV2.Tokenizer` is a report-level singleton.** The moment provider-usage figures + enter the same envelope, that field silently claims the whole report was tiktoken-counted. + It must be scoped to the deterministic sections, not merely supplemented. +- **Provenance is section-level, not per-figure.** FR-013 requires measured and estimated + figures to coexist inside one block, so new row types need their own provenance field. +- **tiktoken fetches its vocabulary over the network** unless `TIKTOKEN_CACHE_DIR` is set; + nothing in the repo or CI sets it. SC-004 (an outsider reproduces the deterministic figures) + is not currently achievable on a restricted network. +- **`RetryRateForArm` returns 0.0 for unknown arms**, indistinguishable from a measured 0.0. + Mixing a measured rate with a defaulted one under a single section badge is a live hazard + for FR-013. +- **SC-011 has no enforcement.** "Reports are never committed" is gitignore plus convention; + no CI job or test fails on a committed report. + +## Project Structure + +### Documentation (this feature) + +```text +specs/103-token-bench/ +├── plan.md # This file +├── spec.md # Feature specification +├── research.md # Phase 0 decisions + evidence +├── data-model.md # Phase 1 entities +├── quickstart.md # Phase 1 operator walkthrough +├── contracts/ +│ ├── report-v2-additions.md # additive report blocks + per-row provenance +│ ├── mode-matrix.md # the 5 valid cells + 7 skip reasons +│ └── replay-input.md # activity JSONL contract consumed by replay +├── checklists/ +│ └── requirements.md +└── tasks.md # /speckit.tasks output — NOT created here +``` + +### Source Code (repository root) + +```text +bench/ +├── replaycorpus/ # NEW — activity JSONL loader, grouping, usability flags +│ ├── load.go +│ ├── group.go # work_session_id grouping; parent_id ↔ request_id join +│ └── flags.go # truncated / bodies-missing / sensitive / unreplayable +├── replay.go # NEW — deterministic cost recomputation over the matrix +├── modematrix.go # NEW — the 5 valid cells, skip reasons, cell→URL mapping +├── agentloop.go # NEW — live-loop driver + MCPMark meta.json ingestion +├── reportv2.go # EXTEND — additive optional blocks; per-row provenance +├── live_report.go # EXTEND — scope Tokenizer to deterministic sections +├── session.go # EXTEND — measured rates supersede armRetryRates +└── cmd/bench/main.go # EXTEND — -replay and -agent-loop branches + +internal/contracts/activity.go # EXTEND — add request_bytes / response_bytes +internal/httpapi/activity.go # EXTEND — copy them in the export projection + +specs/083-discovery-profiler/contracts/report-v2.schema.json # EXTEND — declare new blocks +.github/workflows/bench.yml # EXTEND — TIKTOKEN_CACHE_DIR; assert no committed reports +``` + +**Structure Decision**: The feature lives almost entirely inside the existing `bench/` tree, +using its established seams. `bench/replaycorpus/` is a new sibling of `bench/corpusio/` +rather than an addition to it, because `corpusio` is scoped by its own doc comment to public +tool-retrieval corpora and produces `Corpus`/`GoldenSet` values — a session trace is neither. +`replay.go`, `modematrix.go` and `agentloop.go` sit in package `bench` (not a subpackage) +because they need `Tokenizer`, `ProxyToolsForMode` and the `EncodingArm` interface, and +`bench` cannot import `bench/arms` (a documented cycle); they cross that boundary with the +same structural interface `RunArms` already uses. + +The only production-code change is two additive DTO fields plus their projection — deliberately +minimal, and justified because without them FR-002 degrades from accurate annotation to +wholesale exclusion. + +## Phase 1: Design & Contracts + +Artifacts produced: [data-model.md](./data-model.md), [contracts/](./contracts/), +[quickstart.md](./quickstart.md). + +**Post-design constitution re-check: PASS.** The design adds no production request-path code, +introduces no new abstraction beyond one loader package that mirrors an existing sibling, and +adds no module dependency. Principle IV is satisfied by the bodies-off default rather than by +a masking layer, which is the simpler of the two available designs. + +## Open decisions for the operator (not blockers) + +These need a human, and none blocks starting US1: + +1. **Pinned model and spend ceiling** for the live loop. US1 delivers alone without it. +2. **Whether cache-read tokens count toward "tokens per completed task."** FR-014 requires + tracking them separately; FR-018 does not say which composite the headline uses. This is a + definition decision, not a discovery. +3. **Whether to publish the measured tiktoken→Claude divergence.** The repo's caveat says up + to ~60%; Anthropic's guidance says ~15–20% on typical text. Neither is sourced in-repo and + they differ threefold. `count_tokens` settles it with no inference spend, and doing so turns + FR-022's tolerance into a measured number. diff --git a/specs/103-token-bench/quickstart.md b/specs/103-token-bench/quickstart.md new file mode 100644 index 000000000..1817142a5 --- /dev/null +++ b/specs/103-token-bench/quickstart.md @@ -0,0 +1,128 @@ +# Quickstart: token-bench (spec 103) + +Two independent halves. **Start with the deterministic one** — it needs no model spend, no +credentials and no network, and it delivers value alone. + +--- + +## Half 1 — Deterministic replay (US1) + +### 1. Export a real workload + +From a machine that has been using mcpproxy for real work: + +```bash +mcpproxy activity export --format json > ~/replay-corpus.jsonl +``` + +**Bodies stay off.** The headline is computable from pre-truncation byte sizes, and the export +path does not mask — a bodies-on export is raw and unmasked by design. + +> The file is real user traffic. Keep it outside the repository, never commit it, and delete +> it when finished. It is deliberately gitignored nowhere, because it should never be inside +> the repo in the first place. + +### 2. Recompute cost across the matrix + +```bash +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out bench/results +``` + +Produces the `replay` block in `bench/results/report.json` plus the dashboard section: per +mode cell, what this workload would have cost, with the exclusion accounting beside it. + +### 3. Read the exclusion report first, not the headline + +The exclusion counts tell you whether the headline is trustworthy. A corpus that is 80% +truncated produces a real number over a small slice, and the report says so rather than +pretending otherwise. + +### 4. Verify determinism + +```bash +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/run-a +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/run-b +diff /tmp/run-a/report.json /tmp/run-b/report.json && echo "byte-identical (SC-002)" +``` + +### Offline note + +The tokenizer downloads its vocabulary on first use unless a cache directory is set: + +```bash +export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" +``` + +Set this before claiming an offline run. Without it, a reproducer on a restricted network +fails at step one — which is what SC-004 requires to work. + +--- + +## Half 2 — Live agent loop (US2) + +**Costs model spend.** Do not start until a model and a ceiling are pinned. + +### 1. Stand up the full fleet on one proxy + +Configure mcpproxy with **all** suite services at once — filesystem, postgres, and the +credentialed ones if used — even while running a single service's tasks. A single server's +toolset is too small a fleet for proxy modes to differ from baseline, and the full fleet is +also the honest baseline: the same agent, the same tasks, all tools loaded directly. + +Enable code execution if the `code_exec` cell is in scope; without it that cell is degenerate +and is skipped with a reason. + +Leave the instance running. **The whole matrix crosses on one instance** — cells are selected +by endpoint URL, and only the two serialization axes need config. + +### 2. Point the suite at mcpproxy + +MCPMark is pinned by commit SHA. Patch its single MCP factory with an env-gated branch that +returns an HTTP MCP server at mcpproxy's URL with the API key header. That is the only change +the suite needs — `list_tools()` and `call_tool()` are the only calls its agent loop makes +against the server object. + +### 3. Start with the credential-free core + +Filesystem (30 tasks) and postgres (21) need no third-party credentials and are fully local. +That is the clean starting point. GitHub and Notion add real-fleet realism but **perform real +writes** and must be run so they cannot damage real data. + +### 4. Run each cell at k >= 4 + +A single agentic run is noise. Every model-dependent figure is an average over at least four +runs with its spread, or it is not a headline. + +### 5. Ingest the results + +The suite's per-task metadata carries the pass verdict, token usage split, and turn count; its +trajectory files are what make corrective-versus-infrastructure retry classification possible. +These feed tokens-per-completed-task, completion rate, first-attempt success and retry rates. + +--- + +## Reading the report honestly + +- **Check the accounting source on every figure.** Deterministic figures come from the local + tokenizer; live figures come from provider-reported usage. They are never summed — a + cross-source aggregate is withheld with a stated reason instead. +- **Check provenance per row**, not just the section badge. A section can legitimately hold a + measured figure beside an estimated one. +- **A withheld figure is a result**, not a bug. It means the honest number could not be + computed from what was measured. +- **Completion rate sits beside cost.** A mode that costs less and completes less is not a + saving, and the report marks it as a regression regardless of its token figure. +- **Every percentage carries its fleet shape.** A percentage without one is not publishable. + +## Before publishing + +1. Recompute the achievable ceiling **for each fleet shape** — never carry a previous + ceiling forward. That is the error spec 102 made. +2. Measure the tokenizer's divergence from the pinned model using the token-counting endpoint + (no inference spend) rather than quoting either of the two contradictory figures currently + in circulation. +3. Confirm no report was committed: + ```bash + git status --porcelain bench/results + ``` + Empty output, or the run is not publishable. diff --git a/specs/103-token-bench/research.md b/specs/103-token-bench/research.md new file mode 100644 index 000000000..303d46804 --- /dev/null +++ b/specs/103-token-bench/research.md @@ -0,0 +1,231 @@ +# Phase 0 Research: token-bench (spec 103) + +Five topics, resolved 2026-08-31 against the tree at `b7d803e76` and against the suites' +current published code. Every decision below carries the evidence that settled it. + +--- + +## 1. How to attach to the existing `bench/` harness + +**Decision**: Use the "new measurement mode" seam twice — a flag on `cmd/bench`, a driver file +in package `bench`, and an additive report block. Add no new arm, no new binary, no new +package tree beyond one loader. + +**Rationale**: `bench/` has three distinct extension seams and only one of them fits: + +| Seam | For | Fits spec 103? | +|---|---|---| +| New **arm** (`bench/arms/`) | one deterministic way of rendering *tool definitions* into context text | **No** — spec 103 measures modes, not renderings. The renderings already exist. | +| New **corpus** (`bench/corpusio/` + an `OfflineSection`) | one more frozen tool universe to score | **No** — replay crosses the mode matrix; it is not one more corpus | +| New **measurement mode** (flag + driver + report block) | a new kind of measurement | **Yes, twice** | + +The precedent to copy is spec 085's flip gates: a `-flip-gates` flag, a driver +(`bench/flipgate.go`) parameterised over a function type so the arithmetic is unit-testable +with no proxy running, production transport isolated in `bench/mcpcaller.go`, and the result +hung off the live report. `-live` is the same shape at larger scale. + +**Alternatives considered**: adding replay as an arm (rejected — an arm renders tool +definitions and must satisfy the arm-interface contract including determinism over a corpus, +which a session trace is not); adding it as a corpus (rejected — a corpus is one tool universe +scored by many arms, whereas replay is one workload scored across the mode matrix). + +**Report extension cost**: additive optional fields with `omitempty`, plus matching optional +properties in `specs/083-discovery-profiler/contracts/report-v2.schema.json`, plus one +provenance key per block. **No `report_version` bump** — that is reserved for changing an +existing field's meaning or shape, and spec 103 changes none. Precedent: `LatencyV2`'s +`rest_search`/`mcp_discovery` were added beside the flat fields the same way. + +> The schema has no `additionalProperties: false`, so an undeclared block would validate +> silently. Do not rely on that — the schema file is the reviewed contract and new blocks must +> be declared in it. + +**Finding that becomes work**: SC-011 ("no generated report is committed") is enforced only by +`bench/.gitignore` and convention. There is no CI job or test that fails on a committed +report. The cheap gate is a CI step asserting `git status --porcelain bench/results` is empty, +mirroring `scripts/verify-oas.sh`. + +--- + +## 2. What replay can consume, and the privacy posture it forces + +**Decision**: Consume JSONL from `GET /api/v1/activity/export?format=json` (equivalently +`mcpproxy activity export --format json`) in a new `bench/replaycorpus/` package. **Default to +bodies-off.** Add `request_bytes` / `response_bytes` to the export contract. + +**Rationale**: + +- **CSV is unusable** — it drops `work_session_id`, arguments, response and all byte fields. +- **Nothing in `bench/` references activity today**, so the loader is purely additive. +- **The export path never masks.** `maskActivityPayloads` is wired into the list and detail + handlers only. Bodies-on export is raw and unmasked, by design — it is the + compliance/incident-response surface. This is why bodies-off is the default and bodies-on is + an explicit opt-in with a warning. +- **`has_sensitive_data` is set asynchronously *after* the record is persisted.** A freshly + exported record can be sensitive but not yet flagged. Exclude-by-flag is therefore a + best-effort reducer and must never be documented as a guarantee. + +**The one required backend change**: `request_bytes` and `response_bytes` exist on the storage +record and are explicitly documented as *measured pre-truncation* — but they are absent from +the export contract entirely. Adding them (two fields on the DTO, two lines in the export +projection) turns FR-002 from "exclude every truncated record" into "annotate it accurately", +which materially raises how much real traffic the headline can use. + +**Alternatives considered**: reading BBolt directly (rejected — bypasses the contract, couples +the harness to storage internals, and defeats the privacy gating entirely); building a new +capture mechanism (rejected — explicitly out of scope in the spec, and unnecessary). + +--- + +## 3. The mode matrix: 5 valid cells, not 12 + +**Decision**: A matrix cell is **one MCP endpoint plus the one serialization axis that governs +that endpoint's surface** — not a 3×2×2 config product. + +| # | Routing mode / endpoint | `tool_response_mode` | `direct_tool_response_mode` | Notes | +|---|---|---|---|---| +| 1 | `retrieve_tools` — `/mcp/call` | full | n/a | today's default | +| 2 | `retrieve_tools` — `/mcp/call` | compact | n/a | spec 085 | +| 3 | `direct` — `/mcp/all` | n/a | full | pre-102 direct | +| 4 | `direct` — `/mcp/all` | n/a | deferred | spec 102 | +| 5 | `code_execution` — `/mcp/code` | forced full | n/a | needs `enable_code_execution: true` | + +Plus the existing `baseline` arm as the FR-020 denominator. + +**The 7 skipped combinations and their FR-017 reasons**: + +- `code_execution × compact` (×2 across the direct axis) — **forced**: the code-execution + surface overwrites the response mode with `full` and blanks the detail parameter; `detail` + is not even in that surface's schema. +- `direct × {full, compact}` on the *discovery* axis (×2) — **inapplicable**: + `tool_response_mode` has exactly one consumer, inside the `retrieve_tools` handler. The + direct surface has no `retrieve_tools` tool at all. +- `retrieve_tools × deferred`, `code_execution × deferred` (×2) — **inapplicable**: + `direct_tool_response_mode` is read only when building the direct listing. +- `code_execution` with `enable_code_execution: false` — **degenerate**: the surface can + discover tools and call none of them. + +**Crossing the matrix is cheap.** All three routing-mode servers are built at startup and all +three endpoints are permanently mounted regardless of config — the source comments this +explicitly. A cell is therefore selected **by URL on one long-lived proxy instance**; only the +two serialization axes need config, and each affects only its own surface. + +**Reuse the existing skip shape** rather than inventing one: `ArmResult.Skipped` / +`SkipReason` and the `SkippedArmResult` constructor already implement "a skipped row with a +reason, never a zero", which is exactly what FR-017 asks for. + +**This directly simplifies the spec**: FR-015/FR-016/FR-017 describe a matrix that is 5 rows +plus 7 documented skips, not a 12-cell product. + +--- + +## 4. Public agent-loop suite + +**Decision**: Adopt **MCPMark** (`eval-sys/mcpmark`, Apache-2.0), SHA-pinned. Adopt +**MCP-Atlas** only as a *corpus donor* for fleet-shape work, not as an agent loop. + +**Both previously-unverified questions resolve in MCPMark's favour**: + +1. **Per-task token counts**: yes. Its per-task `meta.json` carries + `execution_result.success`, `token_usage{input, output, total, reasoning}`, `turn_count` + and `agent_execution_time`; `messages.json` holds the full trajectory, which is what makes + corrective-vs-infrastructure retry classification possible. Its aggregator already emits + `pass@1{avg,std}`, `pass@k`, `pass^k` and per-run cost. +2. **Can it point at a single endpoint?** Yes — the load-bearing question, and the answer is + that MCPMark has exactly **one** MCP factory seam. Pointing it at mcpproxy is a single + env-gated `elif` branch returning an HTTP MCP server at mcpproxy's URL. Its transport is + the official Python SDK streamable-HTTP — the same transport `bench/mcpcaller.go` already + drives. `list_tools()` and `call_tool()` are the only two calls its agent loop makes. + +**Consequence**: the spec's fallback (build an in-repo task set) is **not needed**. + +**Fleet shaping matters for honesty.** Stock MCPMark connects to one server per run, so the +per-task fleet is a single server's toolset — far too small for mcpproxy to show its +asymptote. Configure mcpproxy with *all* MCPMark services simultaneously while running one +service's tasks. This is a pure mcpproxy-config change and it is also the honest FR-020 +baseline: the same agent, the same tasks, all tools loaded directly. + +**Credential-free core**: filesystem (30 tasks) + postgres (21) need zero third-party +credentials and are fully local — the FR-027-clean starting point. GitHub (23) and Notion (28) +add real-fleet realism but perform real writes and need credentials. + +**Alternatives considered**: MCP-Atlas as the loop (rejected — its value is its 36-server / +220-tool *shape*, ideal for FR-024's fleet-shape decomposition, but it is not an agent-loop +harness); τ²-bench (rejected — needs an MCP bridge that does not exist); BFCL / ToolBench / +API-Bank (rejected — not MCP); LiveMCPBench as a loop (rejected — stale since 2025-08, though +its committed 527-tool snapshot remains useful as a corpus). + +--- + +## 5. Token accounting for a live loop + +**Decision**: Keep tiktoken (`cl100k_base`) as the *sole* source for everything deterministic. +Read provider-reported `usage` **directly off the model response** in the agent driver for the +live loop. **Never sum the two.** + +**Rationale**: The harness already solved "two token shapes must not be mixed into one +number", for schemas: when either side lacks schema counts, the savings ratio is **withheld +with a stated reason** rather than computed. The break-even computation carries the same rule +in its own doc comment and errors rather than fabricating a verdict. Spec 103's never-summed +constraint is that same rule on a new axis, so it is enforceable with machinery that already +exists — the extension is a new section and a source label, not new aggregation logic. + +Provider `usage` maps 1:1 onto FR-014's required split (input / output / cache-read) with no +derivation, so reading response fields is strictly simpler than proxying. An LLM proxy is kept +only as the fallback for a closed-box suite whose loop cannot be edited — which MCPMark, being +patchable at one seam, is not. + +**Alternatives considered**: an LLM proxy as the primary path (rejected — an extra hop and a +second failure mode on a run that costs real money, and it sees nothing the response object +does not already carry; a silently dropped record would understate cost *in the project's +favour*, the same failure class FR-002 forbids for truncation); feeding provider-derived retry +rates into the existing session estimator (rejected — that formula's other terms are tiktoken +counts, so it would produce exactly the summed hybrid the spec forbids; a measured session +cost must be a new row type sourced wholly from provider usage); reusing +`internal/server/tokens/savings.go` (rejected — a production/Web-UI path whose "query result" +is an arbitrary prefix of the tool list rather than a ranking, and whose "all tools" side is a +hand-built JSON shape rather than the live builders; reusing it would reintroduce exactly the +drift IC-002 forbids). + +**There is no existing code anywhere in the repo that reads a provider usage field.** Every +token count in the tree is produced by the local tiktoken wrapper. Cache-token handling is +entirely greenfield — no `cache_read` concept exists anywhere. + +### Two hazards this creates + +- **`ReportV2.Tokenizer` is a report-level singleton** naming one estimator. Once provider + usage enters the same envelope, that field silently claims the whole report was + tiktoken-counted. It must be **scoped** to the deterministic sections, and the live block + must carry its own accounting-source field (provider + pinned model). +- **`RetryRateForArm` returns 0.0 for unknown arms**, indistinguishable from a measured 0.0. + With FR-013 replacing assumed rates only where measurements exist, one table could mix a + measured rate and a defaulted one under a single section-level `estimated` badge. Session + cost rows need a **per-row** provenance field. + +--- + +## Cross-cutting: reproducibility hazards found + +1. **tiktoken downloads its vocabulary at runtime** from a remote host unless + `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` is set. Nothing in the repo or in + `bench.yml` sets it, so the documented "offline estimator" property holds only after a warm + cache. **An outside reproducer on a restricted network fails at step one** — which defeats + SC-004. Fix: set the cache dir in CI and document it in the reproduction procedure. +2. **The tokenizer caveat is unsourced and internally contradicted.** The repo states up to + ~60% underestimation versus Claude; Anthropic's own guidance says ~15–20% on typical text + (more on code). The two differ threefold and neither is sourced in-repo. Publishing + FR-022's numeric tolerance on top of either is a credibility risk. Fix: measure it with + `count_tokens` against the frozen corpora for the pinned model — no inference spend, and it + doubles as a genuine `measured` figure for the per-fleet-shape ceiling. +3. **The two savings numbers have never been compared.** The Web-UI/status figure and the + bench headline are independently implemented over the same fleet. The spec's own + Assumptions say a contradiction between measurements is a finding to investigate. Running + both against one live proxy is cheap and should happen early. + +## Deferred to the operator + +- Pinned model and spend ceiling for the live loop (US1 is unblocked without it). +- Whether cache-read tokens count toward the headline composite (a definition decision). +- Whether the `playwright_webarena` task subset (21 of 127) needs a self-hosted instance. +- The exact corrective-vs-infrastructure retry boundary. Whatever rule is chosen **must be + applied identically to the baseline arm and the proxy arms**, or the comparison is biased + toward whichever arm carries richer error signal. From c5f7dd98a1dce6c240e0965da9a5b97917dadbb6 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:23:33 +0300 Subject: [PATCH 04/12] =?UTF-8?q?fix(specs):=20103=20=E2=80=94=20the=207?= =?UTF-8?q?=20extra=20combinations=20are=20redundant,=20not=20impossible?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review round 2 died on tool-permission errors before writing its findings, but its progress output surfaced two real defects, both verified independently against the tree and both fixed here. 1. The matrix framing was wrong. The 3-axis product is 12 combinations that COLLAPSE ONTO 5 distinct behaviours — the extra 7 are configurable and behaviourally redundant, not "structurally impossible". Each is now a skip-with-reason row that also names the valid cell it collapses onto, and the reason codes distinguish an ignored axis (no consumer on that surface) from a genuine override (code_execution forcing full). Verified: effectiveToolResponseMode has one consumer at internal/server/mcp.go:1585, effectiveDirectToolResponseMode one at internal/server/mcp_routing.go:100. 2. "A cell is selected by endpoint URL, not by a config change" was overstated. URL selects the ROUTING-MODE axis only; the two serialization axes still require config. Whether those fields hot-reload or need a fresh instance per value is now recorded as OPEN and must be settled before /speckit.tasks, because it decides whether the matrix crosses on one long-lived proxy or on several — a change to the runner's shape. Note on the review itself: the run auto-rejected its own attempts to read outside the repo (GOMODCACHE) and hit a ripgrep record-size limit, then exited 0 having printed no findings body. Per standing practice that is a FAILED run, not a clean review, so it is not counted as a completed round. --- .../103-token-bench/contracts/mode-matrix.md | 64 ++++++++++++++----- specs/103-token-bench/data-model.md | 13 ++-- specs/103-token-bench/plan.md | 26 +++++--- 3 files changed, 71 insertions(+), 32 deletions(-) diff --git a/specs/103-token-bench/contracts/mode-matrix.md b/specs/103-token-bench/contracts/mode-matrix.md index 022a646f6..33a570071 100644 --- a/specs/103-token-bench/contracts/mode-matrix.md +++ b/specs/103-token-bench/contracts/mode-matrix.md @@ -1,6 +1,7 @@ # Contract: the mode matrix -**5 valid cells + 1 baseline. 7 structurally impossible combinations, each a skip-with-reason +**5 distinct behaviours + 1 baseline. The 3-axis product's other 7 combinations are +configurable but redundant, each a skip-with-reason row.** This file is the source of truth for FR-015, FR-016 and FR-017. ## Valid cells @@ -17,27 +18,58 @@ row.** This file is the source of truth for FR-015, FR-016 and FR-017. `baseline` is not an mcpproxy mode. It is the same agent doing the same tasks with every upstream tool inline, and it is what every percentage is measured against. -## Skipped combinations (FR-017) +## The other 7 combinations of the 3-axis product (FR-017) -| Combination | Reason code | Why | +The naive product is 3 routing modes x 2 discovery serializations x 2 direct serializations = +**12 combinations, which collapse onto 5 distinct behaviours.** + +They are **not impossible to configure** — they are *configurable and behaviourally redundant*, +because on each surface one or both serialization axes have no consumer. Calling them +"impossible" would be wrong; rendering them as zeros would be worse. Each is a +**skip-with-reason row naming the cell it collapses onto.** + +| Configured combination | Collapses onto | Reason code | |---|---|---| -| `code_execution` × `compact` | `forced` | The code-execution surface overwrites the response mode with `full` and blanks the detail parameter; `detail` is not in that surface's schema. | -| `code_execution` × `compact` × direct axis | `forced` | Same cause, second axis. | -| `direct` × `full` (discovery axis) | `inapplicable` | `tool_response_mode` has exactly one consumer, inside the retrieve_tools handler. The direct surface has no `retrieve_tools` tool. | -| `direct` × `compact` (discovery axis) | `inapplicable` | Same cause. | -| `retrieve_tools` × `deferred` | `inapplicable` | `direct_tool_response_mode` is read only when building the direct listing. | -| `code_execution` × `deferred` | `inapplicable` | Same cause. | -| `code_execution` with `enable_code_execution: false` | `degenerate` | The surface can discover tools and call none of them. | +| `retrieve_tools` x full x **deferred** | `retrieve_full` | `axis-ignored` | +| `retrieve_tools` x compact x **deferred** | `retrieve_compact` | `axis-ignored` | +| `direct` x **compact** x full | `direct_full` | `axis-ignored` | +| `direct` x **compact** x deferred | `direct_deferred` | `axis-ignored` | +| `code_execution` x **compact** x full | `code_exec` | `forced-full` | +| `code_execution` x full x **deferred** | `code_exec` | `axis-ignored` | +| `code_execution` x **compact** x **deferred** | `code_exec` | `forced-full` + `axis-ignored` | + +### Why each axis is ignored where it is + +- **`tool_response_mode` has exactly one consumer**: `effectiveToolResponseMode`, called from + the `retrieve_tools` handler (`internal/server/mcp.go:1585`). Neither the direct nor the + code-execution surface reads it — the direct surface has no `retrieve_tools` tool at all. +- **`direct_tool_response_mode` has exactly one consumer**: `effectiveDirectToolResponseMode`, + read when building the direct listing (`internal/server/mcp_routing.go:100`). Only the + direct surface consults it. +- **`code_execution` forces full**: that surface overwrites the response mode with `full` and + blanks the detail parameter; `detail` is not in its schema. That is a genuine *override*, + distinct from an ignored axis, hence its own reason code. + +### Separately: a degenerate configuration + +`code_execution` with `enable_code_execution: false` is not part of the product above. It is a +configuration in which the surface can discover tools and call none of them. Skip it with +reason `degenerate`. ## Rules -1. **A cell is selected by endpoint URL, not by a config change.** All three routing-mode - servers are built at startup and all three endpoints stay mounted regardless of config. - The whole matrix therefore crosses on ONE long-lived proxy instance. -2. **Only the two serialization axes need config**, and each affects only its own surface. -3. **An inapplicable axis is recorded as not-applicable**, never as a default value — a +1. **The routing-mode axis is selected by endpoint URL — no config change, no restart.** All + three routing-mode servers are built at startup and all three endpoints stay mounted + regardless of config (`internal/server/server.go:2514-2536`). +2. **The two serialization axes DO require config**, and each affects only its own surface. + A cell is therefore (URL, serialization-config) — **not URL alone**. Whether those two + fields hot-reload, or whether each value needs a fresh instance, is OPEN and must be + settled before tasks are written: it decides whether the matrix crosses on one long-lived + proxy or on several. +3. **An ignored axis is recorded as not-applicable**, never as a default value — a default would imply a measurement that was never taken. -4. **A skipped cell carries a reason code and never renders as a zero.** Reuse the existing +4. **A skipped row carries a reason code AND the cell it collapses onto, and never renders + as a zero.** Reuse the existing skipped-row shape (`Skipped` / `SkipReason` plus the skipped-result constructor) rather than inventing a second one. 5. **Capability toggles** (batching, stored scripts, validate-before-dispatch) are binary diff --git a/specs/103-token-bench/data-model.md b/specs/103-token-bench/data-model.md index 9a1080fec..6af4353d5 100644 --- a/specs/103-token-bench/data-model.md +++ b/specs/103-token-bench/data-model.md @@ -63,23 +63,24 @@ One recorded tool call inside a session. ## ModeCell -One point in the matrix. **There are exactly 5 valid cells plus a baseline**; see +One point in the matrix. **There are exactly 5 distinct behaviours plus a baseline**; see [contracts/mode-matrix.md](./contracts/mode-matrix.md). | Field | Meaning | |---|---| | `id` | Stable cell identifier used as a report key. | -| `endpoint` | Which mounted MCP endpoint this cell is served from — this is how a cell is selected, not a config change. | +| `endpoint` | Which mounted MCP endpoint serves this cell. Selects the routing-mode axis with no config change; the serialization axes still need config. | | `discovery_serialization` | full / compact / not-applicable. | | `direct_serialization` | full / deferred / not-applicable. | | `capabilities` | Binary conditions applicable to this cell (batching, stored scripts, validate-before-dispatch). | -| `skipped`, `skip_reason` | Populated for the 7 structurally impossible combinations. | +| `skipped`, `skip_reason`, `collapses_onto` | Populated for the 7 redundant combinations of the 3-axis product; each names the valid cell it collapses onto. | **Validation rules** -- An inapplicable axis MUST be recorded as not-applicable, never as a default value — a - default would imply a measurement that was never taken (FR-017). -- A skipped cell MUST carry a reason and MUST NOT be rendered as a zero. +- An ignored axis MUST be recorded as not-applicable, never as a default value — a default + would imply a measurement that was never taken (FR-017). +- A skipped row MUST carry a reason code AND the cell it collapses onto, and MUST NOT be + rendered as a zero. These combinations are configurable and redundant, NOT impossible. - Cell identity MUST be stable across runs so results are comparable over time (FR-028). --- diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index e8dcc6c4c..300eb02ab 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -18,13 +18,15 @@ contract fix that makes the first of them honest. The research pass changed three things materially versus the spec's assumptions, all in the direction of less work: -- **The mode matrix is 5 cells, not 12.** The three axes are not a free product; two of them - each govern exactly one surface. Seven combinations are structurally impossible and get - skip-with-reason rows. -- **The matrix crosses on ONE long-lived proxy instance.** All three routing-mode servers are - built at startup and permanently mounted, so a cell is selected by URL, not by a config - change plus restart. Only the two serialization axes need config, and only for their own - surface. +- **The mode matrix is 5 distinct behaviours, not 12.** The three axes are not a free product: + each serialization axis has exactly one consumer, so the other 7 combinations are + configurable but **behaviourally redundant** — each collapses onto one of the 5. They are + reported as skip-with-reason rows naming the cell they collapse onto, never as zeros and + never as "impossible". +- **The routing-mode axis costs nothing to cross.** All three routing-mode servers are built + at startup and permanently mounted, so that axis is selected by endpoint URL with no config + change and no restart. The two serialization axes still require config; whether they + hot-reload is an open question that changes the runner's shape (see below). - **MCPMark needs one `elif` to point at mcpproxy**, and already emits per-task token usage, pass verdicts, turn counts and pass^k. The spec's fallback plan (build an in-repo task set) is not needed. @@ -58,7 +60,7 @@ and must run offline; no recorded session content may reach a report or a third |---|---| | **I. Performance at Scale** | N/A to production paths. The harness adds no code to a request path. The one production-touching change (Phase 0 item 3 below) adds two integers to an export DTO. | | **II. Actor-Based Concurrency** | No new concurrency in production code. The harness is a batch tool. | -| **III. Configuration-Driven** | PASS, and strengthened: the matrix is crossed by URL + existing config fields, adding no benchmark-only configuration to the product. | +| **III. Configuration-Driven** | PASS: the matrix is crossed by endpoint URL plus existing config fields, adding no benchmark-only configuration to the product. | | **IV. Security by Default** | **The binding gate for this feature.** Replay inputs are raw user traffic; the export path deliberately does not mask. Design must default to bodies-off and must never transmit recorded content off-box. See Security Design below. | | **V. TDD** | PASS. Every unit below is specified test-first; the deterministic half is fully unit-testable with no proxy, following the `flipgate.go` precedent of parameterising the driver over a function type. | | **VI. Documentation Hygiene** | PASS. `bench/README.md` is the feature's user-facing doc and must gain the replay and live-loop sections; `CLAUDE.md` needs no change (no new commands at the top level, no architecture change). | @@ -89,8 +91,9 @@ Five topics were resolved; the load-bearing outcomes: 1. **Extension seams** — three exist. Spec 103 uses the "new measurement mode" seam twice (flag on `cmd/bench` + driver in package `bench` + report block), following the spec-085 flip-gate precedent. **No new arm is needed**, and no new binary. -2. **The valid matrix is 5 cells**, with 7 documented skip reasons; cells are selected by - endpoint URL on one running proxy. +2. **The matrix is 5 distinct behaviours**; the other 7 combinations of the 3-axis product + are configurable but redundant and collapse onto them. The routing-mode axis is selected by + endpoint URL with no restart; the serialization axes still need config. 3. **One backend change is required**: `request_bytes` / `response_bytes` exist on the storage record, explicitly measured pre-truncation, but are absent from the export contract. Adding them turns FR-002 from "exclude truncated records" into "annotate them accurately", which @@ -187,6 +190,9 @@ a masking layer, which is the simpler of the two available designs. These need a human, and none blocks starting US1: +0. **Can the two serialization fields be hot-reloaded, or does each value need a fresh + instance?** This decides whether the matrix crosses on one long-lived proxy or on several, + and it is the one open item that changes the runner's shape. Verify before `/speckit.tasks`. 1. **Pinned model and spend ceiling** for the live loop. US1 delivers alone without it. 2. **Whether cache-read tokens count toward "tokens per completed task."** FR-014 requires tracking them separately; FR-018 does not say which composite the headline uses. This is a From 34dd92cbac033993ba4f9c351ea5e7010392948e Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:32:54 +0300 Subject: [PATCH 05/12] =?UTF-8?q?fix(specs):=20103=20plan=20=E2=80=94=20by?= =?UTF-8?q?te=20counts=20are=20not=20token=20counts=20(review=20round=202)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nine findings from the re-run cross-model review, all verified against the tree before applying. One invalidated a load-bearing design claim. 1. HIGH — the privacy design rested on a false premise. It claimed bodies-off still yields an accurate cost because pre-truncation byte sizes are recorded. But request_bytes/response_bytes are BYTE LENGTHS, and tokenizing requires the text, so they can only support an estimate. Worse, internal retrieve_tools activity emission carries no byte counts at all, so discovery-response cost is unrecoverable from the activity log by any route. The bodies-off default survives, for a better reason: TOOL-SURFACE cost — the term the modes actually change, and US1's headline — is computed from the fleet's tool definitions and the call sequence, not from recorded content, so it is fully measured with bodies off. Response cost is the part that needs text: estimated with bodies off, measured only with bodies on, and the two are never presented interchangeably. 2. HIGH — the proposed SC-011 gate did not work. `git status --porcelain bench/results` cannot see ignored files and would stay silent about an already-tracked report. Replaced with `test -z "$(git ls-files bench/results)"`. 3. HIGH — the report contract contradicted itself: it required a version bump whenever an existing field's meaning changes, then proposed narrowing the document-level tokenizer field's meaning without one. Resolved additively — the existing field keeps its meaning and each new block carries its own accounting_source. 4. The "exactly one consumer" claim for direct_tool_response_mode was false: it has three production call sites (listing, drift detection, reload logging). All three concern the direct surface, so the 12-to-5 collapse holds, but the claim is now stated as one SURFACE, not one call site. 5. Serialization hot-reload was recorded as an open question blocking tasks. It is not open: tool_response_mode reads the live snapshot per call and direct_tool_response_mode rebuilds on config.reloaded. The matrix crosses on one instance with a config apply between serialization cells. Blocker removed. 6. The package placement needed a dependency rule. bench/corpusio imports bench, so a bench/replaycorpus mirroring it could not be imported by bench/replay.go without a second cycle. Now stated explicitly: either replaycorpus imports nothing from bench, or orchestration stays in cmd/bench. 7. Added an explicit compose-don't-reimplement rule for modematrix.go, naming the existing pieces it must reach through structural interfaces. 8. Sensitive-data wording overstated: there is no persisted HasSensitiveData field; it is derived at export from detection metadata added asynchronously after initial persistence. 9. The quickstart contradicted itself on offline operation. Setting the tokenizer cache variable names a cache but does not fill one, so a genuinely offline reproduction needs it pre-populated. --- .../103-token-bench/contracts/mode-matrix.md | 25 ++++++---- .../103-token-bench/contracts/replay-input.md | 25 +++++++--- .../contracts/report-v2-additions.md | 20 ++++++-- specs/103-token-bench/data-model.md | 19 +++++--- specs/103-token-bench/plan.md | 48 ++++++++++++------- specs/103-token-bench/quickstart.md | 36 +++++++++----- specs/103-token-bench/research.md | 25 +++++++--- 7 files changed, 136 insertions(+), 62 deletions(-) diff --git a/specs/103-token-bench/contracts/mode-matrix.md b/specs/103-token-bench/contracts/mode-matrix.md index 33a570071..f8fe584de 100644 --- a/specs/103-token-bench/contracts/mode-matrix.md +++ b/specs/103-token-bench/contracts/mode-matrix.md @@ -40,12 +40,15 @@ because on each surface one or both serialization axes have no consumer. Calling ### Why each axis is ignored where it is -- **`tool_response_mode` has exactly one consumer**: `effectiveToolResponseMode`, called from - the `retrieve_tools` handler (`internal/server/mcp.go:1585`). Neither the direct nor the - code-execution surface reads it — the direct surface has no `retrieve_tools` tool at all. -- **`direct_tool_response_mode` has exactly one consumer**: `effectiveDirectToolResponseMode`, - read when building the direct listing (`internal/server/mcp_routing.go:100`). Only the - direct surface consults it. +- **`tool_response_mode` governs exactly one surface**: resolved by + `effectiveToolResponseMode`, whose single production call is in the `retrieve_tools` handler + (`internal/server/mcp.go:1584-1585`). Neither the direct nor the code-execution surface reads + it — the direct surface has no `retrieve_tools` tool at all. +- **`direct_tool_response_mode` governs exactly one surface**: resolved by + `effectiveDirectToolResponseMode`. Note it has THREE production call sites, not one — + building the listing (`mcp_routing.go:100`), detecting serialization drift (`:1040-1045`) + and logging a reload (`:1071-1076`) — but all three concern the direct surface only, so the + collapse holds. The claim is one *surface*, not one *call site*. - **`code_execution` forces full**: that surface overwrites the response mode with `full` and blanks the detail parameter; `detail` is not in its schema. That is a genuine *override*, distinct from an ignored axis, hence its own reason code. @@ -62,10 +65,12 @@ reason `degenerate`. three routing-mode servers are built at startup and all three endpoints stay mounted regardless of config (`internal/server/server.go:2514-2536`). 2. **The two serialization axes DO require config**, and each affects only its own surface. - A cell is therefore (URL, serialization-config) — **not URL alone**. Whether those two - fields hot-reload, or whether each value needs a fresh instance, is OPEN and must be - settled before tasks are written: it decides whether the matrix crosses on one long-lived - proxy or on several. + A cell is therefore (URL, serialization-config) — **not URL alone**. + **Both hot-reload, so the matrix still crosses on ONE long-lived instance**, with a config + apply between serialization cells: `tool_response_mode` is read from the live snapshot on + every call (`internal/server/profile_resolver.go:49-64`, documented as taking effect "on + the very next call, without reconstructing the server"), and `direct_tool_response_mode` is + rebuilt on the `config.reloaded` event (`internal/server/server.go:569-597`). 3. **An ignored axis is recorded as not-applicable**, never as a default value — a default would imply a measurement that was never taken. 4. **A skipped row carries a reason code AND the cell it collapses onto, and never renders diff --git a/specs/103-token-bench/contracts/replay-input.md b/specs/103-token-bench/contracts/replay-input.md index ca37f1a1c..624c33612 100644 --- a/specs/103-token-bench/contracts/replay-input.md +++ b/specs/103-token-bench/contracts/replay-input.md @@ -17,7 +17,7 @@ field, so it can neither group units of work nor account tokens. | `tool_name`, `server_name` | Resolves the call against a mode's tool surface. | | `status` | success / error / blocked / rejected. | | `response_truncated` | Marks a record whose stored content was cut. | -| `request_bytes`, `response_bytes` | **Measured pre-truncation.** The accurate cost basis. | +| `request_bytes`, `response_bytes` | **Byte lengths measured pre-truncation — not token counts.** Basis for an explicitly-estimated response cost only. | | `has_sensitive_data` | Best-effort exclusion signal — see limitation below. | ### Backend change this contract requires @@ -26,18 +26,29 @@ field, so it can neither group units of work nor account tokens. measured pre-truncation, but are **absent from the export contract**. They must be added to the export DTO and copied in the export projection. -Without them, FR-002 can only *exclude* truncated records. With them, it can *annotate them -accurately*, which materially increases how much real traffic the headline can use. This is -the only production-code change the feature requires. +Without them, FR-002 can only *exclude* truncated records. With them, a truncated record can +carry an explicitly-estimated response cost instead of being dropped. + +**They do NOT make response cost measurable.** Tokenizing requires the text; a byte length +yields an estimate at best. See the cost split below. + +**Known gap**: internal `retrieve_tools` activity emission carries no byte counts at all, so +discovery-response cost is not recoverable from the activity log by any means. Report it as +out of reach rather than estimating it. ## Privacy contract -1. **Default is bodies-off.** The headline is computable from byte sizes alone. +1. **Default is bodies-off, and the headline still works.** Tool-surface cost — the term the + modes actually change — is computed from the fleet's tool definitions and the call + sequence, not from recorded content, so it is fully `measured` with bodies off. + Response cost is the part that needs text: with bodies off it is reported as an explicit + `estimated` figure from byte length, or omitted. It is never labelled `measured`. 2. **Bodies-on is a separate, explicit opt-in** and prints a warning. The export path does **not** mask: masking is wired into the list and detail handlers only, so a bodies-on export is raw and unmasked by design — it is the compliance surface. -3. **`has_sensitive_data` is not a guarantee.** It is written asynchronously *after* the - record is persisted, so a freshly exported record may be sensitive but not yet flagged. +3. **`has_sensitive_data` is not a guarantee.** There is no persisted sensitivity field: the + flag is derived at export from detection metadata added asynchronously after initial + persistence, so a freshly exported record may be sensitive but not yet flagged. Exclude-by-flag is a best-effort reducer and must be documented as such wherever relied on. 4. **Nothing crosses the loader boundary but counts.** Sessions and calls surrender sizes, statuses and derived measurements. No content reaches a report, a dashboard, or any diff --git a/specs/103-token-bench/contracts/report-v2-additions.md b/specs/103-token-bench/contracts/report-v2-additions.md index facaa1632..ea3459453 100644 --- a/specs/103-token-bench/contracts/report-v2-additions.md +++ b/specs/103-token-bench/contracts/report-v2-additions.md @@ -39,8 +39,17 @@ The report envelope carries **one** tokenizer identity for the whole document. O provider-reported usage enters the same envelope, that field silently claims the entire report was tokenizer-counted. -It must be **scoped to the deterministic sections**, with the live block naming its own -accounting source. Supplementing is not enough — the existing field becomes false otherwise. +**Do not redefine the existing field.** This contract states that a version bump is required +whenever an existing field's meaning changes, and narrowing the document-level tokenizer to +"the deterministic sections only" would be exactly such a change — the two rules contradict. + +Resolve it additively instead: leave the existing field's meaning intact (it names the +deterministic estimator, which remains true of every section that has one), and give each new +block its own explicit `accounting_source` naming either that estimator or the provider plus +pinned model. A reader then never has to infer scope from a document-level field. + +If a future change really must narrow the existing field, bump the version — that is what the +rule is for. ### 2. Add per-row provenance @@ -67,8 +76,11 @@ measured rate and a defaulted one under a single `estimated` badge. ## Enforcement gaps to close - **`SC-011` has no gate today.** "Reports are never committed" is gitignore plus convention; - no CI job or test fails on a committed report. Add a CI step asserting the results directory - is clean, mirroring the existing OAS verification script. + no CI job or test fails on a committed report. + **`git status --porcelain bench/results` is NOT a valid gate** — porcelain does not list + ignored files, and it would also stay silent about a report that is already tracked. The + correct assertion is that no file under the results directory is tracked at all: + `test -z "$(git ls-files bench/results)"`. - **`SC-004` is not currently achievable offline.** The tokenizer fetches its vocabulary over the network unless a cache directory env var is set, and nothing in the repo or CI sets it. Set it in CI and document it in the reproduction procedure. diff --git a/specs/103-token-bench/data-model.md b/specs/103-token-bench/data-model.md index 6af4353d5..210eeb2c7 100644 --- a/specs/103-token-bench/data-model.md +++ b/specs/103-token-bench/data-model.md @@ -28,9 +28,10 @@ One unit of real recorded work, reassembled from exported activity JSONL. - A session with `truncated` calls MUST NOT contribute a silent token total. It either contributes an annotated figure derived from pre-truncation byte sizes, or is excluded and counted in the exclusion report (FR-002, FR-003, SC-008). -- `sensitive` is a **best-effort** reducer. The flag is written asynchronously after the - record is persisted, so its absence does not prove a record is clean. This limitation MUST - be stated wherever the flag is relied upon (Principle IV). +- `sensitive` is a **best-effort** reducer. There is no persisted sensitivity field; the flag + is derived at export from detection metadata that is added asynchronously after initial + persistence, so its absence does not prove a record is clean. This limitation MUST be stated + wherever the flag is relied upon (Principle IV). - A session MUST NOT carry its call contents into any report structure. Only counts, sizes and derived measurements propagate (FR-006). @@ -46,16 +47,20 @@ One recorded tool call inside a session. | Field | Meaning | |---|---| | `tool_name`, `server_name` | Identity, used to resolve the call against a mode's tool surface. | -| `request_bytes`, `response_bytes` | **Measured pre-truncation.** The reason the backend contract change matters: these permit an accurate response-cost figure without reading a body. | +| `request_bytes`, `response_bytes` | **Byte lengths, measured pre-truncation — NOT token counts.** They support an explicitly-estimated response cost when bodies are absent; they cannot produce a measured one, because tokenizing requires the text. | | `response_truncated` | Whether the stored content was cut. | | `status` | success / error / blocked / rejected. | | `parent_id` | Links a code-execution sub-call to the call that issued it. | -| `has_sensitive_data` | Best-effort flag (see above). | +| `has_sensitive_data` | Derived at export from detection metadata, which is added asynchronously after initial persistence. Best-effort (see above). | **Validation rules** -- When bodies are absent (the default), token cost MUST be derived from the byte sizes, and - the resulting figure labelled as byte-derived rather than content-derived. +- **Tool-surface cost does not depend on recorded content at all** — it is computed from the + fleet's tool definitions under each mode plus the call sequence. It is fully `measured` with + bodies off. This is the term the modes actually change, and it is US1's headline. +- **Response cost DOES require the text.** With bodies off it may only be reported as an + explicit `estimated` figure derived from byte length, never as `measured`; with bodies on it + becomes `measured`. The two MUST NOT be presented interchangeably. - A call whose `parent_id` resolves MUST be attributed to its parent's session, so code-execution sub-calls are neither double-counted nor orphaned. diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index 300eb02ab..54935f352 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -25,8 +25,9 @@ direction of less work: never as "impossible". - **The routing-mode axis costs nothing to cross.** All three routing-mode servers are built at startup and permanently mounted, so that axis is selected by endpoint URL with no config - change and no restart. The two serialization axes still require config; whether they - hot-reload is an open question that changes the runner's shape (see below). + change and no restart. The two serialization axes still require config, but **both + hot-reload**, so the whole matrix still crosses on one long-lived instance with a config + apply between serialization cells. - **MCPMark needs one `elif` to point at mcpproxy**, and already emits per-task token usage, pass verdicts, turn counts and pass^k. The spec's fallback plan (build an in-repo task set) is not needed. @@ -79,9 +80,13 @@ Three findings drive this and each becomes a design rule: exported record can be sensitive but not yet flagged. → **Rule**: exclude-by-flag is a best-effort reducer and MUST NOT be described as a guarantee, in code comments or in the published methodology. -3. **Bodies are not needed for the headline.** Byte sizes recorded pre-truncation give an - accurate response-cost figure without reading content (see Phase 0 item 3). - → **Rule**: the default configuration produces the headline without ever loading a body. +3. **Bodies are not needed for the headline — but not for the reason first assumed.** + Tool-surface cost, the term the modes actually change, is computed from the fleet's tool + definitions and the call sequence, so it needs no recorded content at all. + The recorded byte sizes are byte *lengths*, not token counts, so they support only an + explicitly-estimated response cost; measured response cost requires bodies. + → **Rule**: the default configuration produces the measured headline without ever loading a + body, and any response-cost figure taken with bodies off is badged `estimated`. ## Phase 0: Research @@ -94,10 +99,12 @@ Five topics were resolved; the load-bearing outcomes: 2. **The matrix is 5 distinct behaviours**; the other 7 combinations of the 3-axis product are configurable but redundant and collapse onto them. The routing-mode axis is selected by endpoint URL with no restart; the serialization axes still need config. -3. **One backend change is required**: `request_bytes` / `response_bytes` exist on the storage - record, explicitly measured pre-truncation, but are absent from the export contract. Adding - them turns FR-002 from "exclude truncated records" into "annotate them accurately", which - materially improves the headline's coverage. +3. **One backend change is worth making**: `request_bytes` / `response_bytes` exist on the + storage record, measured pre-truncation, but are absent from the export contract. Adding + them lets a truncated record carry an explicitly-estimated response cost instead of being + dropped. They are byte lengths, **not** token counts, so they do not make response cost + measurable — and internal `retrieve_tools` emission carries no byte counts at all, so + discovery-response cost is unrecoverable from the activity log by any route. 4. **MCPMark is adopted**, SHA-pinned, with one `elif` in its single MCP factory. Its per-task `meta.json` feeds FR-010/011/012/018 directly. 5. **Token accounting stays split**: tiktoken for everything deterministic; provider `usage` @@ -169,12 +176,24 @@ rather than an addition to it, because `corpusio` is scoped by its own doc comme tool-retrieval corpora and produces `Corpus`/`GoldenSet` values — a session trace is neither. `replay.go`, `modematrix.go` and `agentloop.go` sit in package `bench` (not a subpackage) because they need `Tokenizer`, `ProxyToolsForMode` and the `EncodingArm` interface, and -`bench` cannot import `bench/arms` (a documented cycle); they cross that boundary with the -same structural interface `RunArms` already uses. +`bench` cannot import `bench/arms` (a real cycle — arms import `bench`); they cross that +boundary with the same structural interface `RunArms` already uses. + +**Dependency rule for `bench/replaycorpus/`**: the existing `bench/corpusio/` imports `bench`, +so if `replaycorpus` mirrors that arrangement then `bench/replay.go` **cannot** import it +without creating a second cycle. Either define the replay domain types wholly inside +`replaycorpus` with no import of `bench`, or keep replay orchestration in `cmd/bench`. This +must be settled before any file is created. + +**`modematrix.go` composes, it does not reimplement.** The pieces it needs already exist — +mode constants and proxy catalogs (`bench/tokens.go`), full rendering +(`bench/arms/baseline.go`), compact measurement (`bench/flipgate.go`), deferred-direct +rendering (`bench/arms/directdeferred.go`) and MCP transport (`bench/mcpcaller.go`). It must +reach them through passed structural interfaces, never by re-deriving a serialization. The only production-code change is two additive DTO fields plus their projection — deliberately -minimal, and justified because without them FR-002 degrades from accurate annotation to -wholesale exclusion. +minimal, and justified because without them a truncated record must be dropped rather than +estimated. ## Phase 1: Design & Contracts @@ -190,9 +209,6 @@ a masking layer, which is the simpler of the two available designs. These need a human, and none blocks starting US1: -0. **Can the two serialization fields be hot-reloaded, or does each value need a fresh - instance?** This decides whether the matrix crosses on one long-lived proxy or on several, - and it is the one open item that changes the runner's shape. Verify before `/speckit.tasks`. 1. **Pinned model and spend ceiling** for the live loop. US1 delivers alone without it. 2. **Whether cache-read tokens count toward "tokens per completed task."** FR-014 requires tracking them separately; FR-018 does not say which composite the headline uses. This is a diff --git a/specs/103-token-bench/quickstart.md b/specs/103-token-bench/quickstart.md index 1817142a5..6ccee5d9c 100644 --- a/specs/103-token-bench/quickstart.md +++ b/specs/103-token-bench/quickstart.md @@ -1,7 +1,8 @@ # Quickstart: token-bench (spec 103) -Two independent halves. **Start with the deterministic one** — it needs no model spend, no -credentials and no network, and it delivers value alone. +Two independent halves. **Start with the deterministic one** — it needs no model spend and no +credentials, and it delivers value alone. It runs offline *once the tokenizer vocabulary is +cached*; see "Offline note" below, which is a real prerequisite rather than a footnote. --- @@ -15,8 +16,10 @@ From a machine that has been using mcpproxy for real work: mcpproxy activity export --format json > ~/replay-corpus.jsonl ``` -**Bodies stay off.** The headline is computable from pre-truncation byte sizes, and the export -path does not mask — a bodies-on export is raw and unmasked by design. +**Bodies stay off.** The headline — per-mode tool-surface cost — is computed from the fleet's +tool definitions and the call sequence, not from recorded content, so it is fully measured +with bodies off. The export path does not mask, so a bodies-on export is raw and unmasked by +design; take one only if you specifically need measured *response* cost, and delete it after. > The file is real user traffic. Keep it outside the repository, never commit it, and delete > it when finished. It is deliberately gitignored nowhere, because it should never be inside @@ -31,13 +34,21 @@ go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out bench/results Produces the `replay` block in `bench/results/report.json` plus the dashboard section: per mode cell, what this workload would have cost, with the exclusion accounting beside it. -### 3. Read the exclusion report first, not the headline +### 3. Know which figures you just got + +- **Tool-surface cost per mode: `measured`.** No recorded content was needed. +- **Response cost: `estimated` or absent.** Tokenizing needs the text; a byte length only + supports an estimate. Re-run with bodies on if you need it measured. +- **Discovery-response cost: unavailable.** Internal `retrieve_tools` activity emission + carries no byte counts, so it is not recoverable from the activity log at all. + +### 4. Read the exclusion report before the headline The exclusion counts tell you whether the headline is trustworthy. A corpus that is 80% truncated produces a real number over a small slice, and the report says so rather than pretending otherwise. -### 4. Verify determinism +### 5. Verify determinism ```bash go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/run-a @@ -53,8 +64,10 @@ The tokenizer downloads its vocabulary on first use unless a cache directory is export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" ``` -Set this before claiming an offline run. Without it, a reproducer on a restricted network -fails at step one — which is what SC-004 requires to work. +Setting the variable only names a cache; it does not fill one. A first run still fetches the +vocabulary, so a genuinely offline reproduction needs the cache pre-populated (warm it once +with network access, or vendor/restore a known-good cache in CI). Without that, a reproducer +on a restricted network fails at step one — which is exactly what SC-004 requires to work. --- @@ -121,8 +134,9 @@ These feed tokens-per-completed-task, completion rate, first-attempt success and 2. Measure the tokenizer's divergence from the pinned model using the token-counting endpoint (no inference spend) rather than quoting either of the two contradictory figures currently in circulation. -3. Confirm no report was committed: +3. Confirm no report is tracked: ```bash - git status --porcelain bench/results + test -z "$(git ls-files bench/results)" && echo "clean (SC-011)" ``` - Empty output, or the run is not publishable. + Use `git ls-files`, not `git status --porcelain` — porcelain does not list ignored files and + would stay silent about an already-tracked report. diff --git a/specs/103-token-bench/research.md b/specs/103-token-bench/research.md index 303d46804..a71290703 100644 --- a/specs/103-token-bench/research.md +++ b/specs/103-token-bench/research.md @@ -64,11 +64,18 @@ bodies-off.** Add `request_bytes` / `response_bytes` to the export contract. exported record can be sensitive but not yet flagged. Exclude-by-flag is therefore a best-effort reducer and must never be documented as a guarantee. -**The one required backend change**: `request_bytes` and `response_bytes` exist on the storage -record and are explicitly documented as *measured pre-truncation* — but they are absent from -the export contract entirely. Adding them (two fields on the DTO, two lines in the export -projection) turns FR-002 from "exclude every truncated record" into "annotate it accurately", -which materially raises how much real traffic the headline can use. +**The one worthwhile backend change**: `request_bytes` and `response_bytes` exist on the +storage record, documented as *measured pre-truncation*, but are absent from the export +contract entirely. Adding them (two DTO fields, two lines in the export projection) lets a +truncated record carry an explicitly-estimated response cost instead of being dropped. + +**Correction applied after review**: an earlier draft claimed these give an *accurate* cost +basis without bodies. They do not — they are byte lengths, and tokenizing requires the text. +What actually survives bodies-off is the **tool-surface** cost, which is derived from the +fleet's tool definitions and the call sequence rather than from any recorded content, and +which is the term the modes change. Response cost with bodies off is `estimated` at best. +Separately, internal `retrieve_tools` emission carries no byte counts, so discovery-response +cost is unrecoverable from the activity log by any route. **Alternatives considered**: reading BBolt directly (rejected — bypasses the contract, couples the harness to storage internals, and defeats the privacy gating entirely); building a new @@ -106,8 +113,12 @@ Plus the existing `baseline` arm as the FR-020 denominator. **Crossing the matrix is cheap.** All three routing-mode servers are built at startup and all three endpoints are permanently mounted regardless of config — the source comments this -explicitly. A cell is therefore selected **by URL on one long-lived proxy instance**; only the -two serialization axes need config, and each affects only its own surface. +explicitly — so the routing-mode axis is selected **by URL**, with no restart. The two +serialization axes do need config, but **both hot-reload**: `tool_response_mode` is read from +the live snapshot on every call (documented as taking effect "on the very next call, without +reconstructing the server") and `direct_tool_response_mode` is rebuilt on the `config.reloaded` +event. So the whole matrix crosses on ONE long-lived instance, with a config apply between +serialization cells. **Reuse the existing skip shape** rather than inventing one: `ArmResult.Skipped` / `SkipReason` and the `SkippedArmResult` constructor already implement "a skipped row with a From 919dec2e3659d1f797193277afad107c177a1a6c Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:38:58 +0300 Subject: [PATCH 06/12] =?UTF-8?q?fix(specs):=20103=20plan=20=E2=80=94=20sc?= =?UTF-8?q?ope=20replay=20per=20cell,=20not=20per=20feature=20(review=20ro?= =?UTF-8?q?und=203)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight findings, all verified before applying. Two corrected an OVER-correction made in round 2: fixing the byte-count error, I replaced one absolute claim with another. Round 3 caught both. 1. "Tool-surface cost carries the headline, so bodies-off works" was too broad. It holds for three of five cells, not all five. For direct_full, direct_deferred and code_exec, what the mode changes IS the static tool-surface payload, computed from the fleet's tool definitions with no recorded content — measurable bodies-off. For retrieve_full and retrieve_compact, what the mode changes IS the retrieve_tools response body, so those two require an explicit bodies-on run and are reported as unavailable otherwise, never as zero. 2. "Discovery-response cost is unrecoverable by any route" was false. The retrieve_tools handler writes the FULL response to the activity log (internal/server/mcp.go:2061-2064), so bodies-on does carry it. It is unavailable only in the bodies-off configuration. 3. SC-002 asks for byte-identical replay reports, but the report carries a wall-clock generated_at stamp, so the documented diff would always fail. The quickstart now excludes that field and the plan records the decision as required before the determinism test is written. 4. The chosen suite records input/output/total/reasoning per task and has NO cache-read field, so FR-014's cache-read axis cannot come from the suite's report. It must be captured in the driver from the provider response or declared out of reach for suite-driven runs. 5-7. Round-2 fixes were applied to the contracts but not to their mirrors in plan.md and research.md, leaving three contradictions: the tokenizer field was still described as needing to be narrowed, the obsolete porcelain SC-011 gate survived in research.md, and the offline remedy still said "set the env var" without the population step. All three now match the contracts. 8. The replay package dependency was left as a choice to settle later while the structure already committed to bench/replay.go. Decided: bench/replaycorpus imports nothing from bench, defines its own domain types, and bench/replay.go imports it. Orchestrating from cmd/bench is rejected — it would put measurement logic outside the package holding every other measurement. Also recorded, because the export carries no fleet snapshot: every replay figure scores a recorded workload against TODAY'S fleet. Internally valid across modes, but not a historical reconstruction, and reports must say so. --- .../103-token-bench/contracts/replay-input.md | 33 +++++++++--- .../contracts/report-v2-additions.md | 7 ++- specs/103-token-bench/data-model.md | 18 ++++--- specs/103-token-bench/plan.md | 50 +++++++++++-------- specs/103-token-bench/quickstart.md | 21 +++++--- specs/103-token-bench/research.md | 46 +++++++++++------ 6 files changed, 120 insertions(+), 55 deletions(-) diff --git a/specs/103-token-bench/contracts/replay-input.md b/specs/103-token-bench/contracts/replay-input.md index 624c33612..898d78e3f 100644 --- a/specs/103-token-bench/contracts/replay-input.md +++ b/specs/103-token-bench/contracts/replay-input.md @@ -32,17 +32,34 @@ carry an explicitly-estimated response cost instead of being dropped. **They do NOT make response cost measurable.** Tokenizing requires the text; a byte length yields an estimate at best. See the cost split below. -**Known gap**: internal `retrieve_tools` activity emission carries no byte counts at all, so -discovery-response cost is not recoverable from the activity log by any means. Report it as -out of reach rather than estimating it. +**Known gap, precisely scoped**: internal `retrieve_tools` activity emission carries no byte +counts, so with bodies OFF its response cost cannot even be estimated. It is *not* unrecoverable +in general — the handler writes the FULL response to the activity log +(`internal/server/mcp.go:2061-2064`), so a bodies-on export does carry it. Report it as +unavailable in the bodies-off configuration, not as unavailable outright. ## Privacy contract -1. **Default is bodies-off, and the headline still works.** Tool-surface cost — the term the - modes actually change — is computed from the fleet's tool definitions and the call - sequence, not from recorded content, so it is fully `measured` with bodies off. - Response cost is the part that needs text: with bodies off it is reported as an explicit - `estimated` figure from byte length, or omitted. It is never labelled `measured`. +1. **Default is bodies-off. Be precise about what that can and cannot measure.** + + | Mode cell | What the mode changes | Measurable bodies-off? | + |---|---|---| + | `direct_full`, `direct_deferred` | the static tool-surface payload | **Yes** — computed from the fleet's tool definitions, no recorded content needed | + | `retrieve_full`, `retrieve_compact` | the per-call `retrieve_tools` **response** | **No** — that response is the cost, and it is a body | + | `code_exec` | the static surface | **Yes**, same as direct | + + So bodies-off covers three of five cells fully. The two `retrieve_tools` cells need bodies, + because the thing their mode changes IS a response body. + + **What the recording contributes** is the *workload shape* — call sequence, tool mix, call + counts, real fleet size — which is what makes the numbers about real usage rather than a + frozen corpus. The tool definitions themselves come from the live fleet, not the recording. + + **Consequence to state in every replay report**: the export carries no fleet snapshot, so a + replay scores the recorded workload against **today's** fleet, not the fleet as it stood + when the session was recorded. If the fleet has changed, the comparison is still internally + valid across modes but is not a historical reconstruction. + 2. **Bodies-on is a separate, explicit opt-in** and prints a warning. The export path does **not** mask: masking is wired into the list and detail handlers only, so a bodies-on export is raw and unmasked by design — it is the compliance surface. diff --git a/specs/103-token-bench/contracts/report-v2-additions.md b/specs/103-token-bench/contracts/report-v2-additions.md index ea3459453..fe7c526da 100644 --- a/specs/103-token-bench/contracts/report-v2-additions.md +++ b/specs/103-token-bench/contracts/report-v2-additions.md @@ -83,4 +83,9 @@ measured rate and a defaulted one under a single `estimated` badge. `test -z "$(git ls-files bench/results)"`. - **`SC-004` is not currently achievable offline.** The tokenizer fetches its vocabulary over the network unless a cache directory env var is set, and nothing in the repo or CI sets it. - Set it in CI and document it in the reproduction procedure. + Setting the variable is necessary but **not sufficient** — it names a cache without filling + one. CI must also populate or restore that cache, and the reproduction procedure must tell + an outsider to warm it once with network access. +- **`SC-002` (byte-identical reruns) collides with the report's wall-clock `generated_at` + stamp.** Replay must pin or exclude that field, or every determinism check fails on it + alone. diff --git a/specs/103-token-bench/data-model.md b/specs/103-token-bench/data-model.md index 210eeb2c7..5166ca701 100644 --- a/specs/103-token-bench/data-model.md +++ b/specs/103-token-bench/data-model.md @@ -55,12 +55,18 @@ One recorded tool call inside a session. **Validation rules** -- **Tool-surface cost does not depend on recorded content at all** — it is computed from the - fleet's tool definitions under each mode plus the call sequence. It is fully `measured` with - bodies off. This is the term the modes actually change, and it is US1's headline. -- **Response cost DOES require the text.** With bodies off it may only be reported as an - explicit `estimated` figure derived from byte length, never as `measured`; with bodies on it - becomes `measured`. The two MUST NOT be presented interchangeably. +- **Tool-surface cost does not depend on recorded content** — it is computed from the fleet's + tool definitions under each mode. It is fully `measured` with bodies off, and it is what the + direct and code-execution cells' modes change. +- **The two `retrieve_tools` cells are NOT measurable with bodies off**, because what their + mode changes is the `retrieve_tools` response body itself. They require an explicit + bodies-on run and MUST be reported as unavailable otherwise, never as zero. +- **Response cost generally requires the text.** With bodies off it may only be an explicit + `estimated` figure derived from byte length; with bodies on it becomes `measured`. The two + MUST NOT be presented interchangeably. +- **The export carries no fleet snapshot**, so every replay figure is against the currently + configured fleet. Reports MUST state this: it is internally valid across modes but is not a + historical reconstruction. - A call whose `parent_id` resolves MUST be attributed to its parent's session, so code-execution sub-calls are neither double-counted nor orphaned. diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index 54935f352..547966841 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -9,8 +9,9 @@ Add two measurement capabilities to the existing `bench/` harness, and one small contract fix that makes the first of them honest. 1. **US1 — deterministic replay** (`-replay`): load exported activity JSONL, group it into - units of work, and recompute what that real workload would have cost under every valid - mode. No model, byte-reproducible. + units of work, and recompute what that real workload would have cost under each mode cell. + No model. Reproducible **modulo the report's `generated_at` stamp**, which must be excluded + or pinned before any byte-identical comparison (see Risks). 2. **US2 — live agent loop**: run MCPMark against mcpproxy under each mode, capturing provider-reported token usage and per-task pass verdicts, to produce tokens per *completed* task, first-attempt success and retry rates. @@ -80,13 +81,13 @@ Three findings drive this and each becomes a design rule: exported record can be sensitive but not yet flagged. → **Rule**: exclude-by-flag is a best-effort reducer and MUST NOT be described as a guarantee, in code comments or in the published methodology. -3. **Bodies are not needed for the headline — but not for the reason first assumed.** - Tool-surface cost, the term the modes actually change, is computed from the fleet's tool - definitions and the call sequence, so it needs no recorded content at all. - The recorded byte sizes are byte *lengths*, not token counts, so they support only an - explicitly-estimated response cost; measured response cost requires bodies. - → **Rule**: the default configuration produces the measured headline without ever loading a - body, and any response-cost figure taken with bodies off is badged `estimated`. +3. **Bodies-off covers three of the five cells, not all five.** For the two direct cells and + the code-execution cell, what the mode changes is the static tool-surface payload, computed + from the fleet's tool definitions with no recorded content. For the two `retrieve_tools` + cells, what the mode changes IS the response body, so those need bodies-on. The recorded + byte sizes are byte *lengths*, not token counts, and support only an estimate. + → **Rule**: bodies-off is the default and yields measured figures for three cells; the two + `retrieve_tools` cells are reported as requiring an explicit bodies-on run. ## Phase 0: Research @@ -103,8 +104,9 @@ Five topics were resolved; the load-bearing outcomes: storage record, measured pre-truncation, but are absent from the export contract. Adding them lets a truncated record carry an explicitly-estimated response cost instead of being dropped. They are byte lengths, **not** token counts, so they do not make response cost - measurable — and internal `retrieve_tools` emission carries no byte counts at all, so - discovery-response cost is unrecoverable from the activity log by any route. + measurable. Internal `retrieve_tools` emission carries no byte counts either — but its FULL + response IS written to the activity log, so its cost is recoverable with bodies on and + unavailable only in the bodies-off configuration. 4. **MCPMark is adopted**, SHA-pinned, with one `elif` in its single MCP factory. Its per-task `meta.json` feeds FR-010/011/012/018 directly. 5. **Token accounting stays split**: tiktoken for everything deterministic; provider `usage` @@ -114,13 +116,20 @@ Five topics were resolved; the load-bearing outcomes: ### Risks carried into Phase 1 - **`ReportV2.Tokenizer` is a report-level singleton.** The moment provider-usage figures - enter the same envelope, that field silently claims the whole report was tiktoken-counted. - It must be scoped to the deterministic sections, not merely supplemented. + enter the same envelope, a reader could take that field to describe them too. Resolve it + **additively** — leave the existing field's meaning intact and give every new block its own + `accounting_source` — rather than by narrowing the existing field, which would itself require + a version bump. See contracts/report-v2-additions.md. - **Provenance is section-level, not per-figure.** FR-013 requires measured and estimated figures to coexist inside one block, so new row types need their own provenance field. - **tiktoken fetches its vocabulary over the network** unless `TIKTOKEN_CACHE_DIR` is set; - nothing in the repo or CI sets it. SC-004 (an outsider reproduces the deterministic figures) - is not currently achievable on a restricted network. + nothing in the repo or CI sets it. Setting the variable only *names* a cache — a first run + still downloads — so an offline reproduction additionally needs the cache **populated** + (warmed once with network, or restored/vendored in CI). Without that, SC-004 is not + achievable on a restricted network. +- **The report carries a wall-clock `generated_at` stamp.** SC-002 asks for byte-identical + replay reports; two runs will differ on that field alone unless replay pins or excludes it. + Decide which before writing the determinism test. - **`RetryRateForArm` returns 0.0 for unknown arms**, indistinguishable from a measured 0.0. Mixing a measured rate with a defaulted one under a single section badge is a live hazard for FR-013. @@ -179,11 +188,12 @@ because they need `Tokenizer`, `ProxyToolsForMode` and the `EncodingArm` interfa `bench` cannot import `bench/arms` (a real cycle — arms import `bench`); they cross that boundary with the same structural interface `RunArms` already uses. -**Dependency rule for `bench/replaycorpus/`**: the existing `bench/corpusio/` imports `bench`, -so if `replaycorpus` mirrors that arrangement then `bench/replay.go` **cannot** import it -without creating a second cycle. Either define the replay domain types wholly inside -`replaycorpus` with no import of `bench`, or keep replay orchestration in `cmd/bench`. This -must be settled before any file is created. +**Dependency rule for `bench/replaycorpus/` — decided, not deferred**: the existing +`bench/corpusio/` imports `bench`, so a `replaycorpus` that mirrored it could not be imported +by `bench/replay.go` without a second cycle. **Decision: `bench/replaycorpus` imports nothing +from `bench`.** It defines its own replay domain types, and `bench/replay.go` imports it. The +alternative (orchestrating from `cmd/bench`) is rejected because it would put measurement +logic outside the package that holds every other measurement. **`modematrix.go` composes, it does not reimplement.** The pieces it needs already exist — mode constants and proxy catalogs (`bench/tokens.go`), full rendering diff --git a/specs/103-token-bench/quickstart.md b/specs/103-token-bench/quickstart.md index 6ccee5d9c..85d0f52b6 100644 --- a/specs/103-token-bench/quickstart.md +++ b/specs/103-token-bench/quickstart.md @@ -36,11 +36,16 @@ mode cell, what this workload would have cost, with the exclusion accounting bes ### 3. Know which figures you just got -- **Tool-surface cost per mode: `measured`.** No recorded content was needed. -- **Response cost: `estimated` or absent.** Tokenizing needs the text; a byte length only - supports an estimate. Re-run with bodies on if you need it measured. -- **Discovery-response cost: unavailable.** Internal `retrieve_tools` activity emission - carries no byte counts, so it is not recoverable from the activity log at all. +- **`direct_full`, `direct_deferred`, `code_exec`: `measured`.** What their mode changes is + the static tool-surface payload, computed from the fleet's tool definitions — no recorded + content needed. +- **`retrieve_full`, `retrieve_compact`: NOT measurable here.** What their mode changes IS the + `retrieve_tools` response body, so these two cells require a bodies-on run. +- **Response cost generally: `estimated` or absent.** Tokenizing needs the text; a byte length + supports only an estimate. +- **The fleet is today's fleet.** The export carries no fleet snapshot, so this scores a + recorded workload against the currently configured servers — internally valid across modes, + but not a historical reconstruction. ### 4. Read the exclusion report before the headline @@ -53,7 +58,10 @@ pretending otherwise. ```bash go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/run-a go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/run-b -diff /tmp/run-a/report.json /tmp/run-b/report.json && echo "byte-identical (SC-002)" +# generated_at is a wall-clock stamp and differs between runs by design. +diff <(jq 'del(.generated_at)' /tmp/run-a/report.json) \ + <(jq 'del(.generated_at)' /tmp/run-b/report.json) \ + && echo "byte-identical modulo generated_at (SC-002)" ``` ### Offline note @@ -62,6 +70,7 @@ The tokenizer downloads its vocabulary on first use unless a cache directory is ```bash export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/warm # warms it, needs network once ``` Setting the variable only names a cache; it does not fill one. A first run still fetches the diff --git a/specs/103-token-bench/research.md b/specs/103-token-bench/research.md index a71290703..b976839bd 100644 --- a/specs/103-token-bench/research.md +++ b/specs/103-token-bench/research.md @@ -41,8 +41,9 @@ existing field's meaning or shape, and spec 103 changes none. Precedent: `Latenc **Finding that becomes work**: SC-011 ("no generated report is committed") is enforced only by `bench/.gitignore` and convention. There is no CI job or test that fails on a committed -report. The cheap gate is a CI step asserting `git status --porcelain bench/results` is empty, -mirroring `scripts/verify-oas.sh`. +report. The gate must assert that nothing under the results directory is **tracked** — +`test -z "$(git ls-files bench/results)"`. A `git status --porcelain` check does NOT work: +porcelain omits ignored files and would stay silent about an already-committed report. --- @@ -69,13 +70,21 @@ storage record, documented as *measured pre-truncation*, but are absent from the contract entirely. Adding them (two DTO fields, two lines in the export projection) lets a truncated record carry an explicitly-estimated response cost instead of being dropped. -**Correction applied after review**: an earlier draft claimed these give an *accurate* cost -basis without bodies. They do not — they are byte lengths, and tokenizing requires the text. -What actually survives bodies-off is the **tool-surface** cost, which is derived from the -fleet's tool definitions and the call sequence rather than from any recorded content, and -which is the term the modes change. Response cost with bodies off is `estimated` at best. -Separately, internal `retrieve_tools` emission carries no byte counts, so discovery-response -cost is unrecoverable from the activity log by any route. +**Correction applied after review, then corrected again**: an early draft claimed these give +an *accurate* cost basis without bodies. They do not — they are byte lengths, and tokenizing +needs the text. A second draft over-corrected, claiming tool-surface cost carries the headline +for all five cells. The precise position: + +- **Three cells (both direct cells and code-execution) are measurable bodies-off**, because + what their mode changes is the static tool-surface payload, computed from the fleet's tool + definitions rather than from any recording. +- **The two `retrieve_tools` cells need bodies-on**, because what their mode changes IS the + `retrieve_tools` response body. +- **`retrieve_tools` response cost is recoverable with bodies on** — the handler writes the + FULL response to the activity log (`internal/server/mcp.go:2061-2064`). It is unavailable + only in the bodies-off configuration, not unrecoverable in general. +- **The export carries no fleet snapshot**, so a replay scores a recorded workload against + today's fleet. Internally valid across modes; not a historical reconstruction. **Alternatives considered**: reading BBolt directly (rejected — bypasses the contract, couples the harness to storage internals, and defeats the privacy gating entirely); building a new @@ -181,7 +190,13 @@ constraint is that same rule on a new axis, so it is enforceable with machinery exists — the extension is a new section and a source label, not new aggregation logic. Provider `usage` maps 1:1 onto FR-014's required split (input / output / cache-read) with no -derivation, so reading response fields is strictly simpler than proxying. An LLM proxy is kept +derivation, so reading response fields is strictly simpler than proxying. + +**Gap to close at task time**: the suite's own per-task output records input / output / total / +reasoning — it has **no cache-read field**. So FR-014's cache-read axis cannot come from the +suite's report; it must be captured in the driver from the provider response, or explicitly +declared out of reach for suite-driven runs. The single MCP-factory patch alone does not +establish FR-014 coverage. An LLM proxy is kept only as the fallback for a closed-box suite whose loop cannot be edited — which MCPMark, being patchable at one seam, is not. @@ -204,9 +219,10 @@ entirely greenfield — no `cache_read` concept exists anywhere. ### Two hazards this creates - **`ReportV2.Tokenizer` is a report-level singleton** naming one estimator. Once provider - usage enters the same envelope, that field silently claims the whole report was - tiktoken-counted. It must be **scoped** to the deterministic sections, and the live block - must carry its own accounting-source field (provider + pinned model). + usage enters the same envelope a reader could take that field to describe it too. Resolve + **additively**: leave the existing field's meaning intact and give each new block its own + accounting-source field (provider + pinned model). Narrowing the existing field would itself + be a meaning change and would require a version bump. - **`RetryRateForArm` returns 0.0 for unknown arms**, indistinguishable from a measured 0.0. With FR-013 replacing assumed rates only where measurements exist, one table could mix a measured rate and a defaulted one under a single section-level `estimated` badge. Session @@ -220,7 +236,9 @@ entirely greenfield — no `cache_read` concept exists anywhere. `TIKTOKEN_CACHE_DIR` / `DATA_GYM_CACHE_DIR` is set. Nothing in the repo or in `bench.yml` sets it, so the documented "offline estimator" property holds only after a warm cache. **An outside reproducer on a restricted network fails at step one** — which defeats - SC-004. Fix: set the cache dir in CI and document it in the reproduction procedure. + SC-004. Fix: setting the variable is necessary but NOT sufficient — it names a cache without + filling one. CI must also **populate or restore** the cache (a cache-restore step, or a + vendored copy), and the reproduction procedure must tell an outsider to warm it once. 2. **The tokenizer caveat is unsourced and internally contradicted.** The repo states up to ~60% underestimation versus Claude; Anthropic's own guidance says ~15–20% on typical text (more on code). The two differ threefold and neither is sourced in-repo. Publishing From cba61c7d9cf81960090759f4497c3584d5e05503 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:43:41 +0300 Subject: [PATCH 07/12] =?UTF-8?q?fix(specs):=20103=20plan=20=E2=80=94=20se?= =?UTF-8?q?parate=20menu=20cost=20from=20complete=20workload=20cost=20(rou?= =?UTF-8?q?nd=204)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five findings, all verified. The first is the THIRD consecutive round in which a fix introduced the next round's defect, so the correction this time is structural rather than another restatement. 1. HIGH — code_exec was wrongly classed as fully measurable bodies-off. Its menu is static, but the surface also registers retrieve_tools with a handler (internal/server/mcp_routing.go:670-676), and those responses are bodies. The artifacts now split TWO different questions that the previous three drafts kept conflating: - MENU cost (what the agent pays to see the tools): measured bodies-off for ALL FIVE cells. - COMPLETE WORKLOAD cost (menu plus every response consumed): measured bodies-off for the two direct cells ONLY. code_exec is partial; retrieve_full and retrieve_compact need bodies-on. Stating it as a per-cell table with both columns removes the ambiguity that produced the last three corrections. 2. The byte-coverage gap was not scoped precisely. Code-execution sub-calls emit both byte counts as zero (internal/server/mcp_code_execution.go:811-818), so a truncated sub-call cannot use the byte-length estimate either and must fall to exclusion accounting rather than contributing zero. 3. The tokenizer contradiction survived in the implementation tree, which still said live_report.go would "scope Tokenizer to deterministic sections" — the very narrowing the contract forbids. Replaced with per-block accounting_source. 4. The proposed SC-011 gate was placed in bench.yml, which triggers only on v* tags and workflow_dispatch and therefore cannot block a PR. Placement is now called out alongside the assertion: a PR-triggered required workflow, or an explicit statement that SC-011 is post-merge detection only. 5. "The recording contributes real fleet size" contradicted the adjacent, correct statement that no fleet snapshot exists. It contributes recorded CALL SHAPE, evaluated against today's fleet. --- .../103-token-bench/contracts/replay-input.md | 57 +++++++++++-------- .../contracts/report-v2-additions.md | 3 + specs/103-token-bench/data-model.md | 7 ++- specs/103-token-bench/plan.md | 21 ++++--- specs/103-token-bench/quickstart.md | 12 ++-- specs/103-token-bench/research.md | 9 ++- 6 files changed, 65 insertions(+), 44 deletions(-) diff --git a/specs/103-token-bench/contracts/replay-input.md b/specs/103-token-bench/contracts/replay-input.md index 898d78e3f..1f4e37f42 100644 --- a/specs/103-token-bench/contracts/replay-input.md +++ b/specs/103-token-bench/contracts/replay-input.md @@ -32,33 +32,42 @@ carry an explicitly-estimated response cost instead of being dropped. **They do NOT make response cost measurable.** Tokenizing requires the text; a byte length yields an estimate at best. See the cost split below. -**Known gap, precisely scoped**: internal `retrieve_tools` activity emission carries no byte -counts, so with bodies OFF its response cost cannot even be estimated. It is *not* unrecoverable -in general — the handler writes the FULL response to the activity log -(`internal/server/mcp.go:2061-2064`), so a bodies-on export does carry it. Report it as -unavailable in the bodies-off configuration, not as unavailable outright. +**Known gaps in byte coverage** — records where even the byte-length estimate is unavailable, +so they must fall to exclusion accounting rather than silently contributing zero: -## Privacy contract - -1. **Default is bodies-off. Be precise about what that can and cannot measure.** - - | Mode cell | What the mode changes | Measurable bodies-off? | - |---|---|---| - | `direct_full`, `direct_deferred` | the static tool-surface payload | **Yes** — computed from the fleet's tool definitions, no recorded content needed | - | `retrieve_full`, `retrieve_compact` | the per-call `retrieve_tools` **response** | **No** — that response is the cost, and it is a body | - | `code_exec` | the static surface | **Yes**, same as direct | +- **Internal `retrieve_tools` emission carries no byte counts.** With bodies off its response + cost cannot even be estimated. It is *not* unrecoverable in general — the handler writes the + FULL response to the activity log (`internal/server/mcp.go:2061-2064`), so a bodies-on export + carries it. +- **Code-execution sub-calls record both byte counts as zero** + (`internal/server/mcp_code_execution.go:811-818`). A truncated sub-call therefore cannot use + the byte-length estimate either, and must be counted as excluded. - So bodies-off covers three of five cells fully. The two `retrieve_tools` cells need bodies, - because the thing their mode changes IS a response body. - - **What the recording contributes** is the *workload shape* — call sequence, tool mix, call - counts, real fleet size — which is what makes the numbers about real usage rather than a - frozen corpus. The tool definitions themselves come from the live fleet, not the recording. +## Privacy contract - **Consequence to state in every replay report**: the export carries no fleet snapshot, so a - replay scores the recorded workload against **today's** fleet, not the fleet as it stood - when the session was recorded. If the fleet has changed, the comparison is still internally - valid across modes but is not a historical reconstruction. +1. **Default is bodies-off. Measurability is per cell, and splits into two different questions: + the MENU cost (what the agent pays to see the tools) and the COMPLETE WORKLOAD cost (menu + plus every response the workload consumed).** + + | Mode cell | Menu cost, bodies-off | Complete workload cost, bodies-off | Why | + |---|---|---|---| + | `direct_full` | measured | measured | the mode changes only the static tools/list payload | + | `direct_deferred` | measured | measured | same | + | `code_exec` | measured | **NO** | this surface also registers `retrieve_tools` (`internal/server/mcp_routing.go:670-676`), whose responses are bodies | + | `retrieve_full` | measured | **NO** | the mode changes the `retrieve_tools` response body itself | + | `retrieve_compact` | measured | **NO** | same | + + So bodies-off gives a measured MENU cost for all five cells, and a measured COMPLETE + WORKLOAD cost for only the two direct cells. Three cells require an explicit bodies-on run + for their complete figure and MUST be reported as unavailable otherwise, never as zero. + + **What the recording contributes** is the *recorded call shape* — call sequence, tool mix + and call counts — evaluated against today's fleet. It does NOT contribute fleet size: the + export carries no fleet snapshot, and the tool definitions come from the live fleet. + + **Consequence to state in every replay report**: a replay scores a recorded workload against + **today's** fleet, not the fleet as it stood when the session was recorded. Internally valid + across modes; not a historical reconstruction. 2. **Bodies-on is a separate, explicit opt-in** and prints a warning. The export path does **not** mask: masking is wired into the list and detail handlers only, so a bodies-on diff --git a/specs/103-token-bench/contracts/report-v2-additions.md b/specs/103-token-bench/contracts/report-v2-additions.md index fe7c526da..1af0b4557 100644 --- a/specs/103-token-bench/contracts/report-v2-additions.md +++ b/specs/103-token-bench/contracts/report-v2-additions.md @@ -81,6 +81,9 @@ measured rate and a defaulted one under a single `estimated` badge. ignored files, and it would also stay silent about a report that is already tracked. The correct assertion is that no file under the results directory is tracked at all: `test -z "$(git ls-files bench/results)"`. + **Placement matters as much as the assertion**: `bench.yml` triggers only on `v*` tags and + `workflow_dispatch`, so a gate there cannot block a pull request. Put it in a PR-triggered + required workflow, or state plainly that SC-011 remains post-merge detection only. - **`SC-004` is not currently achievable offline.** The tokenizer fetches its vocabulary over the network unless a cache directory env var is set, and nothing in the repo or CI sets it. Setting the variable is necessary but **not sufficient** — it names a cache without filling diff --git a/specs/103-token-bench/data-model.md b/specs/103-token-bench/data-model.md index 5166ca701..70bc2d3ec 100644 --- a/specs/103-token-bench/data-model.md +++ b/specs/103-token-bench/data-model.md @@ -58,9 +58,10 @@ One recorded tool call inside a session. - **Tool-surface cost does not depend on recorded content** — it is computed from the fleet's tool definitions under each mode. It is fully `measured` with bodies off, and it is what the direct and code-execution cells' modes change. -- **The two `retrieve_tools` cells are NOT measurable with bodies off**, because what their - mode changes is the `retrieve_tools` response body itself. They require an explicit - bodies-on run and MUST be reported as unavailable otherwise, never as zero. +- **Three cells cannot yield a complete workload figure with bodies off**: both + `retrieve_tools` cells (their mode changes the response body itself) and `code_exec` (its + surface also serves `retrieve_tools`). Menu cost is still measured for all five. The three + MUST be reported as unavailable for the complete figure, never as zero. - **Response cost generally requires the text.** With bodies off it may only be an explicit `estimated` figure derived from byte length; with bodies on it becomes `measured`. The two MUST NOT be presented interchangeably. diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index 547966841..ae801ae7c 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -81,13 +81,14 @@ Three findings drive this and each becomes a design rule: exported record can be sensitive but not yet flagged. → **Rule**: exclude-by-flag is a best-effort reducer and MUST NOT be described as a guarantee, in code comments or in the published methodology. -3. **Bodies-off covers three of the five cells, not all five.** For the two direct cells and - the code-execution cell, what the mode changes is the static tool-surface payload, computed - from the fleet's tool definitions with no recorded content. For the two `retrieve_tools` - cells, what the mode changes IS the response body, so those need bodies-on. The recorded - byte sizes are byte *lengths*, not token counts, and support only an estimate. - → **Rule**: bodies-off is the default and yields measured figures for three cells; the two - `retrieve_tools` cells are reported as requiring an explicit bodies-on run. +3. **Bodies-off measurability is per cell, and splits menu cost from complete workload cost.** + Bodies-off gives a measured MENU cost for all five cells, but a measured COMPLETE WORKLOAD + cost for only the two direct cells. `code_exec` is partial — its menu is static, but the + surface also registers `retrieve_tools`, whose responses are bodies — and both + `retrieve_tools` cells need bodies-on because the mode changes the response body itself. + The recorded byte sizes are byte *lengths*, not token counts, and support only an estimate. + → **Rule**: bodies-off is the default; the three cells that cannot yield a complete figure + from it are reported as unavailable, never as zero. ## Phase 0: Research @@ -168,7 +169,7 @@ bench/ ├── modematrix.go # NEW — the 5 valid cells, skip reasons, cell→URL mapping ├── agentloop.go # NEW — live-loop driver + MCPMark meta.json ingestion ├── reportv2.go # EXTEND — additive optional blocks; per-row provenance -├── live_report.go # EXTEND — scope Tokenizer to deterministic sections +├── live_report.go # EXTEND — per-block accounting_source (do NOT narrow Tokenizer) ├── session.go # EXTEND — measured rates supersede armRetryRates └── cmd/bench/main.go # EXTEND — -replay and -agent-loop branches @@ -176,7 +177,9 @@ internal/contracts/activity.go # EXTEND — add request_bytes / response_byte internal/httpapi/activity.go # EXTEND — copy them in the export projection specs/083-discovery-profiler/contracts/report-v2.schema.json # EXTEND — declare new blocks -.github/workflows/bench.yml # EXTEND — TIKTOKEN_CACHE_DIR; assert no committed reports +.github/workflows/bench.yml # EXTEND — TIKTOKEN_CACHE_DIR set AND cache populated + # SC-011 gate — bench.yml runs only on v* tags and + # workflow_dispatch, so a gate there cannot block a PR ``` **Structure Decision**: The feature lives almost entirely inside the existing `bench/` tree, diff --git a/specs/103-token-bench/quickstart.md b/specs/103-token-bench/quickstart.md index 85d0f52b6..9d793d6eb 100644 --- a/specs/103-token-bench/quickstart.md +++ b/specs/103-token-bench/quickstart.md @@ -36,11 +36,13 @@ mode cell, what this workload would have cost, with the exclusion accounting bes ### 3. Know which figures you just got -- **`direct_full`, `direct_deferred`, `code_exec`: `measured`.** What their mode changes is - the static tool-surface payload, computed from the fleet's tool definitions — no recorded - content needed. -- **`retrieve_full`, `retrieve_compact`: NOT measurable here.** What their mode changes IS the - `retrieve_tools` response body, so these two cells require a bodies-on run. +- **Menu cost: `measured` for all five cells.** Computed from the fleet's tool definitions — + no recorded content needed. +- **Complete workload cost: `measured` for `direct_full` and `direct_deferred` only.** +- **`code_exec`: partial.** Its menu is static, but the surface also serves `retrieve_tools`, + whose responses are bodies — so its complete figure needs a bodies-on run. +- **`retrieve_full`, `retrieve_compact`: need bodies-on.** What their mode changes IS the + `retrieve_tools` response body. - **Response cost generally: `estimated` or absent.** Tokenizing needs the text; a byte length supports only an estimate. - **The fleet is today's fleet.** The export carries no fleet snapshot, so this scores a diff --git a/specs/103-token-bench/research.md b/specs/103-token-bench/research.md index b976839bd..cf8236209 100644 --- a/specs/103-token-bench/research.md +++ b/specs/103-token-bench/research.md @@ -75,9 +75,12 @@ an *accurate* cost basis without bodies. They do not — they are byte lengths, needs the text. A second draft over-corrected, claiming tool-surface cost carries the headline for all five cells. The precise position: -- **Three cells (both direct cells and code-execution) are measurable bodies-off**, because - what their mode changes is the static tool-surface payload, computed from the fleet's tool - definitions rather than from any recording. +- **Separate MENU cost from COMPLETE WORKLOAD cost.** Bodies-off yields a measured menu cost + for all five cells, but a measured complete-workload cost for only the two direct cells, + whose mode changes nothing but the static tools/list payload. +- **`code_exec` is PARTIAL, not measurable**: its menu is static, but the surface also + registers `retrieve_tools` (`internal/server/mcp_routing.go:670-676`) whose responses are + bodies, so its complete workload cost needs bodies-on. - **The two `retrieve_tools` cells need bodies-on**, because what their mode changes IS the `retrieve_tools` response body. - **`retrieve_tools` response cost is recoverable with bodies on** — the handler writes the From 1a43b5ca0f3ee004b88f5a425c073fd75b75b6d3 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:47:22 +0300 Subject: [PATCH 08/12] =?UTF-8?q?fix(specs):=20103=20plan=20=E2=80=94=20re?= =?UTF-8?q?play=20needs=20a=20fleet=20input=20and=20yields=20deltas,=20not?= =?UTF-8?q?=20absolutes=20(round=205)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five findings, three High, all verified. Two expose real design holes rather than wording problems. 1. HIGH — a fleet input is MANDATORY and was missing from the whole design. A menu cost is a property of the tool definitions the agent was shown, and the activity export carries no fleet snapshot. ProxyToolsForMode returns only the built-in proxy tools, not the upstream fleet, so under `direct` the menu is unavailable without one. The documented `-replay ` invocation could therefore compute nothing. Replay now requires pairing with a frozen fleet corpus or a live proxy, and a recording-only run is an error rather than a degraded run. The quickstart commands are corrected accordingly. 2. HIGH — no cell yields an absolute complete-workload cost bodies-off. The previous draft claimed the two direct cells did. Complete workload includes every consumed response and that text is absent, so what bodies-off actually supports is (a) menu cost per cell and (b) the cross-mode DELTA between the two direct cells, whose identical call responses cancel out of the comparison. That delta is the honest bodies-off headline; absolutes are unavailable. 3. HIGH — the loader boundary and bodies-on measurement contradicted each other. ReplayCall carries no response content and no content may leave replaycorpus, yet tokenization was placed downstream in bench/replay.go. Resolved by tokenizing INSIDE the loader and emitting counts, so no text ever crosses the boundary. The two rules can now both hold. 4. Bodies-on does not recover what a truncated retrieve_tools call actually COST the agent: the activity log stores the full pre-truncation response while the agent consumed the truncated text. Tokenizing the stored text would OVERSTATE the agent's cost, so such calls must be excluded or the truncation re-applied. 5. "Provider usage maps 1:1 to input/output/cache-read" is provider-specific and unverified while the provider is unpinned — and is contradicted for the suite's own per-task output, which has no cache-read field. Scoped accordingly. --- .../103-token-bench/contracts/replay-input.md | 51 +++++++++++-------- specs/103-token-bench/data-model.md | 15 ++++-- specs/103-token-bench/plan.md | 25 +++++---- specs/103-token-bench/quickstart.md | 34 +++++++++---- specs/103-token-bench/research.md | 27 ++++++---- 5 files changed, 96 insertions(+), 56 deletions(-) diff --git a/specs/103-token-bench/contracts/replay-input.md b/specs/103-token-bench/contracts/replay-input.md index 1f4e37f42..d0a9c668c 100644 --- a/specs/103-token-bench/contracts/replay-input.md +++ b/specs/103-token-bench/contracts/replay-input.md @@ -42,44 +42,55 @@ so they must fall to exclusion accounting rather than silently contributing zero - **Code-execution sub-calls record both byte counts as zero** (`internal/server/mcp_code_execution.go:811-818`). A truncated sub-call therefore cannot use the byte-length estimate either, and must be counted as excluded. +- **A truncated `retrieve_tools` record does not tell you what the agent PAID.** The activity + log stores the FULL pre-truncation response, while the agent consumed the truncated text. So + even bodies-on recovers the wrong quantity for such calls: they must be excluded, or the + truncation re-applied before counting. Silently tokenizing the stored full text would + OVERSTATE the agent's cost. ## Privacy contract -1. **Default is bodies-off. Measurability is per cell, and splits into two different questions: - the MENU cost (what the agent pays to see the tools) and the COMPLETE WORKLOAD cost (menu - plus every response the workload consumed).** +1. **Replay requires a FLEET INPUT as well as the recording.** A menu cost is a property of the + tool definitions the agent was shown, and the export carries no fleet snapshot. Built-in + proxy tools alone are not the menu — under `direct` the menu is the whole upstream fleet. + So `-replay ` on its own can compute nothing; it must be paired with either a frozen + fleet corpus (the existing committed corpora already serve this) or a live proxy to read + the current fleet from. **A recording-only invocation is an error, not a degraded run.** - | Mode cell | Menu cost, bodies-off | Complete workload cost, bodies-off | Why | +2. **Default is bodies-off. What that yields, per cell:** + + | Mode cell | Menu cost | Absolute complete-workload cost | Cross-mode delta | |---|---|---|---| - | `direct_full` | measured | measured | the mode changes only the static tools/list payload | - | `direct_deferred` | measured | measured | same | - | `code_exec` | measured | **NO** | this surface also registers `retrieve_tools` (`internal/server/mcp_routing.go:670-676`), whose responses are bodies | - | `retrieve_full` | measured | **NO** | the mode changes the `retrieve_tools` response body itself | - | `retrieve_compact` | measured | **NO** | same | + | `direct_full` | measured (needs fleet input) | **NO** — response text absent | **measured** vs `direct_deferred` | + | `direct_deferred` | measured (needs fleet input) | **NO** — response text absent | **measured** vs `direct_full` | + | `code_exec` | measured (needs fleet input) | **NO** | not comparable bodies-off | + | `retrieve_full` | measured (needs fleet input) | **NO** | **NO** — the mode changes the response body | + | `retrieve_compact` | measured (needs fleet input) | **NO** | **NO** — same | - So bodies-off gives a measured MENU cost for all five cells, and a measured COMPLETE - WORKLOAD cost for only the two direct cells. Three cells require an explicit bodies-on run - for their complete figure and MUST be reported as unavailable otherwise, never as zero. + **No cell has a measured ABSOLUTE complete-workload cost bodies-off**, because complete + workload includes every consumed response and that text is absent. What bodies-off does give + is (a) menu cost per cell, and (b) the cross-mode DELTA between the two direct cells, whose + identical call responses cancel out of the comparison. That delta is the honest bodies-off + headline; an absolute figure is not available. - **What the recording contributes** is the *recorded call shape* — call sequence, tool mix - and call counts — evaluated against today's fleet. It does NOT contribute fleet size: the - export carries no fleet snapshot, and the tool definitions come from the live fleet. + **What the recording contributes** is the *recorded call shape* — call sequence, tool mix and + call counts — evaluated against the supplied fleet input. It does not contribute fleet size. **Consequence to state in every replay report**: a replay scores a recorded workload against - **today's** fleet, not the fleet as it stood when the session was recorded. Internally valid + the SUPPLIED fleet, not the fleet as it stood when the session was recorded. Internally valid across modes; not a historical reconstruction. -2. **Bodies-on is a separate, explicit opt-in** and prints a warning. The export path does +3. **Bodies-on is a separate, explicit opt-in** and prints a warning. The export path does **not** mask: masking is wired into the list and detail handlers only, so a bodies-on export is raw and unmasked by design — it is the compliance surface. -3. **`has_sensitive_data` is not a guarantee.** There is no persisted sensitivity field: the +4. **`has_sensitive_data` is not a guarantee.** There is no persisted sensitivity field: the flag is derived at export from detection metadata added asynchronously after initial persistence, so a freshly exported record may be sensitive but not yet flagged. Exclude-by-flag is a best-effort reducer and must be documented as such wherever relied on. -4. **Nothing crosses the loader boundary but counts.** Sessions and calls surrender sizes, +5. **Nothing crosses the loader boundary but counts.** Sessions and calls surrender sizes, statuses and derived measurements. No content reaches a report, a dashboard, or any third-party service including model providers. -5. **Inputs live outside the repository**, are never committed, and the documented procedure +6. **Inputs live outside the repository**, are never committed, and the documented procedure tells an operator how to delete them. ## Loader output diff --git a/specs/103-token-bench/data-model.md b/specs/103-token-bench/data-model.md index 70bc2d3ec..a9a6518ff 100644 --- a/specs/103-token-bench/data-model.md +++ b/specs/103-token-bench/data-model.md @@ -34,6 +34,10 @@ One unit of real recorded work, reassembled from exported activity JSONL. wherever the flag is relied upon (Principle IV). - A session MUST NOT carry its call contents into any report structure. Only counts, sizes and derived measurements propagate (FR-006). +- **Therefore tokenization of response bodies happens INSIDE the loader**, not downstream: a + bodies-on run tokenizes within `replaycorpus` and emits token counts, so no text ever + crosses the boundary. A design that hands bodies to the scoring layer to tokenize would + violate the invariant above — the two cannot both hold. **State transitions**: `loaded → grouped → classified → {scored | excluded}`. Exclusion is terminal and always counted. @@ -58,10 +62,13 @@ One recorded tool call inside a session. - **Tool-surface cost does not depend on recorded content** — it is computed from the fleet's tool definitions under each mode. It is fully `measured` with bodies off, and it is what the direct and code-execution cells' modes change. -- **Three cells cannot yield a complete workload figure with bodies off**: both - `retrieve_tools` cells (their mode changes the response body itself) and `code_exec` (its - surface also serves `retrieve_tools`). Menu cost is still measured for all five. The three - MUST be reported as unavailable for the complete figure, never as zero. +- **NO cell yields an absolute complete-workload cost with bodies off**, because complete + workload includes every consumed response and that text is absent. Bodies-off yields menu + cost per cell (given a fleet input) and the cross-mode DELTA between the two direct cells, + whose identical call responses cancel. Absolute figures MUST be reported as unavailable, + never as zero. +- **Menu cost requires a FLEET INPUT** — a frozen fleet corpus or a live proxy. The export + carries no fleet snapshot, so a recording-only run computes nothing. - **Response cost generally requires the text.** With bodies off it may only be an explicit `estimated` figure derived from byte length; with bodies on it becomes `measured`. The two MUST NOT be presented interchangeably. diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index ae801ae7c..51aeee4fb 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -8,10 +8,12 @@ Add two measurement capabilities to the existing `bench/` harness, and one small backend contract fix that makes the first of them honest. -1. **US1 — deterministic replay** (`-replay`): load exported activity JSONL, group it into - units of work, and recompute what that real workload would have cost under each mode cell. - No model. Reproducible **modulo the report's `generated_at` stamp**, which must be excluded - or pinned before any byte-identical comparison (see Risks). +1. **US1 — deterministic replay** (`-replay` PLUS a fleet input): load exported activity JSONL, + group it into units of work, and recompute menu cost per mode cell and the direct-cell delta + for that real workload shape. **A fleet input is mandatory** — a frozen fleet corpus or a + live proxy — because a menu is a property of the tool definitions and the export carries no + fleet snapshot. No model. Reproducible **modulo the report's `generated_at` stamp**, which + must be excluded or pinned before any byte-identical comparison (see Risks). 2. **US2 — live agent loop**: run MCPMark against mcpproxy under each mode, capturing provider-reported token usage and per-task pass verdicts, to produce tokens per *completed* task, first-attempt success and retry rates. @@ -81,14 +83,15 @@ Three findings drive this and each becomes a design rule: exported record can be sensitive but not yet flagged. → **Rule**: exclude-by-flag is a best-effort reducer and MUST NOT be described as a guarantee, in code comments or in the published methodology. -3. **Bodies-off measurability is per cell, and splits menu cost from complete workload cost.** - Bodies-off gives a measured MENU cost for all five cells, but a measured COMPLETE WORKLOAD - cost for only the two direct cells. `code_exec` is partial — its menu is static, but the - surface also registers `retrieve_tools`, whose responses are bodies — and both - `retrieve_tools` cells need bodies-on because the mode changes the response body itself. +3. **Bodies-off yields menu costs and one cross-mode delta — never an absolute workload cost.** + Complete workload includes every consumed response, and that text is absent bodies-off, so + NO cell has a measured absolute figure. What it does give is menu cost per cell, plus the + delta between the two direct cells, whose identical call responses cancel in the comparison. The recorded byte sizes are byte *lengths*, not token counts, and support only an estimate. - → **Rule**: bodies-off is the default; the three cells that cannot yield a complete figure - from it are reported as unavailable, never as zero. + → **Rule**: bodies-off is the default and reports menu costs plus the direct-cell delta; + absolute workload figures are reported unavailable, never as zero. + → **Rule**: response tokenization happens inside the loader, so no body text ever reaches + the scoring layer (otherwise the no-content-leaves-the-loader invariant cannot hold). ## Phase 0: Research diff --git a/specs/103-token-bench/quickstart.md b/specs/103-token-bench/quickstart.md index 9d793d6eb..0ddb07e90 100644 --- a/specs/103-token-bench/quickstart.md +++ b/specs/103-token-bench/quickstart.md @@ -28,7 +28,13 @@ design; take one only if you specifically need measured *response* cost, and del ### 2. Recompute cost across the matrix ```bash -go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out bench/results +# A fleet input is MANDATORY — a menu is a property of the tool definitions, and the +# activity export carries no fleet snapshot. Use a frozen corpus: +go run ./bench/cmd/bench \ + -replay ~/replay-corpus.jsonl \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \ + -out bench/results +# ...or point at a live proxy to read today's fleet instead. ``` Produces the `replay` block in `bench/results/report.json` plus the dashboard section: per @@ -36,13 +42,16 @@ mode cell, what this workload would have cost, with the exclusion accounting bes ### 3. Know which figures you just got -- **Menu cost: `measured` for all five cells.** Computed from the fleet's tool definitions — - no recorded content needed. -- **Complete workload cost: `measured` for `direct_full` and `direct_deferred` only.** -- **`code_exec`: partial.** Its menu is static, but the surface also serves `retrieve_tools`, - whose responses are bodies — so its complete figure needs a bodies-on run. -- **`retrieve_full`, `retrieve_compact`: need bodies-on.** What their mode changes IS the - `retrieve_tools` response body. +- **Menu cost: `measured` for all five cells** — from the FLEET INPUT you supplied, not from + the recording. +- **Absolute complete-workload cost: unavailable for every cell.** It includes every consumed + response, and that text is absent bodies-off. +- **Cross-mode delta: `measured` for `direct_full` vs `direct_deferred` only** — their call + responses are identical, so they cancel in the comparison. This is the honest bodies-off + headline. +- **`retrieve_full` / `retrieve_compact` deltas: unavailable bodies-off**, because the mode + changes the `retrieve_tools` response body itself. Same for `code_exec`, whose surface also + serves `retrieve_tools`. - **Response cost generally: `estimated` or absent.** Tokenizing needs the text; a byte length supports only an estimate. - **The fleet is today's fleet.** The export carries no fleet snapshot, so this scores a @@ -58,8 +67,9 @@ pretending otherwise. ### 5. Verify determinism ```bash -go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/run-a -go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/run-b +CORPUS=specs/083-discovery-profiler/datasets/corpus_v2.tools.json +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-a +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-b # generated_at is a wall-clock stamp and differs between runs by design. diff <(jq 'del(.generated_at)' /tmp/run-a/report.json) \ <(jq 'del(.generated_at)' /tmp/run-b/report.json) \ @@ -72,7 +82,9 @@ The tokenizer downloads its vocabulary on first use unless a cache directory is ```bash export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" -go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out /tmp/warm # warms it, needs network once +# warms it; needs network once +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json -out /tmp/warm ``` Setting the variable only names a cache; it does not fill one. A first run still fetches the diff --git a/specs/103-token-bench/research.md b/specs/103-token-bench/research.md index cf8236209..7aa650fe5 100644 --- a/specs/103-token-bench/research.md +++ b/specs/103-token-bench/research.md @@ -75,14 +75,18 @@ an *accurate* cost basis without bodies. They do not — they are byte lengths, needs the text. A second draft over-corrected, claiming tool-surface cost carries the headline for all five cells. The precise position: -- **Separate MENU cost from COMPLETE WORKLOAD cost.** Bodies-off yields a measured menu cost - for all five cells, but a measured complete-workload cost for only the two direct cells, - whose mode changes nothing but the static tools/list payload. -- **`code_exec` is PARTIAL, not measurable**: its menu is static, but the surface also - registers `retrieve_tools` (`internal/server/mcp_routing.go:670-676`) whose responses are - bodies, so its complete workload cost needs bodies-on. -- **The two `retrieve_tools` cells need bodies-on**, because what their mode changes IS the - `retrieve_tools` response body. +- **A fleet input is mandatory.** A menu is a property of the tool definitions, and the export + carries no fleet snapshot, so replay must be paired with a frozen fleet corpus or a live + proxy. A recording-only invocation computes nothing. +- **No cell yields an absolute complete-workload cost bodies-off.** Complete workload includes + every consumed response, and that text is absent. Bodies-off gives menu cost per cell plus + the cross-mode DELTA between the two direct cells, whose identical call responses cancel. +- **`code_exec` and both `retrieve_tools` cells have no bodies-off delta either**: the + code-execution surface also registers `retrieve_tools` + (`internal/server/mcp_routing.go:670-676`), and the `retrieve_tools` cells' mode changes the + response body itself. +- **Tokenization of bodies must happen inside the loader**, so no text crosses the boundary + that the privacy invariant draws. - **`retrieve_tools` response cost is recoverable with bodies on** — the handler writes the FULL response to the activity log (`internal/server/mcp.go:2061-2064`). It is unavailable only in the bodies-off configuration, not unrecoverable in general. @@ -192,8 +196,11 @@ in its own doc comment and errors rather than fabricating a verdict. Spec 103's constraint is that same rule on a new axis, so it is enforceable with machinery that already exists — the extension is a new section and a source label, not new aggregation logic. -Provider `usage` maps 1:1 onto FR-014's required split (input / output / cache-read) with no -derivation, so reading response fields is strictly simpler than proxying. +For the provider this project would pin, `usage` maps 1:1 onto FR-014's required split +(input / output / cache-read) with no derivation, so reading response fields is simpler than +proxying. **This is provider-specific and unverified until the provider is pinned** — the +mapping must be re-checked against whichever provider is chosen, and it does NOT hold for the +suite's own per-task output (see the gap below). **Gap to close at task time**: the suite's own per-task output records input / output / total / reasoning — it has **no cache-read field**. So FR-014's cache-read axis cannot come from the From 5ccd7e584272f8bdae2640b0cbc28e9fe1db9493 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:49:43 +0300 Subject: [PATCH 09/12] =?UTF-8?q?fix(specs):=20103=20plan=20=E2=80=94=20tr?= =?UTF-8?q?uncation=20flag=20never=20reaches=20the=20activity=20record=20(?= =?UTF-8?q?round=206)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One finding, and it is a genuine follow-through on round 5 rather than a new error: round 5 established that truncated retrieve_tools records must be excluded or have their truncation re-applied, but nothing in the design could IDENTIFY one. wasTruncated is computed in the retrieve_tools handler (internal/server/mcp.go:2019-2035) and only logged. emitActivityInternalToolCall (:810) has no truncation parameter, so the flag never reaches the activity record. A loader would therefore be unable to tell a truncated record from a complete one, and would silently tokenize the stored full response — overstating what the agent actually paid, which is the exact failure class FR-002 exists to prevent. The plan now names a SECOND backend change (propagate the flag onto internal tool-call activity) and records the fallback explicitly: exclude every internal retrieve_tools record from response-cost accounting and report the count. The fallback must be a deliberate decision, never a silent default. Findings per round on this artifact set: 7, 9, 8, 5, 5, 1 — converging. --- .../103-token-bench/contracts/replay-input.md | 20 ++++++++++++++----- specs/103-token-bench/plan.md | 20 ++++++++++++------- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/specs/103-token-bench/contracts/replay-input.md b/specs/103-token-bench/contracts/replay-input.md index d0a9c668c..c14512266 100644 --- a/specs/103-token-bench/contracts/replay-input.md +++ b/specs/103-token-bench/contracts/replay-input.md @@ -42,11 +42,21 @@ so they must fall to exclusion accounting rather than silently contributing zero - **Code-execution sub-calls record both byte counts as zero** (`internal/server/mcp_code_execution.go:811-818`). A truncated sub-call therefore cannot use the byte-length estimate either, and must be counted as excluded. -- **A truncated `retrieve_tools` record does not tell you what the agent PAID.** The activity - log stores the FULL pre-truncation response, while the agent consumed the truncated text. So - even bodies-on recovers the wrong quantity for such calls: they must be excluded, or the - truncation re-applied before counting. Silently tokenizing the stored full text would - OVERSTATE the agent's cost. +- **A truncated `retrieve_tools` record does not tell you what the agent PAID, and today you + cannot even identify one.** The activity log stores the FULL pre-truncation response while + the agent consumed truncated text, so tokenizing the stored text OVERSTATES the agent's cost. + Worse, the truncation is not recorded: `wasTruncated` is computed in the handler + (`internal/server/mcp.go:2019-2035`) and only logged — `emitActivityInternalToolCall` + (`:810`) has no truncation parameter, so the flag never reaches the activity record. + + **This makes a SECOND backend change necessary** for the exclusion rule to be implementable: + propagate the truncation flag onto internal tool-call activity. Without it the loader cannot + distinguish a truncated `retrieve_tools` record from a complete one and would silently + overstate. + + **Fallback if that change is not wanted**: exclude ALL internal `retrieve_tools` records from + response-cost accounting and report the exclusion count. Blunt, but honest — and it must be + an explicit decision, not a silent default. ## Privacy contract diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index 51aeee4fb..adcfd5bce 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -104,13 +104,17 @@ Five topics were resolved; the load-bearing outcomes: 2. **The matrix is 5 distinct behaviours**; the other 7 combinations of the 3-axis product are configurable but redundant and collapse onto them. The routing-mode axis is selected by endpoint URL with no restart; the serialization axes still need config. -3. **One backend change is worth making**: `request_bytes` / `response_bytes` exist on the - storage record, measured pre-truncation, but are absent from the export contract. Adding - them lets a truncated record carry an explicitly-estimated response cost instead of being - dropped. They are byte lengths, **not** token counts, so they do not make response cost - measurable. Internal `retrieve_tools` emission carries no byte counts either — but its FULL - response IS written to the activity log, so its cost is recoverable with bodies on and - unavailable only in the bodies-off configuration. +3. **Two backend changes are needed, both small.** + (a) `request_bytes` / `response_bytes` exist on the storage record, measured pre-truncation, + but are absent from the export contract. Adding them lets a truncated record carry an + explicitly-estimated response cost instead of being dropped. They are byte lengths, **not** + token counts, so they do not make response cost measurable. + (b) **Truncation is not recorded for internal tool calls.** `wasTruncated` is computed in the + `retrieve_tools` handler and only logged; `emitActivityInternalToolCall` has no truncation + parameter, so the flag never reaches the activity record. Without propagating it the + exclusion rule for truncated `retrieve_tools` records is unimplementable and the loader would + silently overstate agent cost. The fallback — excluding every internal `retrieve_tools` + record from response-cost accounting — must be an explicit decision, not a silent default. 4. **MCPMark is adopted**, SHA-pinned, with one `elif` in its single MCP factory. Its per-task `meta.json` feeds FR-010/011/012/018 directly. 5. **Token accounting stays split**: tiktoken for everything deterministic; provider `usage` @@ -178,6 +182,8 @@ bench/ internal/contracts/activity.go # EXTEND — add request_bytes / response_bytes internal/httpapi/activity.go # EXTEND — copy them in the export projection +internal/server/mcp.go # EXTEND — propagate wasTruncated to internal tool-call + # activity (or accept blanket exclusion) specs/083-discovery-profiler/contracts/report-v2.schema.json # EXTEND — declare new blocks .github/workflows/bench.yml # EXTEND — TIKTOKEN_CACHE_DIR set AND cache populated From 91e877e85954ee7069d095158c15f4ea8202639e Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:50:08 +0300 Subject: [PATCH 10/12] =?UTF-8?q?docs(specs):=20103=20research=20=E2=80=94?= =?UTF-8?q?=20record=20the=20truncation-flag=20gap=20alongside=20the=20byt?= =?UTF-8?q?e-fields=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the round-6 fix: plan.md and contracts/replay-input.md already carry the second backend change, research.md did not. --- specs/103-token-bench/research.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/specs/103-token-bench/research.md b/specs/103-token-bench/research.md index 7aa650fe5..563ca4caa 100644 --- a/specs/103-token-bench/research.md +++ b/specs/103-token-bench/research.md @@ -70,6 +70,15 @@ storage record, documented as *measured pre-truncation*, but are absent from the contract entirely. Adding them (two DTO fields, two lines in the export projection) lets a truncated record carry an explicitly-estimated response cost instead of being dropped. +**A second backend change turns out to be necessary too.** Truncation is not recorded for +internal tool calls: `wasTruncated` is computed in the `retrieve_tools` handler +(`internal/server/mcp.go:2019-2035`) and only logged, and `emitActivityInternalToolCall` +(`:810`) has no truncation parameter. So a truncated `retrieve_tools` record is +indistinguishable from a complete one, and tokenizing its stored full response would overstate +what the agent paid — the exact failure class FR-002 exists to prevent. Either propagate the +flag onto internal tool-call activity, or exclude every internal `retrieve_tools` record from +response-cost accounting as an explicit, reported decision. + **Correction applied after review, then corrected again**: an early draft claimed these give an *accurate* cost basis without bodies. They do not — they are byte lengths, and tokenizing needs the text. A second draft over-corrected, claiming tool-surface cost carries the headline From 3dfd724bc2ee4b0460231ea582444c2b37ef246f Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:51:35 +0300 Subject: [PATCH 11/12] =?UTF-8?q?fix(specs):=20103=20plan=20=E2=80=94=20re?= =?UTF-8?q?concile=20scope=20statements=20with=20the=20second=20backend=20?= =?UTF-8?q?change=20(round=207)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One finding. The round-6 fix added a second production change (propagating the already-computed wasTruncated flag onto internal tool-call activity) but three earlier statements still said the byte DTO fields were the ONLY production change and that no request-path code is touched. Both were now false. Corrected in all three places, including the Constitution Check row for Principle I: the truncation propagation IS on the retrieve_tools request path, though it passes a value the handler already holds and adds no computation or I/O. The blanket-exclusion fallback is recorded as the alternative that removes the request-path touch entirely, at the cost of coverage — an explicit trade, not a default. --- specs/103-token-bench/plan.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index adcfd5bce..26a971c05 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -62,7 +62,7 @@ and must run offline; no recorded session content may reach a report or a third | Principle | Assessment | |---|---| -| **I. Performance at Scale** | N/A to production paths. The harness adds no code to a request path. The one production-touching change (Phase 0 item 3 below) adds two integers to an export DTO. | +| **I. Performance at Scale** | Two production-touching changes, both trivial. (a) Two integers added to an export DTO — off the request path. (b) Propagating an already-computed `wasTruncated` boolean into the internal tool-call activity emit, which IS on the `retrieve_tools` request path: it passes a value the handler already has, adds no computation and no I/O. If even that is unwanted, the blanket-exclusion fallback removes it entirely at the cost of coverage. | | **II. Actor-Based Concurrency** | No new concurrency in production code. The harness is a batch tool. | | **III. Configuration-Driven** | PASS: the matrix is crossed by endpoint URL plus existing config fields, adding no benchmark-only configuration to the product. | | **IV. Security by Default** | **The binding gate for this feature.** Replay inputs are raw user traffic; the export path deliberately does not mask. Design must default to bodies-off and must never transmit recorded content off-box. See Security Design below. | @@ -213,19 +213,26 @@ mode constants and proxy catalogs (`bench/tokens.go`), full rendering rendering (`bench/arms/directdeferred.go`) and MCP transport (`bench/mcpcaller.go`). It must reach them through passed structural interfaces, never by re-deriving a serialization. -The only production-code change is two additive DTO fields plus their projection — deliberately -minimal, and justified because without them a truncated record must be dropped rather than -estimated. +The production-code changes are two, both deliberately minimal: the additive DTO fields plus +their export projection, and propagation of the already-computed truncation flag onto internal +tool-call activity. The first is justified because without it a truncated record must be +dropped rather than estimated; the second because without it a truncated `retrieve_tools` +record cannot be identified at all, and the loader would silently overstate agent cost. + +If the second is rejected, the blanket-exclusion fallback stands in — at the cost of excluding +every internal `retrieve_tools` record from response-cost accounting. That trade must be made +explicitly and reported, not defaulted into. ## Phase 1: Design & Contracts Artifacts produced: [data-model.md](./data-model.md), [contracts/](./contracts/), [quickstart.md](./quickstart.md). -**Post-design constitution re-check: PASS.** The design adds no production request-path code, -introduces no new abstraction beyond one loader package that mirrors an existing sibling, and -adds no module dependency. Principle IV is satisfied by the bodies-off default rather than by -a masking layer, which is the simpler of the two available designs. +**Post-design constitution re-check: PASS.** The design touches the request path in exactly one +place — passing an already-computed boolean into an existing activity emit — and otherwise adds +no production code. It introduces no new abstraction beyond one loader package that mirrors an +existing sibling, and no module dependency. Principle IV is satisfied by the bodies-off default +rather than by a masking layer, which is the simpler of the two available designs. ## Open decisions for the operator (not blockers) From 947cb3416086dd4c70d6b0be06e4307f2e9efddc Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 09:53:32 +0300 Subject: [PATCH 12/12] =?UTF-8?q?fix(specs):=20103=20plan=20=E2=80=94=20po?= =?UTF-8?q?st-design=20re-check=20contradicted=20its=20own=20scope=20line?= =?UTF-8?q?=20(round=208)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One wording finding: the post-design constitution re-check said the design 'otherwise adds no production code' while the paragraph above it documents the export DTO fields and their projection, which are production code. Restated as touching no other REQUEST-PATH code, with the off-request-path footprint named explicitly. --- specs/103-token-bench/plan.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/103-token-bench/plan.md b/specs/103-token-bench/plan.md index 26a971c05..ad10a0d0b 100644 --- a/specs/103-token-bench/plan.md +++ b/specs/103-token-bench/plan.md @@ -229,8 +229,9 @@ Artifacts produced: [data-model.md](./data-model.md), [contracts/](./contracts/) [quickstart.md](./quickstart.md). **Post-design constitution re-check: PASS.** The design touches the request path in exactly one -place — passing an already-computed boolean into an existing activity emit — and otherwise adds -no production code. It introduces no new abstraction beyond one loader package that mirrors an +place — passing an already-computed boolean into an existing activity emit — and touches no +other request-path code. Its remaining production footprint is the additive export DTO fields +and their projection, both off the request path. It introduces no new abstraction beyond one loader package that mirrors an existing sibling, and no module dependency. Principle IV is satisfied by the bodies-off default rather than by a masking layer, which is the simpler of the two available designs.