From 71a632dffe5efe218b91a6d344af7f1c45e26841 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 10:52:12 +0300 Subject: [PATCH 1/4] =?UTF-8?q?docs(specs):=20103-token-bench=20tasks=20?= =?UTF-8?q?=E2=80=94=2055=20tasks,=20US1=20shippable=20alone?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task breakdown from the reviewed spec and plan. Tests are INCLUDED rather than optional: constitution Principle V requires them before implementation. 55 tasks across 8 phases: Setup (3), Foundational (11), US1 (14), US2 (13), US4 (4), US3 (5), US5 (1), Polish (4). 22 marked parallelizable. MVP is Phase 1 + Phase 2 + US1: real-workload menu costs and the direct-cell cross-mode delta with ZERO model spend. That alone replaces frozen-corpus numbers with numbers from real fleets, and it does not depend on the pinned-model decision that gates US2. Sequencing notes carried from the design: - Phase 1 fixes two reproducibility defects BEFORE anything depends on them: the tokenizer cache must be populated, not merely named, and the SC-011 no-committed-reports gate cannot live in bench.yml, which triggers only on v* tags and workflow_dispatch and so can never block a PR. - Phase 2 carries both production changes (export byte fields; truncation-flag propagation), each preceded by its failing test, plus the additive report blocks and the 5-cell matrix. - US1 and US2 are independent once Phase 2 lands; US4 needs US1's fleet plumbing; US3 needs both P1 stories to have produced figures; US5 needs US3, because publishing before reproducibility is the failure this feature exists to avoid. Five standing warnings are restated at the end of the file, each traceable to a defect the review found: never let a truncated record contribute silently; never sum across accounting sources; never publish a percentage without its fleet shape; never present a bodies-off figure as an absolute complete-workload cost; and a recording-only run is an error, not a degraded run. ROADMAP.md regenerated — the new tasks.md adds 103-token-bench to the per-spec progress table at 0/55. --- ROADMAP.md | 1 + specs/103-token-bench/tasks.md | 335 +++++++++++++++++++++++++++++++++ 2 files changed, 336 insertions(+) create mode 100644 specs/103-token-bench/tasks.md diff --git a/ROADMAP.md b/ROADMAP.md index a3cb75f58..49c022b09 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -910,3 +910,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | | [101-tpa-db](./specs/101-tpa-db/) | — | — | | [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) | +| [103-token-bench](./specs/103-token-bench/) | `drafted` | 0/55 (0%) | diff --git a/specs/103-token-bench/tasks.md b/specs/103-token-bench/tasks.md new file mode 100644 index 000000000..b46e1c25d --- /dev/null +++ b/specs/103-token-bench/tasks.md @@ -0,0 +1,335 @@ +# Tasks: Token-efficiency benchmark — measured savings, published results + +**Input**: Design documents from `/specs/103-token-bench/` +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, contracts/ + +**Tests**: INCLUDED. The project constitution (Principle V) requires tests before implementation +for all features, so test tasks are not optional here. + +**Organization**: Grouped by user story. US1 alone is a shippable MVP. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: parallelizable (different files, no dependency on an incomplete task) +- **[Story]**: US1–US5, mapping to spec.md's user stories + +## Path Conventions + +Single Go project. The feature lives in the existing `bench/` tree, with two small production +changes in `internal/`. Paths below are repo-relative and exact. + +--- + +## Phase 1: Setup + +**Purpose**: Make the deterministic half reproducible before anything depends on it. + +- [ ] T001 Set `TIKTOKEN_CACHE_DIR` in `.github/workflows/bench.yml` AND add a cache + restore/save step so the vocabulary is present offline — setting the variable alone only + names a cache, it does not fill one (research.md "Cross-cutting: reproducibility hazards") +- [ ] T002 [P] Add a PR-triggered required workflow step asserting no report is tracked: + `test -z "$(git ls-files bench/results)"`. It must NOT live in `.github/workflows/bench.yml`, + which triggers only on `v*` tags and `workflow_dispatch` and so cannot block a PR (SC-011) +- [ ] T003 [P] Document the replay + agent-loop entry points in `bench/README.md`, including the + mandatory fleet input and the bodies-off/bodies-on distinction (Principle VI) + +--- + +## Phase 2: Foundational (blocking prerequisites) + +**Purpose**: Contract and report groundwork every story depends on. **No user story can start +until this phase completes.** + +**Production changes (both small, both required before US1 can be honest):** + +- [ ] T004 Add `RequestBytes` / `ResponseBytes` to the export DTO in + `internal/contracts/activity.go` and copy them in `storageToContractActivityForExport` + in `internal/httpapi/activity.go`. They exist on the storage record as pre-truncation + measurements but never reach the export (contracts/replay-input.md) +- [ ] T005 Write a failing test asserting the export projection carries both byte fields, in + `internal/httpapi/activity_test.go` — before T004 (TDD, Principle V) +- [ ] T006 Propagate the already-computed `wasTruncated` into the internal tool-call activity + emit: extend `emitActivityInternalToolCall` in `internal/server/mcp.go:810` and pass the + value the `retrieve_tools` handler already holds at `internal/server/mcp.go:2019-2035`. + **Without this a truncated `retrieve_tools` record cannot be identified and the loader + would silently overstate agent cost.** If this is rejected, record the blanket-exclusion + fallback decision in `bench/README.md` instead — it must be explicit, never a default +- [ ] T007 Write a failing test asserting a truncated `retrieve_tools` call produces an activity + record flagged truncated, in `internal/server/mcp_activity_test.go` — before T006 + +**Report envelope (additive, no `report_version` bump):** + +- [ ] T008 [P] Add a `provenance` field to every new row type and constrain it to + `measured|computed|estimated` in `bench/reportv2.go`. Section-level provenance cannot + express FR-013, which lets measured and estimated figures coexist in one block +- [ ] T009 [P] Add a per-block `accounting_source` field in `bench/reportv2.go`. **Do NOT narrow + the existing document-level `Tokenizer` field** — that would be a meaning change requiring + a version bump (contracts/report-v2-additions.md) +- [ ] T010 Declare the new optional blocks (`replay`, `agent_loop`, `payload_decomposition`) in + `specs/083-discovery-profiler/contracts/report-v2.schema.json`. The schema has no + `additionalProperties: false`, so an undeclared block would validate silently — declare + them anyway, the schema file is the reviewed contract +- [ ] T011 Extend `bench/reportv2_test.go` to assert the new blocks validate against the schema + +**Mode matrix:** + +- [ ] T012 Write failing tests for the matrix in `bench/modematrix_test.go`: exactly 5 distinct + cells; the 7 redundant combinations each produce a skip row naming the cell it collapses + onto; `code_execution` + `enable_code_execution:false` yields reason `degenerate` +- [ ] T013 Implement `bench/modematrix.go` per `contracts/mode-matrix.md`. **Compose existing + pieces through structural interfaces — do not re-derive any serialization.** Reuse mode + constants and proxy catalogs (`bench/tokens.go`), full rendering + (`bench/arms/baseline.go`), compact measurement (`bench/flipgate.go`), deferred-direct + rendering (`bench/arms/directdeferred.go`), MCP transport (`bench/mcpcaller.go`). Reuse + the existing skip shape (`ArmResult.Skipped`/`SkipReason`, `SkippedArmResult`) +- [ ] T014 Map a cell to its endpoint URL. The routing-mode axis needs no config change — all + three routing-mode servers are mounted at startup (`internal/server/server.go:2514-2536`). + The two serialization axes need config but both hot-reload, so the matrix crosses on one + long-lived instance with a config apply between serialization cells + +**Checkpoint**: Contracts, report envelope and matrix ready — user stories can begin. + +--- + +## Phase 3: User Story 1 — Recompute real workload cost (Priority: P1) 🎯 MVP + +**Goal**: Given exported activity plus a fleet input, report per-cell menu cost and the +direct-cell cross-mode delta for a real workload shape. No model, deterministic. + +**Independent Test**: Supply an export and a frozen corpus; confirm per-cell menu cost and the +`direct_full` vs `direct_deferred` delta appear, each badged `measured`, with an exclusion +report beside them — and that two runs are identical modulo `generated_at`. + +### Tests for User Story 1 ⚠️ (write first, confirm they fail) + +- [ ] T015 [P] [US1] Loader tests in `bench/replaycorpus/load_test.go`: JSONL decoding; CSV + rejected; grouping by `work_session_id`; `parent_id` ↔ `request_id` join for + code-execution sub-calls +- [ ] T016 [P] [US1] Usability-flag tests in `bench/replaycorpus/flags_test.go`: truncated, + bodies-missing, sensitive and unreplayable are each detected, and every exclusion is + counted — **assert a truncated record never contributes silently** (FR-002, SC-008) +- [ ] T017 [P] [US1] Privacy tests in `bench/replaycorpus/privacy_test.go`: no body text + escapes the package; tokenization happens inside the loader and only counts cross the + boundary (FR-006, SC-009) +- [ ] T018 [P] [US1] Test that a recording-only invocation (no fleet input) is a hard error, + not a degraded run, in `bench/replay_test.go` +- [ ] T019 [P] [US1] Test that no cell reports an absolute complete-workload cost bodies-off, + and that only the `direct_full` vs `direct_deferred` delta is produced, in + `bench/replay_test.go` +- [ ] T020 [P] [US1] Determinism test in `bench/replay_test.go`: two runs over the same inputs + are byte-identical after `generated_at` is excluded (SC-002) + +### Implementation for User Story 1 + +- [ ] T021 [P] [US1] Create `bench/replaycorpus/load.go` — decode `contracts.ActivityRecord` + JSONL from `mcpproxy activity export --format json`. **This package MUST import nothing + from `bench`**, or `bench/replay.go` cannot import it without a cycle (`bench/corpusio` + imports `bench`, which is why mirroring it would fail) +- [ ] T022 [P] [US1] Create `bench/replaycorpus/group.go` — group by `work_session_id`, join + code-execution sub-calls via `parent_id` ↔ `request_id` so they are neither + double-counted nor orphaned +- [ ] T023 [US1] Create `bench/replaycorpus/flags.go` — compute usability flags once at load. + Handle the two byte-coverage gaps: internal `retrieve_tools` records carry no byte counts, + and code-execution sub-calls emit both as zero + (`internal/server/mcp_code_execution.go:811-818`); both fall to exclusion accounting +- [ ] T024 [US1] In `bench/replaycorpus/tokenize.go`, tokenize inside the loader for bodies-on runs and emit counts only, so no text + crosses the boundary. Exclude (or re-apply truncation to) truncated `retrieve_tools` + records — the log stores the FULL pre-truncation response while the agent consumed + truncated text, so tokenizing it as-is would OVERSTATE cost +- [ ] T025 [US1] Create `bench/replay.go` — per-cell menu cost from the fleet input, plus the + direct-cell delta. Reuse `arms.CanonicalToolText` / `CountToolWithSchema` for the tool + surface and the `bench/respcost.go` span-partition technique for responses, so components + sum exactly +- [ ] T026 [US1] Add the `-replay` flag plus a mandatory fleet input to `bench/cmd/bench/main.go` + as a top-level branch beside the profiler check, emitting a `replay` block into `ReportV2` + (not an `OfflineSection` — replay crosses the matrix, it is not one more corpus) +- [ ] T027 [US1] Exclude or pin `generated_at` for replay reports in `bench/replay.go` so SC-002's + determinism check is meaningful +- [ ] T028 [US1] Render the `replay` block in the dashboard template in `bench/report.go`, showing the exclusion report and + stating that figures are scored against **today's** fleet, not the fleet as recorded + +**Checkpoint**: US1 is independently shippable and delivers real-workload numbers with no model spend. + +--- + +## Phase 4: User Story 2 — Measure whether a mode helps the agent succeed (Priority: P1) + +**Goal**: Tokens per *completed* task, completion rate, first-attempt success and retry rates +per cell, from a live agent loop under a pinned model. + +**Independent Test**: Run a fixed task set under at least two cells; confirm all four figures +appear per cell, badged `measured`, averaged over k≥4 runs with spread. + +**⚠️ Blocked on an operator decision**: pinned model + spend ceiling. US1 does not depend on this. + +### Tests for User Story 2 ⚠️ + +- [ ] T029 [P] [US2] Classification tests in `bench/agentloop_test.go` for the binding + Definitions: first-attempt success, corrective vs infrastructure retry, unit of work, + task completion. **The same rule must apply to the baseline arm and the proxy arms**, or + the comparison is biased toward whichever carries richer error signal +- [ ] T030 [P] [US2] In `bench/agentloop_test.go`, test that `completion: no-signal` excludes a record + from completion-dependent figures rather than counting it as success or failure +- [ ] T031 [P] [US2] In `bench/agentloop_test.go`, test that a model-dependent figure with `runs < 4` is + refused as a headline (FR-021) +- [ ] T032 [P] [US2] In `bench/agentloop_test.go`, test that a mode with lower token cost but lower + completion is flagged a regression by the completion threshold and is NOT reported as a saving (SC-007) — drive it + with a deliberately degraded mode +- [ ] T033 [P] [US2] Test that a cross-accounting-source aggregate is WITHHELD with a stated + reason, never computed, mirroring the existing `AuthoritativeHeadline`/`withholdHeadline` + pattern in `bench/live_report.go` + +### Implementation for User Story 2 + +- [ ] T034 [US2] Create `bench/agentloop.go` — driver parameterised over a function type so the + arithmetic is unit-testable with no suite running, following the `bench/flipgate.go` + precedent +- [ ] T035 [US2] Ingest the suite's per-task `meta.json` (`execution_result.success`, + `token_usage`, `turn_count`) and `messages.json` trajectories for retry classification +- [ ] T036 [US2] Capture provider-reported `usage` in the driver for the input/output/cache-read + split. **The suite's own output has no cache-read field**, so that axis must come from the + driver or be explicitly declared out of reach (research.md) +- [ ] T037 [US2] Emit the `agent_loop` block from `bench/agentloop.go` with its own `accounting_source` + (provider + pinned model), never summed with tokenizer-sourced figures +- [ ] T038 [US2] Supersede the assumed per-arm retry defaults where a measurement exists, and + add a per-row provenance field to session-cost rows. `RetryRateForArm` returns `0.0` for + unknown arms (`bench/session.go:49-62`), indistinguishable from a measured `0.0` — the + per-row badge is what prevents a silent mix +- [ ] T039 [US2] Patch the pinned suite's single MCP factory — `src/agents/mcpmark_agent.py` + `_create_mcp_server()` in the external MCPMark clone, NOT a file in this repo — with an + env-gated branch returning an HTTP MCP server at mcpproxy's URL with the API-key header. + Pin by commit SHA and record the SHA in `bench/README.md` (FR-028) +- [ ] T040 [US2] In the benchmark's mcpproxy config (a scratch `mcp_config.json`, never committed), configure ALL suite services simultaneously while running one + service's tasks — a single server's toolset is too small a fleet to show the asymptote, + and the full fleet is also the honest FR-020 baseline +- [ ] T041 [US2] Start with the credential-free core (filesystem + postgres, 51 local tasks); record the chosen scope in `bench/README.md`. + Any service performing real writes must be run so it cannot damage real data (FR-027) + +**Checkpoint**: US1 and US2 both work independently. + +--- + +## Phase 5: User Story 4 — Explain the 29.7% shortfall (Priority: P2) + +**Goal**: Attribute tool-definition payload to names, descriptions, annotations and schemas +across fleet shapes, and confirm or correct spec 102's conclusion. + +**Independent Test**: Produce a decomposition for at least two fleet shapes with a ceiling +recomputed per corpus and an explicit `confirmed`/`corrected` verdict. + +### Tests for User Story 4 ⚠️ + +- [ ] T042 [P] [US4] Test that the four shares sum to the whole payload, in + `bench/payloaddecomp_test.go` +- [ ] T043 [P] [US4] In `bench/payloaddecomp_test.go`, test that the achievable ceiling is recomputed per + corpus and never carried forward as a constant — that carry-forward is precisely the error spec 102 made + +### Implementation for User Story 4 + +- [ ] T044 [US4] Implement the decomposition in `bench/payloaddecomp.go` over at least two fleet + shapes (the 45-tool reference corpus and the 527-tool snapshot) +- [ ] T045 [US4] Emit the `payload_decomposition` block from `bench/payloaddecomp.go` with an explicit `spec102_verdict` + (`confirmed` or `corrected`, with the delta) + +**Checkpoint**: The shortfall has an evidenced answer. + +--- + +## Phase 6: User Story 3 — An outsider reproduces the numbers (Priority: P2) + +**Goal**: Every published figure is reproducible from a documented, pinned procedure. + +**Independent Test**: Someone with no prior context follows the procedure and reproduces the +deterministic figures exactly, the model-dependent ones within the stated tolerance. + +- [ ] T046 [P] [US3] Measure the tokenizer's divergence from the pinned model with the provider's + token-counting endpoint (no inference spend) and record it in `bench/README.md` as + FR-022's numeric tolerance. **The repo currently states ~60% while the provider's guidance says ~15–20%** — + neither is sourced, and they differ threefold +- [ ] T047 [US3] In `bench/reportv2.go`, mark figures derived from private recorded sessions as NOT + independently reproducible; they must never be the sole support for a published claim (FR-030) +- [ ] T048 [US3] In `bench/replay.go` and `bench/agentloop.go`, retain raw per-run records under + `bench/results/` and reference them from the report so a headline traces to its inputs, without embedding session contents (FR-029) +- [ ] T049 [US3] In `bench/reportv2.go`, mark partial or interrupted runs as partial and block them from + publication (FR-032) +- [ ] T050 [US3] Write the reproduction procedure into `bench/README.md`, including warming the + tokenizer cache and supplying a fleet input + +**Checkpoint**: The numbers survive outside scrutiny. + +--- + +## Phase 7: User Story 5 — Publish (Priority: P3) + +**Goal**: Results reach developers deciding whether to adopt. + +- [ ] T051 [US5] Draft the write-up for `mcpproxy.app/blog` (separate repository): measured + figures, the fleet shapes they hold for, known limitations, the reproduction procedure, + and **plainly where savings do not materialise** (FR-031, SC-012). Extend the existing + 2026-03-19 "BM25 vs Embeddings vs Lua" post rather than duplicating it + +--- + +## Phase 8: Polish & Cross-Cutting + +- [ ] T052 [P] Compare the Web-UI/status savings figure (`internal/server/tokens/savings.go`, surfaced via + `internal/runtime/runtime.go`) against the bench headline over one live fleet. They are independently implemented over the same data and have never been + compared; the spec's own Assumptions say a contradiction is a finding to investigate +- [ ] T053 [P] Verify no generated report became tracked: `git ls-files bench/results` is empty +- [ ] T054 Full gates: `go build ./cmd/mcpproxy` and `-tags server`; + `go test -race -count=1 ./internal/... ./bench/...`; + `/opt/homebrew/bin/golangci-lint run --config .github/.golangci.yml ./...`; + `./scripts/test-api-e2e.sh`; `make swagger` diff-clean +- [ ] T055 Cross-model review of the full diff (opencode `gpt-5.6-sol`) staged into `.review-tmp/`, + ≤10 rounds per the standing cap; verify each finding against the tree before fixing + +--- + +## Dependencies + +```text +Phase 1 (Setup) ──► Phase 2 (Foundational) ──┬──► Phase 3 US1 (P1) ──► Phase 5 US4 (P2) + │ └──► Phase 6 US3 (P2) + └──► Phase 4 US2 (P1) ──► Phase 6 US3 (P2) + │ + └──► Phase 7 US5 (P3) + │ + Phase 8 Polish +``` + +- **US1 and US2 are independent of each other** and can proceed in parallel once Phase 2 lands. +- **US4 depends on US1** only for the fleet-input plumbing. +- **US3 depends on both P1 stories** having produced figures worth reproducing. +- **US5 depends on US3** — publishing before reproducibility is what this feature exists to avoid. + +## Parallel opportunities + +- **Phase 1**: T002, T003 together. +- **Phase 2**: T008 and T009 together (both `bench/reportv2.go`, so coordinate); T005/T007 are + independent test-first tasks. +- **US1**: T015–T020 all in parallel (distinct test files); then T021 and T022 in parallel. +- **US2**: T029–T033 all in parallel. +- **US4**: T042 and T043 in parallel. +- **Polish**: T052 and T053 in parallel. + +## Implementation strategy + +**MVP = Phase 1 + Phase 2 + Phase 3 (US1).** That yields real-workload menu costs and the +direct-cell delta with **zero model spend**, and it is independently useful: it replaces the +frozen-corpus numbers with numbers from real fleets. + +**Increment 2 = US2**, gated on the pinned-model decision. This is where the feature's actual +claim — tokens per *completed* task — becomes measurable. + +**Increment 3 = US4, then US3, then US5.** + +## Warnings carried from the design + +1. **Never let a truncated record contribute silently.** It understates cost *in the project's + favour*, which is the failure this whole feature exists to prevent. +2. **Never sum across accounting sources.** Withhold with a stated reason instead. +3. **Never publish a percentage without its fleet shape.** +4. **Never present a bodies-off figure as an absolute complete-workload cost.** Bodies-off + yields menu costs and one cross-mode delta. +5. **A recording-only run is an error**, not a degraded run — without a fleet input there is no + menu to cost. From 9b2f3c8378d58589f3c9c891ccec9f564f0797ee Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 10:57:07 +0300 Subject: [PATCH 2/4] =?UTF-8?q?fix(specs):=20103=20tasks=20=E2=80=94=20clo?= =?UTF-8?q?se=20four=20coverage=20gaps=20found=20in=20review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ten findings from the cross-model review of the task list, all verified against the spec and contracts. Four were genuine COVERAGE gaps, not wording. 1. FR-020's baseline arm was never implemented. T044 configured the full fleet in mcpproxy, but nothing ran the same agent over the same tasks with all tools loaded directly, BYPASSING mcpproxy — the denominator every published percentage is measured against. The deterministic `baseline` renderer used by the offline arms is a different thing and does not substitute. Added T038. 2. SC-010 had no execution task. Patching the suite, configuring the fleet and choosing tasks were all present; actually RUNNING it across two mode cells and recording comparable results was not. Added T045, which also carries the k>=4 requirement and the credential-free starting scope. 3. FR-023's cost-versus-outcome view was uncovered. Emitting the agent_loop block is not the same as rendering cost against completion outcome, which is what lets a reader see which modes are worth their savings. Added T042. 4. FR-016's capability conditions (batching, stored scripts, validate-before-dispatch as binary conditions over applicable cells) were neither tested nor explicitly implemented. Added T013 and folded them into T014. Ordering and fidelity fixes: 5. TDD ordering was reversed for both backend changes — the tests said "before" the implementation but were listed after it. Swapped: T004/T005 now precede T006/T007. 6. The privacy posture was documented but never implemented or tested. Bodies-off as the DEFAULT, bodies-on as an explicit opt-in with a printed warning, and refusing an input path inside the working tree are now T025, asserted by T018. Deletion instructions are T031. 7. `accounting_source` was added as a field but only ever populated for agent_loop. A field that exists without values does not satisfy the contract, so replay (T027) and payload_decomposition (T049) now populate it, and T011 asserts every emitted block carries one. 8. Several `[P]` markers were wrong: T019-T021 share bench/replay_test.go, T032-T036 share bench/agentloop_test.go, T046-T047 share bench/payloaddecomp_test.go. Parallel means DIFFERENT FILES, so those markers are removed and the parallel-opportunities section is corrected — it had claimed T015-T020 were "distinct test files" when three were not. 9. T005 cited internal/server/mcp_activity_test.go, which does not exist; it now explicitly creates internal/server/mcp_activity_truncation_test.go. 10. T052's "retain raw records" conflicted with the never-committed rule. It now specifies run-local retention under the gitignored results directory, with report references degrading to "records not retained" rather than dangling. 55 tasks -> 59. ROADMAP.md regenerated. --- ROADMAP.md | 2 +- specs/103-token-bench/tasks.md | 262 +++++++++++++++++++-------------- 2 files changed, 152 insertions(+), 112 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 49c022b09..3e4d53f2a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -910,4 +910,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | | [101-tpa-db](./specs/101-tpa-db/) | — | — | | [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) | -| [103-token-bench](./specs/103-token-bench/) | `drafted` | 0/55 (0%) | +| [103-token-bench](./specs/103-token-bench/) | `drafted` | 0/59 (0%) | diff --git a/specs/103-token-bench/tasks.md b/specs/103-token-bench/tasks.md index b46e1c25d..5be14bd85 100644 --- a/specs/103-token-bench/tasks.md +++ b/specs/103-token-bench/tasks.md @@ -40,52 +40,61 @@ changes in `internal/`. Paths below are repo-relative and exact. **Purpose**: Contract and report groundwork every story depends on. **No user story can start until this phase completes.** -**Production changes (both small, both required before US1 can be honest):** - -- [ ] T004 Add `RequestBytes` / `ResponseBytes` to the export DTO in +**Production changes (both small, both required before US1 can be honest). Tests first:** + +- [ ] T004 Write a FAILING test asserting the export projection carries both byte fields, in + `internal/httpapi/activity_test.go` (TDD, Principle V — must precede T006) +- [ ] T005 Write a FAILING test asserting a truncated `retrieve_tools` call produces an activity + record flagged truncated. Create `internal/server/mcp_activity_truncation_test.go` — no + such file exists today (must precede T007) +- [ ] T006 Add `RequestBytes` / `ResponseBytes` to the export DTO in `internal/contracts/activity.go` and copy them in `storageToContractActivityForExport` in `internal/httpapi/activity.go`. They exist on the storage record as pre-truncation - measurements but never reach the export (contracts/replay-input.md) -- [ ] T005 Write a failing test asserting the export projection carries both byte fields, in - `internal/httpapi/activity_test.go` — before T004 (TDD, Principle V) -- [ ] T006 Propagate the already-computed `wasTruncated` into the internal tool-call activity + measurements but never reach the export (contracts/replay-input.md). Turns T004 green +- [ ] T007 Propagate the already-computed `wasTruncated` into the internal tool-call activity emit: extend `emitActivityInternalToolCall` in `internal/server/mcp.go:810` and pass the value the `retrieve_tools` handler already holds at `internal/server/mcp.go:2019-2035`. **Without this a truncated `retrieve_tools` record cannot be identified and the loader - would silently overstate agent cost.** If this is rejected, record the blanket-exclusion - fallback decision in `bench/README.md` instead — it must be explicit, never a default -- [ ] T007 Write a failing test asserting a truncated `retrieve_tools` call produces an activity - record flagged truncated, in `internal/server/mcp_activity_test.go` — before T006 + would silently overstate agent cost.** If rejected, record the blanket-exclusion fallback + decision in `bench/README.md` — it must be explicit, never a default. Turns T005 green **Report envelope (additive, no `report_version` bump):** -- [ ] T008 [P] Add a `provenance` field to every new row type and constrain it to +- [ ] T008 Add a `provenance` field to every new row type and constrain it to `measured|computed|estimated` in `bench/reportv2.go`. Section-level provenance cannot express FR-013, which lets measured and estimated figures coexist in one block -- [ ] T009 [P] Add a per-block `accounting_source` field in `bench/reportv2.go`. **Do NOT narrow +- [ ] T009 Add a per-block `accounting_source` field in `bench/reportv2.go`. **Do NOT narrow the existing document-level `Tokenizer` field** — that would be a meaning change requiring - a version bump (contracts/report-v2-additions.md) + a version bump (contracts/report-v2-additions.md). Same file as T008, so not parallel - [ ] T010 Declare the new optional blocks (`replay`, `agent_loop`, `payload_decomposition`) in `specs/083-discovery-profiler/contracts/report-v2.schema.json`. The schema has no `additionalProperties: false`, so an undeclared block would validate silently — declare them anyway, the schema file is the reviewed contract - [ ] T011 Extend `bench/reportv2_test.go` to assert the new blocks validate against the schema + AND that every emitted block carries a populated `accounting_source` — a field that exists + but is never set does not satisfy the contract **Mode matrix:** -- [ ] T012 Write failing tests for the matrix in `bench/modematrix_test.go`: exactly 5 distinct +- [ ] T012 Write FAILING tests for the matrix in `bench/modematrix_test.go`: exactly 5 distinct cells; the 7 redundant combinations each produce a skip row naming the cell it collapses onto; `code_execution` + `enable_code_execution:false` yields reason `degenerate` -- [ ] T013 Implement `bench/modematrix.go` per `contracts/mode-matrix.md`. **Compose existing - pieces through structural interfaces — do not re-derive any serialization.** Reuse mode - constants and proxy catalogs (`bench/tokens.go`), full rendering - (`bench/arms/baseline.go`), compact measurement (`bench/flipgate.go`), deferred-direct - rendering (`bench/arms/directdeferred.go`), MCP transport (`bench/mcpcaller.go`). Reuse - the existing skip shape (`ArmResult.Skipped`/`SkipReason`, `SkippedArmResult`) -- [ ] T014 Map a cell to its endpoint URL. The routing-mode axis needs no config change — all - three routing-mode servers are mounted at startup (`internal/server/server.go:2514-2536`). - The two serialization axes need config but both hot-reload, so the matrix crosses on one - long-lived instance with a config apply between serialization cells +- [ ] T013 Write FAILING tests in `bench/modematrix_test.go` for the FR-016 capability + conditions — batching, stored scripts and validate-before-dispatch as BINARY conditions + over the cells where each is available, with the report enumerating applicable rows. + Same file as T012, so not parallel +- [ ] T014 Implement `bench/modematrix.go` per `contracts/mode-matrix.md`, including the FR-016 + capability conditions. **Compose existing pieces through structural interfaces — do not + re-derive any serialization.** Reuse mode constants and proxy catalogs + (`bench/tokens.go`), full rendering (`bench/arms/baseline.go`), compact measurement + (`bench/flipgate.go`), deferred-direct rendering (`bench/arms/directdeferred.go`), MCP + transport (`bench/mcpcaller.go`). Reuse the existing skip shape + (`ArmResult.Skipped`/`SkipReason`, `SkippedArmResult`) +- [ ] T015 Map a cell to its endpoint URL in `bench/modematrix.go`. The routing-mode axis needs + no config change — all three routing-mode servers are mounted at startup + (`internal/server/server.go:2514-2536`). The two serialization axes need config but both + hot-reload, so the matrix crosses on one long-lived instance with a config apply between + serialization cells **Checkpoint**: Contracts, report envelope and matrix ready — user stories can begin. @@ -102,51 +111,61 @@ report beside them — and that two runs are identical modulo `generated_at`. ### Tests for User Story 1 ⚠️ (write first, confirm they fail) -- [ ] T015 [P] [US1] Loader tests in `bench/replaycorpus/load_test.go`: JSONL decoding; CSV +- [ ] T016 [P] [US1] Loader tests in `bench/replaycorpus/load_test.go`: JSONL decoding; CSV rejected; grouping by `work_session_id`; `parent_id` ↔ `request_id` join for code-execution sub-calls -- [ ] T016 [P] [US1] Usability-flag tests in `bench/replaycorpus/flags_test.go`: truncated, +- [ ] T017 [P] [US1] Usability-flag tests in `bench/replaycorpus/flags_test.go`: truncated, bodies-missing, sensitive and unreplayable are each detected, and every exclusion is counted — **assert a truncated record never contributes silently** (FR-002, SC-008) -- [ ] T017 [P] [US1] Privacy tests in `bench/replaycorpus/privacy_test.go`: no body text - escapes the package; tokenization happens inside the loader and only counts cross the - boundary (FR-006, SC-009) -- [ ] T018 [P] [US1] Test that a recording-only invocation (no fleet input) is a hard error, - not a degraded run, in `bench/replay_test.go` -- [ ] T019 [P] [US1] Test that no cell reports an absolute complete-workload cost bodies-off, - and that only the `direct_full` vs `direct_deferred` delta is produced, in - `bench/replay_test.go` -- [ ] T020 [P] [US1] Determinism test in `bench/replay_test.go`: two runs over the same inputs - are byte-identical after `generated_at` is excluded (SC-002) +- [ ] T018 [P] [US1] Privacy tests in `bench/replaycorpus/privacy_test.go`: no body text escapes + the package; tokenization happens inside the loader and only counts cross the boundary; + **bodies-off is the DEFAULT**; bodies-on requires an explicit opt-in and emits a warning + (FR-006, SC-009, contracts/replay-input.md) +- [ ] T019 [US1] In `bench/replay_test.go`: a recording-only invocation (no fleet input) is a + hard error, not a degraded run +- [ ] T020 [US1] In `bench/replay_test.go`: no cell reports an absolute complete-workload cost + bodies-off, and only the `direct_full` vs `direct_deferred` delta is produced. Same file + as T019, so not parallel +- [ ] T021 [US1] In `bench/replay_test.go`: two runs over the same inputs are byte-identical + after `generated_at` is excluded (SC-002). Same file as T019/T020, so not parallel ### Implementation for User Story 1 -- [ ] T021 [P] [US1] Create `bench/replaycorpus/load.go` — decode `contracts.ActivityRecord` +- [ ] T022 [P] [US1] Create `bench/replaycorpus/load.go` — decode `contracts.ActivityRecord` JSONL from `mcpproxy activity export --format json`. **This package MUST import nothing from `bench`**, or `bench/replay.go` cannot import it without a cycle (`bench/corpusio` imports `bench`, which is why mirroring it would fail) -- [ ] T022 [P] [US1] Create `bench/replaycorpus/group.go` — group by `work_session_id`, join +- [ ] T023 [P] [US1] Create `bench/replaycorpus/group.go` — group by `work_session_id`, join code-execution sub-calls via `parent_id` ↔ `request_id` so they are neither double-counted nor orphaned -- [ ] T023 [US1] Create `bench/replaycorpus/flags.go` — compute usability flags once at load. +- [ ] T024 [US1] Create `bench/replaycorpus/flags.go` — compute usability flags once at load. Handle the two byte-coverage gaps: internal `retrieve_tools` records carry no byte counts, and code-execution sub-calls emit both as zero (`internal/server/mcp_code_execution.go:811-818`); both fall to exclusion accounting -- [ ] T024 [US1] In `bench/replaycorpus/tokenize.go`, tokenize inside the loader for bodies-on runs and emit counts only, so no text - crosses the boundary. Exclude (or re-apply truncation to) truncated `retrieve_tools` - records — the log stores the FULL pre-truncation response while the agent consumed - truncated text, so tokenizing it as-is would OVERSTATE cost -- [ ] T025 [US1] Create `bench/replay.go` — per-cell menu cost from the fleet input, plus the +- [ ] T025 [US1] Enforce the privacy posture in `bench/replaycorpus/load.go`: bodies-off by + DEFAULT, bodies-on only via explicit opt-in that prints a warning, and refuse an input + path inside the repository working tree (replay inputs live outside it and are never + committed). Turns T018 green +- [ ] T026 [US1] In `bench/replaycorpus/tokenize.go`, tokenize inside the loader for bodies-on + runs and emit counts only, so no text crosses the boundary. Exclude (or re-apply + truncation to) truncated `retrieve_tools` records — the log stores the FULL pre-truncation + response while the agent consumed truncated text, so tokenizing it as-is would OVERSTATE + cost +- [ ] T027 [US1] Create `bench/replay.go` — per-cell menu cost from the fleet input, plus the direct-cell delta. Reuse `arms.CanonicalToolText` / `CountToolWithSchema` for the tool surface and the `bench/respcost.go` span-partition technique for responses, so components - sum exactly -- [ ] T026 [US1] Add the `-replay` flag plus a mandatory fleet input to `bench/cmd/bench/main.go` + sum exactly. Populate the `replay` block's `accounting_source` as the deterministic + tokenizer +- [ ] T028 [US1] Add the `-replay` flag plus a mandatory fleet input to `bench/cmd/bench/main.go` as a top-level branch beside the profiler check, emitting a `replay` block into `ReportV2` (not an `OfflineSection` — replay crosses the matrix, it is not one more corpus) -- [ ] T027 [US1] Exclude or pin `generated_at` for replay reports in `bench/replay.go` so SC-002's - determinism check is meaningful -- [ ] T028 [US1] Render the `replay` block in the dashboard template in `bench/report.go`, showing the exclusion report and - stating that figures are scored against **today's** fleet, not the fleet as recorded +- [ ] T029 [US1] Exclude or pin `generated_at` for replay reports in `bench/replay.go` so + SC-002's determinism check is meaningful +- [ ] T030 [US1] Render the `replay` block in the dashboard template in `bench/report.go`, + showing the exclusion report and stating that figures are scored against **today's** fleet, + not the fleet as recorded +- [ ] T031 [US1] Document in `bench/README.md` how to delete a replay input when finished + (contracts/replay-input.md privacy rule 6) **Checkpoint**: US1 is independently shippable and delivers real-workload numbers with no model spend. @@ -164,46 +183,56 @@ appear per cell, badged `measured`, averaged over k≥4 runs with spread. ### Tests for User Story 2 ⚠️ -- [ ] T029 [P] [US2] Classification tests in `bench/agentloop_test.go` for the binding - Definitions: first-attempt success, corrective vs infrastructure retry, unit of work, - task completion. **The same rule must apply to the baseline arm and the proxy arms**, or - the comparison is biased toward whichever carries richer error signal -- [ ] T030 [P] [US2] In `bench/agentloop_test.go`, test that `completion: no-signal` excludes a record - from completion-dependent figures rather than counting it as success or failure -- [ ] T031 [P] [US2] In `bench/agentloop_test.go`, test that a model-dependent figure with `runs < 4` is +- [ ] T032 [US2] Classification tests in `bench/agentloop_test.go` for the binding Definitions: + first-attempt success, corrective vs infrastructure retry, unit of work, task completion. + **The same rule must apply to the baseline arm and the proxy arms**, or the comparison is + biased toward whichever carries richer error signal +- [ ] T033 [US2] In `bench/agentloop_test.go`: `completion: no-signal` excludes a record from + completion-dependent figures rather than counting it as success or failure. Same file as + T032, so not parallel +- [ ] T034 [US2] In `bench/agentloop_test.go`: a model-dependent figure with `runs < 4` is refused as a headline (FR-021) -- [ ] T032 [P] [US2] In `bench/agentloop_test.go`, test that a mode with lower token cost but lower - completion is flagged a regression by the completion threshold and is NOT reported as a saving (SC-007) — drive it - with a deliberately degraded mode -- [ ] T033 [P] [US2] Test that a cross-accounting-source aggregate is WITHHELD with a stated - reason, never computed, mirroring the existing `AuthoritativeHeadline`/`withholdHeadline` - pattern in `bench/live_report.go` +- [ ] T035 [US2] In `bench/agentloop_test.go`: a mode with lower token cost but lower completion + is flagged a regression by the completion threshold and is NOT reported as a saving + (SC-007) — drive it with a deliberately degraded mode +- [ ] T036 [US2] In `bench/agentloop_test.go`: a cross-accounting-source aggregate is WITHHELD + with a stated reason, never computed, mirroring the existing + `AuthoritativeHeadline`/`withholdHeadline` pattern in `bench/live_report.go` ### Implementation for User Story 2 -- [ ] T034 [US2] Create `bench/agentloop.go` — driver parameterised over a function type so the +- [ ] T037 [US2] Create `bench/agentloop.go` — driver parameterised over a function type so the arithmetic is unit-testable with no suite running, following the `bench/flipgate.go` precedent -- [ ] T035 [US2] Ingest the suite's per-task `meta.json` (`execution_result.success`, +- [ ] T038 [US2] Implement the FR-020 BASELINE ARM in `bench/agentloop.go`: the same agent + running the same tasks with every upstream tool loaded directly, **bypassing mcpproxy + entirely**. This is the denominator every published percentage is measured against, and + without it no saving can be quoted. It is NOT the deterministic `baseline` renderer used + by the offline arms +- [ ] T039 [US2] Ingest the suite's per-task `meta.json` (`execution_result.success`, `token_usage`, `turn_count`) and `messages.json` trajectories for retry classification -- [ ] T036 [US2] Capture provider-reported `usage` in the driver for the input/output/cache-read +- [ ] T040 [US2] Capture provider-reported `usage` in the driver for the input/output/cache-read split. **The suite's own output has no cache-read field**, so that axis must come from the driver or be explicitly declared out of reach (research.md) -- [ ] T037 [US2] Emit the `agent_loop` block from `bench/agentloop.go` with its own `accounting_source` - (provider + pinned model), never summed with tokenizer-sourced figures -- [ ] T038 [US2] Supersede the assumed per-arm retry defaults where a measurement exists, and - add a per-row provenance field to session-cost rows. `RetryRateForArm` returns `0.0` for - unknown arms (`bench/session.go:49-62`), indistinguishable from a measured `0.0` — the - per-row badge is what prevents a silent mix -- [ ] T039 [US2] Patch the pinned suite's single MCP factory — `src/agents/mcpmark_agent.py` +- [ ] T041 [US2] Emit the `agent_loop` block from `bench/agentloop.go` with its own populated + `accounting_source` (provider + pinned model), never summed with tokenizer-sourced figures +- [ ] T042 [US2] Implement the FR-023 COST-VERSUS-OUTCOME VIEW in `bench/report.go`: cost + plotted against completion outcome per cell, so a reader sees which modes are worth their + savings rather than only which are cheapest. Completion rate must sit beside cost at equal + prominence (FR-018) +- [ ] T043 [US2] Patch the pinned suite's single MCP factory — `src/agents/mcpmark_agent.py` `_create_mcp_server()` in the external MCPMark clone, NOT a file in this repo — with an env-gated branch returning an HTTP MCP server at mcpproxy's URL with the API-key header. Pin by commit SHA and record the SHA in `bench/README.md` (FR-028) -- [ ] T040 [US2] In the benchmark's mcpproxy config (a scratch `mcp_config.json`, never committed), configure ALL suite services simultaneously while running one - service's tasks — a single server's toolset is too small a fleet to show the asymptote, - and the full fleet is also the honest FR-020 baseline -- [ ] T041 [US2] Start with the credential-free core (filesystem + postgres, 51 local tasks); record the chosen scope in `bench/README.md`. - Any service performing real writes must be run so it cannot damage real data (FR-027) +- [ ] T044 [US2] In the benchmark's mcpproxy config (a scratch `mcp_config.json`, never + committed), configure ALL suite services simultaneously while running one service's tasks + — a single server's toolset is too small a fleet to show the asymptote, and the full fleet + is also the honest FR-020 baseline +- [ ] T045 [US2] RUN the public suite against mcpproxy across at least two mode cells plus the + baseline arm, at k>=4 runs each, and record comparable results (SC-010). Start with the + credential-free core (filesystem + postgres, 51 local tasks); any service performing real + writes must be run so it cannot damage real data (FR-027). Record the executed scope and + the pinned model in `bench/README.md` **Checkpoint**: US1 and US2 both work independently. @@ -219,17 +248,18 @@ recomputed per corpus and an explicit `confirmed`/`corrected` verdict. ### Tests for User Story 4 ⚠️ -- [ ] T042 [P] [US4] Test that the four shares sum to the whole payload, in - `bench/payloaddecomp_test.go` -- [ ] T043 [P] [US4] In `bench/payloaddecomp_test.go`, test that the achievable ceiling is recomputed per - corpus and never carried forward as a constant — that carry-forward is precisely the error spec 102 made +- [ ] T046 [US4] In `bench/payloaddecomp_test.go`: the four shares sum to the whole payload +- [ ] T047 [US4] In `bench/payloaddecomp_test.go`: the achievable ceiling is recomputed per + corpus and never carried forward as a constant — that carry-forward is precisely the error + spec 102 made. Same file as T046, so not parallel ### Implementation for User Story 4 -- [ ] T044 [US4] Implement the decomposition in `bench/payloaddecomp.go` over at least two fleet +- [ ] T048 [US4] Implement the decomposition in `bench/payloaddecomp.go` over at least two fleet shapes (the 45-tool reference corpus and the 527-tool snapshot) -- [ ] T045 [US4] Emit the `payload_decomposition` block from `bench/payloaddecomp.go` with an explicit `spec102_verdict` - (`confirmed` or `corrected`, with the delta) +- [ ] T049 [US4] Emit the `payload_decomposition` block from `bench/payloaddecomp.go` with a + populated `accounting_source` and an explicit `spec102_verdict` (`confirmed` or + `corrected`, with the delta) **Checkpoint**: The shortfall has an evidenced answer. @@ -242,17 +272,21 @@ recomputed per corpus and an explicit `confirmed`/`corrected` verdict. **Independent Test**: Someone with no prior context follows the procedure and reproduces the deterministic figures exactly, the model-dependent ones within the stated tolerance. -- [ ] T046 [P] [US3] Measure the tokenizer's divergence from the pinned model with the provider's - token-counting endpoint (no inference spend) and record it in `bench/README.md` as - FR-022's numeric tolerance. **The repo currently states ~60% while the provider's guidance says ~15–20%** — - neither is sourced, and they differ threefold -- [ ] T047 [US3] In `bench/reportv2.go`, mark figures derived from private recorded sessions as NOT - independently reproducible; they must never be the sole support for a published claim (FR-030) -- [ ] T048 [US3] In `bench/replay.go` and `bench/agentloop.go`, retain raw per-run records under - `bench/results/` and reference them from the report so a headline traces to its inputs, without embedding session contents (FR-029) -- [ ] T049 [US3] In `bench/reportv2.go`, mark partial or interrupted runs as partial and block them from - publication (FR-032) -- [ ] T050 [US3] Write the reproduction procedure into `bench/README.md`, including warming the +- [ ] T050 [P] [US3] Measure the tokenizer's divergence from the pinned model with the + provider's token-counting endpoint (no inference spend) and record it in `bench/README.md` + as FR-022's numeric tolerance. **The repo currently states ~60% while the provider's + guidance says ~15–20%** — neither is sourced, and they differ threefold +- [ ] T051 [US3] In `bench/reportv2.go`, mark figures derived from private recorded sessions as + NOT independently reproducible; they must never be the sole support for a published claim + (FR-030) +- [ ] T052 [US3] In `bench/replay.go` and `bench/agentloop.go`, retain raw per-run records under + the gitignored `bench/results/` and reference them from the report by RUN-LOCAL path, with + an explicit note that they are not durable across a results cleanup. A report reference + must degrade to "records not retained" rather than dangling (FR-029, and consistent with + SC-011's never-committed rule) +- [ ] T053 [US3] In `bench/reportv2.go`, mark partial or interrupted runs as partial and block + them from publication (FR-032) +- [ ] T054 [US3] Write the reproduction procedure into `bench/README.md`, including warming the tokenizer cache and supplying a fleet input **Checkpoint**: The numbers survive outside scrutiny. @@ -263,7 +297,7 @@ deterministic figures exactly, the model-dependent ones within the stated tolera **Goal**: Results reach developers deciding whether to adopt. -- [ ] T051 [US5] Draft the write-up for `mcpproxy.app/blog` (separate repository): measured +- [ ] T055 [US5] Draft the write-up for `mcpproxy.app/blog` (separate repository): measured figures, the fleet shapes they hold for, known limitations, the reproduction procedure, and **plainly where savings do not materialise** (FR-031, SC-012). Extend the existing 2026-03-19 "BM25 vs Embeddings vs Lua" post rather than duplicating it @@ -272,16 +306,18 @@ deterministic figures exactly, the model-dependent ones within the stated tolera ## Phase 8: Polish & Cross-Cutting -- [ ] T052 [P] Compare the Web-UI/status savings figure (`internal/server/tokens/savings.go`, surfaced via - `internal/runtime/runtime.go`) against the bench headline over one live fleet. They are independently implemented over the same data and have never been - compared; the spec's own Assumptions say a contradiction is a finding to investigate -- [ ] T053 [P] Verify no generated report became tracked: `git ls-files bench/results` is empty -- [ ] T054 Full gates: `go build ./cmd/mcpproxy` and `-tags server`; +- [ ] T056 [P] Compare the Web-UI/status savings figure (`internal/server/tokens/savings.go`, + surfaced via `internal/runtime/runtime.go`) against the bench headline over one live + fleet. They are independently implemented over the same data and have never been compared; + the spec's own Assumptions say a contradiction is a finding to investigate +- [ ] T057 [P] Verify no generated report became tracked: `git ls-files bench/results` is empty +- [ ] T058 Full gates: `go build ./cmd/mcpproxy` and `-tags server`; `go test -race -count=1 ./internal/... ./bench/...`; `/opt/homebrew/bin/golangci-lint run --config .github/.golangci.yml ./...`; `./scripts/test-api-e2e.sh`; `make swagger` diff-clean -- [ ] T055 Cross-model review of the full diff (opencode `gpt-5.6-sol`) staged into `.review-tmp/`, - ≤10 rounds per the standing cap; verify each finding against the tree before fixing +- [ ] T059 Cross-model review of the full diff (opencode `gpt-5.6-sol`) staged into + `.review-tmp/`, ≤10 rounds per the standing cap; verify each finding against the tree + before fixing --- @@ -304,13 +340,17 @@ Phase 1 (Setup) ──► Phase 2 (Foundational) ──┬──► Phase 3 US1 ## Parallel opportunities +`[P]` means **different files**. Tasks editing the same file are deliberately NOT marked +parallel, even when logically independent. + - **Phase 1**: T002, T003 together. -- **Phase 2**: T008 and T009 together (both `bench/reportv2.go`, so coordinate); T005/T007 are - independent test-first tasks. -- **US1**: T015–T020 all in parallel (distinct test files); then T021 and T022 in parallel. -- **US2**: T029–T033 all in parallel. -- **US4**: T042 and T043 in parallel. -- **Polish**: T052 and T053 in parallel. +- **Phase 2**: T004 and T005 together (different test files). T008–T009 share + `bench/reportv2.go` and T012–T013 share `bench/modematrix_test.go` — both sequential. +- **US1**: T016, T017, T018 together (three distinct `bench/replaycorpus/*_test.go` files). + T019–T021 all edit `bench/replay_test.go` — sequential. Then T022 and T023 together. +- **US2**: T032–T036 all edit `bench/agentloop_test.go` — sequential. +- **US4**: T046–T047 share `bench/payloaddecomp_test.go` — sequential. +- **Polish**: T056 and T057 together. ## Implementation strategy From b1b955957f47671a02848aa8ebe81c112833aba6 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 10:59:46 +0300 Subject: [PATCH 3/4] =?UTF-8?q?fix(specs):=20103=20tasks=20=E2=80=94=20res?= =?UTF-8?q?tore=20FR-013,=20add=20FR-004=20labelling,=20tests=20before=20i?= =?UTF-8?q?mpl=20(round=202)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four findings, all genuine. One was a regression I introduced in the round-1 fix. 1. FR-013 had NO implementation task. The previous revision's US2 rewrite, which added the FR-020 baseline arm, silently dropped the task that makes measured retry and success rates supersede the literature-derived armRetryRates defaults — the plan requires it explicitly. Restored as a test plus an implementation task in bench/session.go, including the per-row provenance field on session-cost rows. Without it, an unknown arm's defaulted 0.0 stays indistinguishable from a measured 0.0 under one section badge. 2. FR-004 had no implementation or test. Replay output must be labelled a COUNTERFACTUAL over recorded traffic rather than observed agent behaviour — the Replay Boundary is the spec's most load-bearing constraint, and nothing enforced it. Added the labelling task and an assertion that no replay figure may be emitted without the marker. 3. TDD ordering was still wrong in two places. The report/schema tests followed their implementation, and the US3 report behaviours (not-independently- reproducible marking, partial-run blocking) had no preceding tests. Both now lead with a failing test. 4. One same-file [P] conflict survived: the tokenizer-divergence task and the reproduction-procedure task both write bench/README.md. Marker removed and the parallel-opportunities section corrected. Also replaced every intra-file task-ID cross-reference with a descriptive phrase, so renumbering can no longer silently invalidate a dependency note — which is how finding 1 slipped through in the first place. 59 tasks -> 64. ROADMAP.md regenerated. --- ROADMAP.md | 2 +- specs/103-token-bench/tasks.md | 141 +++++++++++++++++++-------------- 2 files changed, 82 insertions(+), 61 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 3e4d53f2a..c4f4d82fa 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -910,4 +910,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | | [101-tpa-db](./specs/101-tpa-db/) | — | — | | [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) | -| [103-token-bench](./specs/103-token-bench/) | `drafted` | 0/59 (0%) | +| [103-token-bench](./specs/103-token-bench/) | `drafted` | 0/64 (0%) | diff --git a/specs/103-token-bench/tasks.md b/specs/103-token-bench/tasks.md index 5be14bd85..4116340ee 100644 --- a/specs/103-token-bench/tasks.md +++ b/specs/103-token-bench/tasks.md @@ -43,36 +43,38 @@ until this phase completes.** **Production changes (both small, both required before US1 can be honest). Tests first:** - [ ] T004 Write a FAILING test asserting the export projection carries both byte fields, in - `internal/httpapi/activity_test.go` (TDD, Principle V — must precede T006) + `internal/httpapi/activity_test.go` (TDD, Principle V — must precede the DTO change below) - [ ] T005 Write a FAILING test asserting a truncated `retrieve_tools` call produces an activity record flagged truncated. Create `internal/server/mcp_activity_truncation_test.go` — no - such file exists today (must precede T007) + such file exists today (must precede the truncation-propagation task below) - [ ] T006 Add `RequestBytes` / `ResponseBytes` to the export DTO in `internal/contracts/activity.go` and copy them in `storageToContractActivityForExport` in `internal/httpapi/activity.go`. They exist on the storage record as pre-truncation - measurements but never reach the export (contracts/replay-input.md). Turns T004 green + measurements but never reach the export (contracts/replay-input.md). Turns the export-projection test green - [ ] T007 Propagate the already-computed `wasTruncated` into the internal tool-call activity emit: extend `emitActivityInternalToolCall` in `internal/server/mcp.go:810` and pass the value the `retrieve_tools` handler already holds at `internal/server/mcp.go:2019-2035`. **Without this a truncated `retrieve_tools` record cannot be identified and the loader would silently overstate agent cost.** If rejected, record the blanket-exclusion fallback - decision in `bench/README.md` — it must be explicit, never a default. Turns T005 green + decision in `bench/README.md` — it must be explicit, never a default. Turns the truncation test green -**Report envelope (additive, no `report_version` bump):** +**Report envelope (additive, no `report_version` bump). Tests first:** -- [ ] T008 Add a `provenance` field to every new row type and constrain it to +- [ ] T008 Write FAILING tests in `bench/reportv2_test.go` asserting: the new blocks validate + against the schema; every emitted block carries a populated `accounting_source`; and each + new row type carries a `provenance` value from the closed enum. A field that exists but is + never set does not satisfy the contract +- [ ] T009 Add a `provenance` field to every new row type and constrain it to `measured|computed|estimated` in `bench/reportv2.go`. Section-level provenance cannot express FR-013, which lets measured and estimated figures coexist in one block -- [ ] T009 Add a per-block `accounting_source` field in `bench/reportv2.go`. **Do NOT narrow +- [ ] T010 Add a per-block `accounting_source` field in `bench/reportv2.go`. **Do NOT narrow the existing document-level `Tokenizer` field** — that would be a meaning change requiring - a version bump (contracts/report-v2-additions.md). Same file as T008, so not parallel -- [ ] T010 Declare the new optional blocks (`replay`, `agent_loop`, `payload_decomposition`) in + a version bump (contracts/report-v2-additions.md). Same file as the provenance task, so + not parallel +- [ ] T011 Declare the new optional blocks (`replay`, `agent_loop`, `payload_decomposition`) in `specs/083-discovery-profiler/contracts/report-v2.schema.json`. The schema has no `additionalProperties: false`, so an undeclared block would validate silently — declare them anyway, the schema file is the reviewed contract -- [ ] T011 Extend `bench/reportv2_test.go` to assert the new blocks validate against the schema - AND that every emitted block carries a populated `accounting_source` — a field that exists - but is never set does not satisfy the contract **Mode matrix:** @@ -82,7 +84,7 @@ until this phase completes.** - [ ] T013 Write FAILING tests in `bench/modematrix_test.go` for the FR-016 capability conditions — batching, stored scripts and validate-before-dispatch as BINARY conditions over the cells where each is available, with the report enumerating applicable rows. - Same file as T012, so not parallel + Same file as the task above, so not parallel - [ ] T014 Implement `bench/modematrix.go` per `contracts/mode-matrix.md`, including the FR-016 capability conditions. **Compose existing pieces through structural interfaces — do not re-derive any serialization.** Reuse mode constants and proxy catalogs @@ -125,9 +127,9 @@ report beside them — and that two runs are identical modulo `generated_at`. hard error, not a degraded run - [ ] T020 [US1] In `bench/replay_test.go`: no cell reports an absolute complete-workload cost bodies-off, and only the `direct_full` vs `direct_deferred` delta is produced. Same file - as T019, so not parallel + as the task above, so not parallel - [ ] T021 [US1] In `bench/replay_test.go`: two runs over the same inputs are byte-identical - after `generated_at` is excluded (SC-002). Same file as T019/T020, so not parallel + after `generated_at` is excluded (SC-002). Same file as the task above, so not parallel ### Implementation for User Story 1 @@ -145,7 +147,7 @@ report beside them — and that two runs are identical modulo `generated_at`. - [ ] T025 [US1] Enforce the privacy posture in `bench/replaycorpus/load.go`: bodies-off by DEFAULT, bodies-on only via explicit opt-in that prints a warning, and refuse an input path inside the repository working tree (replay inputs live outside it and are never - committed). Turns T018 green + committed). Turns the privacy test green - [ ] T026 [US1] In `bench/replaycorpus/tokenize.go`, tokenize inside the loader for bodies-on runs and emit counts only, so no text crosses the boundary. Exclude (or re-apply truncation to) truncated `retrieve_tools` records — the log stores the FULL pre-truncation @@ -161,10 +163,15 @@ report beside them — and that two runs are identical modulo `generated_at`. (not an `OfflineSection` — replay crosses the matrix, it is not one more corpus) - [ ] T029 [US1] Exclude or pin `generated_at` for replay reports in `bench/replay.go` so SC-002's determinism check is meaningful -- [ ] T030 [US1] Render the `replay` block in the dashboard template in `bench/report.go`, - showing the exclusion report and stating that figures are scored against **today's** fleet, - not the fleet as recorded -- [ ] T031 [US1] Document in `bench/README.md` how to delete a replay input when finished +- [ ] T030 [US1] Label every replay figure as a COUNTERFACTUAL over recorded traffic, not + observed agent behaviour, in `bench/replay.go` — FR-004 forbids presenting it as observed, + and the Replay Boundary is the spec's most load-bearing constraint +- [ ] T031 [US1] Assert that labelling in `bench/replay_test.go`: no replay figure may be emitted + without the counterfactual marker +- [ ] T032 [US1] Render the `replay` block in the dashboard template in `bench/report.go`, + showing the exclusion report, the counterfactual label, and that figures are scored against + **today's** fleet, not the fleet as recorded +- [ ] T033 [US1] Document in `bench/README.md` how to delete a replay input when finished (contracts/replay-input.md privacy rule 6) **Checkpoint**: US1 is independently shippable and delivers real-workload numbers with no model spend. @@ -183,52 +190,60 @@ appear per cell, badged `measured`, averaged over k≥4 runs with spread. ### Tests for User Story 2 ⚠️ -- [ ] T032 [US2] Classification tests in `bench/agentloop_test.go` for the binding Definitions: +- [ ] T034 [US2] Classification tests in `bench/agentloop_test.go` for the binding Definitions: first-attempt success, corrective vs infrastructure retry, unit of work, task completion. **The same rule must apply to the baseline arm and the proxy arms**, or the comparison is biased toward whichever carries richer error signal -- [ ] T033 [US2] In `bench/agentloop_test.go`: `completion: no-signal` excludes a record from +- [ ] T035 [US2] In `bench/agentloop_test.go`: `completion: no-signal` excludes a record from completion-dependent figures rather than counting it as success or failure. Same file as - T032, so not parallel -- [ ] T034 [US2] In `bench/agentloop_test.go`: a model-dependent figure with `runs < 4` is + the task above, so not parallel +- [ ] T036 [US2] In `bench/agentloop_test.go`: a model-dependent figure with `runs < 4` is refused as a headline (FR-021) -- [ ] T035 [US2] In `bench/agentloop_test.go`: a mode with lower token cost but lower completion +- [ ] T037 [US2] In `bench/agentloop_test.go`: a mode with lower token cost but lower completion is flagged a regression by the completion threshold and is NOT reported as a saving (SC-007) — drive it with a deliberately degraded mode -- [ ] T036 [US2] In `bench/agentloop_test.go`: a cross-accounting-source aggregate is WITHHELD +- [ ] T038 [US2] In `bench/agentloop_test.go`: a cross-accounting-source aggregate is WITHHELD with a stated reason, never computed, mirroring the existing `AuthoritativeHeadline`/`withholdHeadline` pattern in `bench/live_report.go` ### Implementation for User Story 2 -- [ ] T037 [US2] Create `bench/agentloop.go` — driver parameterised over a function type so the +- [ ] T039 [US2] Create `bench/agentloop.go` — driver parameterised over a function type so the arithmetic is unit-testable with no suite running, following the `bench/flipgate.go` precedent -- [ ] T038 [US2] Implement the FR-020 BASELINE ARM in `bench/agentloop.go`: the same agent +- [ ] T040 [US2] Implement the FR-020 BASELINE ARM in `bench/agentloop.go`: the same agent running the same tasks with every upstream tool loaded directly, **bypassing mcpproxy entirely**. This is the denominator every published percentage is measured against, and without it no saving can be quoted. It is NOT the deterministic `baseline` renderer used by the offline arms -- [ ] T039 [US2] Ingest the suite's per-task `meta.json` (`execution_result.success`, +- [ ] T041 [US2] Ingest the suite's per-task `meta.json` (`execution_result.success`, `token_usage`, `turn_count`) and `messages.json` trajectories for retry classification -- [ ] T040 [US2] Capture provider-reported `usage` in the driver for the input/output/cache-read +- [ ] T042 [US2] Capture provider-reported `usage` in the driver for the input/output/cache-read split. **The suite's own output has no cache-read field**, so that axis must come from the driver or be explicitly declared out of reach (research.md) -- [ ] T041 [US2] Emit the `agent_loop` block from `bench/agentloop.go` with its own populated +- [ ] T043 [US2] Emit the `agent_loop` block from `bench/agentloop.go` with its own populated `accounting_source` (provider + pinned model), never summed with tokenizer-sourced figures -- [ ] T042 [US2] Implement the FR-023 COST-VERSUS-OUTCOME VIEW in `bench/report.go`: cost +- [ ] T044 [US2] Write a FAILING test in `bench/session_test.go`: where a measured retry rate + exists it supersedes the assumed default, and where none exists the row stays badged + `estimated`. `RetryRateForArm` returns `0.0` for unknown arms (`bench/session.go:49-62`), + indistinguishable from a measured `0.0` — this test is what prevents the silent mix +- [ ] T045 [US2] Implement FR-013 in `bench/session.go`: measured success and retry rates + supersede the literature-derived `armRetryRates` defaults wherever a measurement exists, + and add a per-row provenance field to session-cost rows so measured and estimated rows are + distinguishable inside one table +- [ ] T046 [US2] Implement the FR-023 COST-VERSUS-OUTCOME VIEW in `bench/report.go`: cost plotted against completion outcome per cell, so a reader sees which modes are worth their savings rather than only which are cheapest. Completion rate must sit beside cost at equal prominence (FR-018) -- [ ] T043 [US2] Patch the pinned suite's single MCP factory — `src/agents/mcpmark_agent.py` +- [ ] T047 [US2] Patch the pinned suite's single MCP factory — `src/agents/mcpmark_agent.py` `_create_mcp_server()` in the external MCPMark clone, NOT a file in this repo — with an env-gated branch returning an HTTP MCP server at mcpproxy's URL with the API-key header. Pin by commit SHA and record the SHA in `bench/README.md` (FR-028) -- [ ] T044 [US2] In the benchmark's mcpproxy config (a scratch `mcp_config.json`, never +- [ ] T048 [US2] In the benchmark's mcpproxy config (a scratch `mcp_config.json`, never committed), configure ALL suite services simultaneously while running one service's tasks — a single server's toolset is too small a fleet to show the asymptote, and the full fleet is also the honest FR-020 baseline -- [ ] T045 [US2] RUN the public suite against mcpproxy across at least two mode cells plus the +- [ ] T049 [US2] RUN the public suite against mcpproxy across at least two mode cells plus the baseline arm, at k>=4 runs each, and record comparable results (SC-010). Start with the credential-free core (filesystem + postgres, 51 local tasks); any service performing real writes must be run so it cannot damage real data (FR-027). Record the executed scope and @@ -248,16 +263,16 @@ recomputed per corpus and an explicit `confirmed`/`corrected` verdict. ### Tests for User Story 4 ⚠️ -- [ ] T046 [US4] In `bench/payloaddecomp_test.go`: the four shares sum to the whole payload -- [ ] T047 [US4] In `bench/payloaddecomp_test.go`: the achievable ceiling is recomputed per +- [ ] T050 [US4] In `bench/payloaddecomp_test.go`: the four shares sum to the whole payload +- [ ] T051 [US4] In `bench/payloaddecomp_test.go`: the achievable ceiling is recomputed per corpus and never carried forward as a constant — that carry-forward is precisely the error - spec 102 made. Same file as T046, so not parallel + spec 102 made. Same file as the task above, so not parallel ### Implementation for User Story 4 -- [ ] T048 [US4] Implement the decomposition in `bench/payloaddecomp.go` over at least two fleet +- [ ] T052 [US4] Implement the decomposition in `bench/payloaddecomp.go` over at least two fleet shapes (the 45-tool reference corpus and the 527-tool snapshot) -- [ ] T049 [US4] Emit the `payload_decomposition` block from `bench/payloaddecomp.go` with a +- [ ] T053 [US4] Emit the `payload_decomposition` block from `bench/payloaddecomp.go` with a populated `accounting_source` and an explicit `spec102_verdict` (`confirmed` or `corrected`, with the delta) @@ -272,21 +287,24 @@ recomputed per corpus and an explicit `confirmed`/`corrected` verdict. **Independent Test**: Someone with no prior context follows the procedure and reproduces the deterministic figures exactly, the model-dependent ones within the stated tolerance. -- [ ] T050 [P] [US3] Measure the tokenizer's divergence from the pinned model with the +- [ ] T054 [US3] Write FAILING tests in `bench/reportv2_test.go` for the US3 report behaviours: + a private-session-derived figure is marked not-independently-reproducible; a partial run is + marked partial and refused for publication +- [ ] T055 [US3] Measure the tokenizer's divergence from the pinned model with the provider's token-counting endpoint (no inference spend) and record it in `bench/README.md` as FR-022's numeric tolerance. **The repo currently states ~60% while the provider's guidance says ~15–20%** — neither is sourced, and they differ threefold -- [ ] T051 [US3] In `bench/reportv2.go`, mark figures derived from private recorded sessions as +- [ ] T056 [US3] In `bench/reportv2.go`, mark figures derived from private recorded sessions as NOT independently reproducible; they must never be the sole support for a published claim (FR-030) -- [ ] T052 [US3] In `bench/replay.go` and `bench/agentloop.go`, retain raw per-run records under +- [ ] T057 [US3] In `bench/replay.go` and `bench/agentloop.go`, retain raw per-run records under the gitignored `bench/results/` and reference them from the report by RUN-LOCAL path, with an explicit note that they are not durable across a results cleanup. A report reference must degrade to "records not retained" rather than dangling (FR-029, and consistent with SC-011's never-committed rule) -- [ ] T053 [US3] In `bench/reportv2.go`, mark partial or interrupted runs as partial and block +- [ ] T058 [US3] In `bench/reportv2.go`, mark partial or interrupted runs as partial and block them from publication (FR-032) -- [ ] T054 [US3] Write the reproduction procedure into `bench/README.md`, including warming the +- [ ] T059 [US3] Write the reproduction procedure into `bench/README.md`, including warming the tokenizer cache and supplying a fleet input **Checkpoint**: The numbers survive outside scrutiny. @@ -297,7 +315,7 @@ deterministic figures exactly, the model-dependent ones within the stated tolera **Goal**: Results reach developers deciding whether to adopt. -- [ ] T055 [US5] Draft the write-up for `mcpproxy.app/blog` (separate repository): measured +- [ ] T060 [US5] Draft the write-up for `mcpproxy.app/blog` (separate repository): measured figures, the fleet shapes they hold for, known limitations, the reproduction procedure, and **plainly where savings do not materialise** (FR-031, SC-012). Extend the existing 2026-03-19 "BM25 vs Embeddings vs Lua" post rather than duplicating it @@ -306,16 +324,16 @@ deterministic figures exactly, the model-dependent ones within the stated tolera ## Phase 8: Polish & Cross-Cutting -- [ ] T056 [P] Compare the Web-UI/status savings figure (`internal/server/tokens/savings.go`, +- [ ] T061 [P] Compare the Web-UI/status savings figure (`internal/server/tokens/savings.go`, surfaced via `internal/runtime/runtime.go`) against the bench headline over one live fleet. They are independently implemented over the same data and have never been compared; the spec's own Assumptions say a contradiction is a finding to investigate -- [ ] T057 [P] Verify no generated report became tracked: `git ls-files bench/results` is empty -- [ ] T058 Full gates: `go build ./cmd/mcpproxy` and `-tags server`; +- [ ] T062 [P] Verify no generated report became tracked: `git ls-files bench/results` is empty +- [ ] T063 Full gates: `go build ./cmd/mcpproxy` and `-tags server`; `go test -race -count=1 ./internal/... ./bench/...`; `/opt/homebrew/bin/golangci-lint run --config .github/.golangci.yml ./...`; `./scripts/test-api-e2e.sh`; `make swagger` diff-clean -- [ ] T059 Cross-model review of the full diff (opencode `gpt-5.6-sol`) staged into +- [ ] T064 Cross-model review of the full diff (opencode `gpt-5.6-sol`) staged into `.review-tmp/`, ≤10 rounds per the standing cap; verify each finding against the tree before fixing @@ -341,16 +359,19 @@ Phase 1 (Setup) ──► Phase 2 (Foundational) ──┬──► Phase 3 US1 ## Parallel opportunities `[P]` means **different files**. Tasks editing the same file are deliberately NOT marked -parallel, even when logically independent. - -- **Phase 1**: T002, T003 together. -- **Phase 2**: T004 and T005 together (different test files). T008–T009 share - `bench/reportv2.go` and T012–T013 share `bench/modematrix_test.go` — both sequential. -- **US1**: T016, T017, T018 together (three distinct `bench/replaycorpus/*_test.go` files). - T019–T021 all edit `bench/replay_test.go` — sequential. Then T022 and T023 together. -- **US2**: T032–T036 all edit `bench/agentloop_test.go` — sequential. -- **US4**: T046–T047 share `bench/payloaddecomp_test.go` — sequential. -- **Polish**: T056 and T057 together. +parallel, even when logically independent — several markers were removed on review for exactly +that reason. + +- **Phase 1**: the SC-011 gate and the README documentation task run together. +- **Phase 2**: the two failing backend tests run together (different test files). The two + `bench/reportv2.go` tasks and the two `bench/modematrix_test.go` tasks are sequential. +- **US1**: the three `bench/replaycorpus/*_test.go` test tasks run together; everything touching + `bench/replay_test.go` is sequential. The two loader implementation files run together. +- **US2**: everything in `bench/agentloop_test.go` is sequential. +- **US4**: both `bench/payloaddecomp_test.go` tasks are sequential. +- **US3**: nothing is parallel — the tokenizer-divergence task and the reproduction-procedure + task both write `bench/README.md`. +- **Polish**: the savings-comparison and tracked-report checks run together. ## Implementation strategy From b60fdb8a9e41dda9c6173f45f6a967e00a051e39 Mon Sep 17 00:00:00 2001 From: Algis Dumbris Date: Mon, 31 Aug 2026 11:01:23 +0300 Subject: [PATCH 4/4] =?UTF-8?q?fix(specs):=20103=20tasks=20=E2=80=94=20FR-?= =?UTF-8?q?004=20test=20before=20implementation=20(round=203)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One finding: the FR-004 pair I added in round 2 put the implementation before its assertion, inverting the test-first order the constitution requires. Swapped, so the failing assertion in bench/replay_test.go now precedes the labelling change in bench/replay.go. Round 3 confirmed everything else: all FRs and SCs covered, no [P] marker on tasks sharing a file, and no brittle task-ID cross-references remain. --- specs/103-token-bench/tasks.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/103-token-bench/tasks.md b/specs/103-token-bench/tasks.md index 4116340ee..89a9ce2ee 100644 --- a/specs/103-token-bench/tasks.md +++ b/specs/103-token-bench/tasks.md @@ -163,11 +163,12 @@ report beside them — and that two runs are identical modulo `generated_at`. (not an `OfflineSection` — replay crosses the matrix, it is not one more corpus) - [ ] T029 [US1] Exclude or pin `generated_at` for replay reports in `bench/replay.go` so SC-002's determinism check is meaningful -- [ ] T030 [US1] Label every replay figure as a COUNTERFACTUAL over recorded traffic, not +- [ ] T030 [US1] Write a FAILING assertion in `bench/replay_test.go`: no replay figure may be + emitted without the counterfactual marker (FR-004) +- [ ] T031 [US1] Label every replay figure as a COUNTERFACTUAL over recorded traffic, not observed agent behaviour, in `bench/replay.go` — FR-004 forbids presenting it as observed, - and the Replay Boundary is the spec's most load-bearing constraint -- [ ] T031 [US1] Assert that labelling in `bench/replay_test.go`: no replay figure may be emitted - without the counterfactual marker + and the Replay Boundary is the spec's most load-bearing constraint. Turns the assertion + above green - [ ] T032 [US1] Render the `replay` block in the dashboard template in `bench/report.go`, showing the exclusion report, the counterfactual label, and that figures are scored against **today's** fleet, not the fleet as recorded