Skip to content

Commit 1041e4e

Browse files
authored
docs(specs): 103-token-bench tasks — 55 tasks, US1 shippable alone (#1139)
* docs(specs): 103-token-bench tasks — 55 tasks, US1 shippable alone Task breakdown from the reviewed spec and plan. Tests are INCLUDED rather than optional: constitution Principle V requires them before implementation. 55 tasks across 8 phases: Setup (3), Foundational (11), US1 (14), US2 (13), US4 (4), US3 (5), US5 (1), Polish (4). 22 marked parallelizable. MVP is Phase 1 + Phase 2 + US1: real-workload menu costs and the direct-cell cross-mode delta with ZERO model spend. That alone replaces frozen-corpus numbers with numbers from real fleets, and it does not depend on the pinned-model decision that gates US2. Sequencing notes carried from the design: - Phase 1 fixes two reproducibility defects BEFORE anything depends on them: the tokenizer cache must be populated, not merely named, and the SC-011 no-committed-reports gate cannot live in bench.yml, which triggers only on v* tags and workflow_dispatch and so can never block a PR. - Phase 2 carries both production changes (export byte fields; truncation-flag propagation), each preceded by its failing test, plus the additive report blocks and the 5-cell matrix. - US1 and US2 are independent once Phase 2 lands; US4 needs US1's fleet plumbing; US3 needs both P1 stories to have produced figures; US5 needs US3, because publishing before reproducibility is the failure this feature exists to avoid. Five standing warnings are restated at the end of the file, each traceable to a defect the review found: never let a truncated record contribute silently; never sum across accounting sources; never publish a percentage without its fleet shape; never present a bodies-off figure as an absolute complete-workload cost; and a recording-only run is an error, not a degraded run. ROADMAP.md regenerated — the new tasks.md adds 103-token-bench to the per-spec progress table at 0/55. * fix(specs): 103 tasks — close four coverage gaps found in review Ten findings from the cross-model review of the task list, all verified against the spec and contracts. Four were genuine COVERAGE gaps, not wording. 1. FR-020's baseline arm was never implemented. T044 configured the full fleet in mcpproxy, but nothing ran the same agent over the same tasks with all tools loaded directly, BYPASSING mcpproxy — the denominator every published percentage is measured against. The deterministic `baseline` renderer used by the offline arms is a different thing and does not substitute. Added T038. 2. SC-010 had no execution task. Patching the suite, configuring the fleet and choosing tasks were all present; actually RUNNING it across two mode cells and recording comparable results was not. Added T045, which also carries the k>=4 requirement and the credential-free starting scope. 3. FR-023's cost-versus-outcome view was uncovered. Emitting the agent_loop block is not the same as rendering cost against completion outcome, which is what lets a reader see which modes are worth their savings. Added T042. 4. FR-016's capability conditions (batching, stored scripts, validate-before-dispatch as binary conditions over applicable cells) were neither tested nor explicitly implemented. Added T013 and folded them into T014. Ordering and fidelity fixes: 5. TDD ordering was reversed for both backend changes — the tests said "before" the implementation but were listed after it. Swapped: T004/T005 now precede T006/T007. 6. The privacy posture was documented but never implemented or tested. Bodies-off as the DEFAULT, bodies-on as an explicit opt-in with a printed warning, and refusing an input path inside the working tree are now T025, asserted by T018. Deletion instructions are T031. 7. `accounting_source` was added as a field but only ever populated for agent_loop. A field that exists without values does not satisfy the contract, so replay (T027) and payload_decomposition (T049) now populate it, and T011 asserts every emitted block carries one. 8. Several `[P]` markers were wrong: T019-T021 share bench/replay_test.go, T032-T036 share bench/agentloop_test.go, T046-T047 share bench/payloaddecomp_test.go. Parallel means DIFFERENT FILES, so those markers are removed and the parallel-opportunities section is corrected — it had claimed T015-T020 were "distinct test files" when three were not. 9. T005 cited internal/server/mcp_activity_test.go, which does not exist; it now explicitly creates internal/server/mcp_activity_truncation_test.go. 10. T052's "retain raw records" conflicted with the never-committed rule. It now specifies run-local retention under the gitignored results directory, with report references degrading to "records not retained" rather than dangling. 55 tasks -> 59. ROADMAP.md regenerated. * fix(specs): 103 tasks — restore FR-013, add FR-004 labelling, tests before impl (round 2) Four findings, all genuine. One was a regression I introduced in the round-1 fix. 1. FR-013 had NO implementation task. The previous revision's US2 rewrite, which added the FR-020 baseline arm, silently dropped the task that makes measured retry and success rates supersede the literature-derived armRetryRates defaults — the plan requires it explicitly. Restored as a test plus an implementation task in bench/session.go, including the per-row provenance field on session-cost rows. Without it, an unknown arm's defaulted 0.0 stays indistinguishable from a measured 0.0 under one section badge. 2. FR-004 had no implementation or test. Replay output must be labelled a COUNTERFACTUAL over recorded traffic rather than observed agent behaviour — the Replay Boundary is the spec's most load-bearing constraint, and nothing enforced it. Added the labelling task and an assertion that no replay figure may be emitted without the marker. 3. TDD ordering was still wrong in two places. The report/schema tests followed their implementation, and the US3 report behaviours (not-independently- reproducible marking, partial-run blocking) had no preceding tests. Both now lead with a failing test. 4. One same-file [P] conflict survived: the tokenizer-divergence task and the reproduction-procedure task both write bench/README.md. Marker removed and the parallel-opportunities section corrected. Also replaced every intra-file task-ID cross-reference with a descriptive phrase, so renumbering can no longer silently invalidate a dependency note — which is how finding 1 slipped through in the first place. 59 tasks -> 64. ROADMAP.md regenerated. * fix(specs): 103 tasks — FR-004 test before implementation (round 3) One finding: the FR-004 pair I added in round 2 put the implementation before its assertion, inverting the test-first order the constitution requires. Swapped, so the failing assertion in bench/replay_test.go now precedes the labelling change in bench/replay.go. Round 3 confirmed everything else: all FRs and SCs covered, no [P] marker on tasks sharing a file, and no brittle task-ID cross-references remain.
1 parent 286885a commit 1041e4e

2 files changed

Lines changed: 398 additions & 0 deletions

File tree

‎ROADMAP.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -910,3 +910,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—`
910910
| [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — |
911911
| [101-tpa-db](./specs/101-tpa-db/) | — | — |
912912
| [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) |
913+
| [103-token-bench](./specs/103-token-bench/) | `drafted` | 0/64 (0%) |

0 commit comments

Comments
 (0)