Repository navigation
Commit 1041e4e
authored
docs(specs): 103-token-bench tasks — 55 tasks, US1 shippable alone (#1139)
* docs(specs): 103-token-bench tasks — 55 tasks, US1 shippable alone
Task breakdown from the reviewed spec and plan. Tests are INCLUDED rather than
optional: constitution Principle V requires them before implementation.
55 tasks across 8 phases: Setup (3), Foundational (11), US1 (14), US2 (13),
US4 (4), US3 (5), US5 (1), Polish (4). 22 marked parallelizable.
MVP is Phase 1 + Phase 2 + US1: real-workload menu costs and the direct-cell
cross-mode delta with ZERO model spend. That alone replaces frozen-corpus
numbers with numbers from real fleets, and it does not depend on the pinned-model
decision that gates US2.
Sequencing notes carried from the design:
- Phase 1 fixes two reproducibility defects BEFORE anything depends on them: the
tokenizer cache must be populated, not merely named, and the SC-011
no-committed-reports gate cannot live in bench.yml, which triggers only on v*
tags and workflow_dispatch and so can never block a PR.
- Phase 2 carries both production changes (export byte fields; truncation-flag
propagation), each preceded by its failing test, plus the additive report
blocks and the 5-cell matrix.
- US1 and US2 are independent once Phase 2 lands; US4 needs US1's fleet plumbing;
US3 needs both P1 stories to have produced figures; US5 needs US3, because
publishing before reproducibility is the failure this feature exists to avoid.
Five standing warnings are restated at the end of the file, each traceable to a
defect the review found: never let a truncated record contribute silently; never
sum across accounting sources; never publish a percentage without its fleet
shape; never present a bodies-off figure as an absolute complete-workload cost;
and a recording-only run is an error, not a degraded run.
ROADMAP.md regenerated — the new tasks.md adds 103-token-bench to the per-spec
progress table at 0/55.
* fix(specs): 103 tasks — close four coverage gaps found in review
Ten findings from the cross-model review of the task list, all verified against
the spec and contracts. Four were genuine COVERAGE gaps, not wording.
1. FR-020's baseline arm was never implemented. T044 configured the full fleet in
mcpproxy, but nothing ran the same agent over the same tasks with all tools
loaded directly, BYPASSING mcpproxy — the denominator every published
percentage is measured against. The deterministic `baseline` renderer used by
the offline arms is a different thing and does not substitute. Added T038.
2. SC-010 had no execution task. Patching the suite, configuring the fleet and
choosing tasks were all present; actually RUNNING it across two mode cells and
recording comparable results was not. Added T045, which also carries the k>=4
requirement and the credential-free starting scope.
3. FR-023's cost-versus-outcome view was uncovered. Emitting the agent_loop block
is not the same as rendering cost against completion outcome, which is what
lets a reader see which modes are worth their savings. Added T042.
4. FR-016's capability conditions (batching, stored scripts,
validate-before-dispatch as binary conditions over applicable cells) were
neither tested nor explicitly implemented. Added T013 and folded them into
T014.
Ordering and fidelity fixes:
5. TDD ordering was reversed for both backend changes — the tests said "before"
the implementation but were listed after it. Swapped: T004/T005 now precede
T006/T007.
6. The privacy posture was documented but never implemented or tested. Bodies-off
as the DEFAULT, bodies-on as an explicit opt-in with a printed warning, and
refusing an input path inside the working tree are now T025, asserted by T018.
Deletion instructions are T031.
7. `accounting_source` was added as a field but only ever populated for
agent_loop. A field that exists without values does not satisfy the contract,
so replay (T027) and payload_decomposition (T049) now populate it, and T011
asserts every emitted block carries one.
8. Several `[P]` markers were wrong: T019-T021 share bench/replay_test.go,
T032-T036 share bench/agentloop_test.go, T046-T047 share
bench/payloaddecomp_test.go. Parallel means DIFFERENT FILES, so those markers
are removed and the parallel-opportunities section is corrected — it had
claimed T015-T020 were "distinct test files" when three were not.
9. T005 cited internal/server/mcp_activity_test.go, which does not exist; it now
explicitly creates internal/server/mcp_activity_truncation_test.go.
10. T052's "retain raw records" conflicted with the never-committed rule. It now
specifies run-local retention under the gitignored results directory, with
report references degrading to "records not retained" rather than dangling.
55 tasks -> 59. ROADMAP.md regenerated.
* fix(specs): 103 tasks — restore FR-013, add FR-004 labelling, tests before impl (round 2)
Four findings, all genuine. One was a regression I introduced in the round-1 fix.
1. FR-013 had NO implementation task. The previous revision's US2 rewrite, which
added the FR-020 baseline arm, silently dropped the task that makes measured
retry and success rates supersede the literature-derived armRetryRates
defaults — the plan requires it explicitly. Restored as a test plus an
implementation task in bench/session.go, including the per-row provenance
field on session-cost rows. Without it, an unknown arm's defaulted 0.0 stays
indistinguishable from a measured 0.0 under one section badge.
2. FR-004 had no implementation or test. Replay output must be labelled a
COUNTERFACTUAL over recorded traffic rather than observed agent behaviour —
the Replay Boundary is the spec's most load-bearing constraint, and nothing
enforced it. Added the labelling task and an assertion that no replay figure
may be emitted without the marker.
3. TDD ordering was still wrong in two places. The report/schema tests followed
their implementation, and the US3 report behaviours (not-independently-
reproducible marking, partial-run blocking) had no preceding tests. Both now
lead with a failing test.
4. One same-file [P] conflict survived: the tokenizer-divergence task and the
reproduction-procedure task both write bench/README.md. Marker removed and the
parallel-opportunities section corrected.
Also replaced every intra-file task-ID cross-reference with a descriptive phrase,
so renumbering can no longer silently invalidate a dependency note — which is how
finding 1 slipped through in the first place.
59 tasks -> 64. ROADMAP.md regenerated.
* fix(specs): 103 tasks — FR-004 test before implementation (round 3)
One finding: the FR-004 pair I added in round 2 put the implementation before its
assertion, inverting the test-first order the constitution requires. Swapped, so
the failing assertion in bench/replay_test.go now precedes the labelling change
in bench/replay.go.
Round 3 confirmed everything else: all FRs and SCs covered, no [P] marker on
tasks sharing a file, and no brittle task-ID cross-references remain.1 parent 286885a commit 1041e4e
2 files changed
Lines changed: 398 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
910 | 910 | | |
911 | 911 | | |
912 | 912 | | |
| 913 | + | |
0 commit comments