Skip to content

docs(specs): 103-token-bench tasks — 55 tasks, US1 shippable alone - #1139

Merged
Dumbris merged 4 commits into
mainfrom
103-token-bench-tasks
Aug 31, 2026
Merged

Dumbris merged 4 commits into
mainfrom
103-token-bench-tasks

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member

What

/speckit.tasks output for spec 103, following the spec and plan merged in #1137.

55 tasks across 8 phases — Setup (3), Foundational (11), US1 (14), US2 (13), US4 (4), US3 (5), US5 (1), Polish (4). 22 parallelizable.

Tests are included, not optional: constitution Principle V requires them before implementation, so each production change is preceded by its failing test.

MVP

Phase 1 + Phase 2 + US1. That delivers real-workload menu costs and the direct_full vs direct_deferred cross-mode delta with zero model spend, replacing frozen-corpus numbers with numbers from real fleets. It does not depend on the pinned-model decision that gates US2.

Sequencing that came out of the review

Phase 1 fixes two reproducibility defects before anything depends on them. The tokenizer cache must be populated, not merely named — setting TIKTOKEN_CACHE_DIR alone still downloads on first run, which defeats SC-004 offline. And the SC-011 no-committed-reports gate cannot live in bench.yml, which triggers only on v* tags and workflow_dispatch and so could never block a PR.

Phase 2 carries both production changes, each preceded by a failing test:

  • export byte fields (internal/contracts/activity.go, internal/httpapi/activity.go)
  • truncation-flag propagation (internal/server/mcp.go) — without it a truncated retrieve_tools record cannot be identified and the loader would silently overstate agent cost

Plus the additive report blocks (no report_version bump) and the 5-cell mode matrix.

Dependencies: US1 and US2 are independent once Phase 2 lands. US4 needs US1's fleet plumbing. US3 needs both P1 stories to have produced figures. US5 needs US3 — publishing before reproducibility is the exact failure this feature exists to avoid.

Five standing warnings

Restated at the end of the file, each traceable to a defect the 9-round review found:

  1. Never let a truncated record contribute silently — it understates cost in the project's favour.
  2. Never sum across accounting sources; withhold with a stated reason.
  3. Never publish a percentage without its fleet shape.
  4. Never present a bodies-off figure as an absolute complete-workload cost.
  5. A recording-only run is an error, not a degraded run — without a fleet input there is no menu to cost.

Note

ROADMAP.md regenerated: the new tasks.md adds 103-token-bench to the per-spec progress table at 0/55.

Task breakdown from the reviewed spec and plan. Tests are INCLUDED rather than
optional: constitution Principle V requires them before implementation.

55 tasks across 8 phases: Setup (3), Foundational (11), US1 (14), US2 (13),
US4 (4), US3 (5), US5 (1), Polish (4). 22 marked parallelizable.

MVP is Phase 1 + Phase 2 + US1: real-workload menu costs and the direct-cell
cross-mode delta with ZERO model spend. That alone replaces frozen-corpus
numbers with numbers from real fleets, and it does not depend on the pinned-model
decision that gates US2.

Sequencing notes carried from the design:
- Phase 1 fixes two reproducibility defects BEFORE anything depends on them: the
  tokenizer cache must be populated, not merely named, and the SC-011
  no-committed-reports gate cannot live in bench.yml, which triggers only on v*
  tags and workflow_dispatch and so can never block a PR.
- Phase 2 carries both production changes (export byte fields; truncation-flag
  propagation), each preceded by its failing test, plus the additive report
  blocks and the 5-cell matrix.
- US1 and US2 are independent once Phase 2 lands; US4 needs US1's fleet plumbing;
  US3 needs both P1 stories to have produced figures; US5 needs US3, because
  publishing before reproducibility is the failure this feature exists to avoid.

Five standing warnings are restated at the end of the file, each traceable to a
defect the review found: never let a truncated record contribute silently; never
sum across accounting sources; never publish a percentage without its fleet
shape; never present a bodies-off figure as an absolute complete-workload cost;
and a recording-only run is an error, not a degraded run.

ROADMAP.md regenerated — the new tasks.md adds 103-token-bench to the per-spec
progress table at 0/55.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: b60fdb8
Status: ✅  Deploy successful!
Preview URL: https://8bb5c8ed.mcpproxy-docs.pages.dev
Branch Preview URL: https://103-token-bench-tasks.mcpproxy-docs.pages.dev

View logs

Ten findings from the cross-model review of the task list, all verified against
the spec and contracts. Four were genuine COVERAGE gaps, not wording.

1. FR-020's baseline arm was never implemented. T044 configured the full fleet in
   mcpproxy, but nothing ran the same agent over the same tasks with all tools
   loaded directly, BYPASSING mcpproxy — the denominator every published
   percentage is measured against. The deterministic `baseline` renderer used by
   the offline arms is a different thing and does not substitute. Added T038.

2. SC-010 had no execution task. Patching the suite, configuring the fleet and
   choosing tasks were all present; actually RUNNING it across two mode cells and
   recording comparable results was not. Added T045, which also carries the k>=4
   requirement and the credential-free starting scope.

3. FR-023's cost-versus-outcome view was uncovered. Emitting the agent_loop block
   is not the same as rendering cost against completion outcome, which is what
   lets a reader see which modes are worth their savings. Added T042.

4. FR-016's capability conditions (batching, stored scripts,
   validate-before-dispatch as binary conditions over applicable cells) were
   neither tested nor explicitly implemented. Added T013 and folded them into
   T014.

Ordering and fidelity fixes:

5. TDD ordering was reversed for both backend changes — the tests said "before"
   the implementation but were listed after it. Swapped: T004/T005 now precede
   T006/T007.

6. The privacy posture was documented but never implemented or tested. Bodies-off
   as the DEFAULT, bodies-on as an explicit opt-in with a printed warning, and
   refusing an input path inside the working tree are now T025, asserted by T018.
   Deletion instructions are T031.

7. `accounting_source` was added as a field but only ever populated for
   agent_loop. A field that exists without values does not satisfy the contract,
   so replay (T027) and payload_decomposition (T049) now populate it, and T011
   asserts every emitted block carries one.

8. Several `[P]` markers were wrong: T019-T021 share bench/replay_test.go,
   T032-T036 share bench/agentloop_test.go, T046-T047 share
   bench/payloaddecomp_test.go. Parallel means DIFFERENT FILES, so those markers
   are removed and the parallel-opportunities section is corrected — it had
   claimed T015-T020 were "distinct test files" when three were not.

9. T005 cited internal/server/mcp_activity_test.go, which does not exist; it now
   explicitly creates internal/server/mcp_activity_truncation_test.go.

10. T052's "retain raw records" conflicted with the never-committed rule. It now
    specifies run-local retention under the gitignored results directory, with
    report references degrading to "records not retained" rather than dangling.

55 tasks -> 59. ROADMAP.md regenerated.
…efore impl (round 2)

Four findings, all genuine. One was a regression I introduced in the round-1 fix.

1. FR-013 had NO implementation task. The previous revision's US2 rewrite, which
   added the FR-020 baseline arm, silently dropped the task that makes measured
   retry and success rates supersede the literature-derived armRetryRates
   defaults — the plan requires it explicitly. Restored as a test plus an
   implementation task in bench/session.go, including the per-row provenance
   field on session-cost rows. Without it, an unknown arm's defaulted 0.0 stays
   indistinguishable from a measured 0.0 under one section badge.

2. FR-004 had no implementation or test. Replay output must be labelled a
   COUNTERFACTUAL over recorded traffic rather than observed agent behaviour —
   the Replay Boundary is the spec's most load-bearing constraint, and nothing
   enforced it. Added the labelling task and an assertion that no replay figure
   may be emitted without the marker.

3. TDD ordering was still wrong in two places. The report/schema tests followed
   their implementation, and the US3 report behaviours (not-independently-
   reproducible marking, partial-run blocking) had no preceding tests. Both now
   lead with a failing test.

4. One same-file [P] conflict survived: the tokenizer-divergence task and the
   reproduction-procedure task both write bench/README.md. Marker removed and the
   parallel-opportunities section corrected.

Also replaced every intra-file task-ID cross-reference with a descriptive phrase,
so renumbering can no longer silently invalidate a dependency note — which is how
finding 1 slipped through in the first place.

59 tasks -> 64. ROADMAP.md regenerated.
One finding: the FR-004 pair I added in round 2 put the implementation before its
assertion, inverting the test-first order the constitution requires. Swapped, so
the failing assertion in bench/replay_test.go now precedes the labelling change
in bench/replay.go.

Round 3 confirmed everything else: all FRs and SCs covered, no [P] marker on
tasks sharing a file, and no brittle task-ID cross-references remain.
@Dumbris

Dumbris commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

Task review complete — 4 rounds, 15 findings, READY TO IMPLEMENT

Reviewed with opencode gpt-5.6-sol. Findings per round: 10 → 4 → 1 → 0.

55 → 64 tasks. The growth is the point: four of round 1's findings were coverage gaps — work the plan requires that no task covered.

Coverage gaps found

Gap Why it mattered
FR-020 baseline arm never implemented Nothing ran the same agent over the same tasks with all tools loaded directly, bypassing mcpproxy — the denominator every published percentage is measured against. The deterministic baseline renderer used by the offline arms is a different thing and doesn't substitute.
SC-010 had no execution task Patching the suite, configuring the fleet and choosing tasks were all present; actually running it across two cells was not.
FR-023 cost-versus-outcome view Emitting a data block isn't rendering cost against completion — which is the whole point of not rewarding a false saving.
FR-016 capability conditions Batching, stored scripts and validate-before-dispatch were neither tested nor implemented.

A regression the review caught in my own fix

Round 1's fix rewrote US2 to add the baseline arm and, in doing so, silently dropped the FR-013 task — the one making measured retry rates supersede the literature-derived defaults.

Root cause: tasks cross-referenced each other by ID (Same file as T032), and renumbering invalidated those notes with no signal. Beyond restoring the task, every intra-file task-ID reference is now a descriptive phrase, so renumbering can't silently break a dependency note again.

Ordering and hygiene

  • TDD ordering was inverted in five places across rounds — both backend changes, the report/schema tests, the US3 report behaviours, and the FR-004 pair. All now lead with a failing test, as constitution Principle V requires.
  • Four [P] markers sat on tasks sharing a file (bench/replay_test.go, bench/agentloop_test.go, bench/payloaddecomp_test.go, bench/README.md). [P] means different files; the parallel-opportunities section had also claimed six test tasks were "distinct test files" when three were not.
  • FR-004 had no implementation or test. Replay output must be labelled a counterfactual over recorded traffic — the Replay Boundary is the spec's most load-bearing constraint, and nothing enforced it.

Final: 64 tasks, 8 phases, 8 parallelizable, every FR and SC covered, no brittle ID references.

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: 103-token-bench-tasks

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (26 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (21 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 33370964926 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Dumbris
Dumbris merged commit 1041e4e into main Aug 31, 2026
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants