Repository navigation
Commit 4e91517
authored
feat(bench): complete spec 103 token-bench — live agent loop, reproducibility, publication draft (#1153)
* feat(bench): complete spec 103 — live agent loop, reproducibility, write-up (60/64)
US2, US3 and the US5 draft. Four units built in parallel, then an adversarial
pass found five defects and three vacuous tests; all are fixed here.
US2 — the live agent loop (bench/agentloop.go). Fully unit-testable with no
suite, no model and no spend: the driver is parameterised over function types
the way flipgate.go is, so every rule that decides what gets PUBLISHED is
exercised by fakes. Classification implements the spec's binding Definitions and
sees only an attempt sequence — no cell, no endpoint — so provably the same rule
applies to the baseline arm and the proxy arms. The FR-020 baseline is enforced
with two sentinels: a missing baseline is an error (a saving measured against
nothing is fabricated, not smaller) and a "baseline" cell carrying an mcpproxy
endpoint is a sixth proxy cell wearing the denominator's name.
US3 — reproducibility. Private-recording provenance, partial-run blocking,
run-local record retention that degrades to "records not retained" rather than
dangling, and the reproduction procedure in the README including that setting
TIKTOKEN_CACHE_DIR only NAMES a cache while a first run still downloads.
DEFECTS FROM THE ADVERSARIAL PASS:
1. The regression rule fabricated a claim. cellHasCompletionFigure walked raw
records while the completion rate it guarded came from an aggregate that
drops a run WHOLESALE if any unit is partial. The two disagreed, so a cell
where nothing was counted published regression:true and a "100.0 percentage
points below baseline" reason — in the same row that said nothing was
measured, with the JSON and the HTML disagreeing about it. Now gated on the
aggregate, and on Withheld.
2. runs<4 was not enforced where it mattered. SavingVsBaseline — documented as
the only sanctioned way to turn two cells into a percentage — never read
Runs, so a caller could publish a 90% saving off a SINGLE run. It now refuses
below MinHeadlineRuns, and the withheld check moved BEFORE the regression
check so an unmeasured cell is never described as a regression.
3. A cell that measured nothing rendered 0.0% first-attempt success, 0 retries
and ±0.0% spread, badged "measured". Only the two cost columns were inside
the withheld guard. All derived columns now render em-dashes, and a
single-run spread renders "spread undefined" rather than ±0.0%.
4. THE STRUCTURAL ONE: US2/US3 were libraries with NO production callers.
cmd/bench never set ReportV2.AgentLoop, .RunStatus or .Inputs, so every
replay report silently read as independently reproducible by omission.
ReplayReport now stamps PrivateRecordingInputs — an outsider cannot obtain
the operator's recording, and saying so is the difference between an honest
limitation and an implied guarantee.
5. Three vacuous dashboard assertions, each proven to pass with the behaviour
deleted: they matched STATIC TEMPLATE PROSE elsewhere on the page — the
legend sentence contains "regression", the footnote contains "runs", and the
page-wide legend supplies both provenance badges. Scoped to the row that must
carry the mark via a new rowFor helper, and mutation-verified.
Also: .pre-commit-config.yaml excludes .patch/.diff from trailing-whitespace.
A unified diff encodes meaning in leading whitespace — a blank context line IS a
lone space — and the hook silently rewrote 4 context lines of
bench/mcpmark/mcpproxy_arm.patch, producing a patch git apply would reject.
T049 and T055 are NOT ticked and are not done. T049 requires running a paid
suite against third-party services; T055 requires an API key for the
token-counting endpoint. Both are built up to that boundary and blocked only on
spend and credentials.
Gates: go build ./... and -tags server clean; go test ./bench/... green; -race
green; golangci-lint v2 0 issues; serveredition tests green; make swagger
diff-clean; git ls-files bench/results empty.
* fix(bench): enforce the never-sum and regression rules where they are bypassable (T064)
Four findings from the opencode gpt-5.6-sol review, all verified and fixed. Two
were High and both were the same shape: an invariant stated in one place and
unenforced in the place that actually produces published numbers.
1. THE NEVER-SUM RULE WAS NOT ENFORCED ON THE LIVE PATH. AggregateTokenFigures
was exercised by tests but no production path called it, while aggregateCell
added token integers whose origin nothing checked and AgentLoopBlockFor then
stamped the caller's provider onto the result — so a tokenizer-derived figure
mixed into a provider run produced a plausible hybrid wearing a provider
label. Two reviewers found this independently; the first pass raised it as a
caveat and I did not act on it, which is why it survived to a second.
UnitRecord now carries its own AccountingSource and aggregateCell refuses to
add a record that disagrees with the aggregate. A cell whose records mixed
sources is WITHHELD outright rather than published as a partial total, since
a partial total presented as the cell's cost understates it.
2. SavingVsBaseline TRUSTED A MUTABLE FLAG. It is documented as the only
sanctioned way to turn two cells into a percentage, but it read cell.Regression
rather than deriving the regression from the completion rates sitting on both
cells — so a block unmarshalled from a report.json or assembled by hand, with
Regression:false and a 50-point completion deficit, returned a saving. It now
derives the verdict, and the threshold it uses is serialised on the block so a
figure recomputed from a saved report uses the number that run applied rather
than whatever the default has since become.
3. Unmeasured figures serialised as measured zeros. FirstAttemptSuccessPct and
SpreadPct are now POINTERS: with no determinate intents there is no rate, and
with fewer than two per-run figures the spread is undefined. A serialised 0
rendered "0.0%" and "±0.0%" — a claim of perfect consistency from a single
run, which is the opposite of what FR-021 asks the field to convey.
4. Per-row session provenance existed but never reached the report. ReportV2
still carried []SessionCostEstimate, so the row type that distinguishes a
measured 0.0 retry rate from a defaulted one could not serialise or render.
The envelope, the live path and the dashboard table now use SessionCostRow,
and the table shows the retry-rate source and run count per row instead of
asserting "these are estimates" in a footnote.
Both High fixes are mutation-verified: disabling the mixed-source withholding or
the derived regression fails its test, and both pass reverted.
One test was comparing pointer ADDRESSES after the nullable change and would
have silently always differed; it now compares values.
T063 and T064 ticked. Gates: go build ./... and -tags server clean; go test
./bench/... green; -race green; golangci-lint v2 0 issues; all three schema
validation tests PASS.
* fix(ci): warm the tiktoken vocabulary before the parallel test sweep
Unit Tests (windows-latest) failed on TestBaseline_CountToolWithSchemaParity in
bench/arms with:
load tiktoken encoding "cl100k_base": failed to rename temporary file:
... Access is denied.
Not a generic flake, and not a regression from this branch — it is the
reproducibility hazard spec 103 documents, biting CI. tiktoken-go downloads the
BPE on first use and finishes with an atomic rename into a shared cache dir.
`go test ./...` runs packages in parallel, so several test binaries race to
download the same file; on Windows the loser fails its whole package on the
rename. The test that fails is therefore whichever package lost the race, which
is why it lands on files nobody touched.
Setting TIKTOKEN_CACHE_DIR alone does NOT fix this — it names a cache, and an
empty one still races. Populating it first does, because every later tokenizer
construction is a cache hit that renames nothing.
Adds bench/cmd/warmtiktoken and a step that runs it before both test steps, with
TIKTOKEN_CACHE_DIR set for the sweep. The command doubles as the SC-004 offline
prerequisite: an outsider reproducing the deterministic figures needs the
vocabulary present, and "run this" is easier to follow than a paragraph
explaining that the env var is necessary but not sufficient.
The ubuntu failure in the same run was TestFastPathDoesNotOvertakeAQueue
(internal/upstream/limiter) — a separate, documented, load-sensitive flake that
this branch does not touch and that clears on rerun.1 parent 607a80e commit 4e91517
22 files changed
Lines changed: 5530 additions & 60 deletions
File tree
- .github/workflows
- bench
- cmd/warmtiktoken
- mcpmark
- specs/103-token-bench
- findings
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
123 | 140 | | |
124 | 141 | | |
125 | 142 | | |
126 | 143 | | |
| 144 | + | |
| 145 | + | |
127 | 146 | | |
128 | 147 | | |
129 | 148 | | |
130 | 149 | | |
| 150 | + | |
| 151 | + | |
131 | 152 | | |
132 | 153 | | |
133 | 154 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
10 | 16 | | |
11 | 17 | | |
12 | 18 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
772 | 772 | | |
773 | 773 | | |
774 | 774 | | |
775 | | - | |
| 775 | + | |
776 | 776 | | |
777 | 777 | | |
778 | 778 | | |
| |||
914 | 914 | | |
915 | 915 | | |
916 | 916 | | |
917 | | - | |
| 917 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
465 | 465 | | |
466 | 466 | | |
467 | 467 | | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
468 | 561 | | |
469 | 562 | | |
470 | 563 | | |
471 | 564 | | |
472 | 565 | | |
473 | | - | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
474 | 570 | | |
475 | 571 | | |
476 | 572 | | |
| |||
588 | 684 | | |
589 | 685 | | |
590 | 686 | | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
0 commit comments