Skip to content

Commit 4e91517

Browse files
authored
feat(bench): complete spec 103 token-bench — live agent loop, reproducibility, publication draft (#1153)
* feat(bench): complete spec 103 — live agent loop, reproducibility, write-up (60/64) US2, US3 and the US5 draft. Four units built in parallel, then an adversarial pass found five defects and three vacuous tests; all are fixed here. US2 — the live agent loop (bench/agentloop.go). Fully unit-testable with no suite, no model and no spend: the driver is parameterised over function types the way flipgate.go is, so every rule that decides what gets PUBLISHED is exercised by fakes. Classification implements the spec's binding Definitions and sees only an attempt sequence — no cell, no endpoint — so provably the same rule applies to the baseline arm and the proxy arms. The FR-020 baseline is enforced with two sentinels: a missing baseline is an error (a saving measured against nothing is fabricated, not smaller) and a "baseline" cell carrying an mcpproxy endpoint is a sixth proxy cell wearing the denominator's name. US3 — reproducibility. Private-recording provenance, partial-run blocking, run-local record retention that degrades to "records not retained" rather than dangling, and the reproduction procedure in the README including that setting TIKTOKEN_CACHE_DIR only NAMES a cache while a first run still downloads. DEFECTS FROM THE ADVERSARIAL PASS: 1. The regression rule fabricated a claim. cellHasCompletionFigure walked raw records while the completion rate it guarded came from an aggregate that drops a run WHOLESALE if any unit is partial. The two disagreed, so a cell where nothing was counted published regression:true and a "100.0 percentage points below baseline" reason — in the same row that said nothing was measured, with the JSON and the HTML disagreeing about it. Now gated on the aggregate, and on Withheld. 2. runs<4 was not enforced where it mattered. SavingVsBaseline — documented as the only sanctioned way to turn two cells into a percentage — never read Runs, so a caller could publish a 90% saving off a SINGLE run. It now refuses below MinHeadlineRuns, and the withheld check moved BEFORE the regression check so an unmeasured cell is never described as a regression. 3. A cell that measured nothing rendered 0.0% first-attempt success, 0 retries and ±0.0% spread, badged "measured". Only the two cost columns were inside the withheld guard. All derived columns now render em-dashes, and a single-run spread renders "spread undefined" rather than ±0.0%. 4. THE STRUCTURAL ONE: US2/US3 were libraries with NO production callers. cmd/bench never set ReportV2.AgentLoop, .RunStatus or .Inputs, so every replay report silently read as independently reproducible by omission. ReplayReport now stamps PrivateRecordingInputs — an outsider cannot obtain the operator's recording, and saying so is the difference between an honest limitation and an implied guarantee. 5. Three vacuous dashboard assertions, each proven to pass with the behaviour deleted: they matched STATIC TEMPLATE PROSE elsewhere on the page — the legend sentence contains "regression", the footnote contains "runs", and the page-wide legend supplies both provenance badges. Scoped to the row that must carry the mark via a new rowFor helper, and mutation-verified. Also: .pre-commit-config.yaml excludes .patch/.diff from trailing-whitespace. A unified diff encodes meaning in leading whitespace — a blank context line IS a lone space — and the hook silently rewrote 4 context lines of bench/mcpmark/mcpproxy_arm.patch, producing a patch git apply would reject. T049 and T055 are NOT ticked and are not done. T049 requires running a paid suite against third-party services; T055 requires an API key for the token-counting endpoint. Both are built up to that boundary and blocked only on spend and credentials. Gates: go build ./... and -tags server clean; go test ./bench/... green; -race green; golangci-lint v2 0 issues; serveredition tests green; make swagger diff-clean; git ls-files bench/results empty. * fix(bench): enforce the never-sum and regression rules where they are bypassable (T064) Four findings from the opencode gpt-5.6-sol review, all verified and fixed. Two were High and both were the same shape: an invariant stated in one place and unenforced in the place that actually produces published numbers. 1. THE NEVER-SUM RULE WAS NOT ENFORCED ON THE LIVE PATH. AggregateTokenFigures was exercised by tests but no production path called it, while aggregateCell added token integers whose origin nothing checked and AgentLoopBlockFor then stamped the caller's provider onto the result — so a tokenizer-derived figure mixed into a provider run produced a plausible hybrid wearing a provider label. Two reviewers found this independently; the first pass raised it as a caveat and I did not act on it, which is why it survived to a second. UnitRecord now carries its own AccountingSource and aggregateCell refuses to add a record that disagrees with the aggregate. A cell whose records mixed sources is WITHHELD outright rather than published as a partial total, since a partial total presented as the cell's cost understates it. 2. SavingVsBaseline TRUSTED A MUTABLE FLAG. It is documented as the only sanctioned way to turn two cells into a percentage, but it read cell.Regression rather than deriving the regression from the completion rates sitting on both cells — so a block unmarshalled from a report.json or assembled by hand, with Regression:false and a 50-point completion deficit, returned a saving. It now derives the verdict, and the threshold it uses is serialised on the block so a figure recomputed from a saved report uses the number that run applied rather than whatever the default has since become. 3. Unmeasured figures serialised as measured zeros. FirstAttemptSuccessPct and SpreadPct are now POINTERS: with no determinate intents there is no rate, and with fewer than two per-run figures the spread is undefined. A serialised 0 rendered "0.0%" and "±0.0%" — a claim of perfect consistency from a single run, which is the opposite of what FR-021 asks the field to convey. 4. Per-row session provenance existed but never reached the report. ReportV2 still carried []SessionCostEstimate, so the row type that distinguishes a measured 0.0 retry rate from a defaulted one could not serialise or render. The envelope, the live path and the dashboard table now use SessionCostRow, and the table shows the retry-rate source and run count per row instead of asserting "these are estimates" in a footnote. Both High fixes are mutation-verified: disabling the mixed-source withholding or the derived regression fails its test, and both pass reverted. One test was comparing pointer ADDRESSES after the nullable change and would have silently always differed; it now compares values. T063 and T064 ticked. Gates: go build ./... and -tags server clean; go test ./bench/... green; -race green; golangci-lint v2 0 issues; all three schema validation tests PASS. * fix(ci): warm the tiktoken vocabulary before the parallel test sweep Unit Tests (windows-latest) failed on TestBaseline_CountToolWithSchemaParity in bench/arms with: load tiktoken encoding "cl100k_base": failed to rename temporary file: ... Access is denied. Not a generic flake, and not a regression from this branch — it is the reproducibility hazard spec 103 documents, biting CI. tiktoken-go downloads the BPE on first use and finishes with an atomic rename into a shared cache dir. `go test ./...` runs packages in parallel, so several test binaries race to download the same file; on Windows the loser fails its whole package on the rename. The test that fails is therefore whichever package lost the race, which is why it lands on files nobody touched. Setting TIKTOKEN_CACHE_DIR alone does NOT fix this — it names a cache, and an empty one still races. Populating it first does, because every later tokenizer construction is a cache hit that renames nothing. Adds bench/cmd/warmtiktoken and a step that runs it before both test steps, with TIKTOKEN_CACHE_DIR set for the sweep. The command doubles as the SC-004 offline prerequisite: an outsider reproducing the deterministic figures needs the vocabulary present, and "run this" is easier to follow than a paragraph explaining that the env var is necessary but not sufficient. The ubuntu failure in the same run was TestFastPathDoesNotOvertakeAQueue (internal/upstream/limiter) — a separate, documented, load-sensitive flake that this branch does not touch and that clears on rerun.
1 parent 607a80e commit 4e91517

22 files changed

Lines changed: 5530 additions & 60 deletions

‎.github/workflows/unit-tests.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,14 +120,35 @@ jobs:
120120
New-Item -ItemType Directory -Force -Path web/frontend
121121
Copy-Item -Recurse frontend/dist web/frontend/
122122
123+
# Warm the tiktoken vocabulary ONCE, before the parallel test run.
124+
#
125+
# tiktoken-go downloads the BPE on first use into a shared temp dir and
126+
# finishes with an atomic rename. `go test ./...` runs packages in
127+
# parallel, so several test binaries race to download the same file, and
128+
# on Windows the loser hits "Access is denied" on the rename and the
129+
# package fails — observed as TestBaseline_CountToolWithSchemaParity
130+
# flaking in bench/arms with no code change anywhere near it.
131+
#
132+
# Setting TIKTOKEN_CACHE_DIR alone does NOT fix this: it only names a
133+
# cache, and an empty one still races. Populating it first does, because
134+
# every later tokenizer construction is a cache hit with no rename.
135+
- name: Warm the tiktoken vocabulary cache
136+
run: go run ./bench/cmd/warmtiktoken
137+
env:
138+
TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken
139+
123140
- name: Run unit tests (Windows)
124141
if: matrix.os == 'windows-latest'
125142
shell: pwsh
126143
run: go test -v -race -timeout 10m '-coverprofile=coverage.out' -covermode=atomic '-skip=E2E|Binary|MCPProtocol|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint' ./...
144+
env:
145+
TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken
127146

128147
- name: Run unit tests (Unix)
129148
if: matrix.os != 'windows-latest'
130149
run: go test -v -race -coverprofile=coverage.out -skip "E2E|Binary|MCPProtocol|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint" ./...
150+
env:
151+
TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken
131152

132153
- name: Generate coverage report (Windows)
133154
if: matrix.os == 'windows-latest'

‎.pre-commit-config.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,12 @@ repos:
77
rev: v5.0.0
88
hooks:
99
- id: trailing-whitespace
10+
# A unified diff encodes meaning in leading whitespace: an unchanged
11+
# context line begins with a single space, and a blank context line IS
12+
# that lone space. Stripping it turns valid patches into ones git apply
13+
# rejects — observed on bench/mcpmark/mcpproxy_arm.patch, where the hook
14+
# silently rewrote 4 context lines.
15+
exclude: '\.(patch|diff)$'
1016
- id: end-of-file-fixer
1117
- id: check-merge-conflict
1218

‎ROADMAP.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -772,7 +772,7 @@ graph LR
772772
| Web UI + macOS app UX audit | In progress | P0 | — | | |
773773
| Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | |
774774
| Action log / transparency — info at a glance | In progress | P1 | — | | |
775-
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 37/64 (58%) | [103-token-bench](./specs/103-token-bench/) | #1141 |
775+
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | #1141 |
776776
| Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | |
777777
| Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | |
778778
| Planning/docs truth automation | In progress | P2 | — | | |
@@ -914,4 +914,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—`
914914
| [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — |
915915
| [101-tpa-db](./specs/101-tpa-db/) | — | — |
916916
| [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) |
917-
| [103-token-bench](./specs/103-token-bench/) | `in-flight` | 37/64 (58%) |
917+
| [103-token-bench](./specs/103-token-bench/) | `shipped` | 62/64 (97%) |

‎bench/README.md‎

Lines changed: 111 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -465,12 +465,108 @@ offline reproduction needs the cache pre-populated — warm it once with network
465465
access, or restore a known-good copy (which is what `.github/workflows/bench.yml`
466466
does via its `actions/cache` step).
467467

468+
### Reproducing a published figure (SC-004)
469+
470+
Someone with no prior context reproduces the deterministic figures **exactly**,
471+
and the model-dependent ones within the stated numeric tolerance (FR-022 — see
472+
the tokenizer caveat under "Known limitations"), by following these steps and
473+
nothing else.
474+
475+
**1. Populate the tokenizer cache. Naming it is not filling it.**
476+
477+
```bash
478+
export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken"
479+
go run ./bench/cmd/bench \
480+
-corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
481+
-out /tmp/warm # downloads the vocabulary once — needs network
482+
ls "$TIKTOKEN_CACHE_DIR" # non-empty ⇒ every later run is genuinely offline
483+
```
484+
485+
`TIKTOKEN_CACHE_DIR` only *names* a directory. A first run with the variable set
486+
and the directory empty still reaches the network, so a machine "configured for
487+
offline" but never warmed fails at the first token count rather than at
488+
configuration time — and it fails the same way on an air-gapped reviewer's
489+
laptop as it does in a network-restricted CI job. Warm it once with network
490+
access, or restore a known-good copy (which is what `.github/workflows/bench.yml`
491+
does via its `actions/cache` step).
492+
493+
**2. Supply a fleet input. There is no default, and there is no fallback.**
494+
495+
A menu cost is a property of the tool definitions the agent was shown. A
496+
recording carries the *call shape* — sequence, tool mix, call counts — and no
497+
fleet snapshot whatsoever, so a recording-only invocation has nothing to price:
498+
499+
```bash
500+
go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out bench/results
501+
# bench: replay requires a fleet input as well as a recording: a menu is a
502+
# property of the tool definitions and the activity export carries no fleet
503+
# snapshot, so a recording on its own computes nothing — pass a frozen corpus
504+
# or a live-proxy catalog (this is an error, not a degraded run)
505+
# exit status 1
506+
```
507+
508+
That is a **hard error, not a degraded run**. The alternative — quietly scoring
509+
against an empty or assumed fleet — would produce a plausible-looking number
510+
with no fleet shape behind it, which is exactly the failure every percentage in
511+
this report carries a `fleet_shape` to prevent. Pass the frozen corpus for a
512+
reproducible figure, or a live `-proxy` for today's fleet.
513+
514+
**3. Run the pinned command, twice.** The deterministic half is byte-identical
515+
across runs once the wall-clock stamp is removed — the `diff` recipe under
516+
"Verify determinism" above is the check, and it is what SC-002 means. If two
517+
runs on the same machine differ by anything other than `generated_at`, stop:
518+
nothing downstream is worth comparing yet.
519+
520+
**4. Read the three honesty marks before quoting anything.** They are fields in
521+
`report.json`, not prose in this file, because a report travels without its
522+
README:
523+
524+
| Field | Says | If it is missing |
525+
|-------|------|------------------|
526+
| `run_status.completeness` | `complete`, or `partial` with a reason and the unmeasured cells named | Undeclared completeness — the run is **not publishable** as a complete comparison (FR-032) |
527+
| `<block>.inputs.independently_reproducible` | whether an outsider can obtain the inputs at all | Undeclared availability — treated as **not** reproducible, and blocks publication (FR-030) |
528+
| `<block>.records` | where this block's raw per-run records sit, as a path relative to the run directory | No `records` key at all blocks publication; `retention: not_retained` does not — the figure is real, just no longer traceable to its inputs (FR-029) |
529+
530+
`bench.ReportV2.PublicationCheck()` applies all three and returns blockers plus
531+
caveats. A partial run blocks. An undeclared input blocks. A report in which
532+
*every* block rests on a private recording blocks — see step 5.
533+
534+
**5. Know what you cannot reproduce.** A replay run over someone's own exported
535+
activity is scored from **private recorded sessions**: the recording is raw user
536+
traffic, it is never published (FR-006), and no procedure exists that would hand
537+
it to you. Such a block is marked `independently_reproducible: false` with the
538+
limitation stated in the document, and it may **never be the sole support for a
539+
published claim** — a report whose only figures come from one operator's
540+
recording is a claim nobody outside can check, however honestly each row is
541+
labelled. Publish it beside a block an outsider *can* reproduce (the committed
542+
corpora, or a pinned public task suite), and keep the caveat attached.
543+
544+
**6. Raw records are run-local and not durable.** They are written under the
545+
gitignored `bench/results/` tree — the same tree SC-011 forbids committing and
546+
the same tree a cleanup empties. The report references them by a path relative
547+
to its own directory, never an absolute one: an absolute path would carry the
548+
operator's filesystem into a published document and would dangle on every other
549+
machine. Copy the run directory elsewhere before relying on those records; once
550+
they are gone the reference degrades to `records not retained` rather than
551+
pointing at nothing.
552+
553+
Each mark is written by the run that produces the block: the replay entry point
554+
sets `replay.inputs` (`private-recording` for an operator's own export) and
555+
`replay.records`; the agent loop sets its own pair plus the report-level
556+
`run_status`, derived from the cells it planned against the cells it actually
557+
measured. `WriteJSON` re-checks every records reference against the directory it
558+
is writing into, so a reference cannot be published pointing at a file that is
559+
already gone.
560+
468561
### Before publishing a replay or agent-loop number
469562

470563
1. Recompute the achievable ceiling **for each fleet shape**; never carry a
471564
previous fleet's ceiling forward.
472565
2. Quote the fleet shape beside every percentage.
473-
3. Confirm no report is tracked:
566+
3. Confirm the run is not `partial`, and that no private-recording figure is
567+
standing alone — `PublicationCheck()` answers both, and returns the caveats
568+
that must travel with the number even when it publishes.
569+
4. Confirm no report is tracked:
474570
```bash
475571
test -z "$(git ls-files bench/results)" && echo "clean (SC-011)"
476572
```
@@ -588,3 +684,17 @@ headline numbers (image drift can change the tool corpus).
588684

589685
Methodology questions / disputes: open an issue in `smart-mcp-proxy/mcpproxy-go`
590686
and tag the maintainers, or comment on the roadmap benchmark ticket (MCP-42).
687+
688+
Warm it with the one-line command rather than by hand — the same step CI runs
689+
before its parallel test sweep:
690+
691+
```bash
692+
export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken"
693+
go run ./bench/cmd/warmtiktoken
694+
```
695+
696+
Why a dedicated step: `go test ./...` runs packages in parallel, and several
697+
test binaries race to download the same vocabulary into one cache dir. On
698+
Windows the loser fails its whole package on the atomic rename — it surfaced as
699+
`bench/arms` flaking with no code change near it. Naming the cache does not fix
700+
that; populating it first does.

0 commit comments

Comments
 (0)