Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,14 +120,35 @@ jobs:
New-Item -ItemType Directory -Force -Path web/frontend
Copy-Item -Recurse frontend/dist web/frontend/

# Warm the tiktoken vocabulary ONCE, before the parallel test run.
#
# tiktoken-go downloads the BPE on first use into a shared temp dir and
# finishes with an atomic rename. `go test ./...` runs packages in
# parallel, so several test binaries race to download the same file, and
# on Windows the loser hits "Access is denied" on the rename and the
# package fails — observed as TestBaseline_CountToolWithSchemaParity
# flaking in bench/arms with no code change anywhere near it.
#
# Setting TIKTOKEN_CACHE_DIR alone does NOT fix this: it only names a
# cache, and an empty one still races. Populating it first does, because
# every later tokenizer construction is a cache hit with no rename.
- name: Warm the tiktoken vocabulary cache
run: go run ./bench/cmd/warmtiktoken
env:
TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken

- name: Run unit tests (Windows)
if: matrix.os == 'windows-latest'
shell: pwsh
run: go test -v -race -timeout 10m '-coverprofile=coverage.out' -covermode=atomic '-skip=E2E|Binary|MCPProtocol|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint' ./...
env:
TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken

- name: Run unit tests (Unix)
if: matrix.os != 'windows-latest'
run: go test -v -race -coverprofile=coverage.out -skip "E2E|Binary|MCPProtocol|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint" ./...
env:
TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken

- name: Generate coverage report (Windows)
if: matrix.os == 'windows-latest'
Expand Down
6 changes: 6 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ repos:
rev: v5.0.0
hooks:
- id: trailing-whitespace
# A unified diff encodes meaning in leading whitespace: an unchanged
# context line begins with a single space, and a blank context line IS
# that lone space. Stripping it turns valid patches into ones git apply
# rejects — observed on bench/mcpmark/mcpproxy_arm.patch, where the hook
# silently rewrote 4 context lines.
exclude: '\.(patch|diff)$'
- id: end-of-file-fixer
- id: check-merge-conflict

Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -772,7 +772,7 @@ graph LR
| Web UI + macOS app UX audit | In progress | P0 | — | | |
| Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | |
| Action log / transparency — info at a glance | In progress | P1 | — | | |
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 37/64 (58%) | [103-token-bench](./specs/103-token-bench/) | #1141 |
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | #1141 |
| Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | |
| Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | |
| Planning/docs truth automation | In progress | P2 | — | | |
Expand Down Expand Up @@ -914,4 +914,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—`
| [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — |
| [101-tpa-db](./specs/101-tpa-db/) | — | — |
| [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) |
| [103-token-bench](./specs/103-token-bench/) | `in-flight` | 37/64 (58%) |
| [103-token-bench](./specs/103-token-bench/) | `shipped` | 62/64 (97%) |
112 changes: 111 additions & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -465,12 +465,108 @@ offline reproduction needs the cache pre-populated — warm it once with network
access, or restore a known-good copy (which is what `.github/workflows/bench.yml`
does via its `actions/cache` step).

### Reproducing a published figure (SC-004)

Someone with no prior context reproduces the deterministic figures **exactly**,
and the model-dependent ones within the stated numeric tolerance (FR-022 — see
the tokenizer caveat under "Known limitations"), by following these steps and
nothing else.

**1. Populate the tokenizer cache. Naming it is not filling it.**

```bash
export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken"
go run ./bench/cmd/bench \
-corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \
-out /tmp/warm # downloads the vocabulary once — needs network
ls "$TIKTOKEN_CACHE_DIR" # non-empty ⇒ every later run is genuinely offline
```

`TIKTOKEN_CACHE_DIR` only *names* a directory. A first run with the variable set
and the directory empty still reaches the network, so a machine "configured for
offline" but never warmed fails at the first token count rather than at
configuration time — and it fails the same way on an air-gapped reviewer's
laptop as it does in a network-restricted CI job. Warm it once with network
access, or restore a known-good copy (which is what `.github/workflows/bench.yml`
does via its `actions/cache` step).

**2. Supply a fleet input. There is no default, and there is no fallback.**

A menu cost is a property of the tool definitions the agent was shown. A
recording carries the *call shape* — sequence, tool mix, call counts — and no
fleet snapshot whatsoever, so a recording-only invocation has nothing to price:

```bash
go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out bench/results
# bench: replay requires a fleet input as well as a recording: a menu is a
# property of the tool definitions and the activity export carries no fleet
# snapshot, so a recording on its own computes nothing — pass a frozen corpus
# or a live-proxy catalog (this is an error, not a degraded run)
# exit status 1
```

That is a **hard error, not a degraded run**. The alternative — quietly scoring
against an empty or assumed fleet — would produce a plausible-looking number
with no fleet shape behind it, which is exactly the failure every percentage in
this report carries a `fleet_shape` to prevent. Pass the frozen corpus for a
reproducible figure, or a live `-proxy` for today's fleet.

**3. Run the pinned command, twice.** The deterministic half is byte-identical
across runs once the wall-clock stamp is removed — the `diff` recipe under
"Verify determinism" above is the check, and it is what SC-002 means. If two
runs on the same machine differ by anything other than `generated_at`, stop:
nothing downstream is worth comparing yet.

**4. Read the three honesty marks before quoting anything.** They are fields in
`report.json`, not prose in this file, because a report travels without its
README:

| Field | Says | If it is missing |
|-------|------|------------------|
| `run_status.completeness` | `complete`, or `partial` with a reason and the unmeasured cells named | Undeclared completeness — the run is **not publishable** as a complete comparison (FR-032) |
| `<block>.inputs.independently_reproducible` | whether an outsider can obtain the inputs at all | Undeclared availability — treated as **not** reproducible, and blocks publication (FR-030) |
| `<block>.records` | where this block's raw per-run records sit, as a path relative to the run directory | No `records` key at all blocks publication; `retention: not_retained` does not — the figure is real, just no longer traceable to its inputs (FR-029) |

`bench.ReportV2.PublicationCheck()` applies all three and returns blockers plus
caveats. A partial run blocks. An undeclared input blocks. A report in which
*every* block rests on a private recording blocks — see step 5.

**5. Know what you cannot reproduce.** A replay run over someone's own exported
activity is scored from **private recorded sessions**: the recording is raw user
traffic, it is never published (FR-006), and no procedure exists that would hand
it to you. Such a block is marked `independently_reproducible: false` with the
limitation stated in the document, and it may **never be the sole support for a
published claim** — a report whose only figures come from one operator's
recording is a claim nobody outside can check, however honestly each row is
labelled. Publish it beside a block an outsider *can* reproduce (the committed
corpora, or a pinned public task suite), and keep the caveat attached.

**6. Raw records are run-local and not durable.** They are written under the
gitignored `bench/results/` tree — the same tree SC-011 forbids committing and
the same tree a cleanup empties. The report references them by a path relative
to its own directory, never an absolute one: an absolute path would carry the
operator's filesystem into a published document and would dangle on every other
machine. Copy the run directory elsewhere before relying on those records; once
they are gone the reference degrades to `records not retained` rather than
pointing at nothing.

Each mark is written by the run that produces the block: the replay entry point
sets `replay.inputs` (`private-recording` for an operator's own export) and
`replay.records`; the agent loop sets its own pair plus the report-level
`run_status`, derived from the cells it planned against the cells it actually
measured. `WriteJSON` re-checks every records reference against the directory it
is writing into, so a reference cannot be published pointing at a file that is
already gone.

### Before publishing a replay or agent-loop number

1. Recompute the achievable ceiling **for each fleet shape**; never carry a
previous fleet's ceiling forward.
2. Quote the fleet shape beside every percentage.
3. Confirm no report is tracked:
3. Confirm the run is not `partial`, and that no private-recording figure is
standing alone — `PublicationCheck()` answers both, and returns the caveats
that must travel with the number even when it publishes.
4. Confirm no report is tracked:
```bash
test -z "$(git ls-files bench/results)" && echo "clean (SC-011)"
```
Expand Down Expand Up @@ -588,3 +684,17 @@ headline numbers (image drift can change the tool corpus).

Methodology questions / disputes: open an issue in `smart-mcp-proxy/mcpproxy-go`
and tag the maintainers, or comment on the roadmap benchmark ticket (MCP-42).

Warm it with the one-line command rather than by hand — the same step CI runs
before its parallel test sweep:

```bash
export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken"
go run ./bench/cmd/warmtiktoken
```

Why a dedicated step: `go test ./...` runs packages in parallel, and several
test binaries race to download the same vocabulary into one cache dir. On
Windows the loser fails its whole package on the atomic rename — it surfaced as
`bench/arms` flaking with no code change near it. Naming the cache does not fix
that; populating it first does.
Loading
Loading