diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml index 5b0090fea..0b3c9b394 100644 --- a/.github/workflows/unit-tests.yml +++ b/.github/workflows/unit-tests.yml @@ -120,14 +120,35 @@ jobs: New-Item -ItemType Directory -Force -Path web/frontend Copy-Item -Recurse frontend/dist web/frontend/ + # Warm the tiktoken vocabulary ONCE, before the parallel test run. + # + # tiktoken-go downloads the BPE on first use into a shared temp dir and + # finishes with an atomic rename. `go test ./...` runs packages in + # parallel, so several test binaries race to download the same file, and + # on Windows the loser hits "Access is denied" on the rename and the + # package fails — observed as TestBaseline_CountToolWithSchemaParity + # flaking in bench/arms with no code change anywhere near it. + # + # Setting TIKTOKEN_CACHE_DIR alone does NOT fix this: it only names a + # cache, and an empty one still races. Populating it first does, because + # every later tokenizer construction is a cache hit with no rename. + - name: Warm the tiktoken vocabulary cache + run: go run ./bench/cmd/warmtiktoken + env: + TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken + - name: Run unit tests (Windows) if: matrix.os == 'windows-latest' shell: pwsh run: go test -v -race -timeout 10m '-coverprofile=coverage.out' -covermode=atomic '-skip=E2E|Binary|MCPProtocol|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint' ./... + env: + TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken - name: Run unit tests (Unix) if: matrix.os != 'windows-latest' run: go test -v -race -coverprofile=coverage.out -skip "E2E|Binary|MCPProtocol|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint" ./... + env: + TIKTOKEN_CACHE_DIR: ${{ runner.temp }}/tiktoken - name: Generate coverage report (Windows) if: matrix.os == 'windows-latest' diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 386a06b64..5ae98bad4 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -7,6 +7,12 @@ repos: rev: v5.0.0 hooks: - id: trailing-whitespace + # A unified diff encodes meaning in leading whitespace: an unchanged + # context line begins with a single space, and a blank context line IS + # that lone space. Stripping it turns valid patches into ones git apply + # rejects — observed on bench/mcpmark/mcpproxy_arm.patch, where the hook + # silently rewrote 4 context lines. + exclude: '\.(patch|diff)$' - id: end-of-file-fixer - id: check-merge-conflict diff --git a/ROADMAP.md b/ROADMAP.md index 84efe9c6d..f65258bd7 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -772,7 +772,7 @@ graph LR | Web UI + macOS app UX audit | In progress | P0 | — | | | | Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | | | Action log / transparency — info at a glance | In progress | P1 | — | | | -| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 37/64 (58%) | [103-token-bench](./specs/103-token-bench/) | #1141 | +| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | #1141 | | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | | Planning/docs truth automation | In progress | P2 | — | | | @@ -914,4 +914,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | | [101-tpa-db](./specs/101-tpa-db/) | — | — | | [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) | -| [103-token-bench](./specs/103-token-bench/) | `in-flight` | 37/64 (58%) | +| [103-token-bench](./specs/103-token-bench/) | `shipped` | 62/64 (97%) | diff --git a/bench/README.md b/bench/README.md index 4bb7bafb9..ac00eb5af 100644 --- a/bench/README.md +++ b/bench/README.md @@ -465,12 +465,108 @@ offline reproduction needs the cache pre-populated — warm it once with network access, or restore a known-good copy (which is what `.github/workflows/bench.yml` does via its `actions/cache` step). +### Reproducing a published figure (SC-004) + +Someone with no prior context reproduces the deterministic figures **exactly**, +and the model-dependent ones within the stated numeric tolerance (FR-022 — see +the tokenizer caveat under "Known limitations"), by following these steps and +nothing else. + +**1. Populate the tokenizer cache. Naming it is not filling it.** + +```bash +export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" +go run ./bench/cmd/bench \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \ + -out /tmp/warm # downloads the vocabulary once — needs network +ls "$TIKTOKEN_CACHE_DIR" # non-empty ⇒ every later run is genuinely offline +``` + +`TIKTOKEN_CACHE_DIR` only *names* a directory. A first run with the variable set +and the directory empty still reaches the network, so a machine "configured for +offline" but never warmed fails at the first token count rather than at +configuration time — and it fails the same way on an air-gapped reviewer's +laptop as it does in a network-restricted CI job. Warm it once with network +access, or restore a known-good copy (which is what `.github/workflows/bench.yml` +does via its `actions/cache` step). + +**2. Supply a fleet input. There is no default, and there is no fallback.** + +A menu cost is a property of the tool definitions the agent was shown. A +recording carries the *call shape* — sequence, tool mix, call counts — and no +fleet snapshot whatsoever, so a recording-only invocation has nothing to price: + +```bash +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -out bench/results +# bench: replay requires a fleet input as well as a recording: a menu is a +# property of the tool definitions and the activity export carries no fleet +# snapshot, so a recording on its own computes nothing — pass a frozen corpus +# or a live-proxy catalog (this is an error, not a degraded run) +# exit status 1 +``` + +That is a **hard error, not a degraded run**. The alternative — quietly scoring +against an empty or assumed fleet — would produce a plausible-looking number +with no fleet shape behind it, which is exactly the failure every percentage in +this report carries a `fleet_shape` to prevent. Pass the frozen corpus for a +reproducible figure, or a live `-proxy` for today's fleet. + +**3. Run the pinned command, twice.** The deterministic half is byte-identical +across runs once the wall-clock stamp is removed — the `diff` recipe under +"Verify determinism" above is the check, and it is what SC-002 means. If two +runs on the same machine differ by anything other than `generated_at`, stop: +nothing downstream is worth comparing yet. + +**4. Read the three honesty marks before quoting anything.** They are fields in +`report.json`, not prose in this file, because a report travels without its +README: + +| Field | Says | If it is missing | +|-------|------|------------------| +| `run_status.completeness` | `complete`, or `partial` with a reason and the unmeasured cells named | Undeclared completeness — the run is **not publishable** as a complete comparison (FR-032) | +| `.inputs.independently_reproducible` | whether an outsider can obtain the inputs at all | Undeclared availability — treated as **not** reproducible, and blocks publication (FR-030) | +| `.records` | where this block's raw per-run records sit, as a path relative to the run directory | No `records` key at all blocks publication; `retention: not_retained` does not — the figure is real, just no longer traceable to its inputs (FR-029) | + +`bench.ReportV2.PublicationCheck()` applies all three and returns blockers plus +caveats. A partial run blocks. An undeclared input blocks. A report in which +*every* block rests on a private recording blocks — see step 5. + +**5. Know what you cannot reproduce.** A replay run over someone's own exported +activity is scored from **private recorded sessions**: the recording is raw user +traffic, it is never published (FR-006), and no procedure exists that would hand +it to you. Such a block is marked `independently_reproducible: false` with the +limitation stated in the document, and it may **never be the sole support for a +published claim** — a report whose only figures come from one operator's +recording is a claim nobody outside can check, however honestly each row is +labelled. Publish it beside a block an outsider *can* reproduce (the committed +corpora, or a pinned public task suite), and keep the caveat attached. + +**6. Raw records are run-local and not durable.** They are written under the +gitignored `bench/results/` tree — the same tree SC-011 forbids committing and +the same tree a cleanup empties. The report references them by a path relative +to its own directory, never an absolute one: an absolute path would carry the +operator's filesystem into a published document and would dangle on every other +machine. Copy the run directory elsewhere before relying on those records; once +they are gone the reference degrades to `records not retained` rather than +pointing at nothing. + +Each mark is written by the run that produces the block: the replay entry point +sets `replay.inputs` (`private-recording` for an operator's own export) and +`replay.records`; the agent loop sets its own pair plus the report-level +`run_status`, derived from the cells it planned against the cells it actually +measured. `WriteJSON` re-checks every records reference against the directory it +is writing into, so a reference cannot be published pointing at a file that is +already gone. + ### Before publishing a replay or agent-loop number 1. Recompute the achievable ceiling **for each fleet shape**; never carry a previous fleet's ceiling forward. 2. Quote the fleet shape beside every percentage. -3. Confirm no report is tracked: +3. Confirm the run is not `partial`, and that no private-recording figure is + standing alone — `PublicationCheck()` answers both, and returns the caveats + that must travel with the number even when it publishes. +4. Confirm no report is tracked: ```bash test -z "$(git ls-files bench/results)" && echo "clean (SC-011)" ``` @@ -588,3 +684,17 @@ headline numbers (image drift can change the tool corpus). Methodology questions / disputes: open an issue in `smart-mcp-proxy/mcpproxy-go` and tag the maintainers, or comment on the roadmap benchmark ticket (MCP-42). + +Warm it with the one-line command rather than by hand — the same step CI runs +before its parallel test sweep: + +```bash +export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" +go run ./bench/cmd/warmtiktoken +``` + +Why a dedicated step: `go test ./...` runs packages in parallel, and several +test binaries race to download the same vocabulary into one cache dir. On +Windows the loser fails its whole package on the atomic rename — it surfaced as +`bench/arms` flaking with no code change near it. Naming the cache does not fix +that; populating it first does. diff --git a/bench/agentloop.go b/bench/agentloop.go new file mode 100644 index 000000000..50b260887 --- /dev/null +++ b/bench/agentloop.go @@ -0,0 +1,1349 @@ +// agentloop.go — Spec 103 US2: the LIVE agent-loop driver. This is where the +// feature's actual claim — tokens per *completed* task — becomes measurable. +// +// # Why this file exists at all +// +// Every other measurement in this harness prices a MENU: how many tokens the +// tool definitions cost an agent before it does any work. That number is real +// and it is reproducible, but it is not the number a user cares about. A mode +// that halves the menu and doubles the number of failed attempts costs more, +// not less. Only a live loop can tell the difference, because the difference +// is made of decisions a model takes — which call it attempts, whether the +// first attempt was right, whether it finished. A recording cannot answer any +// of that (see the Replay Boundary in the spec), which is exactly why US1's +// replay and this file are separate stories with separate accounting. +// +// # Why it is parameterised over function types +// +// A live run costs real money and needs a pinned model, a patched suite and +// credentials. If the arithmetic that decides what gets PUBLISHED could only +// be exercised by such a run, it would never be exercised: the tests would be +// skipped in CI, the classification rules would drift, and the first time +// anybody checked the numbers would be the day they were published. +// +// So the driver takes RunTaskFunc and UsageFunc (the bench/flipgate.go +// precedent, where RetrieveToolsFunc makes the flip-gate maths testable with +// no proxy running). Every rule below — the binding Definitions, the FR-021 +// headline bar, the FR-019 regression flag, the never-sum-across-sources +// refusal — is decided in pure functions over records, and the production +// caller supplies real records instead of fakes. Nothing about the arithmetic +// changes between the two. +// +// # The classification is deliberately source-blind +// +// ClassifyIntents sees an attempt sequence and nothing else — not the cell, +// not the endpoint, not whether mcpproxy was in the path. If the baseline arm +// and the proxy arms were classified by different rules, the comparison would +// be biased toward whichever surface reports failure more legibly, and that +// bias would be invisible in the published figure. One rule, both arms. +// +// # What this file will NOT do +// +// It will not sum a provider-reported figure with a tokenizer-counted one. It +// will not publish a percentage without a fleet shape. It will not treat a +// missing completion verdict as a failure, a missing cache-read count as a +// zero, or a single run as a headline. Each of those would understate cost or +// overstate success in this project's favour, which is the failure class the +// whole feature exists to prevent. +package bench + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "sort" + "strings" +) + +// MinHeadlineRuns is FR-021's bar: a model-dependent figure is an average over +// at least four runs together with a measure of consistency. A single run of a +// stochastic loop is an anecdote, and the difference between two modes is +// routinely smaller than the run-to-run variance of one of them. +const MinHeadlineRuns = 4 + +// DefaultCompletionRegressionThresholdPct is FR-019's "stated threshold", in +// PERCENTAGE POINTS of completion rate below the baseline arm. It is stated as +// a named constant rather than buried in a comparison precisely because FR-019 +// requires the threshold to be stated: a reader must be able to see what bar a +// mode had to clear before its cheapness was called a saving. +const DefaultCompletionRegressionThresholdPct = 5.0 + +// Completion verdicts. TASK COMPLETION is the suite's own pass verdict and is +// NEVER inferred. Where a suite emits no verdict the unit is reported as +// CompletionNoSignal and excluded from completion-dependent figures — not +// counted as a success (which would flatter us) and not counted as a failure +// (which would flatter a competing mode). Silence is not evidence either way. +const ( + CompletionPass = "pass" + CompletionFail = "fail" + CompletionNoSignal = "no-signal" +) + +// Attempt outcomes. +// +// The distinction that carries weight is AttemptOutcomeInfrastructure versus +// AttemptOutcomeError. A transport failure measures the NETWORK, not the mode: +// counting it against a cell would mean a flaky hop during one arm's run shows +// up as that mode confusing the agent. So transport failures are counted on +// their own axis and leave the intent indeterminate rather than scoring it. +// +// AttemptOutcomeUnknown exists because a trajectory can end mid-call. An +// unknown outcome is never silently read as success; it makes its intent +// indeterminate, and the record that carries it is partial. +const ( + AttemptOutcomeOK = "ok" + AttemptOutcomeError = "error" + AttemptOutcomeInfrastructure = "infrastructure" + AttemptOutcomeUnknown = "unknown" +) + +// ErrAgentLoopBaselineMissing is the FR-020 refusal: no baseline arm, no +// denominator, therefore no publishable percentage. It is an error rather than +// a degraded run because there is nothing to degrade TO — a savings figure +// measured against nothing is not a smaller figure, it is a fabricated one. +var ErrAgentLoopBaselineMissing = errors.New("agent loop requires the FR-020 baseline arm") + +// ErrBaselineNotBypassing guards the substance of FR-020 rather than its name. +// The baseline is the same agent doing the same tasks with EVERY upstream tool +// loaded directly, bypassing mcpproxy entirely. A cell called "baseline" that +// points at an mcpproxy endpoint is a sixth proxy cell wearing the +// denominator's name, and every percentage computed against it would be a +// proxy-versus-proxy comparison published as proxy-versus-nothing. +var ErrBaselineNotBypassing = errors.New("the baseline arm must bypass mcpproxy entirely (no endpoint)") + +// ErrAgentLoopModelUnpinned refuses provider-reported usage from an unpinned +// model. Usage numbers are only interpretable against the model that produced +// them: the same task under a different model has a different token shape, so +// an unpinned figure is not comparable to any later run and cannot honour +// FR-028's "a later run is comparable to an earlier one". +var ErrAgentLoopModelUnpinned = errors.New("provider-sourced agent-loop figures require a pinned model") + +// ErrDriverCacheReadMissing fires when the driver's own usage hook returns no +// cache-read count. The hook exists for exactly one reason — the suite's +// per-task output has no cache-read field, so that axis can only come from the +// driver reading provider responses. A hook that returns nothing for it is a +// wiring bug, and treating it as "unavailable" would hide the bug behind the +// legitimate unavailable path. +var ErrDriverCacheReadMissing = errors.New("driver usage hook returned no cache-read count") + +// Attempt is ONE tool call issued toward a given intent — the spec's ATTEMPT, +// not its unit of work. +// +// ArgsFingerprint is what makes the binding Definition's carve-out decidable. +// "A schema-valid call the agent immediately re-issues DIFFERENTLY is not a +// success" needs a notion of "differently", and the arguments are it: a +// re-issue with the same arguments after a transport failure is the network +// being retried, while a re-issue with changed arguments is the agent +// correcting itself, which is the thing a routing/serialization mode can +// actually be blamed for. +type Attempt struct { + // IntentID groups attempts aimed at the same goal. Attempts sharing an + // IntentID form one intent; the tool name is the usual value. + IntentID string `json:"intent_id"` + // Seq orders attempts within an intent (1-based). + Seq int `json:"seq"` + // Outcome is one of the AttemptOutcome* values. + Outcome string `json:"outcome"` + // ArgsFingerprint identifies the arguments this attempt was issued with. + ArgsFingerprint string `json:"args_fingerprint,omitempty"` +} + +// IntentOutcome is the classification of one intent's attempt sequence. +type IntentOutcome struct { + IntentID string `json:"intent_id"` + // FirstAttemptSuccess implements the binding Definition verbatim: the + // first attempt returned a non-error result AND was not followed by a + // corrective retry for the same intent. + FirstAttemptSuccess bool `json:"first_attempt_success"` + // Indeterminate marks an intent whose first attempt failed for + // infrastructure reasons or whose outcome is unknown. Such an intent is + // EXCLUDED from the first-attempt-success denominator rather than counted + // as a failure: it measures the network (or a truncated record), not the + // mode, and scoring it would let network weather move a published figure. + Indeterminate bool `json:"indeterminate"` + // Corrective and infrastructure retries are counted separately (FR-011) + // because only the first says anything about how well the mode serves the + // agent. + RetriesCorrective int `json:"retries_corrective"` + RetriesInfrastructure int `json:"retries_infrastructure"` +} + +// ProviderUsage is provider-reported consumption for one unit of work, with +// the FR-014 axes kept apart. +// +// CacheReadTokens is a POINTER on purpose. The pinned suite's own per-task +// output records input / output / total / reasoning and has NO cache-read +// field, so a suite-derived record genuinely cannot know it — and a 0 there +// would read as "measured, and nothing was served from cache", which in a long +// agent loop is the difference between an honest total and one understated in +// this project's favour. nil means unavailable and is reported as such. +type ProviderUsage struct { + InputTokens int `json:"input_tokens"` + OutputTokens int `json:"output_tokens"` + CacheReadTokens *int `json:"cache_read_tokens"` + // ReasoningTokens is recorded because the suite reports it, and is + // deliberately NOT added into the total: providers report it as a + // component of output, so adding it would double-count. + ReasoningTokens int `json:"reasoning_tokens,omitempty"` + // Responses counts the provider responses this usage was summed from, so + // a partially-captured unit is distinguishable from a genuinely cheap one + // (data-model.md RunRecord rule). + Responses int `json:"responses,omitempty"` +} + +// Total is input + output + cache-read, and reports whether every axis was +// available. A caller that ignores the second return value and publishes the +// first has published an understated cost. +func (u ProviderUsage) Total() (total int, complete bool) { + total = u.InputTokens + u.OutputTokens + if u.CacheReadTokens == nil { + return total, false + } + return total + *u.CacheReadTokens, true +} + +// UnitRecord is the raw outcome of ONE UNIT OF WORK — one task from the fixed +// task set — under one mode cell on one run. It is the traceable unit behind +// every headline figure (FR-029). A unit is never an individual tool call, no +// matter how many attempts it took. +type UnitRecord struct { + CellID string `json:"cell_id"` + UnitID string `json:"unit_id"` + RunIndex int `json:"run_index"` + // Completion is the suite's verdict: pass, fail, or no-signal. + Completion string `json:"completion"` + // Partial marks a run record that did not finish. Partial records are + // excluded from every figure and counted separately (FR-032). + Partial bool `json:"partial,omitempty"` + Attempts []Attempt `json:"attempts,omitempty"` + Usage ProviderUsage `json:"usage"` + // Source names where THIS record's token figures came from. + // + // It lives on the record rather than only on the block because the block's + // source is a LABEL applied at assembly, while the sum it labels is built + // here, record by record. Without a per-record source, aggregateCell adds + // integers whose origin nothing checked and AgentLoopBlockFor then stamps + // the caller's provider onto the result — so a tokenizer-derived figure + // mixed into a provider run produces a plausible hybrid wearing a provider + // label. The never-sum rule has to be checkable at the point of addition, + // not asserted at the point of naming. + // + // Zero value means "unset", which aggregation treats as the block's own + // source: the common case is one driver producing every record, and + // requiring every fake in every test to restate it would be noise. + Source AccountingSource `json:"source,omitzero"` + TurnCount int `json:"turn_count,omitempty"` +} + +// RunTaskFunc runs ONE unit of work under ONE mode cell and returns its raw +// record. It is a function type for the reason RetrieveToolsFunc is +// (flipgate.go): a concrete suite in the signature would pin every test to a +// paid run, and an untestable rule is an unenforced rule. +type RunTaskFunc func(cell ModeCell, unitID string, runIndex int) (*UnitRecord, error) + +// UsageFunc returns the provider-reported usage the DRIVER captured for one +// unit of work, read off the model responses themselves. +// +// It exists because of a specific gap: the suite's per-task report has no +// cache-read field (research.md §5), so FR-014's third axis cannot come from +// the suite at all. Either the driver supplies it here, or the axis is +// recorded as unavailable — never as zero. +type UsageFunc func(cell ModeCell, unitID string, runIndex int) (ProviderUsage, error) + +// AgentLoopDriver is the pair of seams a live run plugs into. +type AgentLoopDriver struct { + // RunTask is mandatory. + RunTask RunTaskFunc + // CaptureUsage is OPTIONAL. When nil, records keep the usage RunTask + // returned — which, for a suite-derived record, has no cache-read count, + // and the resulting cells are marked accordingly rather than zero-filled. + CaptureUsage UsageFunc +} + +// AgentLoopOptions configures one live measurement. +type AgentLoopOptions struct { + // Suite and SuiteVersion pin the task suite (FR-028). SuiteVersion should + // be the commit SHA the run was executed at. + Suite string + SuiteVersion string + // Provider and Model name the accounting source. Both are mandatory: see + // ErrAgentLoopModelUnpinned. + Provider string + Model string + // FleetShape is the tool set every percentage from this run holds for + // (IC-004). A percentage without it does not get published. + FleetShape FleetShape + // Cells is the matrix under measurement. It MUST include the baseline + // cell — that is the denominator, not an optional extra row. + Cells []ModeCell + // Units is the fixed task set, run identically under every cell. + Units []string + // Runs is k, the number of repetitions per cell (FR-021 wants >= 4). + Runs int + // CompletionRegressionThresholdPct overrides the stated FR-019 threshold. + // Zero means DefaultCompletionRegressionThresholdPct. + CompletionRegressionThresholdPct float64 +} + +func (o AgentLoopOptions) regressionThresholdPct() float64 { + if o.CompletionRegressionThresholdPct > 0 { + return o.CompletionRegressionThresholdPct + } + return DefaultCompletionRegressionThresholdPct +} + +// SourcedFigure is one value together with the accounting source that produced +// it. The pairing is the whole point: a bare float64 cannot be checked against +// the never-sum rule, and by the time figures are floats in a slice the source +// is exactly the information that has been lost. +type SourcedFigure struct { + Value float64 + Source AccountingSource +} + +// AggregateResult is an aggregate that either has a value or has a reason it +// does not. It mirrors the harness's existing withhold-rather-than-compute +// pattern (AuthoritativeHeadline / withholdHeadline in live_report.go): when +// the inputs cannot honestly be combined, the number is WITHHELD with a stated +// reason rather than computed and quietly caveated somewhere else. +type AggregateResult struct { + Value float64 `json:"value"` + Source AccountingSource `json:"source"` + Withheld bool `json:"withheld,omitempty"` + WithheldReason string `json:"withheld_reason,omitempty"` +} + +// SavingVerdict is a per-cell saving against the baseline arm, or the stated +// reason there is none to publish. FleetShape travels with it because a +// percentage without its fleet shape does not appear (IC-004/SC-005): the +// figure moves with fleet size, so quoting it bare is quoting it at an +// unstated size. +type SavingVerdict struct { + CellID string `json:"cell_id"` + SavingPct float64 `json:"saving_pct"` + Regression bool `json:"regression,omitempty"` + Withheld bool `json:"withheld,omitempty"` + WithheldReason string `json:"withheld_reason,omitempty"` + FleetShape FleetShape `json:"fleet_shape"` +} + +// AgentLoopAccountingSource builds the block's accounting source (T043). +// +// A provider-sourced block names the provider AND the pinned model, because +// the two together are what makes a usage number mean anything. This is the +// same refusal shape as ErrStubToolSchemas in mcptools.go: the guard returns +// an error rather than a plausible-looking default, so an unpinned run cannot +// produce a report that merely looks comparable to a pinned one. +func AgentLoopAccountingSource(provider, model string) (AccountingSource, error) { + if strings.TrimSpace(provider) == "" { + return AccountingSource{}, errors.New("agent loop: a provider must be named for provider-reported usage") + } + if strings.TrimSpace(model) == "" { + return AccountingSource{}, fmt.Errorf("agent loop: provider %q: %w", provider, ErrAgentLoopModelUnpinned) + } + return AccountingSource{ + Kind: AccountingKindProvider, + Identity: provider, + Model: model, + }, nil +} + +// AggregateTokenFigures sums figures ONLY when every one of them came from the +// same accounting source, and otherwise withholds with a stated reason (T038). +// +// This is data-model.md invariant 2, and it is enforced here rather than left +// to callers because the failure it prevents is silent: tokenizer counts and +// provider-reported usage are both "tokens", both plausible magnitudes, and a +// summed hybrid looks exactly like a real number. There is no downstream check +// that would catch it. Two figures from the same provider but DIFFERENT pinned +// models are also different sources — the same task has a different token +// shape under a different model. +func AggregateTokenFigures(figures []SourcedFigure) AggregateResult { + if len(figures) == 0 { + return AggregateResult{ + Withheld: true, + WithheldReason: "aggregate withheld: no figures supplied", + } + } + + src := figures[0].Source + if src.IsZero() { + return AggregateResult{ + Withheld: true, + WithheldReason: "aggregate withheld: a figure carries no accounting_source, and a figure " + + "whose source is unknown cannot enter any aggregate (data-model.md: accounting_source is mandatory)", + } + } + + total := 0.0 + for _, f := range figures { + if f.Source.IsZero() { + return AggregateResult{ + Withheld: true, + WithheldReason: "aggregate withheld: a figure carries no accounting_source, and a figure " + + "whose source is unknown cannot enter any aggregate (data-model.md: accounting_source is mandatory)", + } + } + if f.Source != src { + return AggregateResult{ + Withheld: true, + WithheldReason: fmt.Sprintf( + "aggregate WITHHELD: figures come from different accounting sources (%s and %s). "+ + "Deterministic tokenizer counts and provider-reported usage are never summed — a "+ + "cross-source total looks like a measurement and is not one. Report each source's "+ + "figure separately instead.", + describeSource(src), describeSource(f.Source)), + } + } + total += f.Value + } + return AggregateResult{Value: total, Source: src} +} + +// describeSource renders an accounting source for a withheld reason. It always +// names the KIND, because the kind is what makes two sources incompatible. +func describeSource(s AccountingSource) string { + switch { + case s.IsZero(): + return "unset" + case s.Model != "": + return fmt.Sprintf("%s:%s/%s", s.Kind, s.Identity, s.Model) + default: + return fmt.Sprintf("%s:%s", s.Kind, s.Identity) + } +} + +// ClassifyIntents implements the spec's binding Definitions over one unit of +// work's attempt sequence, and NOTHING else is in scope for it: no cell, no +// endpoint, no knowledge of whether mcpproxy was in the path. That is the +// point — the same rule must apply to the baseline arm and the proxy arms, or +// the comparison is biased toward whichever carries richer error signal. +// +// The rules, verbatim from the Definitions: +// +// - ATTEMPT: one tool call issued toward a given intent. +// - FIRST-ATTEMPT SUCCESS: the first attempt returned a non-error result AND +// was not followed by a corrective retry for the same intent. A +// schema-valid call the agent immediately re-issues differently is NOT a +// success — hence the arguments comparison rather than a status check. +// - RETRY: a subsequent attempt for the same intent. Transport/infrastructure +// retries are counted SEPARATELY because they measure the network. +// +// A retry is INFRASTRUCTURE only when the attempt it follows failed for +// transport reasons AND the arguments are unchanged: that is the shape of a +// re-send. Change the arguments and the agent has decided the call was wrong, +// which is a correction no matter what failed first. Anything following a +// non-transport outcome is corrective — including a re-issue after a +// successful call, which means the agent was not satisfied by the result. +func ClassifyIntents(attempts []Attempt) []IntentOutcome { + if len(attempts) == 0 { + return nil + } + + order := make([]string, 0, len(attempts)) + byIntent := make(map[string][]Attempt, len(attempts)) + for _, a := range attempts { + if _, seen := byIntent[a.IntentID]; !seen { + order = append(order, a.IntentID) + } + byIntent[a.IntentID] = append(byIntent[a.IntentID], a) + } + + out := make([]IntentOutcome, 0, len(order)) + for _, id := range order { + group := byIntent[id] + sort.SliceStable(group, func(i, j int) bool { return group[i].Seq < group[j].Seq }) + + outcome := IntentOutcome{IntentID: id} + first := group[0] + outcome.Indeterminate = first.Outcome == AttemptOutcomeInfrastructure || + first.Outcome == AttemptOutcomeUnknown + + for i := 1; i < len(group); i++ { + prev := group[i-1] + sameArgs := group[i].ArgsFingerprint == prev.ArgsFingerprint + if prev.Outcome == AttemptOutcomeInfrastructure && sameArgs { + outcome.RetriesInfrastructure++ + continue + } + outcome.RetriesCorrective++ + } + + outcome.FirstAttemptSuccess = !outcome.Indeterminate && + first.Outcome == AttemptOutcomeOK && + outcome.RetriesCorrective == 0 + out = append(out, outcome) + } + return out +} + +// RunAgentLoop drives the fixed task set across every configured cell — the +// FR-020 baseline arm included — and aggregates the result into the report +// block (T039/T040/T043). +func RunAgentLoop(driver AgentLoopDriver, opts AgentLoopOptions) (*AgentLoopBlock, error) { + records, err := CollectAgentLoop(driver, opts) + if err != nil { + return nil, err + } + return AgentLoopBlockFor(records, opts) +} + +// CollectAgentLoop runs cells x runs x units through the driver. +// +// The iteration is cell-major and deterministic so a partially completed run +// is a prefix of a full one rather than an arbitrary subset, and so a reader +// comparing two result sets is comparing the same order of work. +func CollectAgentLoop(driver AgentLoopDriver, opts AgentLoopOptions) ([]UnitRecord, error) { + if driver.RunTask == nil { + return nil, errors.New("agent loop: driver has no RunTask function") + } + if err := validateAgentLoopOptions(opts); err != nil { + return nil, err + } + if len(opts.Units) == 0 { + return nil, errors.New("agent loop: the task set is empty; there is no unit of work to measure") + } + if opts.Runs < 1 { + return nil, fmt.Errorf("agent loop: runs = %d; at least one run is required (FR-021 wants %d for a headline)", + opts.Runs, MinHeadlineRuns) + } + + records := make([]UnitRecord, 0, len(opts.Cells)*opts.Runs*len(opts.Units)) + for _, cell := range opts.Cells { + for run := 0; run < opts.Runs; run++ { + for _, unit := range opts.Units { + rec, err := driver.RunTask(cell, unit, run) + if err != nil { + return nil, fmt.Errorf("agent loop: cell %s, unit %s, run %d: %w", cell.ID, unit, run, err) + } + if rec == nil { + return nil, fmt.Errorf("agent loop: cell %s, unit %s, run %d: driver returned no record", + cell.ID, unit, run) + } + // A mislabelled record would land in the wrong cell and + // silently move a published comparison, so the labels are + // checked rather than trusted. + if rec.CellID != cell.ID || rec.UnitID != unit || rec.RunIndex != run { + return nil, fmt.Errorf( + "agent loop: driver returned a record labelled %s/%s/run%d for %s/%s/run%d", + rec.CellID, rec.UnitID, rec.RunIndex, cell.ID, unit, run) + } + + if driver.CaptureUsage != nil { + usage, err := driver.CaptureUsage(cell, unit, run) + if err != nil { + return nil, fmt.Errorf("agent loop: capture usage for %s/%s/run%d: %w", + cell.ID, unit, run, err) + } + if usage.CacheReadTokens == nil { + return nil, fmt.Errorf("agent loop: %s/%s/run%d: %w", + cell.ID, unit, run, ErrDriverCacheReadMissing) + } + rec.Usage = usage + } + records = append(records, *rec) + } + } + } + return records, nil +} + +// validateAgentLoopOptions enforces the two structural preconditions that +// cannot be recovered from later: the denominator exists, and it is really the +// denominator. +func validateAgentLoopOptions(opts AgentLoopOptions) error { + var baseline *ModeCell + for i := range opts.Cells { + if opts.Cells[i].ID == CellBaseline { + baseline = &opts.Cells[i] + break + } + } + if baseline == nil { + return fmt.Errorf("agent loop: %w — the baseline is the denominator every published "+ + "percentage is measured against (FR-020), not an optional extra row", ErrAgentLoopBaselineMissing) + } + if baseline.Endpoint != "" { + return fmt.Errorf("agent loop: baseline cell carries endpoint %q: %w — the baseline is the same "+ + "agent doing the same tasks with every upstream tool loaded DIRECTLY", + baseline.Endpoint, ErrBaselineNotBypassing) + } + if baseline.RoutingMode != ModeBaseline { + return fmt.Errorf("agent loop: baseline cell has routing mode %q, want %q: %w", + baseline.RoutingMode, ModeBaseline, ErrBaselineNotBypassing) + } + return nil +} + +// cellAggregate accumulates one cell's counted records before they become a +// report row. +type cellAggregate struct { + // source is the accounting source every summed record agreed on; + // mixedSources records that at least one disagreed and was excluded. + source AccountingSource + mixedSources bool + + runs int + partialRuns int + completionEligible int + completed int + tokensEligible int + inputTokens int + outputTokens int + cacheReadTokens int + cacheReadAvailable bool + sawRecord bool + determinateIntents int + firstAttemptSuccesses int + retriesCorrective int + retriesInfrastructure int + perRunCostPerTask []float64 +} + +// AgentLoopBlockFor aggregates raw records into the agent_loop report block +// (T043). +// +// The arithmetic decisions worth stating plainly, because each one is a place +// a friendlier number was available: +// +// - A run in which ANY unit is partial is excluded WHOLESALE, not +// unit-by-unit. A run that died halfway completed a PREFIX of the task set, +// and averaging that prefix biases the figure toward whichever tasks the +// suite happens to order first. +// - Units with no completion signal are excluded from completion-dependent +// figures on BOTH sides of the ratio — their tokens leave the numerator +// with them. Keeping their cost while dropping their (absent) verdict +// would inflate cost per completed task; the reverse would deflate it. +// - Tokens per completed task is a POOLED ratio (total tokens over total +// completions), not a mean of per-run ratios, so a run that completed +// nothing still contributes the money it spent. Spread is measured across +// the per-run ratios that are defined. +// - A cell whose token accounting is missing an axis is not a headline. An +// incomplete total understates cost, and understating cost in this +// project's favour is the exact failure this feature exists to prevent. +func AgentLoopBlockFor(records []UnitRecord, opts AgentLoopOptions) (*AgentLoopBlock, error) { + if err := validateAgentLoopOptions(opts); err != nil { + return nil, err + } + source, err := AgentLoopAccountingSource(opts.Provider, opts.Model) + if err != nil { + return nil, err + } + + known := make(map[string]bool, len(opts.Cells)) + for _, c := range opts.Cells { + known[c.ID] = true + } + byCell := make(map[string][]UnitRecord, len(opts.Cells)) + for _, rec := range records { + if !known[rec.CellID] { + // A record for a cell outside the matrix would vanish from every + // figure without a trace, which is the silent accounting the whole + // report works to avoid. + return nil, fmt.Errorf("agent loop: record for unknown cell %q (unit %s, run %d)", + rec.CellID, rec.UnitID, rec.RunIndex) + } + if rec.Completion != CompletionPass && rec.Completion != CompletionFail && rec.Completion != CompletionNoSignal { + return nil, fmt.Errorf("agent loop: record %s/%s/run%d carries completion %q, not one of {%s,%s,%s} — "+ + "a completion verdict is taken from the suite, never inferred", + rec.CellID, rec.UnitID, rec.RunIndex, rec.Completion, + CompletionPass, CompletionFail, CompletionNoSignal) + } + byCell[rec.CellID] = append(byCell[rec.CellID], rec) + } + + block := &AgentLoopBlock{ + AccountingSource: source, + Suite: opts.Suite, + SuiteVersion: opts.SuiteVersion, + FleetShape: opts.FleetShape, + } + + var baselineCompletion float64 + baselineHasCompletion := false + cells := make([]AgentLoopCell, 0, len(opts.Cells)) + for _, mc := range opts.Cells { + agg := aggregateCell(byCell[mc.ID]) + row := agg.toRow(mc.ID) + if mc.ID == CellBaseline && agg.completionEligible > 0 { + baselineCompletion = row.CompletionRatePct + baselineHasCompletion = true + } + cells = append(cells, row) + } + + // FR-019: the regression flag is decided against the baseline arm, so it + // needs a second pass — the baseline row must exist before any cell can be + // compared to it. + threshold := opts.regressionThresholdPct() + for i := range cells { + if cells[i].CellID == CellBaseline || !baselineHasCompletion { + continue + } + // Gate on the AGGREGATE, not on a second walk of the raw records. + // + // The two disagree, and the disagreement fabricates claims. A run is + // dropped WHOLESALE if any unit in it is partial, so a cell can hold a + // record with a completion verdict while the aggregate counted nothing + // — and the raw-record walk then let a CompletionRatePct of 0, which + // means "nothing was measured", be compared against the baseline and + // reported as a 100-point regression. That is a quantitative claim + // about a cell the same row admits is unmeasured. + // + // Withheld is checked too: a withheld cell has no figure to compare, + // and flagging one produced a report whose JSON said regression:true + // while the HTML rendered "withheld" — the two disagreeing about the + // same cell. + if cells[i].Withheld || !cellCompletionMeasured(byCell[cells[i].CellID]) { + continue + } + if baselineCompletion-cells[i].CompletionRatePct > threshold { + cells[i].Regression = true + } + } + block.CompletionRegressionThresholdPct = threshold + block.Cells = cells + return block, nil +} + +// cellCompletionMeasured reports whether a cell's AGGREGATE produced a +// completion rate — the same number the regression check compares. +// +// It deliberately re-aggregates rather than inspecting raw records. An earlier +// version walked the records looking for any verdict-carrying unit, which +// disagrees with the aggregate whenever a run is dropped for partiality: the +// record exists, the aggregate counted nothing, and the cell's implicit 0 was +// then reported as a catastrophic regression. Asking the aggregate is the only +// way to be sure the figure being compared is one that exists. +func cellCompletionMeasured(records []UnitRecord) bool { + return aggregateCell(records).completionEligible > 0 +} + +// aggregateCell folds one cell's records, run by run. +func aggregateCell(records []UnitRecord) cellAggregate { + agg := cellAggregate{cacheReadAvailable: true} + if len(records) == 0 { + agg.cacheReadAvailable = false + return agg + } + agg.sawRecord = true + + runIndexes := make([]int, 0, len(records)) + byRun := make(map[int][]UnitRecord, len(records)) + for _, r := range records { + if _, seen := byRun[r.RunIndex]; !seen { + runIndexes = append(runIndexes, r.RunIndex) + } + byRun[r.RunIndex] = append(byRun[r.RunIndex], r) + } + sort.Ints(runIndexes) + + for _, idx := range runIndexes { + run := byRun[idx] + partial := false + for _, r := range run { + if r.Partial { + partial = true + break + } + } + if partial { + agg.partialRuns++ + continue + } + + runEligible, runCompleted, runTokens := 0, 0, 0 + for _, r := range run { + // First-attempt success and retries are NOT completion-dependent: + // they describe how the agent fared per intent, so every + // non-partial unit contributes regardless of its verdict. + for _, o := range ClassifyIntents(r.Attempts) { + agg.retriesCorrective += o.RetriesCorrective + agg.retriesInfrastructure += o.RetriesInfrastructure + if o.Indeterminate { + continue + } + agg.determinateIntents++ + if o.FirstAttemptSuccess { + agg.firstAttemptSuccesses++ + } + } + + if r.Completion == CompletionNoSignal { + continue + } + // THE NEVER-SUM RULE, enforced where the addition happens. + // A record whose source differs from the one this aggregate is + // accumulating cannot be added to it: the result would be a + // hybrid, and a hybrid labelled with either source is a false + // claim about both. Recorded and surfaced, never silently dropped + // and never silently summed. + if !r.Source.IsZero() { + if agg.source.IsZero() { + agg.source = r.Source + } else if agg.source != r.Source { + agg.mixedSources = true + continue + } + } + total, complete := r.Usage.Total() + if !complete { + agg.cacheReadAvailable = false + } + runEligible++ + runTokens += total + agg.inputTokens += r.Usage.InputTokens + agg.outputTokens += r.Usage.OutputTokens + if r.Usage.CacheReadTokens != nil { + agg.cacheReadTokens += *r.Usage.CacheReadTokens + } + if r.Completion == CompletionPass { + runCompleted++ + } + } + + if runEligible == 0 { + continue + } + agg.runs++ + agg.completionEligible += runEligible + agg.completed += runCompleted + agg.tokensEligible += runTokens + if runCompleted > 0 { + agg.perRunCostPerTask = append(agg.perRunCostPerTask, float64(runTokens)/float64(runCompleted)) + } + } + return agg +} + +// toRow renders one aggregate as a report row, attaching the withheld reasons +// that keep an incomplete figure from reading as a measured one. +func (a cellAggregate) toRow(cellID string) AgentLoopCell { + // A cell that mixed accounting sources is WITHHELD outright. The records + // that disagreed were already excluded from the sums, so what remains is a + // partial total — and a partial total presented as the cell's cost + // understates it, which is the direction of error this benchmark exists to + // prevent. Withhold and say why. + if a.mixedSources { + return AgentLoopCell{ + CellID: cellID, + Provenance: ProvenanceMeasured, + Runs: a.runs, + PartialRuns: a.partialRuns, + Withheld: true, + WithheldReason: "withheld: this cell's records carried MORE THAN ONE accounting source. " + + "Provider-reported usage and tokenizer counts measure different things and are never " + + "summed, so the records that disagreed were excluded — leaving a partial total that " + + "would understate the cell's cost if published.", + } + } + row := AgentLoopCell{ + CellID: cellID, + // Every row here is an attempted MEASUREMENT of a live run; a row that + // could not be measured says so through Withheld rather than by + // downgrading its provenance, which would suggest an estimate exists. + Provenance: ProvenanceMeasured, + Runs: a.runs, + PartialRuns: a.partialRuns, + RetriesCorrective: a.retriesCorrective, + RetriesInfrastructure: a.retriesInfrastructure, + InputTokens: a.inputTokens, + OutputTokens: a.outputTokens, + CacheReadTokens: a.cacheReadTokens, + } + if a.determinateIntents > 0 { + v := pct(a.firstAttemptSuccesses, a.determinateIntents) + row.FirstAttemptSuccessPct = &v + } + if a.completionEligible > 0 { + row.CompletionRatePct = pct(a.completed, a.completionEligible) + } + if a.completed > 0 { + row.TokensPerCompletedTask = float64(a.tokensEligible) / float64(a.completed) + } + // Only when a spread is DEFINED. Fewer than two per-run figures leaves it + // nil, which renders as "spread undefined" rather than as ±0.0%. + if len(a.perRunCostPerTask) >= 2 { + sp := relativeSpreadPct(a.perRunCostPerTask) + row.SpreadPct = &sp + } + + var reasons []string + switch { + case !a.sawRecord: + reasons = append(reasons, "no run records were supplied for this cell, so nothing about it was measured") + case a.runs == 0: + reasons = append(reasons, "no run completed with a unit carrying a completion verdict; "+ + "every supplied run was partial or verdict-less") + } + if a.sawRecord && a.runs > 0 && a.completed == 0 { + reasons = append(reasons, "no unit of work completed under this cell in any counted run, "+ + "so tokens per completed task is undefined (it is not zero)") + } + if !a.cacheReadAvailable && a.sawRecord { + reasons = append(reasons, "cache-read consumption is UNAVAILABLE for at least one counted unit "+ + "(the suite's per-task output carries no cache-read field, so that axis must come from the "+ + "driver): the token totals here are a LOWER BOUND, not a complete FR-014 accounting") + } + if len(reasons) > 0 { + row.Withheld = true + row.WithheldReason = strings.Join(reasons, "; ") + } + + row.Headline = row.Runs >= MinHeadlineRuns && + len(a.perRunCostPerTask) >= 2 && + a.completed > 0 && + a.cacheReadAvailable && + !row.Withheld + return row +} + +// SavingVsBaseline is the only sanctioned way to turn two agent-loop cells into +// a percentage (FR-019/FR-020/SC-007). +// +// It refuses in three situations, each for a reason that would otherwise be +// invisible in the published number: +// +// 1. There is no usable baseline. FR-020's denominator is the same agent doing +// the same tasks with every tool loaded directly; without it a "saving" is +// measured against nothing. +// 2. The cell is a REGRESSION — its completion rate fell more than the stated +// threshold below the baseline's. A mode that finishes fewer tasks is not +// cheap, it is broken, and describing it as a saving is precisely what +// SC-007 exists to prevent. Its token cost is still reported; it is the +// word "saving" that is refused. +// 3. Either cell's token accounting is incomplete. Comparing a complete total +// against a lower bound produces a percentage of two different quantities. +// +// The comparison inside one block is legitimate because both cells share the +// block's accounting source. Comparing a cell here against a figure from the +// deterministic blocks is NOT, which is what AggregateTokenFigures refuses. +func (b *AgentLoopBlock) SavingVsBaseline(cellID string) SavingVerdict { + verdict := SavingVerdict{CellID: cellID} + if b == nil { + return SavingVerdict{ + CellID: cellID, + Withheld: true, + WithheldReason: "saving withheld: no agent_loop block", + } + } + verdict.FleetShape = b.FleetShape + + var cell, baseline *AgentLoopCell + for i := range b.Cells { + switch b.Cells[i].CellID { + case cellID: + cell = &b.Cells[i] + case CellBaseline: + baseline = &b.Cells[i] + } + } + if cell == nil { + verdict.Withheld = true + verdict.WithheldReason = fmt.Sprintf("saving withheld: no measured cell %q in this block", cellID) + return verdict + } + if baseline == nil || baseline.TokensPerCompletedTask <= 0 { + verdict.Withheld = true + verdict.WithheldReason = "saving WITHHELD: the FR-020 baseline arm produced no tokens-per-completed-task " + + "figure, so there is no denominator. A saving quoted against a missing baseline is measured against nothing." + return verdict + } + if cell.Withheld || baseline.Withheld { + verdict.Withheld = true + verdict.WithheldReason = "saving WITHHELD: a figure on one side of the comparison was itself withheld, " + + "so the ratio would be a claim built on a number the report declines to publish. Checked BEFORE the " + + "regression branch on purpose: an unmeasured cell has no completion rate, and describing its implicit " + + "zero as a regression fabricates a quantitative claim about a cell nothing was counted for. " + + "Cell reason: " + orNone(cell.WithheldReason) + " Baseline reason: " + orNone(baseline.WithheldReason) + return verdict + } + // FR-021: a model-dependent figure needs at least MinHeadlineRuns runs and + // a computable spread. This function is documented as the only sanctioned + // way to turn two cells into a percentage, so the bar has to live HERE — + // enforcing it only on the row's Headline flag left a caller free to + // publish a saving off a single run, which is exactly the single-run + // headline FR-021 exists to forbid. + if cell.Runs < MinHeadlineRuns || baseline.Runs < MinHeadlineRuns { + verdict.Withheld = true + verdict.WithheldReason = fmt.Sprintf( + "saving WITHHELD: FR-021 requires at least %d runs per side and this comparison has %d (cell) and "+ + "%d (baseline). A single agentic run is noise, and its spread is undefined rather than zero.", + MinHeadlineRuns, cell.Runs, baseline.Runs) + return verdict + } + // DERIVE the regression from the completion rates rather than trusting the + // row's flag. + // + // This function is documented as the only sanctioned way to turn two cells + // into a percentage, so every publication rule has to hold for ANY block it + // is handed — including one unmarshalled from a report.json, assembled by a + // caller, or mutated after AgentLoopBlockFor set the flag. Trusting a + // mutable bool meant a block carrying Regression:false with a 50-point + // completion deficit returned a saving; the numbers needed to catch that + // are right here on both cells. + regressed := cell.Regression + if baseline.CompletionRatePct-cell.CompletionRatePct > CompletionRegressionThresholdPct(b) { + regressed = true + } + if regressed { + verdict.Regression = true + verdict.Withheld = true + verdict.WithheldReason = fmt.Sprintf( + "saving WITHHELD — REGRESSION: completion rate %.1f%% is %.1f percentage points below the baseline's "+ + "%.1f%%. A mode that completes fewer tasks must not be described as a saving regardless of its "+ + "token cost (FR-019/SC-007); its cost is reported beside its completion rate instead.", + cell.CompletionRatePct, baseline.CompletionRatePct-cell.CompletionRatePct, baseline.CompletionRatePct) + return verdict + } + if cell.TokensPerCompletedTask <= 0 { + verdict.Withheld = true + verdict.WithheldReason = fmt.Sprintf("saving withheld: cell %q has no tokens-per-completed-task figure", cellID) + return verdict + } + + verdict.SavingPct = (1.0 - cell.TokensPerCompletedTask/baseline.TokensPerCompletedTask) * 100.0 + return verdict +} + +func orNone(s string) string { + if s == "" { + return "(none)." + } + return s + "." +} + +// pct is a percentage guarded against a zero denominator. +func pct(part, whole int) float64 { + if whole == 0 { + return 0 + } + return float64(part) / float64(whole) * 100.0 +} + +// relativeSpreadPct is FR-021's "measure of consistency": the range of the +// per-run values as a percentage of their mean. Range rather than standard +// deviation because at k=4 a standard deviation is barely meaningful, while +// the best-to-worst gap is exactly what a reader needs to judge whether a +// difference between two modes is bigger than the noise inside one of them. +func relativeSpreadPct(values []float64) float64 { + if len(values) < 2 { + return 0 + } + minV, maxV, sum := values[0], values[0], 0.0 + for _, v := range values { + if v < minV { + minV = v + } + if v > maxV { + maxV = v + } + sum += v + } + mean := sum / float64(len(values)) + if mean == 0 { + return 0 + } + return (maxV - minV) / mean * 100.0 +} + +// --------------------------------------------------------------------------- +// MCPMark ingestion (T041) +// --------------------------------------------------------------------------- + +// MCPMarkTokenUsage mirrors the pinned suite's per-task `token_usage` object. +// +// Every field is a POINTER so an absent key is distinguishable from a reported +// zero — the same reason ReplayCellCost.ResponseTokens is a pointer. A missing +// count silently read as 0 would understate cost, and understating cost in +// this project's favour is the failure class FR-002 forbids for truncation and +// that this file must not reintroduce on a new axis. +// +// Both key spellings are accepted. The suite reports input/output/total/ +// reasoning; the `_tokens`-suffixed spelling is common in provider payloads +// that the suite passes through, and refusing to read a file we can obviously +// read would trade a real measurement for a stylistic point. Which spelling +// the pinned SHA actually emits is recorded in bench/README.md alongside the +// pin, and is re-checked whenever the pin moves. +type MCPMarkTokenUsage struct { + Input *int `json:"input"` + Output *int `json:"output"` + Total *int `json:"total"` + Reasoning *int `json:"reasoning"` + InputAlt *int `json:"input_tokens"` + OutputAlt *int `json:"output_tokens"` + TotalAlt *int `json:"total_tokens"` + ReasoningAlt *int `json:"reasoning_tokens"` +} + +func firstNonNil(vals ...*int) *int { + for _, v := range vals { + if v != nil { + return v + } + } + return nil +} + +// MCPMarkMeta is the subset of the suite's per-task meta.json this driver +// reads: the pass verdict, the token usage, and the turn count. +// +// execution_result.success is a POINTER because its absence is NOT a failure. +// TASK COMPLETION is the suite's own verdict; where the suite emits none, the +// unit has NO COMPLETION SIGNAL and is excluded from completion-dependent +// figures. Decoding a missing bool as false would quietly convert "we do not +// know" into "it failed", which is an inference the Definitions forbid. +type MCPMarkMeta struct { + TaskName string `json:"task_name"` + ExecutionResult struct { + Success *bool `json:"success"` + } `json:"execution_result"` + TokenUsage *MCPMarkTokenUsage `json:"token_usage"` + TurnCount *int `json:"turn_count"` + + // SourcePath is the file this was read from, kept so a report can + // reference the raw per-run record it was computed from (FR-029). + SourcePath string `json:"-"` +} + +// LoadMCPMarkMeta reads one per-task meta.json. +// +// A missing or malformed file is a REPORTED ERROR, never a zero-valued record. +// The distinction matters more than it looks: a run over hundreds of tasks +// where a handful of meta files failed to write would, under a zero-filling +// loader, publish a cheaper-and-less-successful-looking number with no trace +// of why — and nothing downstream could recover the difference between "this +// task cost nothing" and "we could not read what it cost". +func LoadMCPMarkMeta(path string) (*MCPMarkMeta, error) { + data, err := os.ReadFile(path) //nolint:gosec // operator-supplied suite output path + if err != nil { + return nil, fmt.Errorf("read mcpmark meta %q: %w", path, err) + } + var meta MCPMarkMeta + if err := json.Unmarshal(data, &meta); err != nil { + return nil, fmt.Errorf("decode mcpmark meta %q: %w", path, err) + } + meta.SourcePath = path + return &meta, nil +} + +// Completion maps the suite's verdict onto the report vocabulary. An absent +// success field is no-signal, never a failure. +func (m *MCPMarkMeta) Completion() string { + switch { + case m.ExecutionResult.Success == nil: + return CompletionNoSignal + case *m.ExecutionResult.Success: + return CompletionPass + default: + return CompletionFail + } +} + +// UnitRecord projects one meta.json onto a UnitRecord. +// +// The resulting usage has NO cache-read count, and that is not an oversight to +// be tidied away with a zero: the suite's output has no such field at all +// (research.md §5). FR-014's third axis has to come from the driver's own +// capture of provider responses, and a record built from meta.json alone +// carries the gap honestly so the report can say so. +func (m *MCPMarkMeta) UnitRecord(cellID, unitID string, runIndex int) (*UnitRecord, error) { + if m == nil { + return nil, errors.New("mcpmark meta: nil") + } + if m.TokenUsage == nil { + return nil, fmt.Errorf("mcpmark meta %q: no token_usage object; a cost figure cannot be produced and "+ + "a zero here would understate cost", m.SourcePath) + } + input := firstNonNil(m.TokenUsage.Input, m.TokenUsage.InputAlt) + output := firstNonNil(m.TokenUsage.Output, m.TokenUsage.OutputAlt) + if input == nil || output == nil { + return nil, fmt.Errorf("mcpmark meta %q: token_usage is missing the input and/or output count; "+ + "defaulting either to zero would understate cost", m.SourcePath) + } + usage := ProviderUsage{ + InputTokens: *input, + OutputTokens: *output, + // Deliberately nil: the suite reports no cache-read figure. + CacheReadTokens: nil, + Responses: 1, + } + if r := firstNonNil(m.TokenUsage.Reasoning, m.TokenUsage.ReasoningAlt); r != nil { + usage.ReasoningTokens = *r + } + + rec := &UnitRecord{ + CellID: cellID, + UnitID: unitID, + RunIndex: runIndex, + Completion: m.Completion(), + Usage: usage, + } + if m.TurnCount != nil { + rec.TurnCount = *m.TurnCount + } + return rec, nil +} + +// mcpMarkMessage is the subset of one trajectory message this driver reads. +type mcpMarkMessage struct { + Role string `json:"role"` + ToolCalls []struct { + ID string `json:"id"` + Function struct { + Name string `json:"name"` + Arguments json.RawMessage `json:"arguments"` + } `json:"function"` + } `json:"tool_calls"` + ToolCallID string `json:"tool_call_id"` + IsError *bool `json:"is_error"` + ErrorType string `json:"error_type"` + Content json.RawMessage `json:"content"` +} + +// LoadMCPMarkTrajectory reads the suite's messages.json and reconstructs the +// ordered ATTEMPT sequence that retry classification needs. +// +// Why the trajectory and not just meta.json: meta.json says whether the task +// passed and what it cost, but first-attempt success and corrective-versus- +// infrastructure retries are properties of the SEQUENCE of calls. Without the +// trajectory, FR-010 and FR-011 are unmeasurable and the whole "does this mode +// help the agent succeed" question collapses back into token counting. +// +// Two classification honesty notes: +// +// - The intent is the tool NAME and the arguments are the fingerprint. This +// is an approximation of "intent" — two calls to the same tool for +// genuinely different purposes read as one intent — and it is the +// approximation that errs toward counting MORE corrective retries, i.e. +// against mcpproxy rather than for it. +// - An error whose type is not recognisably a transport failure is +// classified as a MODE error, not an infrastructure one, for the same +// reason: the conservative direction is the one that does not flatter us. +// +// A trailing tool call with no result marks the record truncated (the caller +// sets Partial); an unanswered call with later calls after it is a malformed +// trajectory and an error, because it means results were dropped rather than +// cut off. +func LoadMCPMarkTrajectory(path string) (attempts []Attempt, truncated bool, err error) { + data, rerr := os.ReadFile(path) //nolint:gosec // operator-supplied suite output path + if rerr != nil { + return nil, false, fmt.Errorf("read mcpmark trajectory %q: %w", path, rerr) + } + var msgs []mcpMarkMessage + if uerr := json.Unmarshal(data, &msgs); uerr != nil { + return nil, false, fmt.Errorf("decode mcpmark trajectory %q: %w", path, uerr) + } + + type pending struct { + index int // position in the attempts slice + issuedAt int // index of the message that issued the call + } + byCallID := make(map[string]pending) + seqByIntent := make(map[string]int) + resolved := make(map[string]bool) + lastIssuedAt := -1 + + for mi, msg := range msgs { + for _, tc := range msg.ToolCalls { + name := tc.Function.Name + if name == "" { + return nil, false, fmt.Errorf("mcpmark trajectory %q: message %d issues a tool call with no name", path, mi) + } + seqByIntent[name]++ + attempts = append(attempts, Attempt{ + IntentID: name, + Seq: seqByIntent[name], + Outcome: AttemptOutcomeUnknown, + ArgsFingerprint: normalizeArguments(tc.Function.Arguments), + }) + if tc.ID != "" { + byCallID[tc.ID] = pending{index: len(attempts) - 1, issuedAt: mi} + } + lastIssuedAt = mi + } + if msg.ToolCallID == "" { + continue + } + p, ok := byCallID[msg.ToolCallID] + if !ok { + return nil, false, fmt.Errorf("mcpmark trajectory %q: message %d is a result for unknown tool call %q", + path, mi, msg.ToolCallID) + } + attempts[p.index].Outcome = classifyResultOutcome(msg) + resolved[msg.ToolCallID] = true + } + + for id, p := range byCallID { + if resolved[id] { + continue + } + if p.issuedAt == lastIssuedAt { + truncated = true + continue + } + return nil, false, fmt.Errorf("mcpmark trajectory %q: tool call %q has no result but later calls follow it; "+ + "results were dropped rather than cut off", path, id) + } + return attempts, truncated, nil +} + +// transportErrorTypes are the error_type values read as INFRASTRUCTURE. The +// list is deliberately short: an unrecognised failure is classified as a mode +// error, which counts against mcpproxy rather than for it. +var transportErrorTypes = map[string]bool{ + "transport": true, + "network": true, + "connection": true, + "timeout": true, +} + +// classifyResultOutcome maps one tool-result message onto an attempt outcome. +func classifyResultOutcome(msg mcpMarkMessage) string { + isErr := msg.IsError != nil && *msg.IsError + if !isErr { + // MCP results carry their own isError flag inside the content object. + var content struct { + IsError *bool `json:"isError"` + } + if json.Unmarshal(msg.Content, &content) == nil && content.IsError != nil { + isErr = *content.IsError + } + } + if !isErr { + return AttemptOutcomeOK + } + if transportErrorTypes[strings.ToLower(strings.TrimSpace(msg.ErrorType))] { + return AttemptOutcomeInfrastructure + } + return AttemptOutcomeError +} + +// normalizeArguments renders a tool call's arguments as a comparable +// fingerprint. The suite may carry them as a JSON-encoded STRING (the OpenAI +// shape) or as an object; both are reduced to their textual form so "same +// arguments" is decidable either way. No argument VALUE ever leaves this +// package in a report — the fingerprint is used only for the equality test. +func normalizeArguments(raw json.RawMessage) string { + if len(raw) == 0 { + return "" + } + var s string + if err := json.Unmarshal(raw, &s); err == nil { + return s + } + return string(raw) +} + +// CompletionRegressionThresholdPct is the stated FR-019 threshold for a block. +// +// A named accessor rather than a bare constant so the value a verdict uses and +// the value AgentLoopBlockFor applied cannot drift apart — they must be the +// same number or a cell can be flagged by one and cleared by the other. +func CompletionRegressionThresholdPct(b *AgentLoopBlock) float64 { + if b != nil && b.CompletionRegressionThresholdPct > 0 { + return b.CompletionRegressionThresholdPct + } + return DefaultCompletionRegressionThresholdPct +} diff --git a/bench/agentloop_test.go b/bench/agentloop_test.go new file mode 100644 index 000000000..ece5302d4 --- /dev/null +++ b/bench/agentloop_test.go @@ -0,0 +1,943 @@ +package bench + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "strings" + "testing" +) + +// Spec 103 US2 — T034-T038: the live agent-loop driver's arithmetic and its +// classification of the spec's BINDING Definitions. +// +// Every test here runs with NO suite, NO model and NO network: the driver is +// parameterised over function types (the bench/flipgate.go precedent) so the +// arithmetic that decides what gets published is exercised by fakes. A test +// that needed a paid run would never be run, and an unrun test gates nothing. + +const ( + testProvider = "test-provider" + testModel = "test-model-2026-01-01" +) + +func testAttempt(intent string, seq int, outcome, args string) Attempt { + return Attempt{IntentID: intent, Seq: seq, Outcome: outcome, ArgsFingerprint: args} +} + +func testUnit(cellID, unitID string, run int, completion string, attempts []Attempt, in, out int, cacheRead *int) UnitRecord { + return UnitRecord{ + CellID: cellID, + UnitID: unitID, + RunIndex: run, + Completion: completion, + Attempts: attempts, + Usage: ProviderUsage{ + InputTokens: in, + OutputTokens: out, + CacheReadTokens: cacheRead, + Responses: 1, + }, + } +} + +func intPtr(v int) *int { return &v } + +// testOpts builds valid options: the FR-020 baseline cell plus the given proxy +// cells, a pinned provider+model, and a fleet shape (IC-004). +func testOpts(proxy ...ModeCell) AgentLoopOptions { + cells := append([]ModeCell{BaselineCell()}, proxy...) + return AgentLoopOptions{ + Suite: "mcpmark", + SuiteVersion: "deadbeef", + Provider: testProvider, + Model: testModel, + FleetShape: FleetShape{ID: "fleet-45", ToolCount: 45}, + Cells: cells, + Units: []string{"t1"}, + Runs: 1, + } +} + +func cellByID(t *testing.T, b *AgentLoopBlock, id string) AgentLoopCell { + t.Helper() + for _, c := range b.Cells { + if c.CellID == id { + return c + } + } + t.Fatalf("block has no cell %q (cells: %+v)", id, b.Cells) + return AgentLoopCell{} +} + +// --------------------------------------------------------------------------- +// T034 — the binding Definitions +// --------------------------------------------------------------------------- + +func TestClassifyIntents_FirstAttemptSuccessDefinition(t *testing.T) { + tests := []struct { + name string + attempts []Attempt + wantSuccess bool + wantIndet bool + wantCorrect int + wantInfra int + }{ + { + name: "single non-error attempt is a first-attempt success", + attempts: []Attempt{testAttempt("read", 1, AttemptOutcomeOK, `{"p":"/a"}`)}, + wantSuccess: true, + }, + { + // The binding Definition's explicit carve-out: a schema-valid call + // the agent immediately re-issues DIFFERENTLY is NOT a success. + name: "non-error call re-issued differently is not a success", + attempts: []Attempt{ + testAttempt("read", 1, AttemptOutcomeOK, `{"p":"/a"}`), + testAttempt("read", 2, AttemptOutcomeOK, `{"p":"/b"}`), + }, + wantSuccess: false, + wantCorrect: 1, + }, + { + name: "error then corrected call is not a first-attempt success", + attempts: []Attempt{ + testAttempt("read", 1, AttemptOutcomeError, `{"p":"/a"}`), + testAttempt("read", 2, AttemptOutcomeOK, `{"path":"/a"}`), + }, + wantSuccess: false, + wantCorrect: 1, + }, + { + // Transport retries measure the network, not the mode, so they are + // counted separately AND leave the intent indeterminate rather than + // scoring against the cell that happened to hit a flaky hop. + name: "identical re-issue after a transport failure is an infrastructure retry", + attempts: []Attempt{ + testAttempt("read", 1, AttemptOutcomeInfrastructure, `{"p":"/a"}`), + testAttempt("read", 2, AttemptOutcomeOK, `{"p":"/a"}`), + }, + wantSuccess: false, + wantIndet: true, + wantInfra: 1, + }, + { + name: "changed arguments after a transport failure is corrective, not infrastructure", + attempts: []Attempt{ + testAttempt("read", 1, AttemptOutcomeInfrastructure, `{"p":"/a"}`), + testAttempt("read", 2, AttemptOutcomeOK, `{"path":"/a"}`), + }, + wantSuccess: false, + wantIndet: true, + wantCorrect: 1, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := ClassifyIntents(tc.attempts) + if len(got) != 1 { + t.Fatalf("ClassifyIntents returned %d intents, want 1: %+v", len(got), got) + } + o := got[0] + if o.FirstAttemptSuccess != tc.wantSuccess { + t.Errorf("FirstAttemptSuccess = %v, want %v", o.FirstAttemptSuccess, tc.wantSuccess) + } + if o.Indeterminate != tc.wantIndet { + t.Errorf("Indeterminate = %v, want %v", o.Indeterminate, tc.wantIndet) + } + if o.RetriesCorrective != tc.wantCorrect { + t.Errorf("RetriesCorrective = %d, want %d", o.RetriesCorrective, tc.wantCorrect) + } + if o.RetriesInfrastructure != tc.wantInfra { + t.Errorf("RetriesInfrastructure = %d, want %d", o.RetriesInfrastructure, tc.wantInfra) + } + }) + } +} + +// The classification must be a property of the ATTEMPT SEQUENCE alone. If the +// baseline arm and the proxy arms were classified by different rules — because +// one carries richer error signal than the other — the comparison would be +// biased toward whichever surface reports failure more legibly, which is the +// single easiest way to fake a favourable result. +func TestClassifyIntents_SameRuleForBaselineAndProxyArms(t *testing.T) { + attempts := []Attempt{ + testAttempt("read", 1, AttemptOutcomeError, `{"p":"/a"}`), + testAttempt("read", 2, AttemptOutcomeOK, `{"path":"/a"}`), + testAttempt("write", 1, AttemptOutcomeOK, `{"path":"/b"}`), + } + + opts := testOpts(mustCell(t, CellRetrieveFull)) + records := []UnitRecord{ + testUnit(CellBaseline, "t1", 0, CompletionPass, attempts, 100, 10, intPtr(0)), + testUnit(CellRetrieveFull, "t1", 0, CompletionPass, attempts, 100, 10, intPtr(0)), + } + + block, err := AgentLoopBlockFor(records, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + base := cellByID(t, block, CellBaseline) + proxy := cellByID(t, block, CellRetrieveFull) + + // Compare VALUES, not pointers. The figure is optional (nil = not + // measured), so a bare != compares addresses and always differs. + if (base.FirstAttemptSuccessPct == nil) != (proxy.FirstAttemptSuccessPct == nil) { + t.Errorf("one arm measured first-attempt success and the other did not: baseline=%v proxy=%v", + base.FirstAttemptSuccessPct, proxy.FirstAttemptSuccessPct) + } else if base.FirstAttemptSuccessPct != nil && + *base.FirstAttemptSuccessPct != *proxy.FirstAttemptSuccessPct { + t.Errorf("first-attempt success differs across arms for identical attempts: baseline %v, proxy %v", + *base.FirstAttemptSuccessPct, *proxy.FirstAttemptSuccessPct) + } + if base.RetriesCorrective != proxy.RetriesCorrective || base.RetriesInfrastructure != proxy.RetriesInfrastructure { + t.Errorf("retry counts differ across arms for identical attempts: baseline %d/%d, proxy %d/%d", + base.RetriesCorrective, base.RetriesInfrastructure, + proxy.RetriesCorrective, proxy.RetriesInfrastructure) + } + // 2 intents, one corrected and one clean. + if base.FirstAttemptSuccessPct == nil { + t.Fatal("FirstAttemptSuccessPct must be measured here — 2 determinate intents were supplied") + } else if *base.FirstAttemptSuccessPct != 50 { + t.Errorf("FirstAttemptSuccessPct = %v, want 50 (1 of 2 intents clean)", *base.FirstAttemptSuccessPct) + } +} + +// UNIT OF WORK is one task, never an individual tool call. A unit with many +// attempts is still exactly one unit in the completion denominator. +func TestAgentLoopBlockFor_UnitOfWorkIsTaskNotToolCall(t *testing.T) { + attempts := []Attempt{ + testAttempt("a", 1, AttemptOutcomeError, `{}`), + testAttempt("a", 2, AttemptOutcomeOK, `{"x":1}`), + testAttempt("b", 1, AttemptOutcomeOK, `{}`), + testAttempt("c", 1, AttemptOutcomeOK, `{}`), + testAttempt("d", 1, AttemptOutcomeOK, `{}`), + } + opts := testOpts() + records := []UnitRecord{ + testUnit(CellBaseline, "t1", 0, CompletionPass, attempts, 100, 10, intPtr(0)), + } + block, err := AgentLoopBlockFor(records, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + base := cellByID(t, block, CellBaseline) + if base.CompletionRatePct != 100 { + t.Errorf("CompletionRatePct = %v, want 100 — one completed task with five attempts is ONE unit of work", + base.CompletionRatePct) + } + if base.TokensPerCompletedTask != 110 { + t.Errorf("TokensPerCompletedTask = %v, want 110 (per TASK, not per call)", base.TokensPerCompletedTask) + } +} + +// --------------------------------------------------------------------------- +// T035 — no completion signal excludes, never assumes +// --------------------------------------------------------------------------- + +func TestAgentLoopBlockFor_NoSignalIsExcludedNotCounted(t *testing.T) { + ok := []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)} + opts := testOpts() + opts.Units = []string{"t1", "t2", "t3"} + records := []UnitRecord{ + testUnit(CellBaseline, "t1", 0, CompletionPass, ok, 100, 0, intPtr(0)), + testUnit(CellBaseline, "t2", 0, CompletionFail, ok, 100, 0, intPtr(0)), + testUnit(CellBaseline, "t3", 0, CompletionNoSignal, ok, 900, 0, intPtr(0)), + } + block, err := AgentLoopBlockFor(records, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + base := cellByID(t, block, CellBaseline) + + // 1 pass of 2 units carrying a verdict. Counting the no-signal unit as a + // failure would give 33.3; counting it as a success would give 66.7. + if base.CompletionRatePct != 50 { + t.Errorf("CompletionRatePct = %v, want 50 — the no-signal unit must be EXCLUDED, not scored", + base.CompletionRatePct) + } + // Its 900 tokens must not enter a completion-dependent figure either. + if base.TokensPerCompletedTask != 200 { + t.Errorf("TokensPerCompletedTask = %v, want 200 — the no-signal unit's tokens must be excluded too", + base.TokensPerCompletedTask) + } + if base.InputTokens != 200 { + t.Errorf("InputTokens = %v, want 200 (the reported totals must reconcile with the headline)", base.InputTokens) + } +} + +// --------------------------------------------------------------------------- +// T036 — FR-021: a model-dependent figure needs runs >= 4 +// --------------------------------------------------------------------------- + +func TestAgentLoopBlockFor_RunsBelowFourIsRefusedAsHeadline(t *testing.T) { + ok := []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)} + build := func(runs int) *AgentLoopBlock { + t.Helper() + opts := testOpts() + opts.Runs = runs + var records []UnitRecord + for r := 0; r < runs; r++ { + // Vary the cost slightly so a spread is computable. + records = append(records, testUnit(CellBaseline, "t1", r, CompletionPass, ok, 100+r, 0, intPtr(0))) + } + block, err := AgentLoopBlockFor(records, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor(runs=%d): %v", runs, err) + } + return block + } + + for _, runs := range []int{1, 2, 3} { + c := cellByID(t, build(runs), CellBaseline) + if c.Runs != runs { + t.Errorf("runs=%d: Runs = %d, want %d", runs, c.Runs, runs) + } + if c.Headline { + t.Errorf("runs=%d: Headline = true, want false — FR-021 requires at least %d runs", + runs, MinHeadlineRuns) + } + } + c := cellByID(t, build(4), CellBaseline) + if !c.Headline { + t.Errorf("runs=4: Headline = false, want true (%+v)", c) + } + if c.SpreadPct == nil { + t.Errorf("runs=4: SpreadPct is nil, but four runs give a DEFINED spread — FR-021 requires a " + + "measure of consistency beside the average") + } else if *c.SpreadPct <= 0 { + t.Errorf("runs=4: SpreadPct = %v, want > 0 — FR-021 requires a measure of consistency beside the average", + *c.SpreadPct) + } +} + +// --------------------------------------------------------------------------- +// T037 — SC-007: cheaper but completing less is a REGRESSION, not a saving +// --------------------------------------------------------------------------- + +func TestAgentLoopBlockFor_DegradedModeIsRegressionNotSaving(t *testing.T) { + ok := []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)} + opts := testOpts(mustCell(t, CellRetrieveCompact)) + opts.Units = []string{"t1", "t2", "t3", "t4"} + opts.Runs = 4 + + var records []UnitRecord + for r := 0; r < 4; r++ { + // Baseline: 4 of 4 completed, 1000 tokens each. + for i, u := range opts.Units { + records = append(records, testUnit(CellBaseline, u, r, CompletionPass, ok, 1000+i, 0, intPtr(0))) + } + // Degraded mode: MUCH cheaper (100 tokens) but completes only 2 of 4. + for i, u := range opts.Units { + verdict := CompletionPass + if i >= 2 { + verdict = CompletionFail + } + records = append(records, testUnit(CellRetrieveCompact, u, r, verdict, ok, 100+i, 0, intPtr(0))) + } + } + + block, err := AgentLoopBlockFor(records, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + degraded := cellByID(t, block, CellRetrieveCompact) + + if !degraded.Regression { + t.Fatalf("degraded cell not flagged as a regression: completion %v vs baseline %v (%+v)", + degraded.CompletionRatePct, cellByID(t, block, CellBaseline).CompletionRatePct, degraded) + } + verdict := block.SavingVsBaseline(CellRetrieveCompact) + if !verdict.Withheld { + t.Errorf("SavingVsBaseline(%s) reported a saving of %v%% for a regressed mode; it must be WITHHELD (FR-019/SC-007)", + CellRetrieveCompact, verdict.SavingPct) + } + if verdict.SavingPct != 0 { + t.Errorf("withheld verdict still carries SavingPct = %v; a withheld figure has no value", verdict.SavingPct) + } + if !strings.Contains(strings.ToLower(verdict.WithheldReason), "regression") { + t.Errorf("WithheldReason = %q, want it to state the regression", verdict.WithheldReason) + } + // Every percentage carries its fleet shape (IC-004). + if verdict.FleetShape.ID != "fleet-45" || verdict.FleetShape.ToolCount != 45 { + t.Errorf("verdict.FleetShape = %+v, want the block's fleet shape", verdict.FleetShape) + } + + // A mode that is cheaper AND keeps completion is a genuine saving — the + // control that proves the regression flag is not simply always set. + var good []UnitRecord + for r := 0; r < 4; r++ { + for i, u := range opts.Units { + good = append(good, testUnit(CellBaseline, u, r, CompletionPass, ok, 1000+i, 0, intPtr(0))) + good = append(good, testUnit(CellRetrieveCompact, u, r, CompletionPass, ok, 500+i, 0, intPtr(0))) + } + } + block2, err := AgentLoopBlockFor(good, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor (control): %v", err) + } + if c := cellByID(t, block2, CellRetrieveCompact); c.Regression { + t.Errorf("non-degraded cell flagged as a regression (%+v)", c) + } + if v := block2.SavingVsBaseline(CellRetrieveCompact); v.Withheld || v.SavingPct <= 0 { + t.Errorf("SavingVsBaseline(control) = %+v, want a real positive saving", v) + } +} + +// --------------------------------------------------------------------------- +// T038 — a cross-accounting-source aggregate is WITHHELD, never computed +// --------------------------------------------------------------------------- + +func TestAggregateTokenFigures_CrossSourceIsWithheld(t *testing.T) { + tokenizer := AccountingSource{Kind: AccountingKindTokenizer, Identity: DefaultEncoding} + provider := AccountingSource{Kind: AccountingKindProvider, Identity: testProvider, Model: testModel} + + mixed := AggregateTokenFigures([]SourcedFigure{ + {Value: 1000, Source: tokenizer}, + {Value: 250, Source: provider}, + }) + if !mixed.Withheld { + t.Fatalf("cross-source aggregate was COMPUTED (%v); it must be withheld with a reason", mixed.Value) + } + if mixed.Value != 0 { + t.Errorf("withheld aggregate carries Value = %v; a withheld figure has no value", mixed.Value) + } + if mixed.WithheldReason == "" { + t.Error("withheld aggregate carries no reason — 'withheld with a stated reason' is the whole rule") + } + for _, want := range []string{AccountingKindTokenizer, AccountingKindProvider} { + if !strings.Contains(mixed.WithheldReason, want) { + t.Errorf("WithheldReason = %q, want it to name the source %q", mixed.WithheldReason, want) + } + } + + // Same source: summed, and the source travels with the result. + same := AggregateTokenFigures([]SourcedFigure{ + {Value: 100, Source: provider}, + {Value: 250, Source: provider}, + }) + if same.Withheld { + t.Fatalf("same-source aggregate was withheld: %q", same.WithheldReason) + } + if same.Value != 350 { + t.Errorf("Value = %v, want 350", same.Value) + } + if same.Source != provider { + t.Errorf("Source = %+v, want %+v", same.Source, provider) + } + + // The same provider on a DIFFERENT model is a different source: usage from + // two models is not one number. + otherModel := provider + otherModel.Model = "some-other-model" + crossModel := AggregateTokenFigures([]SourcedFigure{ + {Value: 100, Source: provider}, + {Value: 100, Source: otherModel}, + }) + if !crossModel.Withheld { + t.Errorf("aggregate across two pinned models was computed (%v); models are not interchangeable", crossModel.Value) + } + + // An unpopulated source cannot enter any aggregate at all. + unset := AggregateTokenFigures([]SourcedFigure{{Value: 5, Source: AccountingSource{}}}) + if !unset.Withheld { + t.Errorf("aggregate over an unset accounting source was computed (%v)", unset.Value) + } +} + +// --------------------------------------------------------------------------- +// T040 — the baseline arm bypasses mcpproxy entirely +// --------------------------------------------------------------------------- + +func TestAgentLoopBlockFor_BaselineArmIsMandatoryAndBypassesProxy(t *testing.T) { + ok := []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)} + + // No baseline cell configured: there is no denominator, so no percentage + // can be published (FR-020). + noBase := testOpts() + noBase.Cells = []ModeCell{mustCell(t, CellRetrieveFull)} + if _, err := AgentLoopBlockFor( + []UnitRecord{testUnit(CellRetrieveFull, "t1", 0, CompletionPass, ok, 10, 0, intPtr(0))}, + noBase, + ); !errors.Is(err, ErrAgentLoopBaselineMissing) { + t.Errorf("AgentLoopBlockFor without a baseline cell: err = %v, want ErrAgentLoopBaselineMissing", err) + } + + // A "baseline" pointed at an mcpproxy endpoint is not a baseline: it is a + // sixth proxy cell wearing the denominator's name. + fake := testOpts() + fake.Cells[0].Endpoint = EndpointDirect + if _, err := AgentLoopBlockFor( + []UnitRecord{testUnit(CellBaseline, "t1", 0, CompletionPass, ok, 10, 0, intPtr(0))}, + fake, + ); !errors.Is(err, ErrBaselineNotBypassing) { + t.Errorf("AgentLoopBlockFor with an endpoint-bearing baseline: err = %v, want ErrBaselineNotBypassing", err) + } + + // A baseline that produced no usable records: every proxy percentage is + // withheld rather than measured against nothing. + opts := testOpts(mustCell(t, CellRetrieveFull)) + block, err := AgentLoopBlockFor( + []UnitRecord{testUnit(CellRetrieveFull, "t1", 0, CompletionPass, ok, 10, 0, intPtr(0))}, + opts, + ) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + v := block.SavingVsBaseline(CellRetrieveFull) + if !v.Withheld || !strings.Contains(strings.ToLower(v.WithheldReason), "baseline") { + t.Errorf("saving against an absent baseline = %+v, want withheld naming the missing baseline", v) + } +} + +// The driver must actually RUN the baseline arm alongside the proxy cells. +func TestRunAgentLoop_DrivesBaselineAndProxyOverTheSameTaskSet(t *testing.T) { + opts := testOpts(mustCell(t, CellRetrieveFull)) + opts.Units = []string{"t1", "t2"} + opts.Runs = 4 + + var seen []string + driver := AgentLoopDriver{ + RunTask: func(cell ModeCell, unitID string, runIndex int) (*UnitRecord, error) { + seen = append(seen, cell.ID+"/"+unitID) + return &UnitRecord{ + CellID: cell.ID, + UnitID: unitID, + RunIndex: runIndex, + Completion: CompletionPass, + Attempts: []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)}, + Usage: ProviderUsage{InputTokens: 100 + runIndex, OutputTokens: 10, CacheReadTokens: intPtr(5), Responses: 1}, + }, nil + }, + } + block, err := RunAgentLoop(driver, opts) + if err != nil { + t.Fatalf("RunAgentLoop: %v", err) + } + if len(seen) != 2*2*4 { + t.Errorf("driver invoked %d times, want %d (2 cells x 2 units x 4 runs)", len(seen), 2*2*4) + } + base := cellByID(t, block, CellBaseline) + if !base.Headline { + t.Errorf("baseline cell is not a headline after 4 runs: %+v", base) + } + if base.CacheReadTokens == 0 { + t.Error("driver-captured cache-read tokens did not reach the block") + } +} + +// --------------------------------------------------------------------------- +// T041 — MCPMark per-task meta.json ingestion +// --------------------------------------------------------------------------- + +func writeMeta(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "meta.json") + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + return path +} + +func TestLoadMCPMarkMeta_HappyPath(t *testing.T) { + path := writeMeta(t, `{ + "task_name": "filesystem/find_file", + "execution_result": {"success": true}, + "token_usage": {"input": 12000, "output": 340, "total": 12340, "reasoning": 0}, + "turn_count": 7 + }`) + meta, err := LoadMCPMarkMeta(path) + if err != nil { + t.Fatalf("LoadMCPMarkMeta: %v", err) + } + rec, err := meta.UnitRecord(CellRetrieveFull, "filesystem/find_file", 0) + if err != nil { + t.Fatalf("UnitRecord: %v", err) + } + if rec.Completion != CompletionPass { + t.Errorf("Completion = %q, want %q", rec.Completion, CompletionPass) + } + if rec.Usage.InputTokens != 12000 || rec.Usage.OutputTokens != 340 { + t.Errorf("usage = %+v, want input 12000 / output 340", rec.Usage) + } + if rec.TurnCount != 7 { + t.Errorf("TurnCount = %d, want 7", rec.TurnCount) + } + // T042: the suite's own output has NO cache-read field. It must arrive as + // unavailable, never as a zero that reads like a measurement. + if rec.Usage.CacheReadTokens != nil { + t.Errorf("CacheReadTokens = %v, want nil — meta.json has no cache-read field", + *rec.Usage.CacheReadTokens) + } +} + +func TestLoadMCPMarkMeta_DefensiveParsing(t *testing.T) { + t.Run("missing file is an error, never a zero", func(t *testing.T) { + if _, err := LoadMCPMarkMeta(filepath.Join(t.TempDir(), "nope.json")); err == nil { + t.Fatal("LoadMCPMarkMeta on a missing file returned no error") + } + }) + t.Run("malformed json is an error, never a zero", func(t *testing.T) { + if _, err := LoadMCPMarkMeta(writeMeta(t, `{"execution_result":`)); err == nil { + t.Fatal("LoadMCPMarkMeta on malformed JSON returned no error") + } + }) + t.Run("absent success verdict is no-signal, not failure", func(t *testing.T) { + meta, err := LoadMCPMarkMeta(writeMeta(t, `{"execution_result": {}, "token_usage": {"input": 1, "output": 2}}`)) + if err != nil { + t.Fatalf("LoadMCPMarkMeta: %v", err) + } + rec, err := meta.UnitRecord(CellRetrieveFull, "t1", 0) + if err != nil { + t.Fatalf("UnitRecord: %v", err) + } + if rec.Completion != CompletionNoSignal { + t.Errorf("Completion = %q, want %q — an absent verdict is NOT a failure", + rec.Completion, CompletionNoSignal) + } + }) + t.Run("absent token_usage is an error, never a zero cost", func(t *testing.T) { + meta, err := LoadMCPMarkMeta(writeMeta(t, `{"execution_result": {"success": false}}`)) + if err != nil { + t.Fatalf("LoadMCPMarkMeta: %v", err) + } + if _, err := meta.UnitRecord(CellRetrieveFull, "t1", 0); err == nil { + t.Fatal("UnitRecord with no token_usage returned no error — a zero cost understates in our favour") + } + }) + t.Run("suffixed key spelling is accepted", func(t *testing.T) { + meta, err := LoadMCPMarkMeta(writeMeta(t, + `{"execution_result": {"success": false}, "token_usage": {"input_tokens": 5, "output_tokens": 6}}`)) + if err != nil { + t.Fatalf("LoadMCPMarkMeta: %v", err) + } + rec, err := meta.UnitRecord(CellRetrieveFull, "t1", 0) + if err != nil { + t.Fatalf("UnitRecord: %v", err) + } + if rec.Usage.InputTokens != 5 || rec.Usage.OutputTokens != 6 { + t.Errorf("usage = %+v, want input 5 / output 6", rec.Usage) + } + if rec.Completion != CompletionFail { + t.Errorf("Completion = %q, want %q", rec.Completion, CompletionFail) + } + }) +} + +func TestLoadMCPMarkTrajectory_ClassifiesRetries(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "messages.json") + msgs := []any{ + map[string]any{"role": "assistant", "tool_calls": []any{ + map[string]any{"id": "c1", "function": map[string]any{"name": "fs__read", "arguments": `{"p":"/a"}`}}, + }}, + map[string]any{"role": "tool", "tool_call_id": "c1", "is_error": true, "content": "unknown argument p"}, + map[string]any{"role": "assistant", "tool_calls": []any{ + map[string]any{"id": "c2", "function": map[string]any{"name": "fs__read", "arguments": `{"path":"/a"}`}}, + }}, + map[string]any{"role": "tool", "tool_call_id": "c2", "content": "ok"}, + } + body, _ := json.Marshal(msgs) + if err := os.WriteFile(path, body, 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + + attempts, truncated, err := LoadMCPMarkTrajectory(path) + if err != nil { + t.Fatalf("LoadMCPMarkTrajectory: %v", err) + } + if truncated { + t.Error("truncated = true for a complete trajectory") + } + if len(attempts) != 2 { + t.Fatalf("got %d attempts, want 2: %+v", len(attempts), attempts) + } + outcomes := ClassifyIntents(attempts) + if len(outcomes) != 1 { + t.Fatalf("got %d intents, want 1: %+v", len(outcomes), outcomes) + } + if outcomes[0].FirstAttemptSuccess { + t.Error("FirstAttemptSuccess = true for an errored first attempt") + } + if outcomes[0].RetriesCorrective != 1 { + t.Errorf("RetriesCorrective = %d, want 1", outcomes[0].RetriesCorrective) + } + + t.Run("a trailing unanswered call marks the record truncated", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "messages.json") + body, _ := json.Marshal([]any{ + map[string]any{"role": "assistant", "tool_calls": []any{ + map[string]any{"id": "c1", "function": map[string]any{"name": "fs__read", "arguments": `{}`}}, + }}, + }) + if err := os.WriteFile(p, body, 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + attempts, truncated, err := LoadMCPMarkTrajectory(p) + if err != nil { + t.Fatalf("LoadMCPMarkTrajectory: %v", err) + } + if !truncated { + t.Error("truncated = false for a trajectory whose last call has no result") + } + if len(attempts) != 1 || attempts[0].Outcome != AttemptOutcomeUnknown { + t.Errorf("attempts = %+v, want one attempt with an unknown outcome", attempts) + } + }) + + t.Run("malformed trajectory is an error", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "messages.json") + if err := os.WriteFile(p, []byte(`{"not":"a list"`), 0o600); err != nil { + t.Fatalf("write fixture: %v", err) + } + if _, _, err := LoadMCPMarkTrajectory(p); err == nil { + t.Fatal("LoadMCPMarkTrajectory on malformed JSON returned no error") + } + }) +} + +// --------------------------------------------------------------------------- +// T042 — input / output / cache-read stay separate; unavailable != zero +// --------------------------------------------------------------------------- + +func TestAgentLoopBlockFor_CacheReadUnavailableIsRecordedNotZeroed(t *testing.T) { + ok := []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)} + opts := testOpts() + opts.Runs = 4 + + var suiteOnly []UnitRecord + for r := 0; r < 4; r++ { + // nil cache-read: exactly what the suite's meta.json yields. + suiteOnly = append(suiteOnly, testUnit(CellBaseline, "t1", r, CompletionPass, ok, 100+r, 10, nil)) + } + block, err := AgentLoopBlockFor(suiteOnly, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + c := cellByID(t, block, CellBaseline) + if !c.Withheld { + t.Fatal("cache-read was unavailable but the cell was not marked withheld — a silent zero") + } + if !strings.Contains(strings.ToLower(c.WithheldReason), "cache") { + t.Errorf("WithheldReason = %q, want it to name the missing cache-read axis", c.WithheldReason) + } + if c.Headline { + t.Error("Headline = true for a cell whose token accounting is missing an axis — that understates cost") + } + if c.InputTokens == 0 || c.OutputTokens == 0 { + t.Errorf("input/output were dropped along with cache-read: %+v", c) + } + + // With the driver supplying the axis, the same cell becomes a headline. + var withCache []UnitRecord + for r := 0; r < 4; r++ { + withCache = append(withCache, testUnit(CellBaseline, "t1", r, CompletionPass, ok, 100+r, 10, intPtr(20))) + } + block2, err := AgentLoopBlockFor(withCache, opts) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + c2 := cellByID(t, block2, CellBaseline) + if c2.Withheld { + t.Errorf("cell withheld despite complete token accounting: %q", c2.WithheldReason) + } + if c2.CacheReadTokens != 80 { + t.Errorf("CacheReadTokens = %d, want 80 (4 runs x 20)", c2.CacheReadTokens) + } + if !c2.Headline { + t.Errorf("Headline = false for a complete 4-run cell: %+v", c2) + } + // Cache reads are real tokens and belong in the cost per completed task. + if c2.TokensPerCompletedTask != 131.5 { + t.Errorf("TokensPerCompletedTask = %v, want 131.5 (mean input 101.5 + output 10 + cache 20)", + c2.TokensPerCompletedTask) + } +} + +func TestRunAgentLoop_DriverUsageHookMustSupplyCacheRead(t *testing.T) { + opts := testOpts() + driver := AgentLoopDriver{ + RunTask: func(cell ModeCell, unitID string, runIndex int) (*UnitRecord, error) { + return &UnitRecord{ + CellID: cell.ID, UnitID: unitID, RunIndex: runIndex, + Completion: CompletionPass, + Attempts: []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)}, + Usage: ProviderUsage{InputTokens: 10, OutputTokens: 1, Responses: 1}, + }, nil + }, + // The hook exists to supply the one axis the suite cannot. Returning + // no cache-read from it is a wiring bug, not an unavailable axis. + CaptureUsage: func(ModeCell, string, int) (ProviderUsage, error) { + return ProviderUsage{InputTokens: 10, OutputTokens: 1, Responses: 1}, nil + }, + } + if _, err := RunAgentLoop(driver, opts); !errors.Is(err, ErrDriverCacheReadMissing) { + t.Errorf("RunAgentLoop: err = %v, want ErrDriverCacheReadMissing", err) + } +} + +// --------------------------------------------------------------------------- +// T043 — the block's accounting source names the provider AND the pinned model +// --------------------------------------------------------------------------- + +func TestAgentLoopAccountingSource_RefusesAnUnpinnedModel(t *testing.T) { + if _, err := AgentLoopAccountingSource(testProvider, ""); !errors.Is(err, ErrAgentLoopModelUnpinned) { + t.Errorf("AgentLoopAccountingSource with no model: err = %v, want ErrAgentLoopModelUnpinned", err) + } + if _, err := AgentLoopAccountingSource("", testModel); err == nil { + t.Error("AgentLoopAccountingSource with no provider returned no error") + } + src, err := AgentLoopAccountingSource(testProvider, testModel) + if err != nil { + t.Fatalf("AgentLoopAccountingSource: %v", err) + } + if src.Kind != AccountingKindProvider || src.Identity != testProvider || src.Model != testModel { + t.Errorf("source = %+v, want provider-usage/%s/%s", src, testProvider, testModel) + } +} + +func TestAgentLoopBlockFor_EmitsPopulatedAccountingSource(t *testing.T) { + ok := []Attempt{testAttempt("a", 1, AttemptOutcomeOK, `{}`)} + opts := testOpts() + block, err := AgentLoopBlockFor( + []UnitRecord{testUnit(CellBaseline, "t1", 0, CompletionPass, ok, 10, 1, intPtr(0))}, + opts, + ) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + if block.AccountingSource.IsZero() { + t.Fatal("agent_loop block emitted with an unpopulated accounting_source") + } + if block.AccountingSource.Model != testModel { + t.Errorf("accounting_source.model = %q, want the pinned model %q", block.AccountingSource.Model, testModel) + } + if block.Suite != "mcpmark" || block.SuiteVersion != "deadbeef" { + t.Errorf("suite pin lost: %q/%q", block.Suite, block.SuiteVersion) + } + if block.FleetShape.ToolCount != 45 { + t.Errorf("FleetShape = %+v, want the configured fleet shape", block.FleetShape) + } + for _, c := range block.Cells { + if !IsValidProvenance(c.Provenance) { + t.Errorf("cell %s carries provenance %q, not in the closed enum", c.CellID, c.Provenance) + } + } + // The whole report envelope must validate. + r2 := &ReportV2{AgentLoop: block} + if err := r2.ValidateAdditiveBlocks(); err != nil { + t.Errorf("ValidateAdditiveBlocks: %v", err) + } + + // An unpinned model is refused at emission, not silently emitted. + bad := testOpts() + bad.Model = "" + if _, err := AgentLoopBlockFor( + []UnitRecord{testUnit(CellBaseline, "t1", 0, CompletionPass, ok, 10, 1, intPtr(0))}, + bad, + ); !errors.Is(err, ErrAgentLoopModelUnpinned) { + t.Errorf("AgentLoopBlockFor with an unpinned model: err = %v, want ErrAgentLoopModelUnpinned", err) + } +} + +func mustCell(t *testing.T, id string) ModeCell { + t.Helper() + c, ok := CellByID(id) + if !ok { + t.Fatalf("no mode cell %q", id) + } + return c +} + +// The never-sum rule must hold WHERE THE ADDITION HAPPENS, not only in the +// standalone helper. +// +// Two reviewers independently found the same gap: AggregateTokenFigures was +// exercised by tests but no production path called it, while aggregateCell +// added token integers whose origin nothing checked and AgentLoopBlockFor then +// stamped the caller's provider onto the result. A tokenizer figure mixed into +// a provider run produced a plausible hybrid wearing a provider label. +func TestAgentLoopBlock_MixedAccountingSourcesIsWithheld(t *testing.T) { + provider := AccountingSource{Kind: AccountingKindProvider, Identity: "anthropic", Model: "pinned-1"} + tokenizer := AccountingSource{Kind: AccountingKindTokenizer, Identity: DefaultEncoding} + + rec := func(unit string, src AccountingSource, in, out int) UnitRecord { + cr := 0 + return UnitRecord{ + CellID: CellDirectFull, UnitID: unit, RunIndex: 0, + Completion: CompletionPass, Source: src, + Usage: ProviderUsage{InputTokens: in, OutputTokens: out, CacheReadTokens: &cr}, + } + } + + directCell, ok := CellByID(CellDirectFull) + if !ok { + t.Fatalf("cell %q must exist in the matrix", CellDirectFull) + } + block, err := AgentLoopBlockFor([]UnitRecord{ + rec("u1", provider, 100, 10), + // Same cell, DIFFERENT source. Adding this to the sum above would + // produce a hybrid the block would then label "provider-usage". + rec("u2", tokenizer, 900, 90), + }, AgentLoopOptions{ + Provider: "anthropic", Model: "pinned-1", + Cells: []ModeCell{BaselineCell(), directCell}, + }) + if err != nil { + t.Fatalf("AgentLoopBlockFor: %v", err) + } + + var cell *AgentLoopCell + for i := range block.Cells { + if block.Cells[i].CellID == CellDirectFull { + cell = &block.Cells[i] + } + } + if cell == nil { + t.Fatal("the direct_full cell must appear in the block") + } + if !cell.Withheld { + t.Fatalf("a cell whose records mixed accounting sources must be WITHHELD, got "+ + "input=%d output=%d", cell.InputTokens, cell.OutputTokens) + } + if !strings.Contains(cell.WithheldReason, "accounting source") { + t.Errorf("the reason must name the cause; got %q", cell.WithheldReason) + } + // And the partial total must not leak out as if it were the cell's cost. + if cell.InputTokens != 0 || cell.OutputTokens != 0 { + t.Errorf("a withheld mixed-source cell must not publish a partial total, got in=%d out=%d", + cell.InputTokens, cell.OutputTokens) + } +} + +// SavingVsBaseline must DERIVE the regression, not trust the row's flag. +// +// It is documented as the only sanctioned way to turn two cells into a +// percentage, so its guards have to hold for any block handed to it — including +// one unmarshalled from a report.json or mutated after assembly. +func TestSavingVsBaseline_DerivesRegressionFromCompletionRates(t *testing.T) { + // A hand-built block, as if loaded from disk: the cell completes half as + // many tasks as the baseline, but its Regression flag says otherwise. + block := &AgentLoopBlock{ + AccountingSource: AccountingSource{Kind: AccountingKindProvider, Identity: "anthropic", Model: "pinned-1"}, + Cells: []AgentLoopCell{ + {CellID: CellBaseline, Runs: 4, CompletionRatePct: 100, TokensPerCompletedTask: 1000, Provenance: ProvenanceMeasured}, + {CellID: CellDirectFull, Runs: 4, CompletionRatePct: 50, TokensPerCompletedTask: 100, + Regression: false, Provenance: ProvenanceMeasured}, + }, + } + + v := block.SavingVsBaseline(CellDirectFull) + if !v.Withheld { + t.Fatalf("a cell completing 50%% against a 100%% baseline must not yield a saving; got %.1f%%", + v.SavingPct) + } + if !v.Regression { + t.Error("the verdict must mark the regression it derived, not echo the row's flag") + } + if v.SavingPct != 0 { + t.Errorf("a withheld verdict must carry no percentage, got %.1f", v.SavingPct) + } +} diff --git a/bench/cmd/warmtiktoken/main.go b/bench/cmd/warmtiktoken/main.go new file mode 100644 index 000000000..4cdcb8d0a --- /dev/null +++ b/bench/cmd/warmtiktoken/main.go @@ -0,0 +1,37 @@ +// Command warmtiktoken populates the tiktoken vocabulary cache before a +// parallel test run. +// +// tiktoken-go downloads the BPE on first use and finishes with an atomic +// rename into a shared cache directory. `go test ./...` runs packages in +// parallel, so several test binaries race to download the same file; on Windows +// the loser hits "Access is denied" on the rename and its whole package fails. +// It surfaced as bench/arms flaking with no code change anywhere near it. +// +// Setting TIKTOKEN_CACHE_DIR alone does not fix that — it names a cache, and an +// empty one still races. Running this once first does, because every later +// construction is a cache hit that renames nothing. +// +// It doubles as the SC-004 offline prerequisite: an outsider reproducing the +// deterministic figures needs the vocabulary present, and "warm the cache" is +// easier to follow as a command than as a paragraph. +package main + +import ( + "fmt" + "os" + + "github.com/smart-mcp-proxy/mcpproxy-go/bench" +) + +func main() { + tk, err := bench.NewTokenizer(bench.DefaultEncoding) + if err != nil { + fmt.Fprintf(os.Stderr, "warmtiktoken: %v\n", err) + os.Exit(1) + } + // Encode something so a lazily-initialised encoder is actually exercised + // rather than merely constructed. + n := tk.Count("warm the cache") + fmt.Printf("warmtiktoken: %s ready (%d tokens for the probe string), cache dir %q\n", + bench.DefaultEncoding, n, os.Getenv("TIKTOKEN_CACHE_DIR")) +} diff --git a/bench/live_report.go b/bench/live_report.go index 75d791ce2..d8b1780ff 100644 --- a/bench/live_report.go +++ b/bench/live_report.go @@ -92,7 +92,7 @@ type LiveReport struct { BreakEven *BreakEvenAnalysis `json:"break_even,omitempty"` // SessionEstimates are the FR-019 estimator rows for the measured live // encoding (baseline_json — the proxy's own full-JSON responses). - SessionEstimates []SessionCostEstimate `json:"session_estimates,omitempty"` + SessionEstimates []SessionCostRow `json:"session_estimates,omitempty"` // MCPDiscoveryLatency aggregates the MCP retrieve_tools call latencies // from the per-query response-cost rows (FR-023). It is a DIFFERENT // surface than Latency, which times REST /api/v1/index/search calls — @@ -251,8 +251,12 @@ func RunLive(ctx context.Context, client *LiveClient, golden *GoldenSet, opts Li rep.BreakEven = be // The measured live encoding is the proxy's own full-JSON // response rendering — the baseline_json arm by definition. - rep.SessionEstimates = EstimateSessionCosts(proxyMenuTokens, - map[string]float64{"baseline_json": respCost.Mean}) + // Rows, not bare estimates: no measured outcomes exist on the + // live path yet, so every row comes back badged estimated — which + // is the honest state, and is now VISIBLE in the table rather + // than asserted in a footnote. + rep.SessionEstimates = EstimateSessionCostRows(proxyMenuTokens, + map[string]float64{"baseline_json": respCost.Mean}, nil) } } diff --git a/bench/mcpmark/README.md b/bench/mcpmark/README.md new file mode 100644 index 000000000..1dccff888 --- /dev/null +++ b/bench/mcpmark/README.md @@ -0,0 +1,299 @@ +# MCPMark ↔ mcpproxy integration (spec 103 US2) + +This directory is the **integration guide**, not a copy of the suite. It holds three +things and nothing else: + +| File | What it is | +|------|------------| +| `mcpproxy_arm.patch` | A reviewable diff against a SHA-pinned MCPMark checkout. One seam, mirrored twice. | +| `mcp_config.example.json` | An **example** mcpproxy config standing the whole suite fleet up at once. The real one is never committed. | +| `README.md` | This file: how to apply the patch, what it does and does not do, and the seams still open before a run can be published. | + +**Why a patch and not a vendored suite.** Vendoring MCPMark would put ~140 task +definitions, their fixtures and their verifiers into this repository, where they +would drift from upstream silently and where "which version produced this number" +becomes unanswerable. A patch against a pinned SHA keeps the comparison honest: +the suite is exactly upstream plus a diff you can read in one sitting. + +--- + +## 1. The pin (FR-028) + +| | | +|---|---| +| Repository | | +| Commit | `cd45b7f57923b9b3985467f5139927575f83141c` | +| Commit date | 2026-06-12 | +| License | Apache-2.0 | +| Task set | **MCPMark Verified** — the default at this commit | +| Python | `requires-python >= 3.11` | + +**Report every figure as "MCPMark Verified @ cd45b7f5".** Upstream made Verified +the default in this very commit and states that results from earlier task +versions are deprecated and *not directly comparable*. A number quoted without +the task-set name is therefore not comparable to anything, including our own +earlier runs. + +Task counts at this commit, counted as `description.md` files under `tasks/`: + +| Service | Tasks | Credentials needed | Real writes | +|---------|------:|--------------------|-------------| +| filesystem | 40 | none | local only | +| postgres | 31 | local server | local only | +| github | 33 | PAT | **yes, remote** | +| notion | 38 | integration token | **yes, remote** | +| playwright | 4 | none | browser only | +| playwright_webarena | 31 | WebArena host | depends on host | + +These differ from the counts in `specs/103-token-bench/research.md` +(filesystem 30, postgres 21, github 23, notion 28). Research counted the +pre-Verified task set; this table counts the pinned commit. **Use this table**, +and re-count after any re-pin — the suite grows. + +--- + +## 2. Applying the patch + +```bash +# 1. Clone and pin. Do this OUTSIDE this repository. +git clone https://github.com/eval-sys/mcpmark ~/bench/mcpmark +cd ~/bench/mcpmark +git checkout --detach cd45b7f57923b9b3985467f5139927575f83141c + +# 2. Dry-run first. If this fails the pin has moved; re-pin deliberately, +# do not force the patch. +git apply --check /path/to/mcpproxy-go/bench/mcpmark/mcpproxy_arm.patch + +# 3. Apply. +git apply /path/to/mcpproxy-go/bench/mcpmark/mcpproxy_arm.patch + +# 4. Confirm it is the only difference from the pin. +git diff --stat # expect: 2 files changed, 132 insertions(+) + +# To revert: git checkout -- src/agents/ +``` + +Install the suite per its own README (it uses `uv`); this patch adds no +dependency — `MCPHttpServer` and the official MCP Python SDK streamable-HTTP +transport it wraps are already in the suite. + +### What the patch changes + +MCPMark builds its MCP server in exactly one place, and defines that place +twice: `_create_mcp_server()` in `src/agents/mcpmark_agent.py` and its mirror in +`src/agents/base_agent.py` (which `src/agents/react_agent.py` inherits). The +patch adds, to both: + +* `_mcpproxy_arm_server()` — returns an `MCPHttpServer` pointed at mcpproxy, or + `None`; +* two lines at the top of `_create_mcp_server()` that return it when it is not + `None`. + +`list_tools()` and `call_tool()` are the only calls the agent loop makes against +the server object, and `MCPHttpServer` satisfies both unchanged. Nothing in the +loop, the task set, the state managers or the verifiers is touched. + +### Why the gate is an env var and not a fork + +The FR-020 baseline arm is *the same agent running the same tasks with every +upstream tool loaded directly*. If the baseline ran unpatched code and the proxy +arms ran patched code, the patch would be a variable inside its own comparison. +With `MCPPROXY_MCP_URL` unset, `_mcpproxy_arm_server()` returns `None` and the +stock factory runs byte-for-byte as upstream — **both arms execute the same +file, and only the environment differs.** + +The gate is the URL rather than a boolean because the URL is *also the cell +selector*. mcpproxy mounts one endpoint per routing mode permanently, at +startup, regardless of config (`internal/server/server.go`, "Routing mode +dedicated endpoints"), so the whole matrix crosses on one long-lived instance: + +| Matrix cell | `MCPPROXY_MCP_URL` path | Config needed | +|---|---|---| +| `retrieve_full` | `/mcp/call` | `tool_response_mode: "full"` | +| `retrieve_compact` | `/mcp/call` | `tool_response_mode: "compact"` | +| `direct_full` | `/mcp/all` | `direct_tool_response_mode: "full"` | +| `direct_deferred` | `/mcp/all` | `direct_tool_response_mode: "deferred"` | +| `code_exec` | `/mcp/code` | `enable_code_execution: true` | +| `baseline` (FR-020) | *(unset)* | — the suite spawns the servers itself | + +Both serialization settings hot-reload, so a cell change is a config apply, not +a restart. See `bench/modematrix.go` for the frozen cell ids and the seven +redundant combinations that are reported as skips. + +### Environment + +| Variable | Meaning | +|---|---| +| `MCPPROXY_MCP_URL` | **The gate.** e.g. `http://127.0.0.1:18080/mcp/call`. Unset ⇒ stock suite behaviour. | +| `MCPPROXY_API_KEY` | Sent as the `X-API-Key` **header**. Required when `require_mcp_auth: true`. | +| `MCPPROXY_MCP_TIMEOUT` | Seconds; defaults to the suite's own `DEFAULT_TIMEOUT` (600). | + +Two details that are easy to get wrong: + +* **Header, never `?apikey=`.** The query parameter is a Web-UI-only convenience + on mcpproxy; using it writes the key into every access log and every recorded + activity URL. +* **The suite's `MCPHttpServer` default timeout is 30 s and covers `initialize`, + `list_tools` *and* every `call_tool`.** Behind a proxy, first contact also + brings up N upstream servers (npx / pipx / docker cold starts), which + routinely exceeds 30 s. A timeout there surfaces as a transport error that a + naive classifier reads as a corrective retry — biasing the exact rates this + benchmark exists to measure. The patch raises the default for that reason; + do not lower it back. + +--- + +## 3. The mcpproxy side — `mcp_config.example.json` + +**This file is an EXAMPLE.** Copy it outside the repository, fill the +`REPLACE_ME…` placeholders, and point mcpproxy at the copy. The real file +carries credentials and machine-specific absolute paths and is **never +committed** — the same rule that keeps generated reports out of the tree +(SC-011) applies to its inputs. + +```bash +cp bench/mcpmark/mcp_config.example.json ~/bench/mcpmark-proxy.json +$EDITOR ~/bench/mcpmark-proxy.json # fill every REPLACE_ME +mcpproxy serve --config ~/bench/mcpmark-proxy.json --data-dir ~/bench/mcpmark-datadir +``` + +### Why all five services at once (T048) + +Stock MCPMark connects to **one** server per run, so the per-task fleet is a +single service's toolset — far too small for any proxy mode to differ +measurably from loading everything inline. Configuring the whole fleet while +running one service's tasks is what puts the measurement in the regime the +proxy exists for, and it is *also the honest baseline*: the same agent, the same +tasks, all those tools loaded directly. + +The upstream versions in the example are pinned to the exact ones the suite +would spawn itself (`@modelcontextprotocol/server-filesystem@2025.12.18`, +`postgres-mcp==0.3.0`, `ghcr.io/github/github-mcp-server:v0.15.0`, +`@notionhq/notion-mcp-server@1.9.1`, `@playwright/mcp@0.0.68`). Pinning them to +anything else would compare two different fleets. + +### Config details that decide whether the run is valid + +* **`quarantine_enabled: false` and `"quarantined": false` per server.** A + quarantined server contributes **no callable tools**, so an unnoticed + quarantine silently shrinks the fleet and inflates every saving. Check the + tool count before believing a number. +* **Unknown keys are silently ignored.** mcpproxy's loader rejects malformed + JSON but accepts unknown fields, so `tools_limitt: 15` costs you a run and no + error. Copy the example; do not retype it. +* **`tools_limit` and `tool_response_limit` are pinned in the example** (15 / + 20000, the product defaults). They cap what `retrieve_tools` returns, so they + are part of the measured configuration, not incidental. +* **`require_mcp_auth: true`** makes `/mcp` reject unauthenticated requests. It + is off by default in the product for client compatibility; turn it on here so + a stray local client cannot join the benchmark instance mid-run. +* **`telemetry.enabled: false`** — benchmark traffic is not product usage. +* **Record the fleet shape actually measured**, not the one configured. If + GitHub or Notion fails to connect (bad token, no Docker), their tools are + absent and the fleet is smaller. Every published percentage carries the fleet + shape it was measured on (IC-004, SC-005). + +### Credential-free core + +`filesystem` + `postgres` + `playwright` need no third-party credentials. That is +the FR-027-clean starting point, and a smaller fleet — say so when reporting it. +GitHub and Notion perform **real writes against real accounts**: run them only +against a throwaway account with a narrowly scoped token, or not at all. + +--- + +## 4. Open seam: per-task service state (read before scheduling a run) + +**The transport patch does not close this, and it is not an oversight — it is +the piece that has to be decided before filesystem or postgres tasks can run +through the proxy at all.** + +MCPMark rebinds its service configuration **per task**, and both credential-free +services do it: + +* **filesystem** — `src/mcp_services/filesystem/filesystem_state_manager.py` + copies the test environment into + `/.mcpmark_backups/backup____` for each + task, hands that path to the agent as `test_directory`, and deletes it + afterwards. Stock MCPMark passes it straight into the filesystem server's + argv, so the server's root *is* the per-task directory. Task instructions then + speak in terms of "the test environment root". +* **postgres** — `src/mcp_services/postgres/postgres_state_manager.py` + `get_service_config_for_agent()` returns a `current_database` created for that + task, which stock MCPMark bakes into `DATABASE_URI`. + +A statically configured mcpproxy fleet cannot follow either. Rooting the +proxied filesystem server at `.mcpmark_backups` (as the example config does, for +want of anything better) makes the *parent of every task directory* the root: +the agent no longer sees the task's own root at the top level, and it can see +other tasks' directories. That is fine for standing the fleet up and counting +tools; it is **not** a valid task run. + +Three ways out, none of them free: + +1. **Re-point per task over the REST API.** `PATCH /api/v1/servers/{id}` + deep-merges (omitted keys preserved) and the config hot-reloads, so the + patched factory — which runs once per task, after state setup, with + `self.service_config` already refreshed — is the right seam to call it from: + + ```bash + curl -sS -X PATCH http://127.0.0.1:18080/api/v1/servers/mcpmark_filesystem \ + -H "X-API-Key: $MCPPROXY_API_KEY" -H 'Content-Type: application/json' \ + -d '{"args":["-y","@modelcontextprotocol/server-filesystem@2025.12.18",""]}' + ``` + + Cost: the upstream restarts and re-indexes between tasks, and the run has to + wait for that. **Not yet exercised — treat the curl as the shape, not as a + verified recipe.** +2. **Run the services that do not rebind.** GitHub and Notion carry static + credentials, so their tasks need no per-task rebinding — but they perform + real remote writes (FR-027) and need credentials, which is exactly what the + credential-free core avoided. +3. **Measure the fleet, not the tasks, through the proxy.** Keep the full fleet + configured for menu-cost measurement and run the *tasks* stock. This is + coherent but it is no longer an agent loop through mcpproxy, so it cannot + produce tokens-per-completed-task, and it is not what US2 asks for. + +Whichever is chosen, say which one produced the numbers. + +--- + +## 5. What this integration cannot give you + +* **Cache-read accounting (FR-014).** The suite's per-task `meta.json` carries + `execution_result.success`, `token_usage{input,output,total,reasoning}` and + `turn_count` — and **no cache-read field**. That axis must come from the + driver reading the provider's own `usage`, or be declared out of reach for + suite-driven runs. The patch does not establish it. +* **A cross-source total.** Deterministic figures come from the tiktoken + tokenizer; live figures come from provider-reported usage. They are never + summed; a cross-source aggregate is *withheld with a stated reason*, the same + rule `bench/live_report.go` already applies to withheld headlines. +* **A headline from one run.** Every model-dependent figure is an average over + k ≥ 4 runs with its spread (FR-021). +* **A saving from a mode that completes less.** Lower cost with a lower + completion rate is a regression, not a saving (FR-019, SC-007). + +## 6. Status + +The patch, the example config and this guide are complete and verified offline. +**No suite run has been executed and no model spend has been incurred** — T049 +(run k ≥ 4 across ≥ 2 cells plus the baseline) is a separate task and is gated +on the pinned-model and spend-ceiling decision, and on §4 above. + +### Verifying this directory without spending anything + +```bash +SHA=cd45b7f57923b9b3985467f5139927575f83141c +git clone https://github.com/eval-sys/mcpmark /tmp/mcpmark && cd /tmp/mcpmark +git checkout --detach $SHA +git apply --check /path/to/bench/mcpmark/mcpproxy_arm.patch # applies clean +git apply /path/to/bench/mcpmark/mcpproxy_arm.patch +python3 -m py_compile src/agents/mcpmark_agent.py src/agents/base_agent.py + +# the example config parses under the production loader +jq -e . /path/to/bench/mcpmark/mcp_config.example.json +mcpproxy doctor --config /path/to/bench/mcpmark/mcp_config.example.json +# ⇒ "doctor requires running daemon" means the CONFIG LOADED; +# "failed to load config file" means it did not. +``` diff --git a/bench/mcpmark/mcp_config.example.json b/bench/mcpmark/mcp_config.example.json new file mode 100644 index 000000000..3fe56a12b --- /dev/null +++ b/bench/mcpmark/mcp_config.example.json @@ -0,0 +1,99 @@ +{ + "_example": "EXAMPLE ONLY — spec 103 T048. Copy this OUTSIDE the repository, fill the REPLACE_ME placeholders, and never commit the copy: the real file carries credentials and absolute paths. See bench/mcpmark/README.md.", + "listen": "127.0.0.1:18080", + "data_dir": "REPLACE_ME_ABSOLUTE_PATH/mcpmark-bench-datadir", + "api_key": "REPLACE_ME_WITH_A_RANDOM_STRING", + "require_mcp_auth": true, + "enable_socket": false, + "enable_tray": false, + "check_server_repo": false, + "quarantine_enabled": false, + "enable_code_execution": true, + "tool_response_mode": "full", + "direct_tool_response_mode": "full", + "tools_limit": 15, + "tool_response_limit": 20000, + "top_k": 5, + "telemetry": { + "enabled": false + }, + "mcpServers": [ + { + "name": "mcpmark_filesystem", + "protocol": "stdio", + "command": "npx", + "args": [ + "-y", + "@modelcontextprotocol/server-filesystem@2025.12.18", + "REPLACE_ME_ABSOLUTE_PATH/mcpmark/.mcpmark_backups" + ], + "enabled": true, + "quarantined": false + }, + { + "name": "mcpmark_postgres", + "protocol": "stdio", + "command": "pipx", + "args": [ + "run", + "postgres-mcp==0.3.0", + "--access-mode=unrestricted" + ], + "env": { + "DATABASE_URI": "postgresql://REPLACE_ME_USER:REPLACE_ME_PASSWORD@127.0.0.1:5432/REPLACE_ME_TASK_DATABASE" + }, + "enabled": true, + "quarantined": false + }, + { + "name": "mcpmark_playwright", + "protocol": "stdio", + "command": "npx", + "args": [ + "-y", + "@playwright/mcp@0.0.68", + "--headless", + "--isolated", + "--no-sandbox", + "--browser", + "chromium", + "--viewport-size", + "1280,720" + ], + "enabled": true, + "quarantined": false + }, + { + "name": "mcpmark_github", + "protocol": "stdio", + "command": "docker", + "args": [ + "run", + "-i", + "--rm", + "-e", + "GITHUB_PERSONAL_ACCESS_TOKEN", + "ghcr.io/github/github-mcp-server:v0.15.0" + ], + "env": { + "GITHUB_PERSONAL_ACCESS_TOKEN": "REPLACE_ME_SCOPED_PAT_FOR_A_THROWAWAY_ACCOUNT" + }, + "enabled": true, + "quarantined": false + }, + { + "name": "mcpmark_notion", + "protocol": "stdio", + "command": "npx", + "args": [ + "-y", + "@notionhq/notion-mcp-server@1.9.1" + ], + "env": { + "OPENAPI_MCP_HEADERS": "{\"Authorization\": \"Bearer REPLACE_ME_NOTION_INTEGRATION_TOKEN\", \"Notion-Version\": \"2022-06-28\"}" + }, + "enabled": true, + "quarantined": false + } + ] +} diff --git a/bench/mcpmark/mcpproxy_arm.patch b/bench/mcpmark/mcpproxy_arm.patch new file mode 100644 index 000000000..e5a03e68a --- /dev/null +++ b/bench/mcpmark/mcpproxy_arm.patch @@ -0,0 +1,196 @@ +mcpproxy benchmark arm for MCPMark — spec 103 US2 (T047) + +Applies to: https://github.com/eval-sys/mcpmark +Pinned commit: cd45b7f57923b9b3985467f5139927575f83141c +Apply with: git -C apply /path/to/mcpproxy_arm.patch + +WHAT THIS CHANGES + One seam, mirrored in the two places MCPMark defines it: the MCP server + factory. An env-gated branch returns an HTTP MCP server pointed at + mcpproxy instead of spawning the service's own stdio/HTTP server. + +WHY IT IS A GATE, NOT A REPLACEMENT + With MCPPROXY_MCP_URL unset the patched files behave exactly as upstream. + That is deliberate: FR-020's baseline arm is the same agent running the + same tasks with all tools loaded directly, so it must run the SAME code, + differing only in environment. A patch that could only be applied for the + proxy arms would make itself a variable in its own comparison. + +WHAT IT DOES NOT CHANGE + Nothing in the agent loop, the task set, the state managers or the + verification scripts. list_tools() and call_tool() are the only calls the + loop makes against the server object, and both are satisfied by + MCPHttpServer unchanged. See bench/mcpmark/README.md for the per-task + service-state seam this patch deliberately does NOT close. + +diff --git a/src/agents/base_agent.py b/src/agents/base_agent.py +index dcf7fcd..6c0aa58 100644 +--- a/src/agents/base_agent.py ++++ b/src/agents/base_agent.py +@@ -5,6 +5,7 @@ from __future__ import annotations + import asyncio + import copy + import json ++import os + import uuid + from abc import ABC, abstractmethod + from typing import Any, Dict, List, Optional, Callable +@@ -162,7 +163,72 @@ class BaseMCPAgent(ABC): + # MCP server management + # ------------------------------------------------------------------ + ++ # ------------------------------------------------------------------ ++ # mcpproxy benchmark arm (external: mcpproxy-go spec 103, bench/mcpmark/) ++ # ------------------------------------------------------------------ ++ ++ def _mcpproxy_arm_server(self): ++ """Return an MCP server pointed at mcpproxy, or None when the arm is off. ++ ++ WHY THIS IS A GATE AND NOT A REPLACEMENT: the benchmark's denominator ++ (spec 103 FR-020) is the same agent running the same tasks with every ++ upstream tool loaded directly. If the baseline ran unpatched code and ++ the proxy arms ran patched code, the patch itself would be a variable ++ in the comparison. With MCPPROXY_MCP_URL unset this returns None and ++ the stock factory below runs byte-for-byte as upstream, so both arms ++ execute the same file and only the environment differs. ++ ++ The gate is the URL rather than a boolean because the URL is also the ++ cell selector: mcpproxy mounts one endpoint per routing mode ++ (/mcp/call, /mcp/all, /mcp/code), all of them permanently, so a matrix ++ cell is chosen by pointing this variable at a different path on the ++ same long-lived instance. ++ ++ Env: ++ MCPPROXY_MCP_URL e.g. http://127.0.0.1:18080/mcp/call (the gate) ++ MCPPROXY_API_KEY mcpproxy REST/MCP API key (optional) ++ MCPPROXY_MCP_TIMEOUT seconds; defaults to the suite's own timeout ++ """ ++ url = os.environ.get("MCPPROXY_MCP_URL", "").strip() ++ if not url: ++ return None ++ ++ headers = {"User-Agent": "MCPMark/1.0 (mcpproxy-bench)"} ++ api_key = os.environ.get("MCPPROXY_API_KEY", "").strip() ++ if api_key: ++ # Header, never ?apikey=. The query parameter is a Web-UI-only ++ # convenience on mcpproxy and would write the key into every ++ # access log and every recorded activity URL. ++ headers["X-API-Key"] = api_key ++ ++ # MCPHttpServer's own default is 30 s, and that one value covers ++ # initialize, list_tools AND every call_tool. Behind a proxy the first ++ # contact also brings up N upstream servers (npx/pipx/docker cold ++ # starts), which routinely exceeds 30 s and would fail the run as a ++ # transport error rather than a task failure -- an infrastructure ++ # retry misread as a corrective one biases the very rates this ++ # benchmark measures. Default to the suite's own DEFAULT_TIMEOUT. ++ try: ++ timeout = int(os.environ.get("MCPPROXY_MCP_TIMEOUT", self.DEFAULT_TIMEOUT)) ++ except ValueError: ++ timeout = self.DEFAULT_TIMEOUT ++ ++ logger.info( ++ "| Using mcpproxy arm: %s (service=%s, auth=%s, timeout=%ss)", ++ url, ++ self.mcp_service, ++ "x-api-key" if api_key else "none", ++ timeout, ++ ) ++ return MCPHttpServer(url=url, headers=headers, timeout=timeout) ++ + async def _create_mcp_server(self) -> Any: ++ # mcpproxy benchmark arm: on when MCPPROXY_MCP_URL is set, otherwise ++ # this is a no-op and the stock dispatch below runs untouched. ++ proxied = self._mcpproxy_arm_server() ++ if proxied is not None: ++ return proxied ++ + if self.mcp_service in self.STDIO_SERVICES: + return self._create_stdio_server() + if self.mcp_service in self.HTTP_SERVICES: +diff --git a/src/agents/mcpmark_agent.py b/src/agents/mcpmark_agent.py +index ef9b5df..baac56f 100644 +--- a/src/agents/mcpmark_agent.py ++++ b/src/agents/mcpmark_agent.py +@@ -7,6 +7,7 @@ Unified agent using LiteLLM for all model interactions with minimal MCP support. + + import asyncio + import json ++import os + import time + from typing import Any, Dict, List, Optional, Callable + from pydantic import AnyUrl +@@ -1099,8 +1100,73 @@ class MCPMarkAgent(BaseMCPAgent): + + # ==================== MCP Server Management ==================== + ++ # ------------------------------------------------------------------ ++ # mcpproxy benchmark arm (external: mcpproxy-go spec 103, bench/mcpmark/) ++ # ------------------------------------------------------------------ ++ ++ def _mcpproxy_arm_server(self): ++ """Return an MCP server pointed at mcpproxy, or None when the arm is off. ++ ++ WHY THIS IS A GATE AND NOT A REPLACEMENT: the benchmark's denominator ++ (spec 103 FR-020) is the same agent running the same tasks with every ++ upstream tool loaded directly. If the baseline ran unpatched code and ++ the proxy arms ran patched code, the patch itself would be a variable ++ in the comparison. With MCPPROXY_MCP_URL unset this returns None and ++ the stock factory below runs byte-for-byte as upstream, so both arms ++ execute the same file and only the environment differs. ++ ++ The gate is the URL rather than a boolean because the URL is also the ++ cell selector: mcpproxy mounts one endpoint per routing mode ++ (/mcp/call, /mcp/all, /mcp/code), all of them permanently, so a matrix ++ cell is chosen by pointing this variable at a different path on the ++ same long-lived instance. ++ ++ Env: ++ MCPPROXY_MCP_URL e.g. http://127.0.0.1:18080/mcp/call (the gate) ++ MCPPROXY_API_KEY mcpproxy REST/MCP API key (optional) ++ MCPPROXY_MCP_TIMEOUT seconds; defaults to the suite's own timeout ++ """ ++ url = os.environ.get("MCPPROXY_MCP_URL", "").strip() ++ if not url: ++ return None ++ ++ headers = {"User-Agent": "MCPMark/1.0 (mcpproxy-bench)"} ++ api_key = os.environ.get("MCPPROXY_API_KEY", "").strip() ++ if api_key: ++ # Header, never ?apikey=. The query parameter is a Web-UI-only ++ # convenience on mcpproxy and would write the key into every ++ # access log and every recorded activity URL. ++ headers["X-API-Key"] = api_key ++ ++ # MCPHttpServer's own default is 30 s, and that one value covers ++ # initialize, list_tools AND every call_tool. Behind a proxy the first ++ # contact also brings up N upstream servers (npx/pipx/docker cold ++ # starts), which routinely exceeds 30 s and would fail the run as a ++ # transport error rather than a task failure -- an infrastructure ++ # retry misread as a corrective one biases the very rates this ++ # benchmark measures. Default to the suite's own DEFAULT_TIMEOUT. ++ try: ++ timeout = int(os.environ.get("MCPPROXY_MCP_TIMEOUT", self.DEFAULT_TIMEOUT)) ++ except ValueError: ++ timeout = self.DEFAULT_TIMEOUT ++ ++ logger.info( ++ "| Using mcpproxy arm: %s (service=%s, auth=%s, timeout=%ss)", ++ url, ++ self.mcp_service, ++ "x-api-key" if api_key else "none", ++ timeout, ++ ) ++ return MCPHttpServer(url=url, headers=headers, timeout=timeout) ++ + async def _create_mcp_server(self) -> Any: + """Create and return an MCP server instance.""" ++ # mcpproxy benchmark arm: on when MCPPROXY_MCP_URL is set, otherwise ++ # this is a no-op and the stock dispatch below runs untouched. ++ proxied = self._mcpproxy_arm_server() ++ if proxied is not None: ++ return proxied ++ + if self.mcp_service in self.STDIO_SERVICES: + return self._create_stdio_server() + elif self.mcp_service in self.HTTP_SERVICES: diff --git a/bench/replay.go b/bench/replay.go index 31e7b49eb..e1967d97a 100644 --- a/bench/replay.go +++ b/bench/replay.go @@ -805,6 +805,19 @@ func (b *ReplayBlock) ValidateExclusionBalance() error { // but empty — replay measures no encoding arms over a corpus; it crosses the // mode matrix, which is why it is a block rather than one more OfflineSection. func ReplayReport(tk *Tokenizer, block *ReplayBlock) *ReportV2 { + // US3 (FR-030): a replay is scored over the operator's OWN recorded + // traffic, which cannot be published, so an outsider cannot reproduce its + // figures. Stamping that here rather than leaving it to the caller is the + // difference between an honest limitation and an implied guarantee — and + // the field existed unpopulated until this wiring, which meant every + // replay report silently read as independently reproducible. + if block != nil && block.Inputs == nil { + block.Inputs = PrivateRecordingInputs( + "Scored over a private activity-log recording that is raw user traffic and is never " + + "committed. The deterministic figures reproduce for anyone holding the SAME recording " + + "and the same fleet input, but an outsider cannot obtain the recording, so these " + + "figures must not be the sole support for a published claim (FR-030).") + } return &ReportV2{ ReportVersion: ReportVersion2, GeneratedAt: ReplayGeneratedAt, diff --git a/bench/replay_test.go b/bench/replay_test.go index cb023197d..e6dcbdeeb 100644 --- a/bench/replay_test.go +++ b/bench/replay_test.go @@ -742,3 +742,30 @@ func TestRunReplay_CountsPartiallyStubbedFleet(t *testing.T) { t.Errorf("tool count %d", block.FleetShape.ToolCount) } } + +// FR-030 — a replay report must declare that its figures are NOT independently +// reproducible. +// +// The field existed unpopulated until it was wired, which meant every replay +// report read as reproducible by omission. An outsider cannot obtain the +// operator's recording, so these figures may not be the sole support for a +// published claim, and the report has to say so rather than leave it inferred. +func TestReplayReport_DeclaresPrivateRecordingProvenance(t *testing.T) { + tk := runnerTokenizer(t) + block, err := RunReplay(tk, replayOptions(t)) + if err != nil { + t.Fatalf("RunReplay: %v", err) + } + + rep := ReplayReport(tk, block) + if rep.Replay.Inputs == nil { + t.Fatal("a replay report must carry input provenance — silence reads as reproducible") + } + if rep.Replay.Inputs.IndependentlyReproducible { + t.Error("a figure scored over a private recording is NOT independently reproducible") + } + if !strings.Contains(strings.ToLower(rep.Replay.Inputs.Limitation), "sole support") { + t.Errorf("the limitation must say the figure cannot be the sole support for a published "+ + "claim; got %q", rep.Replay.Inputs.Limitation) + } +} diff --git a/bench/report.go b/bench/report.go index be948ddaa..06f8ee167 100644 --- a/bench/report.go +++ b/bench/report.go @@ -6,6 +6,7 @@ import ( "html/template" "os" "path/filepath" + "strconv" ) // WriteJSON writes the report as indented JSON to path. @@ -67,6 +68,17 @@ func (r *ReportV2) WriteHTML(path string) error { tmpl, err := template.New("dashboardV2").Funcs(template.FuncMap{ "f1": func(f float64) string { return fmt.Sprintf("%.1f", f) }, "pc1": func(f float64) string { return fmt.Sprintf("%.1f%%", f) }, + // n0 renders a token count at full precision without exponent + // notation — a headline cost printed as "4.1e+04" is unreadable. + "n0": func(f float64) string { return strconv.FormatFloat(f, 'f', -1, 64) }, + // derefF unwraps an optional figure for display. The TEMPLATE must + // branch on nil before calling it — a nil figure means "not measured", + // and printing its zero is the silent-zero defect this guards against. + "derefF": derefF, + // costoutcome builds the FR-023 plot. The arithmetic lives in Go + // (NewCostOutcomeView) so a degenerate axis cannot emit NaN into an + // SVG attribute, where it would fail silently. + "costoutcome": NewCostOutcomeView, // prov looks a section's provenance label up for badge rendering; // missing keys render no badge rather than a wrong one. "prov": func(m map[string]string, key string) string { return m[key] }, @@ -176,6 +188,17 @@ const dashboardV2HTML = ` .stat b { display: block; font-size: 1.4rem; } .notes, .small { font-size: .85rem; opacity: .8; } .warn { color: #c0392b; font-weight: 600; } + /* .hl gives the two FR-018 headline figures — cost and completion — + IDENTICAL weight and size, in the table and in the plot legend. Equal + prominence is a layout property, so it is enforced by one shared class + rather than by two hand-matched declarations that can drift apart. */ + .hl { font-weight: 600; font-size: 1.05rem; } + .plot { max-width: 100%; height: auto; margin: 1rem 0; } + .axis { stroke: #8886; stroke-width: 1; } + .grid { stroke: #8883; stroke-width: 1; stroke-dasharray: 3 3; } + .axlabel { font: 11px system-ui, sans-serif; fill: currentColor; opacity: .7; } + .axtitle { font: 12px system-ui, sans-serif; fill: currentColor; opacity: .85; font-weight: 600; } + .ptlabel { font: 11px system-ui, sans-serif; fill: currentColor; } @@ -242,6 +265,71 @@ const dashboardV2HTML = `

Every figure above scores the recorded call shape against the SUPPLIED fleet — today's fleet, not the fleet as it stood when the sessions were recorded. It is internally valid across mode cells and is not a historical reconstruction. generated_at is pinned rather than stamped so two runs over the same inputs are byte-identical.

{{end}} +{{if .AgentLoop}} +

Cost versus outcome — which modes are worth their savings{{with prov .Provenance "agent_loop"}}{{.}}{{end}}

+

accounting source {{.AgentLoop.AccountingSource.Kind}} · {{.AgentLoop.AccountingSource.Identity}}{{if .AgentLoop.AccountingSource.Model}} · pinned model {{.AgentLoop.AccountingSource.Model}}{{end}}{{if .AgentLoop.Suite}} · suite {{.AgentLoop.Suite}}{{if .AgentLoop.SuiteVersion}} {{.AgentLoop.SuiteVersion}}{{end}}{{end}} · fleet shape {{.AgentLoop.FleetShape.ID}}, {{.AgentLoop.FleetShape.ToolCount}} tools

+
⚠️ These figures come from provider-reported usage under the pinned model above. They are NEVER summed with the tokenizer-counted figures elsewhere in this report: a cross-accounting-source aggregate is withheld with a stated reason rather than computed. Read this section against itself, not against the deterministic sections.
+

The cheapest mode is not automatically the best mode. A mode can be cheap because the agent gave up, so cost is plotted AGAINST the share of tasks completed: up and to the left is better — more tasks finished, fewer tokens spent per finished task.

+ +{{with costoutcome .AgentLoop}} +{{if .HasPlot}} + + Cost versus outcome per mode cell + {{range .CompletionTicks}} + + {{.Label}} + {{end}} + {{range .CostTicks}} + {{.Label}} + {{end}} + + + Tokens per completed task (axis starts at zero) + Completion rate + {{range .Points}} + + {{.CellID}}: {{n0 .TokensPerCompletedTask}} tokens per completed task, {{f1 .CompletionRatePct}}% completed, {{.Runs}} run(s), {{.Provenance}} + + {{.CellID}} + {{end}} + +

Filled marker: a headline-eligible cell (FR-021 — at least four runs). Hollow dashed marker: measured but provisional, published without headline status. Red: a completion regression, which is never reported as a saving however cheap it is.

+{{else}} +

No cell carries a plottable figure — see the exclusions below.

+{{end}} +{{if .Excluded}} +

Not plotted (no honest coordinates — a withheld figure is not a zero): {{range .Excluded}}{{.CellID}} — {{.Reason}}. {{end}}

+{{end}} +{{end}} + + + + + {{range .AgentLoop.Cells}} + + + {{if .Withheld}} + + {{else}} + + + {{end}} + {{if .Withheld}}{{else}} + + {{end}} + + + + + {{end}} + +
Mode cellTokens per completed taskCompletion rateFirst-attempt successCorrective retriesInfra retriesRuns & spreadVerdictProvenance
{{.CellID}}withheld — {{.WithheldReason}}{{n0 .TokensPerCompletedTask}}{{f1 .CompletionRatePct}}%———{{if .FirstAttemptSuccessPct}}{{f1 (derefF .FirstAttemptSuccessPct)}}%{{else}}not measured{{end}}{{.RetriesCorrective}}{{.RetriesInfrastructure}}{{.Runs}} runs · {{if .SpreadPct}}±{{f1 (derefF .SpreadPct)}}%{{else}}spread undefined{{end}}{{if .PartialRuns}}
{{.PartialRuns}} partial run(s) excluded
{{end}}
{{if .Withheld}}—{{else if .Regression}}REGRESSION — completes fewer tasks than the baseline; not a saving{{else if not .Headline}}provisional — {{.Runs}} runs, below the four-run headline bar{{else}}headline{{end}}{{if .Withheld}}—{{else}}{{.Provenance}}{{end}}
+

Cost and completion carry equal weight here by construction (FR-018): a cell is only cheaper in a way worth having if its completion rate holds. A cell badged estimated still rests on an assumed figure rather than a measurement, and is shown beside its measured neighbours rather than hidden behind one section-level badge (FR-013).

+{{end}} + {{if .Corpora}}

Corpora{{with prov .Provenance "corpora"}}{{.}}{{end}}

@@ -332,14 +420,22 @@ const dashboardV2HTML = ` {{if .SessionEstimates}}

Session cost estimates{{with prov .Provenance "session_estimates"}}{{.}}{{end}}

- + {{range .SessionEstimates}} - + + + + + {{end}}
ArmCalls / sessionRetry rateEstimated session tokens
ArmCalls / sessionRetry rateRetry rate sourceRunsEstimated session tokensProvenance
{{.Arm}}{{.CallsPerSession}}{{.RetryRate}}{{.EstimatedTokens}}
{{.Arm}}{{.CallsPerSession}}{{.RetryRate}}{{.RetryRateProvenance}}{{if .MeasuredRuns}}{{.MeasuredRuns}}{{else}}—{{end}}{{.EstimatedTokens}}{{.Provenance}}
-

session_cost = proxy_menu + calls × mean_response(arm) × (1 + retry_rate(arm)); retry rates are literature-derived defaults (research D8), so these rows are estimates, not measurements.

+

session_cost = proxy_menu + calls × mean_response(arm) × (1 + retry_rate(arm)). +A row badged estimated uses a literature-derived retry default +(research D8); one badged measured uses an observed rate over the +run count shown. The per-row badge exists because a defaulted 0.0 and a measured 0.0 are the same number +and a different claim (FR-013).

{{end}} {{if .Latency}} @@ -375,3 +471,221 @@ const dashboardV2HTML = ` ` + +// --------------------------------------------------------------------------- +// FR-023 — the cost-versus-outcome view +// --------------------------------------------------------------------------- +// +// A table sorted by token cost answers "which mode is cheapest". That is the +// wrong question, and answering it well is actively misleading: the cheapest +// mode is often cheap because the agent gave up, retried into a dead end, or +// never finished the task at all. FR-023 therefore asks for cost plotted +// AGAINST completion outcome, so a reader sees which modes are worth their +// savings, and FR-018 requires completion rate to sit beside cost at equal +// prominence rather than in a footnote. +// +// Two axis decisions carry the honesty of the picture: +// +// - The cost axis starts at ZERO. A truncated axis turns a 5% saving into a +// visually dominant one; on a chart whose whole purpose is judging whether +// a saving is worth its outcome, that is the failure mode to design out. +// - The completion axis is FIXED at 0..100%, not scaled to the data. Scaling +// it would make a 92%-vs-90% difference look like a chasm, and would make +// two runs of the report incomparable at a glance. +// +// A cell whose figure was withheld (a cross-accounting-source aggregate, above +// all) is NOT plotted. It has no honest coordinates, and plotting it at the +// origin would read as "free, and never completes anything" — a claim nobody +// measured. It is named in an exclusion list beside the plot instead, the same +// silence-is-never-success posture the replay block's exclusion table takes. + +// Plot geometry, in the SVG's own viewBox units. Kept as constants so the +// axis mapping used to place points is the same one the gridlines are drawn +// with — a plot whose points and axes disagree is worse than no plot. +const ( + costOutcomePlotLeft = 70.0 + costOutcomePlotRight = 610.0 + costOutcomePlotTop = 30.0 + costOutcomePlotBottom = 290.0 +) + +// CostOutcomePoint is one mode cell positioned in the cost/outcome plane. It +// carries both the raw figures (so the table and the plot cannot drift apart) +// and the pre-computed SVG coordinates (so the template does no arithmetic). +type CostOutcomePoint struct { + CellID string + // TokensPerCompletedTask is the FR-018 cost headline, and + // CompletionRatePct is its equal-prominence companion. + TokensPerCompletedTask float64 + CompletionRatePct float64 + // PlotX/PlotY are viewBox coordinates from the axis mappings below. + PlotX float64 + PlotY float64 + // Headline is false when the cell has not met the FR-021 bar; such a + // point is drawn hollow, present but not load-bearing. + Headline bool + // Regression marks a cell that completes materially fewer tasks than the + // baseline. It is drawn in the warning colour and must never be described + // as a saving (FR-019). + Regression bool + // Provenance is the cell's own row provenance (FR-013): a cell still + // resting on an assumption is badged as such beside its measured + // neighbours. + Provenance string + // Runs and SpreadPct travel with the point so the plot's tooltip can say + // how much evidence stands behind it. + Runs int + SpreadPct float64 + // LabelAnchor and LabelDX place the cell name clear of its marker. A + // point in the right half of the plot is labelled to its LEFT, or a long + // cell name on the expensive end of the axis runs off the viewBox and + // the reader loses the identity of the most costly cell — the one they + // most need to identify. + LabelAnchor string + LabelDX float64 +} + +// CostOutcomeExclusion names a cell that could not be plotted, with the reason +// carried through verbatim. An unexplained absence is indistinguishable from +// an oversight. +type CostOutcomeExclusion struct { + CellID string + Reason string +} + +// CostOutcomeTick is one labelled axis gridline, positioned in viewBox units. +type CostOutcomeTick struct { + Label string + Pos float64 +} + +// CostOutcomeView is everything the dashboard needs to draw the FR-023 plot. +// It is built in Go rather than in the template so the axis arithmetic is +// unit-testable and so a degenerate input (one cell, all-zero costs, a nil +// block) cannot produce NaN coordinates inside an SVG attribute, where they +// would fail silently. +type CostOutcomeView struct { + Points []CostOutcomePoint + Excluded []CostOutcomeExclusion + // CostAxisMaxTokens is the zero-based cost axis maximum: the most + // expensive plotted cell plus headroom, never less than the data. + CostAxisMaxTokens float64 + // CompletionAxisMaxPct is fixed at 100 — see the block comment. + CompletionAxisMaxPct float64 + CostTicks []CostOutcomeTick + CompletionTicks []CostOutcomeTick +} + +// PlotX maps a token cost onto the horizontal axis. +func (v CostOutcomeView) PlotX(tokens float64) float64 { + if v.CostAxisMaxTokens <= 0 { + return costOutcomePlotLeft + } + frac := tokens / v.CostAxisMaxTokens + return costOutcomePlotLeft + frac*(costOutcomePlotRight-costOutcomePlotLeft) +} + +// PlotY maps a completion rate onto the vertical axis (0% at the bottom). +func (v CostOutcomeView) PlotY(pct float64) float64 { + if v.CompletionAxisMaxPct <= 0 { + return costOutcomePlotBottom + } + frac := pct / v.CompletionAxisMaxPct + return costOutcomePlotBottom - frac*(costOutcomePlotBottom-costOutcomePlotTop) +} + +// HasPlot reports whether there is anything to draw. A block whose every cell +// was withheld still renders its exclusion list, but no empty axes. +func (v CostOutcomeView) HasPlot() bool { return len(v.Points) > 0 } + +// NewCostOutcomeView builds the plot for one agent-loop block. A nil block +// yields an empty view rather than a panic, so the template can call this +// unconditionally. +func NewCostOutcomeView(b *AgentLoopBlock) CostOutcomeView { + view := CostOutcomeView{CompletionAxisMaxPct: 100} + if b == nil { + return view + } + + maxCost := 0.0 + for i := range b.Cells { + cell := &b.Cells[i] + if cell.Withheld { + reason := cell.WithheldReason + if reason == "" { + reason = "figure withheld, no reason recorded" + } + view.Excluded = append(view.Excluded, CostOutcomeExclusion{CellID: cell.CellID, Reason: reason}) + continue + } + if cell.TokensPerCompletedTask > maxCost { + maxCost = cell.TokensPerCompletedTask + } + view.Points = append(view.Points, CostOutcomePoint{ + CellID: cell.CellID, + TokensPerCompletedTask: cell.TokensPerCompletedTask, + CompletionRatePct: cell.CompletionRatePct, + Headline: cell.Headline, + Regression: cell.Regression, + Provenance: cell.Provenance, + Runs: cell.Runs, + SpreadPct: derefF(cell.SpreadPct), + }) + } + + // Headroom so the rightmost marker is not clipped by the axis edge. The + // floor of 1 keeps an all-zero (or single-cell-at-zero) input finite + // rather than dividing by zero into NaN. + view.CostAxisMaxTokens = maxCost * 1.1 + if view.CostAxisMaxTokens <= 0 { + view.CostAxisMaxTokens = 1 + } + + midX := (costOutcomePlotLeft + costOutcomePlotRight) / 2 + for i := range view.Points { + view.Points[i].PlotX = view.PlotX(view.Points[i].TokensPerCompletedTask) + view.Points[i].PlotY = view.PlotY(view.Points[i].CompletionRatePct) + if view.Points[i].PlotX > midX { + view.Points[i].LabelAnchor, view.Points[i].LabelDX = "end", -12 + } else { + view.Points[i].LabelAnchor, view.Points[i].LabelDX = "start", 12 + } + } + + for _, frac := range []float64{0, 0.25, 0.5, 0.75, 1} { + value := view.CostAxisMaxTokens * frac + view.CostTicks = append(view.CostTicks, CostOutcomeTick{ + Label: shortTokenLabel(value), + Pos: view.PlotX(value), + }) + pct := view.CompletionAxisMaxPct * frac + view.CompletionTicks = append(view.CompletionTicks, CostOutcomeTick{ + Label: fmt.Sprintf("%.0f%%", pct), + Pos: view.PlotY(pct), + }) + } + return view +} + +// shortTokenLabel renders an axis label compactly ("41k") without implying +// more precision than an axis tick carries. +func shortTokenLabel(v float64) string { + switch { + case v >= 1_000_000: + return fmt.Sprintf("%.1fM", v/1_000_000) + case v >= 1000: + return fmt.Sprintf("%.0fk", v/1000) + default: + return fmt.Sprintf("%.0f", v) + } +} + +// derefF renders a nil "undefined" figure as zero for PLOTTING only. Callers +// that display a number must branch on nil instead: a plotted point at zero is +// a position, whereas a printed "0.0%" is a claim. +func derefF(p *float64) float64 { + if p == nil { + return 0 + } + return *p +} diff --git a/bench/reportv2.go b/bench/reportv2.go index 410baa6d3..27ddbe424 100644 --- a/bench/reportv2.go +++ b/bench/reportv2.go @@ -14,6 +14,7 @@ import ( "fmt" "os" "path/filepath" + "strings" ) // ReportVersion2 is the report_version const of the v2 envelope. @@ -440,7 +441,9 @@ type ReplayBlock struct { // sensitive-data flag is derived from detection metadata written // asynchronously AFTER persistence: its absence does not prove a record // is clean. It is a reducer, never a guarantee (Principle IV). - SensitiveFlagBestEffort bool `json:"sensitive_flag_best_effort,omitempty"` + SensitiveFlagBestEffort bool `json:"sensitive_flag_best_effort,omitempty"` + Inputs *InputProvenance `json:"inputs,omitempty"` + Records *RawRecordsRef `json:"records,omitempty"` } // AgentLoopCell is one mode cell measured over a live agent loop (US2). Its @@ -455,8 +458,12 @@ type AgentLoopCell struct { // Runs and SpreadPct are the FR-021 consistency requirement: a // model-dependent figure needs at least four runs plus a spread before // it may be a headline. - Runs int `json:"runs"` - SpreadPct float64 `json:"spread_pct"` + Runs int `json:"runs"` + // SpreadPct is a POINTER: with fewer than two per-run figures the spread is + // UNDEFINED, not zero. Serialising 0 there renders "±0.0%" — a claim of + // perfect consistency from a single run, which is the opposite of what + // FR-021 asks the field to convey. + SpreadPct *float64 `json:"spread_pct"` // PartialRuns counts runs that did not finish. They are excluded from // the figures above and reported, never silently absorbed (FR-032). PartialRuns int `json:"partial_runs,omitempty"` @@ -465,7 +472,11 @@ type AgentLoopCell struct { // fewer tasks is not a saving. TokensPerCompletedTask float64 `json:"tokens_per_completed_task"` CompletionRatePct float64 `json:"completion_rate_pct"` - FirstAttemptSuccessPct float64 `json:"first_attempt_success_pct"` + // FirstAttemptSuccessPct is a POINTER for the same reason: with no + // determinate intents there is no rate, and a serialised 0 reads as "the + // agent never got a call right first time" rather than "nothing was + // measured". + FirstAttemptSuccessPct *float64 `json:"first_attempt_success_pct"` // Corrective and infrastructure retries are counted separately: only the // first kind says anything about how well the mode serves the agent. RetriesCorrective int `json:"retries_corrective"` @@ -494,12 +505,20 @@ type AgentLoopCell struct { // stay separable. type AgentLoopBlock struct { AccountingSource AccountingSource `json:"accounting_source"` + // CompletionRegressionThresholdPct is the stated FR-019 threshold this + // block was evaluated against. Serialised so a reader knows which number + // separated "saving" from "regression", and so a verdict recomputed from a + // loaded report uses the SAME threshold the run applied rather than + // whatever the current default happens to be. + CompletionRegressionThresholdPct float64 `json:"completion_regression_threshold_pct,omitempty"` // Suite and SuiteVersion pin the task suite so a later run is comparable // to an earlier one (FR-028). - Suite string `json:"suite,omitempty"` - SuiteVersion string `json:"suite_version,omitempty"` - FleetShape FleetShape `json:"fleet_shape"` - Cells []AgentLoopCell `json:"cells,omitempty"` + Suite string `json:"suite,omitempty"` + SuiteVersion string `json:"suite_version,omitempty"` + FleetShape FleetShape `json:"fleet_shape"` + Cells []AgentLoopCell `json:"cells,omitempty"` + Inputs *InputProvenance `json:"inputs,omitempty"` + Records *RawRecordsRef `json:"records,omitempty"` } // PayloadDecompositionRow attributes tool-definition cost for ONE fleet shape @@ -534,6 +553,7 @@ type PayloadDecompositionRow struct { type PayloadDecompositionBlock struct { AccountingSource AccountingSource `json:"accounting_source"` Shapes []PayloadDecompositionRow `json:"shapes,omitempty"` + Inputs *InputProvenance `json:"inputs,omitempty"` } // IsValidProvenance reports whether s is one of the three closed-enum values @@ -582,10 +602,24 @@ func (r *ReportV2) ValidateAdditiveBlocks() error { return nil } + // The US3 marks are optional (a report that sets none of them validates + // exactly as before, which is what keeps this feature additive), but a + // mark that IS set and is internally inconsistent is an error here rather + // than a silently-valid document downstream. + if err := r.RunStatus.Validate(); err != nil { + return err + } + if b := r.Replay; b != nil { if err := checkSource("replay", b.AccountingSource); err != nil { return err } + if err := b.Inputs.Validate("replay"); err != nil { + return err + } + if err := b.Records.Validate("replay"); err != nil { + return err + } for i := range b.Cells { if err := checkRow("replay", fmt.Sprintf("cells[%d] (%s)", i, b.Cells[i].CellID), b.Cells[i].Provenance); err != nil { return err @@ -601,6 +635,12 @@ func (r *ReportV2) ValidateAdditiveBlocks() error { if err := checkSource("agent_loop", b.AccountingSource); err != nil { return err } + if err := b.Inputs.Validate("agent_loop"); err != nil { + return err + } + if err := b.Records.Validate("agent_loop"); err != nil { + return err + } for i := range b.Cells { if err := checkRow("agent_loop", fmt.Sprintf("cells[%d] (%s)", i, b.Cells[i].CellID), b.Cells[i].Provenance); err != nil { return err @@ -611,6 +651,9 @@ func (r *ReportV2) ValidateAdditiveBlocks() error { if err := checkSource("payload_decomposition", b.AccountingSource); err != nil { return err } + if err := b.Inputs.Validate("payload_decomposition"); err != nil { + return err + } for i := range b.Shapes { if err := checkRow("payload_decomposition", fmt.Sprintf("shapes[%d] (%s)", i, b.Shapes[i].FleetShape.ID), b.Shapes[i].Provenance); err != nil { return err @@ -624,22 +667,28 @@ func (r *ReportV2) ValidateAdditiveBlocks() error { // v1 report: existing consumers are unaffected (reports are never committed, // Spec 065 CN-003). type ReportV2 struct { - ReportVersion int `json:"report_version"` - GeneratedAt string `json:"generated_at"` - Tokenizer TokenizerInfo `json:"tokenizer"` - Proxy *ProxyInfo `json:"proxy,omitempty"` - Corpora []CorpusDescriptor `json:"corpora"` - Arms []ArmResult `json:"arms"` - ResponseCost *ResponseCostSummary `json:"response_cost,omitempty"` - BreakEven *BreakEvenAnalysis `json:"break_even,omitempty"` - SessionEstimates []SessionCostEstimate `json:"session_estimates,omitempty"` - Latency *LatencyV2 `json:"latency,omitempty"` - Lap *LapVerdict `json:"lap,omitempty"` - Subset *SubsetInfo `json:"subset,omitempty"` + ReportVersion int `json:"report_version"` + GeneratedAt string `json:"generated_at"` + Tokenizer TokenizerInfo `json:"tokenizer"` + Proxy *ProxyInfo `json:"proxy,omitempty"` + Corpora []CorpusDescriptor `json:"corpora"` + Arms []ArmResult `json:"arms"` + ResponseCost *ResponseCostSummary `json:"response_cost,omitempty"` + BreakEven *BreakEvenAnalysis `json:"break_even,omitempty"` + // SessionEstimates carries SessionCostRow, not the bare estimate: FR-013 + // requires a measured retry rate and a defaulted one to be + // distinguishable INSIDE one table, and RetryRateForArm returns 0.0 for an + // unknown arm — indistinguishable from a measured 0.0 without the per-row + // provenance the row type adds. + SessionEstimates []SessionCostRow `json:"session_estimates,omitempty"` + Latency *LatencyV2 `json:"latency,omitempty"` + Lap *LapVerdict `json:"lap,omitempty"` + Subset *SubsetInfo `json:"subset,omitempty"` // Spec 103 blocks. Additive and optional: omitted they do not appear at // all, so existing offline reports and their consumers are byte- // unaffected and no report_version bump is needed. Each names its own // accounting source rather than borrowing the document-level Tokenizer. + RunStatus *RunStatus `json:"run_status,omitempty"` Replay *ReplayBlock `json:"replay,omitempty"` AgentLoop *AgentLoopBlock `json:"agent_loop,omitempty"` PayloadDecomposition *PayloadDecompositionBlock `json:"payload_decomposition,omitempty"` @@ -647,10 +696,27 @@ type ReportV2 struct { } // WriteJSON writes the v2 report as indented JSON into dir/report.json. +// +// Before marshaling it resolves every raw-record reference against dir, the +// run-local root those references are relative to: a reference whose artifact +// is no longer there degrades to "records not retained" instead of being +// written out as a path that resolves to nothing (T057/FR-029). This is the +// only moment both halves are in hand, and the report lands in the same +// gitignored tree the records do — so a reference that was accurate when the +// block was built can already be stale by the time it is serialized. +// +// It stays deterministic (FR-010/SC-002): the same run directory produces the +// same bytes, and no wall-clock time is injected. func (r *ReportV2) WriteJSON(dir string) (string, error) { if err := os.MkdirAll(dir, 0o755); err != nil { return "", fmt.Errorf("mkdir %q: %w", dir, err) } + if r.Replay != nil { + r.Replay.Records.resolve(dir) + } + if r.AgentLoop != nil { + r.AgentLoop.Records.resolve(dir) + } path := filepath.Join(dir, "report.json") data, err := json.MarshalIndent(r, "", " ") if err != nil { @@ -661,3 +727,535 @@ func (r *ReportV2) WriteJSON(dir string) (string, error) { } return path, nil } + +// --------------------------------------------------------------------------- +// US3 — an outsider reproduces the numbers (FR-029/030/032). +// +// The three structures below all answer the same question from different +// angles: what would a reader have to take on trust? Each one exists because +// its absence reads as a guarantee. An unlabelled figure derived from a +// private recording reads as reproducible. A run that stopped after four of +// six cells reads as a complete comparison. A report that references no raw +// records reads as if the records were never needed. None of those readings +// is corrected by a README, because a report travels without one. +// +// All three are ADDITIVE and OPTIONAL: a report that sets none of them +// marshals exactly as before (the contract forbids a report_version bump for +// this feature), and ValidateAdditiveBlocks only checks what is present. The +// strictness lives in PublicationCheck instead — the one moment where "not +// declared" must not be allowed to pass as "fine". +// --------------------------------------------------------------------------- + +// Input availability (FR-030). Every input is either in the repository, +// obtainable by a documented pinned procedure, or a private recording that an +// outsider simply cannot get. +// +// The third value is not a lesser version of the first two. Repository and +// pinned-procedure inputs differ only in where the bytes live; a private +// recording differs in kind, because FR-006 forbids publishing the recording +// and there is no procedure that would hand it over. That is a permanent +// limitation of any figure built on it, and the report says so rather than +// leaving a reader to assume the usual. +const ( + // InputAvailabilityRepository marks inputs committed in this repository — + // the frozen corpora and fixtures. Anyone with a clone has them. + InputAvailabilityRepository = "repository" + // InputAvailabilityPinnedProcedure marks inputs an outsider fetches by + // following a documented, version-pinned procedure (a pinned dataset + // revision, a pinned task suite). Reproducible, just not in-tree. + InputAvailabilityPinnedProcedure = "pinned-procedure" + // InputAvailabilityPrivateRecording marks inputs that are one operator's + // own recorded traffic. Not publishable, therefore not obtainable, + // therefore not independently reproducible. + InputAvailabilityPrivateRecording = "private-recording" +) + +// InputProvenance is one block's FR-030 mark: where its inputs come from and, +// consequently, whether an outsider can reproduce its figures. +// +// IndependentlyReproducible is deliberately redundant with Availability — it +// is exactly `Availability != private-recording`, and Validate enforces that. +// The redundancy is the point: a renderer, a dashboard or an outside reader +// consuming the JSON should not have to know which enum values imply +// reproducibility. The one-word answer is in the document, and the enum +// explains it. A mismatch between the two is a hard error, so the redundancy +// can never drift into a contradiction. +type InputProvenance struct { + Availability string `json:"availability"` + IndependentlyReproducible bool `json:"independently_reproducible"` + // Limitation states, in the report itself, WHY an outsider cannot + // reproduce this block. Mandatory whenever the block is not + // independently reproducible: an unexplained "false" is only marginally + // better than no mark, because the reader cannot tell whether the input + // is private, expired, or merely undocumented. + Limitation string `json:"limitation,omitempty"` + // Procedure names the documented, pinned procedure that obtains the + // inputs. Mandatory for pinned-procedure availability — a procedure that + // is not named is not documented, and FR-030 asks for a documented one. + Procedure string `json:"procedure,omitempty"` +} + +// PrivateRecordingInputs marks a block as built on one operator's own recorded +// sessions (T056/FR-030). It is a constructor rather than a struct literal so +// the not-reproducible flag cannot be forgotten at a call site: the whole +// value of this mark is that it is never omitted on the one block that needs +// it most. +func PrivateRecordingInputs(limitation string) *InputProvenance { + return &InputProvenance{ + Availability: InputAvailabilityPrivateRecording, + IndependentlyReproducible: false, + Limitation: limitation, + } +} + +// ReproducibleInputs marks a block whose inputs an outsider can obtain, either +// from the repository or by following the named pinned procedure. +func ReproducibleInputs(availability, procedure string) *InputProvenance { + return &InputProvenance{ + Availability: availability, + IndependentlyReproducible: true, + Procedure: procedure, + } +} + +// Validate reports whether the mark is internally consistent. block names the +// report block for the error message. +func (p *InputProvenance) Validate(block string) error { + if p == nil { + return nil + } + switch p.Availability { + case InputAvailabilityRepository, InputAvailabilityPinnedProcedure, InputAvailabilityPrivateRecording: + default: + return fmt.Errorf("%s block: inputs.availability %q not in {%s,%s,%s}", + block, p.Availability, InputAvailabilityRepository, + InputAvailabilityPinnedProcedure, InputAvailabilityPrivateRecording) + } + wantReproducible := p.Availability != InputAvailabilityPrivateRecording + if p.IndependentlyReproducible != wantReproducible { + return fmt.Errorf("%s block: inputs.availability %q implies independently_reproducible=%t, got %t", + block, p.Availability, wantReproducible, p.IndependentlyReproducible) + } + if !p.IndependentlyReproducible && p.Limitation == "" { + return fmt.Errorf("%s block: inputs are not independently reproducible but state no limitation", block) + } + if p.Availability == InputAvailabilityPinnedProcedure && p.Procedure == "" { + return fmt.Errorf("%s block: pinned-procedure inputs name no procedure", block) + } + return nil +} + +// Run completeness (FR-032). A run is one harness invocation producing one +// report; it is complete only when every planned cell either produced a figure +// or was skipped for a stated reason. +const ( + RunCompletenessComplete = "complete" + RunCompletenessPartial = "partial" +) + +// RunStatus is the FR-032 mark (T058): whether this report covers the whole +// comparison it set out to make. +// +// Why the three counts rather than one boolean. A cell can be absent for two +// very different reasons, and collapsing them would make the mark useless. A +// DELIBERATE skip — code_exec skipped because code execution is disabled — is +// accounted for, expected, and does not make the comparison partial; the +// harness already reports such cells as skip rows naming what they collapse +// onto. An UNEXPLAINED gap is a cell the run was supposed to measure and did +// not, which is exactly the subset-presented-as-the-whole failure FR-032 +// exists to catch. Planned − measured − skipped is that gap, and MissingCells +// names it, because "5 of 6" tells a reader nothing about which comparison +// they are missing. +// +// Interrupted is separate again: a run killed during the fourth repetition of +// the last cell may have touched every cell and still be partial, since the +// per-cell figures are averages over a repetition count the operator chose. +type RunStatus struct { + Completeness string `json:"completeness"` + // Reason is mandatory on a partial run. A partial with no reason is the + // same silence as no mark at all — the reader learns something is + // missing but not whether it matters. + Reason string `json:"reason,omitempty"` + Interrupted bool `json:"interrupted,omitempty"` + + CellsPlanned int `json:"cells_planned"` + CellsMeasured int `json:"cells_measured"` + CellsSkipped int `json:"cells_skipped,omitempty"` + MissingCells []string `json:"missing_cells,omitempty"` +} + +// DeriveRunStatus computes the completeness label from what the run actually +// did, rather than accepting one the caller asserts. +// +// This direction matters. A producer that stamps its own "complete" stamps it +// from the same optimism that lost the cells; a label derived from the planned +// set and the measured set cannot disagree with them. Validate then re-checks +// the same arithmetic on the way out, so a status that was hand-edited (or +// deserialized from somewhere else) is caught at emission time too. +// +// planned, measured and skipped are cell ids. reason is optional: when the run +// turns out to be partial and the caller supplied none, a factual one is +// synthesized, because a partial run must never reach the report unexplained. +func DeriveRunStatus(planned, measured, skipped []string, interrupted bool, reason string) *RunStatus { + accounted := make(map[string]bool, len(measured)+len(skipped)) + for _, id := range measured { + accounted[id] = true + } + for _, id := range skipped { + accounted[id] = true + } + // Iterate over planned, not over the maps: the missing list must be + // deterministic (FR-010/SC-002) and in the operator's own cell order. + missing := make([]string, 0, len(planned)) + for _, id := range planned { + if !accounted[id] { + missing = append(missing, id) + } + } + + status := &RunStatus{ + Completeness: RunCompletenessComplete, + Interrupted: interrupted, + CellsPlanned: len(planned), + CellsMeasured: len(measured), + CellsSkipped: len(skipped), + Reason: reason, + } + if len(missing) > 0 { + status.MissingCells = missing + } + if len(missing) > 0 || interrupted { + status.Completeness = RunCompletenessPartial + if status.Reason == "" { + switch { + case len(missing) > 0 && interrupted: + status.Reason = fmt.Sprintf("run was interrupted; %d planned cell(s) unmeasured and unaccounted for: %s", + len(missing), strings.Join(missing, ", ")) + case len(missing) > 0: + status.Reason = fmt.Sprintf("%d planned cell(s) unmeasured and unaccounted for: %s", + len(missing), strings.Join(missing, ", ")) + default: + status.Reason = "run was interrupted before it finished" + } + } + } + return status +} + +// IsPartial reports whether this run may not be published as a complete +// comparison. An unset status is NOT treated as complete here — callers that +// need that distinction check for nil themselves, and PublicationCheck does. +func (s *RunStatus) IsPartial() bool { + return s != nil && s.Completeness == RunCompletenessPartial +} + +// Validate re-derives the completeness label from the counts and refuses a +// status that claims more than the counts support. The interesting case is a +// hand-stamped "complete" over an unexplained gap: that is the exact document +// FR-032 forbids, and it is well-formed JSON, so nothing else would catch it. +func (s *RunStatus) Validate() error { + if s == nil { + return nil + } + switch s.Completeness { + case RunCompletenessComplete, RunCompletenessPartial: + default: + return fmt.Errorf("run_status: completeness %q not in {%s,%s}", + s.Completeness, RunCompletenessComplete, RunCompletenessPartial) + } + if s.CellsPlanned <= 0 { + return fmt.Errorf("run_status: cells_planned is %d — a run that planned nothing compares nothing", s.CellsPlanned) + } + if s.CellsMeasured < 0 || s.CellsSkipped < 0 { + return fmt.Errorf("run_status: negative cell counts (measured=%d skipped=%d)", s.CellsMeasured, s.CellsSkipped) + } + gap := s.CellsPlanned - s.CellsMeasured - s.CellsSkipped + if gap < 0 { + return fmt.Errorf("run_status: measured(%d)+skipped(%d) exceeds planned(%d)", + s.CellsMeasured, s.CellsSkipped, s.CellsPlanned) + } + if gap > 0 && len(s.MissingCells) == 0 { + return fmt.Errorf("run_status: %d planned cell(s) unaccounted for but missing_cells is empty — name them", gap) + } + mustBePartial := gap > 0 || s.Interrupted + if mustBePartial && s.Completeness != RunCompletenessPartial { + return fmt.Errorf("run_status: claims %q while %d cell(s) are unaccounted for (interrupted=%t) — that is a subset presented as the whole", + s.Completeness, gap, s.Interrupted) + } + if s.Completeness == RunCompletenessComplete && len(s.MissingCells) > 0 { + return fmt.Errorf("run_status: claims %q but lists %d missing cell(s)", s.Completeness, len(s.MissingCells)) + } + if s.Completeness == RunCompletenessPartial && s.Reason == "" { + return fmt.Errorf("run_status: partial run states no reason") + } + return nil +} + +// Raw per-run record retention (FR-029). +const ( + // RecordsRetained means the artifact exists in this run's output + // directory, at RunLocalPath, right now. + RecordsRetained = "retained" + // RecordsNotRetained means there is no artifact to point at. It is the + // honest terminal state of a reference whose file is gone, and it is + // never a dangling path. + RecordsNotRetained = "not_retained" +) + +// RecordsNotDurableNote is the standing caveat on every retained reference. +// The records live under the gitignored bench/results/ tree — the same tree a +// results cleanup empties, and the same tree SC-011 forbids committing — so +// "retained" means retained in THIS run directory, on THIS machine, until +// someone clears it. Anything stronger would be a promise the harness cannot +// keep. +const RecordsNotDurableNote = "Run-local: retained under the gitignored bench/results/ tree and NOT durable across a results cleanup. Copy it elsewhere before relying on it." + +// RawRecordsRef is the report-side reference to a block's raw per-run records +// (T057/FR-029), so a headline figure can be traced back to its inputs without +// embedding any recorded content in the report (FR-006). +// +// The path is RUN-LOCAL — relative to the directory the report itself was +// written into — for two reasons. An absolute path would carry the operator's +// filesystem layout into a document that may be published, and it would dangle +// on every other machine, which is worse than saying nothing. A relative path +// resolves for anyone holding the run directory and resolves nowhere else, +// which is exactly the scope of the claim. +// +// Durable is always false and Validate enforces it. Like +// InputProvenance.IndependentlyReproducible, the redundancy with the note is +// deliberate: a renderer keys off the boolean, a human reads the note, and +// neither has to infer the other. +type RawRecordsRef struct { + Retention string `json:"retention"` + // RunLocalPath is relative to the report's own directory, and empty + // whenever Retention is not_retained — a not-retained reference that + // still carries a path IS the dangling reference this type replaces. + RunLocalPath string `json:"run_local_path,omitempty"` + Records int `json:"records,omitempty"` + Durable bool `json:"durable"` + // Note carries RecordsNotDurableNote on a retained reference, or the + // reason nothing is retained on a degraded one. + Note string `json:"note"` +} + +// RetainedRecords references a per-run artifact written into this run's own +// output directory. runLocalPath must be relative to that directory. +func RetainedRecords(runLocalPath string, records int) *RawRecordsRef { + return &RawRecordsRef{ + Retention: RecordsRetained, + RunLocalPath: runLocalPath, + Records: records, + Durable: false, + Note: RecordsNotDurableNote, + } +} + +// NotRetainedRecords is the degraded reference: no artifact, a stated reason, +// and deliberately no path. FR-029 wants a traceable figure; when the trace +// does not exist, the report says the trace does not exist. +func NotRetainedRecords(reason string) *RawRecordsRef { + return &RawRecordsRef{ + Retention: RecordsNotRetained, + Durable: false, + Note: reason, + } +} + +// Validate enforces the run-local, non-durable, never-dangling shape. +func (r *RawRecordsRef) Validate(block string) error { + if r == nil { + return nil + } + if r.Durable { + return fmt.Errorf("%s block: records.durable is true — nothing under bench/results/ survives a cleanup", block) + } + if r.Note == "" { + return fmt.Errorf("%s block: records reference carries no note", block) + } + if r.Records < 0 { + return fmt.Errorf("%s block: records count is negative (%d)", block, r.Records) + } + switch r.Retention { + case RecordsRetained: + if r.RunLocalPath == "" { + return fmt.Errorf("%s block: records are retained but no run-local path is given", block) + } + if filepath.IsAbs(r.RunLocalPath) || strings.HasPrefix(r.RunLocalPath, "/") { + return fmt.Errorf("%s block: records path %q is absolute — the reference must be run-local, not a path on the operator's machine", + block, r.RunLocalPath) + } + // Cleaned rather than string-matched: "a/../../x" escapes too, and a + // reference that points outside the run directory is not run-local + // whatever it looks like. + if cleaned := filepath.Clean(r.RunLocalPath); cleaned == ".." || strings.HasPrefix(cleaned, ".."+string(filepath.Separator)) { + return fmt.Errorf("%s block: records path %q escapes the run directory", block, r.RunLocalPath) + } + case RecordsNotRetained: + if r.RunLocalPath != "" { + return fmt.Errorf("%s block: records are not retained but the reference still points at %q — that is the dangling reference it must degrade away from", + block, r.RunLocalPath) + } + default: + return fmt.Errorf("%s block: records.retention %q not in {%s,%s}", + block, r.Retention, RecordsRetained, RecordsNotRetained) + } + return nil +} + +// resolve degrades a retained reference whose artifact is not present in +// reportDir. This runs at write time because that is the only moment the +// report and its records directory are both in hand — and because the report +// is written into the very tree that gets cleaned, a reference that was +// accurate when the block was built can already be stale by the time the +// document is serialized. +// +// Degrading is not a fallback for a bug; it is the specified behaviour. A +// reader who follows a dangling path concludes either that the harness is +// broken or that the figure was never traceable, and both conclusions are +// worse than the truth, which is that the records were cleaned away. +func (r *RawRecordsRef) resolve(reportDir string) { + if r == nil || r.Retention != RecordsRetained || r.RunLocalPath == "" { + return + } + full := filepath.Join(reportDir, filepath.Clean(r.RunLocalPath)) + if _, err := os.Stat(full); err != nil { + missing := r.RunLocalPath + r.Retention = RecordsNotRetained + r.RunLocalPath = "" + r.Records = 0 + r.Note = fmt.Sprintf("records not retained: %q is absent from this run directory (a bench/results cleanup removes it)", missing) + } +} + +// PublicationDecision is the answer to "may this report back a published +// claim?" — a blocker list and a caveat list, both prose meant for the human +// about to publish. +// +// The two lists are not severity levels of one thing. A BLOCKER says the +// document cannot honestly support a published claim at all. A CAVEAT says it +// can, but something must travel with it: notably that a private-recording +// figure stays marked as not independently reproducible even in a report that +// publishes perfectly well. A caveat that quietly disappears once the blocker +// is resolved would defeat the mark entirely, so publication keeps them. +// +// The decision is derived on demand and deliberately NOT stored in the report. +// A stored verdict is a verdict that can go stale against the blocks beside +// it, and a stale "publishable: true" is the single most dangerous field this +// document could carry. +type PublicationDecision struct { + Publishable bool `json:"publishable"` + Blockers []string `json:"blockers,omitempty"` + Caveats []string `json:"caveats,omitempty"` +} + +// PublicationCheck applies FR-030 and FR-032 to a finished report. +// +// It is strict about ABSENCE in a way ValidateAdditiveBlocks deliberately is +// not. Everywhere else in this harness an unset optional field means "this run +// did not produce that", which is fine; at the publication boundary it means +// "nobody said", and "nobody said" must not read as "fine". Undeclared run +// completeness, undeclared input availability and an undeclared records +// reference are each blockers here, while a report that never calls this +// function is unaffected — which is what keeps the addition additive. +// +// The sole-support rule is the substance of FR-030's second sentence. Marking +// a private-recording figure is necessary but not sufficient: a report whose +// EVERY block rests on a private recording is a published claim supported by +// nothing an outsider can check, however honestly each block is labelled. One +// block backed by reproducible inputs is what turns the private figure into +// corroborating detail. +func (r *ReportV2) PublicationCheck() PublicationDecision { + var blockers, caveats []string + + // 1. FR-032: is this a complete comparison at all? + if r.RunStatus == nil { + blockers = append(blockers, "run completeness is not declared: a report that does not say whether its run finished cannot be published as a complete comparison (FR-032)") + } else { + if err := r.RunStatus.Validate(); err != nil { + blockers = append(blockers, "run status is malformed: "+err.Error()) + } + if r.RunStatus.IsPartial() { + blockers = append(blockers, fmt.Sprintf( + "run is PARTIAL (%s): a partial or interrupted run must not be published as a complete comparison (FR-032)", + r.RunStatus.Reason)) + } + } + + // 2. FR-030: what can an outsider actually reproduce? Blocks are walked + // in a fixed order so the decision is deterministic. + type blockInputs struct { + name string + present bool + inputs *InputProvenance + records *RawRecordsRef + // hasRecords marks a block that produces per-run records at all; + // payload decomposition is pure arithmetic over a committed corpus + // and has none to retain. + hasRecords bool + } + blocks := []blockInputs{ + {"replay", r.Replay != nil, nil, nil, true}, + {"agent_loop", r.AgentLoop != nil, nil, nil, true}, + {"payload_decomposition", r.PayloadDecomposition != nil, nil, nil, false}, + } + if r.Replay != nil { + blocks[0].inputs, blocks[0].records = r.Replay.Inputs, r.Replay.Records + } + if r.AgentLoop != nil { + blocks[1].inputs, blocks[1].records = r.AgentLoop.Inputs, r.AgentLoop.Records + } + if r.PayloadDecomposition != nil { + blocks[2].inputs = r.PayloadDecomposition.Inputs + } + + var present, reproducible int + for _, b := range blocks { + if !b.present { + continue + } + present++ + if b.inputs == nil { + blockers = append(blockers, fmt.Sprintf( + "%s block: input availability is not declared — an unset field is not a reproducible input (FR-030)", b.name)) + } else if err := b.inputs.Validate(b.name); err != nil { + blockers = append(blockers, err.Error()) + continue + } else if b.inputs.IndependentlyReproducible { + reproducible++ + } else { + caveats = append(caveats, fmt.Sprintf( + "%s block is not independently reproducible: %s", b.name, b.inputs.Limitation)) + } + + // FR-029: a headline must be traceable to its inputs. "Not retained" + // is an acceptable answer; no answer is not. + if b.hasRecords { + // Validate is nil-safe, so the nil case below is about the + // MEANING of nil (nobody said) rather than about safety. + err := b.records.Validate(b.name) + switch { + case b.records == nil: + blockers = append(blockers, fmt.Sprintf( + "%s block: no raw-record reference — a headline figure must be traceable to its per-run records (FR-029)", b.name)) + case err != nil: + blockers = append(blockers, err.Error()) + case b.records.Retention == RecordsNotRetained: + caveats = append(caveats, fmt.Sprintf( + "%s block: raw per-run records are not retained (%s) — its figures cannot be traced to their inputs", b.name, b.records.Note)) + } + } + } + + switch { + case present == 0: + blockers = append(blockers, "report carries no measured block to publish") + case reproducible == 0: + blockers = append(blockers, "every figure in this report rests on inputs an outsider cannot obtain: a private-recording figure must never be the sole support for a published claim (FR-030)") + } + + return PublicationDecision{ + Publishable: len(blockers) == 0, + Blockers: blockers, + Caveats: caveats, + } +} diff --git a/bench/reportv2_test.go b/bench/reportv2_test.go index 894a717e5..971429a11 100644 --- a/bench/reportv2_test.go +++ b/bench/reportv2_test.go @@ -6,6 +6,7 @@ import ( "os" "os/exec" "path/filepath" + "strings" "testing" ) @@ -76,8 +77,17 @@ func sampleReportV2() *ReportV2 { NaiveFullMenuTokens: 420000, ProxyMenuTokens: 4000, MeanResponseTokens: 11000, BreakEvenCalls: 37.8, }, - SessionEstimates: []SessionCostEstimate{ - {Arm: "baseline_json", CallsPerSession: 3, RetryRate: 0, EstimatedTokens: 37000}, + SessionEstimates: []SessionCostRow{ + { + SessionCostEstimate: SessionCostEstimate{ + Arm: "baseline_json", CallsPerSession: 3, RetryRate: 0, EstimatedTokens: 37000, + }, + // RetryRate 0 with an ESTIMATED source: the defaulted zero. + // The badge is the only thing separating it from a measured + // zero, which is why the row type exists. + Provenance: ProvenanceEstimated, + RetryRateProvenance: ProvenanceEstimated, + }, }, Latency: &LatencyV2{ P50Ms: 4.2, P95Ms: 9.8, P99Ms: 15.1, MaxMs: 22.0, @@ -319,6 +329,10 @@ func sampleReplayBlock() *ReplayBlock { AccountingSource: AccountingSource{Kind: AccountingKindTokenizer, Identity: "cl100k_base"}, Counterfactual: "cost recomputation over recorded traffic scored against the supplied fleet; not observed agent behaviour", BodiesIncluded: false, + // The recording is the operator's own activity export: marked, per + // FR-030, as an input nobody outside the project can obtain. + Inputs: PrivateRecordingInputs("recorded on the maintainer's machine via `mcpproxy activity export`; raw user traffic, never publishable"), + Records: RetainedRecords("records/replay.jsonl", 9), FleetShape: FleetShape{ ID: "corpus_v2@2026-07-14", ToolCount: 45, MeanDefinitionTokens: 444.4, P95DefinitionTokens: 900, @@ -359,18 +373,20 @@ func sampleAgentLoopBlock() *AgentLoopBlock { }, Suite: "mcpmark", SuiteVersion: "v0.3.1", + Inputs: ReproducibleInputs(InputAvailabilityPinnedProcedure, "MCPMark v0.3.1, pinned suite revision; see bench/README.md"), + Records: RetainedRecords("records/agent_loop.jsonl", 7), FleetShape: FleetShape{ID: "corpus_v2@2026-07-14", ToolCount: 45}, Cells: []AgentLoopCell{ { - CellID: "baseline", Provenance: ProvenanceMeasured, Runs: 5, SpreadPct: 8.4, - TokensPerCompletedTask: 41000, CompletionRatePct: 92.0, FirstAttemptSuccessPct: 71.0, + CellID: "baseline", Provenance: ProvenanceMeasured, Runs: 5, SpreadPct: fptr(8.4), + TokensPerCompletedTask: 41000, CompletionRatePct: 92.0, FirstAttemptSuccessPct: fptr(71.0), RetriesCorrective: 6, RetriesInfrastructure: 1, InputTokens: 180000, OutputTokens: 12000, CacheReadTokens: 60000, Headline: true, }, { - CellID: "retrieve_compact", Provenance: ProvenanceEstimated, Runs: 2, SpreadPct: 21.0, - TokensPerCompletedTask: 18000, CompletionRatePct: 74.0, FirstAttemptSuccessPct: 55.0, + CellID: "retrieve_compact", Provenance: ProvenanceEstimated, Runs: 2, SpreadPct: fptr(21.0), + TokensPerCompletedTask: 18000, CompletionRatePct: 74.0, FirstAttemptSuccessPct: fptr(55.0), RetriesCorrective: 11, RetriesInfrastructure: 0, PartialRuns: 1, InputTokens: 70000, OutputTokens: 9000, CacheReadTokens: 21000, Headline: false, Regression: true, @@ -780,3 +796,432 @@ sys.exit(1) }) } } + +// --------------------------------------------------------------------------- +// US3 — an outsider reproduces the numbers (T054/T056/T057/T058). +// +// The three behaviours asserted below are the ones that separate an honest +// limitation from an implied guarantee, so each is tested against the +// MARSHALED document as well as the struct: a mark that exists in Go but never +// reaches the report protects nobody. +// --------------------------------------------------------------------------- + +// TestInputProvenance_PrivateRecordingIsNotIndependentlyReproducible is the +// FR-030 mark (T056). A replay block sourced from an operator's own recording +// cannot be reproduced by an outsider — the recording is private and must stay +// so (FR-006) — and the report has to say that in the document, not in a +// README an aggregator will never read. +func TestInputProvenance_PrivateRecordingIsNotIndependentlyReproducible(t *testing.T) { + p := PrivateRecordingInputs("exported from the maintainer's own mcpproxy activity log; the recording is not publishable") + if p.Availability != InputAvailabilityPrivateRecording { + t.Errorf("availability = %q, want %q", p.Availability, InputAvailabilityPrivateRecording) + } + if p.IndependentlyReproducible { + t.Error("a private-recording input must never be marked independently reproducible") + } + if p.Limitation == "" { + t.Error("a private-recording input must state its limitation") + } + if err := p.Validate("replay"); err != nil { + t.Fatalf("well-formed private-recording provenance rejected: %v", err) + } + + // The converse, so the mark is not vacuously always-false. + pub := ReproducibleInputs(InputAvailabilityRepository, "specs/083-discovery-profiler/datasets/corpus_v2.tools.json") + if !pub.IndependentlyReproducible { + t.Error("a repository input must be marked independently reproducible") + } + if err := pub.Validate("payload_decomposition"); err != nil { + t.Fatalf("well-formed repository provenance rejected: %v", err) + } + + t.Run("mismarked as reproducible", func(t *testing.T) { + bad := PrivateRecordingInputs("private export") + bad.IndependentlyReproducible = true + if err := bad.Validate("replay"); err == nil { + t.Error("a private-recording input claiming independent reproducibility must be rejected") + } + }) + t.Run("limitation not stated", func(t *testing.T) { + bad := PrivateRecordingInputs("") + if err := bad.Validate("replay"); err == nil { + t.Error("a not-reproducible mark with no stated limitation is the implied guarantee this field exists to prevent") + } + }) + t.Run("pinned procedure not named", func(t *testing.T) { + bad := ReproducibleInputs(InputAvailabilityPinnedProcedure, "") + if err := bad.Validate("agent_loop"); err == nil { + t.Error("a 'documented, pinned procedure' that names no procedure is not documented") + } + }) + t.Run("availability outside the enum", func(t *testing.T) { + bad := &InputProvenance{Availability: "somewhere", IndependentlyReproducible: true} + if err := bad.Validate("replay"); err == nil { + t.Error("an availability outside the closed enum must be rejected") + } + }) + + // The mark must survive marshaling onto the block that carries it. + r := sampleReportV2WithBlocks() + r.Replay.Inputs = PrivateRecordingInputs("private activity export") + data, err := json.Marshal(r) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var doc map[string]interface{} + if err := json.Unmarshal(data, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + replay, _ := doc["replay"].(map[string]interface{}) + inputs, ok := replay["inputs"].(map[string]interface{}) + if !ok { + t.Fatal("replay.inputs missing from the emitted report") + } + if repro, _ := inputs["independently_reproducible"].(bool); repro { + t.Error("emitted replay.inputs.independently_reproducible = true for a private recording") + } +} + +// TestPublicationCheck_PrivateFigureIsNeverSoleSupport is the second half of +// FR-030 (T056): the mark alone is not enough, because a marked figure can +// still be the only figure in the document. A report in which EVERY block +// rests on a private recording is refused for publication outright; the same +// private block beside a reproducible one publishes with a caveat. +func TestPublicationCheck_PrivateFigureIsNeverSoleSupport(t *testing.T) { + soleSupport := &ReportV2{ + ReportVersion: ReportVersion2, + RunStatus: DeriveRunStatus([]string{"direct_full", "direct_deferred"}, []string{"direct_full", "direct_deferred"}, nil, false, ""), + Replay: sampleReplayBlock(), + } + soleSupport.Replay.Inputs = PrivateRecordingInputs("maintainer's own activity export") + soleSupport.Replay.Records = RetainedRecords("records/replay.jsonl", 120) + + decision := soleSupport.PublicationCheck() + if decision.Publishable { + t.Error("a report whose only figures come from a private recording must not be publishable (FR-030)") + } + if !containsSubstring(decision.Blockers, "sole support") { + t.Errorf("blockers do not name the sole-support rule: %v", decision.Blockers) + } + + // Add a block an outsider CAN reproduce: the private figure is no longer + // the sole support, so it publishes — carrying its caveat. + corroborated := soleSupport + corroborated.PayloadDecomposition = samplePayloadDecomposition() + corroborated.PayloadDecomposition.Inputs = ReproducibleInputs( + InputAvailabilityRepository, "specs/083-discovery-profiler/datasets/corpus_v2.tools.json") + + decision = corroborated.PublicationCheck() + if !decision.Publishable { + t.Errorf("a private figure corroborated by a reproducible block must be publishable: %v", decision.Blockers) + } + if !containsSubstring(decision.Caveats, "not independently reproducible") { + t.Errorf("the caveat must survive publication, not disappear with the blocker: %v", decision.Caveats) + } +} + +// TestPublicationCheck_UndeclaredInputsAreNotPublishable: an unset field is not +// a reproducible input. Silence is never success anywhere else in this +// harness, and it must not become success at the publication boundary. +func TestPublicationCheck_UndeclaredInputsAreNotPublishable(t *testing.T) { + r := &ReportV2{ + ReportVersion: ReportVersion2, + RunStatus: DeriveRunStatus([]string{"a"}, []string{"a"}, nil, false, ""), + PayloadDecomposition: samplePayloadDecomposition(), // no Inputs set + } + decision := r.PublicationCheck() + if decision.Publishable { + t.Error("a block that never declared its input availability must not be publishable") + } + if !containsSubstring(decision.Blockers, "payload_decomposition") { + t.Errorf("blockers do not name the offending block: %v", decision.Blockers) + } +} + +// TestRunStatus_PartialRunIsMarkedAndBlocked is FR-032 (T058). A run that did +// not measure every planned cell, or that was interrupted, is a comparison +// over a subset; presented as complete it is a comparison over a subset +// presented as the whole. +func TestRunStatus_PartialRunIsMarkedAndBlocked(t *testing.T) { + planned := []string{"baseline", "retrieve_full", "retrieve_compact", "direct_full", "direct_deferred", "code_exec"} + + t.Run("missing cell", func(t *testing.T) { + s := DeriveRunStatus(planned, planned[:4], []string{"code_exec"}, false, "") + if s.Completeness != RunCompletenessPartial { + t.Errorf("completeness = %q, want %q", s.Completeness, RunCompletenessPartial) + } + if len(s.MissingCells) != 1 || s.MissingCells[0] != "direct_deferred" { + t.Errorf("missing cells = %v, want [direct_deferred]", s.MissingCells) + } + if s.Reason == "" { + t.Error("a partial run must carry a reason; an unexplained partial is the silence FR-032 forbids") + } + if err := s.Validate(); err != nil { + t.Fatalf("derived status rejected by its own validation: %v", err) + } + }) + + t.Run("interrupted with every cell measured", func(t *testing.T) { + s := DeriveRunStatus(planned, planned, nil, true, "SIGINT during the third repetition") + if s.Completeness != RunCompletenessPartial { + t.Error("an interrupted run is partial even when every planned cell produced a figure") + } + }) + + t.Run("complete", func(t *testing.T) { + s := DeriveRunStatus(planned, planned[:5], []string{"code_exec"}, false, "") + if s.Completeness != RunCompletenessComplete { + t.Errorf("completeness = %q, want %q — a deliberately skipped cell with a stated reason is not a gap", + s.Completeness, RunCompletenessComplete) + } + if err := s.Validate(); err != nil { + t.Fatalf("complete status rejected: %v", err) + } + }) + + t.Run("hand-stamped complete over a gap", func(t *testing.T) { + s := DeriveRunStatus(planned, planned[:4], nil, false, "") + s.Completeness = RunCompletenessComplete + if err := s.Validate(); err == nil { + t.Error("a status claiming complete while cells are unaccounted for must be rejected") + } + }) + + t.Run("blocked from publication", func(t *testing.T) { + r := sampleReportV2WithBlocks() + r.Replay.Inputs = ReproducibleInputs(InputAvailabilityRepository, "committed corpus") + r.AgentLoop.Inputs = ReproducibleInputs(InputAvailabilityPinnedProcedure, "mcpmark v0.3.1") + r.PayloadDecomposition.Inputs = ReproducibleInputs(InputAvailabilityRepository, "committed corpus") + r.Replay.Records = RetainedRecords("records/replay.jsonl", 9) + r.AgentLoop.Records = RetainedRecords("records/agent_loop.jsonl", 7) + + r.RunStatus = DeriveRunStatus(planned, planned, nil, false, "") + if d := r.PublicationCheck(); !d.Publishable { + t.Fatalf("a complete, reproducible report must be publishable: %v", d.Blockers) + } + + r.RunStatus = DeriveRunStatus(planned, planned[:4], nil, true, "operator interrupted the run") + d := r.PublicationCheck() + if d.Publishable { + t.Error("a partial run must not be publishable as a complete comparison (FR-032)") + } + if !containsSubstring(d.Blockers, "partial") { + t.Errorf("blockers do not name the run as partial: %v", d.Blockers) + } + }) + + t.Run("undeclared completeness", func(t *testing.T) { + r := sampleReportV2WithBlocks() + r.Replay.Inputs = ReproducibleInputs(InputAvailabilityRepository, "committed corpus") + r.RunStatus = nil + if r.PublicationCheck().Publishable { + t.Error("a report that never declared its run completeness must not be publishable") + } + }) + + // The mark must reach the document. + r := sampleReportV2WithBlocks() + r.RunStatus = DeriveRunStatus(planned, planned[:4], nil, true, "operator interrupted the run") + data, err := json.Marshal(r) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var doc map[string]interface{} + if err := json.Unmarshal(data, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + status, ok := doc["run_status"].(map[string]interface{}) + if !ok { + t.Fatal("run_status missing from the emitted report") + } + if c, _ := status["completeness"].(string); c != RunCompletenessPartial { + t.Errorf("emitted run_status.completeness = %q, want %q", c, RunCompletenessPartial) + } +} + +// TestRawRecordsRef_RunLocalAndNotDurable is FR-029 (T057), report side. The +// records live under the gitignored bench/results/ tree, so the reference is +// RUN-LOCAL and explicitly non-durable: an absolute path would leak the +// operator's filesystem into a published document and would dangle on any +// other machine, and a reference with no durability note invites a reader to +// treat a scratch directory as an archive. +func TestRawRecordsRef_RunLocalAndNotDurable(t *testing.T) { + ref := RetainedRecords("records/replay-2026-09-01.jsonl", 120) + if ref.Retention != RecordsRetained { + t.Errorf("retention = %q, want %q", ref.Retention, RecordsRetained) + } + if ref.Durable { + t.Error("no bench/results record is durable — a results cleanup removes it") + } + if ref.Note == "" { + t.Error("a records reference must carry its non-durability note") + } + if err := ref.Validate("replay"); err != nil { + t.Fatalf("well-formed run-local reference rejected: %v", err) + } + + notRetained := NotRetainedRecords("bodies-off run: no per-record artifact was written") + if notRetained.Retention != RecordsNotRetained { + t.Errorf("retention = %q, want %q", notRetained.Retention, RecordsNotRetained) + } + if notRetained.RunLocalPath != "" { + t.Error("a not-retained reference must carry no path — that is precisely the dangling reference it replaces") + } + if err := notRetained.Validate("replay"); err != nil { + t.Fatalf("well-formed not-retained reference rejected: %v", err) + } + + bad := []struct { + name string + ref *RawRecordsRef + }{ + {"absolute path", &RawRecordsRef{Retention: RecordsRetained, RunLocalPath: "/Users/someone/bench/results/replay.jsonl", Note: RecordsNotDurableNote}}, + {"escaping path", &RawRecordsRef{Retention: RecordsRetained, RunLocalPath: "../../secrets.jsonl", Note: RecordsNotDurableNote}}, + {"retained with no path", &RawRecordsRef{Retention: RecordsRetained, Note: RecordsNotDurableNote}}, + {"claimed durable", &RawRecordsRef{Retention: RecordsRetained, RunLocalPath: "records/r.jsonl", Durable: true, Note: RecordsNotDurableNote}}, + {"not retained but still pointing somewhere", &RawRecordsRef{Retention: RecordsNotRetained, RunLocalPath: "records/r.jsonl", Note: "gone"}}, + {"retention outside the enum", &RawRecordsRef{Retention: "archived", RunLocalPath: "records/r.jsonl", Note: RecordsNotDurableNote}}, + {"no note", &RawRecordsRef{Retention: RecordsRetained, RunLocalPath: "records/r.jsonl"}}, + } + for _, tc := range bad { + t.Run(tc.name, func(t *testing.T) { + if err := tc.ref.Validate("replay"); err == nil { + t.Errorf("malformed records reference accepted: %+v", tc.ref) + } + }) + } +} + +// TestRawRecordsRef_DegradesRatherThanDangles is the other half of T057: the +// report is written to the same gitignored tree the records live in, so by the +// time anyone reads it the records may already have been cleaned away. The +// reference must then say "records not retained" — a reader who follows a +// dangling path concludes the harness is broken, or worse, that the figure was +// never traceable at all. +func TestRawRecordsRef_DegradesRatherThanDangles(t *testing.T) { + t.Run("records absent", func(t *testing.T) { + dir := t.TempDir() + r := sampleReportV2WithBlocks() + r.Replay.Records = RetainedRecords("records/replay.jsonl", 120) + + path, err := r.WriteJSON(dir) + if err != nil { + t.Fatalf("WriteJSON: %v", err) + } + raw, err := os.ReadFile(filepath.Clean(path)) + if err != nil { + t.Fatalf("read report: %v", err) + } + var doc map[string]interface{} + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatalf("unmarshal report: %v", err) + } + replay, _ := doc["replay"].(map[string]interface{}) + records, ok := replay["records"].(map[string]interface{}) + if !ok { + t.Fatal("replay.records missing from the emitted report") + } + if got, _ := records["retention"].(string); got != RecordsNotRetained { + t.Errorf("retention = %q, want %q — an absent artifact must degrade, not dangle", got, RecordsNotRetained) + } + if p, _ := records["run_local_path"].(string); p != "" { + t.Errorf("degraded reference still carries a path %q", p) + } + }) + + t.Run("records present", func(t *testing.T) { + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "records"), 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + if err := os.WriteFile(filepath.Join(dir, "records", "replay.jsonl"), []byte("{}\n"), 0o644); err != nil { + t.Fatalf("write records: %v", err) + } + r := sampleReportV2WithBlocks() + r.Replay.Records = RetainedRecords("records/replay.jsonl", 120) + + path, err := r.WriteJSON(dir) + if err != nil { + t.Fatalf("WriteJSON: %v", err) + } + raw, err := os.ReadFile(filepath.Clean(path)) + if err != nil { + t.Fatalf("read report: %v", err) + } + var doc map[string]interface{} + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatalf("unmarshal report: %v", err) + } + replay, _ := doc["replay"].(map[string]interface{}) + records, _ := replay["records"].(map[string]interface{}) + if got, _ := records["retention"].(string); got != RecordsRetained { + t.Errorf("retention = %q, want %q — a present artifact must stay referenced", got, RecordsRetained) + } + if p, _ := records["run_local_path"].(string); p != "records/replay.jsonl" { + t.Errorf("run_local_path = %q, want records/replay.jsonl", p) + } + }) +} + +// TestReportV2_ValidateAdditiveBlocksCoversUS3Structures keeps the new +// structures inside the existing emission-time guard. They are OPTIONAL — a +// report that sets none of them validates exactly as before, which is what +// keeps this change additive — but a malformed one is an error rather than a +// silently-valid document. +func TestReportV2_ValidateAdditiveBlocksCoversUS3Structures(t *testing.T) { + if err := sampleReportV2WithBlocks().ValidateAdditiveBlocks(); err != nil { + t.Fatalf("sample with the US3 fields rejected: %v", err) + } + + t.Run("bad inputs", func(t *testing.T) { + r := sampleReportV2WithBlocks() + r.Replay.Inputs = &InputProvenance{Availability: InputAvailabilityPrivateRecording, IndependentlyReproducible: true} + if err := r.ValidateAdditiveBlocks(); err == nil { + t.Error("a mismarked input provenance must be rejected at emission time") + } + }) + t.Run("bad records", func(t *testing.T) { + r := sampleReportV2WithBlocks() + r.AgentLoop.Records = &RawRecordsRef{Retention: RecordsRetained, RunLocalPath: "/tmp/leak.jsonl", Note: RecordsNotDurableNote} + if err := r.ValidateAdditiveBlocks(); err == nil { + t.Error("an absolute records path must be rejected at emission time") + } + }) + t.Run("bad run status", func(t *testing.T) { + r := sampleReportV2WithBlocks() + r.RunStatus = &RunStatus{Completeness: "mostly", CellsPlanned: 1, CellsMeasured: 1} + if err := r.ValidateAdditiveBlocks(); err == nil { + t.Error("a completeness outside the closed enum must be rejected") + } + }) + t.Run("still additive", func(t *testing.T) { + data, err := json.Marshal(sampleReportV2()) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var doc map[string]interface{} + if err := json.Unmarshal(data, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if _, ok := doc["run_status"]; ok { + t.Error("run_status appeared in a report that never set it — must be omitempty") + } + }) +} + +// containsSubstring reports whether any entry of list contains want. The +// publication decision is prose meant for a human about to publish, so the +// assertions match on the load-bearing phrase rather than on an exact string +// nobody should be pinned to. +func containsSubstring(list []string, want string) bool { + for _, s := range list { + if strings.Contains(s, want) { + return true + } + } + return false +} + +// fptr makes an optional report figure. Nil means NOT MEASURED, which is a +// different claim from a measured zero — see the field docs on AgentLoopCell. +func fptr(f float64) *float64 { return &f } diff --git a/bench/session.go b/bench/session.go index d52971309..9ea2e0353 100644 --- a/bench/session.go +++ b/bench/session.go @@ -103,3 +103,206 @@ func EstimateSessionCosts(proxyMenuTokens int, meanResponseTokensByArm map[strin func roundHalfUp(x float64) int { return int(math.Floor(x + 0.5)) } + +// --------------------------------------------------------------------------- +// FR-013 — a measurement supersedes the assumption, and the reader can tell +// --------------------------------------------------------------------------- +// +// The estimator above prices a session with a retry rate taken from the +// literature (research D8). Once a live agent loop actually runs a mode, that +// assumption is obsolete for that mode — but only for that mode. FR-013 asks +// for exactly that: measured figures REPLACE the defaults wherever a +// measurement exists, and anything still resting on an assumption stays badged +// estimated and is never presented without the distinction. +// +// Why this needs a per-ROW field rather than a section badge. RetryRateForArm +// returns 0.0 for an arm it has never heard of, which is bit-identical to a +// retry rate of 0.0 that a live loop measured. Under one section-level +// "estimated" badge, a table can therefore hold a measured 0.0 beside a +// defaulted 0.0 with nothing on the page separating them, and a reader has no +// way to know which number is evidence. Every row carries its own provenance +// so the two are distinguishable inside one table (contracts/report-v2- +// additions.md, "Add per-row provenance"). +// +// Why a measured row is "computed" and not "measured". Substituting a measured +// retry rate does NOT make the row an observation. The row's token magnitudes +// still come from the deterministic tokenizer; only the behavioural multiplier +// came from the loop. The dashboard's own vocabulary already has the right +// label for that — computed: "arithmetic over measured inputs". Calling the +// row measured would claim the token figure itself was observed under a +// provider, and would invite exactly the cross-accounting-source confusion the +// never-sum rule exists to prevent (data-model.md invariant 2). Note also that +// the retry rate is a dimensionless rate, not a token quantity: applying it to +// tokenizer-counted tokens parameterises the estimate, it does not sum a +// provider figure with a tokenizer figure. + +// MeasuredArmOutcome is what a live agent loop observed for ONE arm (or mode +// cell): its retry rate, its first-attempt success rate, and how many runs +// stand behind them. Runs is not decoration — see HasMeasurement. +type MeasuredArmOutcome struct { + // RetryRate is the measured mean retries per call, on the same scale as + // the armRetryRates defaults it supersedes. + RetryRate float64 + // FirstAttemptSuccessPct is the measured first-attempt success rate + // (FR-010), carried because FR-013 covers success figures as well as + // retries. + FirstAttemptSuccessPct float64 + // Runs is the number of completed runs behind the two figures above. A + // zero here means no run produced them, which HasMeasurement treats as an + // absence rather than a measurement of zero — the same + // indistinguishable-zero hazard one level up. FR-021's four-run bar is a + // PUBLICATION gate applied by the report layer, not a gate on + // superseding: a two-run measurement is still better evidence than a + // literature constant, and the row publishes Runs so the reader can apply + // FR-021 themselves. + Runs int +} + +// HasMeasurement reports whether this outcome is backed by at least one run. +// An outcome that reached the map without a run behind it (an arm the loop was +// configured for but never executed) is an absence, and must not supersede a +// documented default. +func (m MeasuredArmOutcome) HasMeasurement() bool { return m.Runs > 0 } + +// MeasuredOutcomeSource supplies measured outcomes per arm. It is a structural +// interface on purpose: the live agent loop that produces these figures is a +// separate concern with its own accounting source, and the estimator must stay +// unit-testable with a fake and no suite running (the bench/flipgate.go and +// bench/armrun.go precedent). A nil source means "nothing was measured", which +// is the ordinary offline case. +type MeasuredOutcomeSource interface { + MeasuredOutcomeForArm(arm string) (MeasuredArmOutcome, bool) +} + +// MeasuredOutcomes is the trivial map-backed MeasuredOutcomeSource, used by +// callers that already hold the figures and by tests. +type MeasuredOutcomes map[string]MeasuredArmOutcome + +// MeasuredOutcomeForArm implements MeasuredOutcomeSource. It reports ok only +// for an entry that is actually backed by runs, so a zero-value entry cannot +// masquerade as a measured zero. +func (m MeasuredOutcomes) MeasuredOutcomeForArm(arm string) (MeasuredArmOutcome, bool) { + out, ok := m[arm] + if !ok || !out.HasMeasurement() { + return MeasuredArmOutcome{}, false + } + return out, true +} + +// ResolveRetryRate returns the retry rate to price an arm with, together with +// the provenance of that rate: ProvenanceMeasured when a live measurement +// supersedes the default, ProvenanceEstimated when the documented default +// stands. The provenance is the whole point of the second return value — the +// rate alone cannot tell a measured 0.0 from a defaulted 0.0. +func ResolveRetryRate(arm string, measured MeasuredOutcomeSource) (rate float64, provenance string) { + if measured != nil { + if out, ok := measured.MeasuredOutcomeForArm(arm); ok { + return out.RetryRate, ProvenanceMeasured + } + } + return RetryRateForArm(arm), ProvenanceEstimated +} + +// SessionCostRow is a session-cost estimate that knows where its inputs came +// from. It EMBEDS SessionCostEstimate rather than redefining it, so the +// serialized row is a strict superset of the existing shape (the embedded +// fields flatten into the same JSON object) and no existing consumer of a +// session-estimate row breaks. +type SessionCostRow struct { + SessionCostEstimate + + // Provenance is this ROW's label, from the closed measured/computed/ + // estimated enum: computed when a measured retry rate was applied, + // estimated when the row still rests on the literature default. Never + // measured — the token magnitudes are tokenizer-counted (see the block + // comment above). + Provenance string `json:"provenance"` + + // RetryRateProvenance labels the retry rate specifically. It is the field + // that makes a measured 0.0 distinguishable from a defaulted 0.0, and it + // is separate from Provenance because a reader scanning the retry-rate + // column needs the answer in that column. + RetryRateProvenance string `json:"retry_rate_provenance"` + + // MeasuredRuns is how many runs back the measurement, 0 when none. It + // travels with the row so a consumer can apply the FR-021 four-run + // headline bar without re-joining against the agent-loop block. + MeasuredRuns int `json:"measured_runs,omitempty"` + + // FirstAttemptSuccessPct is the measured first-attempt success rate, or + // nil when nothing was measured. It is a POINTER for the same reason + // PayloadDecompositionRow.ShareAnnotationsPct is: a 0 here would read as + // "measured, and nothing ever succeeded on the first attempt", which is a + // far stronger claim than "not measured". nil marshals to null. + FirstAttemptSuccessPct *float64 `json:"first_attempt_success_pct"` +} + +// EstimateSessionCostRow computes one session-cost row, applying a measured +// retry rate for the arm when one exists and the documented default otherwise. +// The formula and rounding policy are unchanged (package comment); only the +// source of retry_rate can differ, and the row says which source it was. +func EstimateSessionCostRow(arm string, proxyMenuTokens int, meanResponseTokens float64, calls int, measured MeasuredOutcomeSource) SessionCostRow { + retry, retryProv := ResolveRetryRate(arm, measured) + raw := float64(proxyMenuTokens) + float64(calls)*meanResponseTokens*(1+retry) + + row := SessionCostRow{ + SessionCostEstimate: SessionCostEstimate{ + Arm: arm, + CallsPerSession: calls, + RetryRate: retry, + EstimatedTokens: roundHalfUp(raw), + }, + Provenance: SessionEstimateProvenance, + RetryRateProvenance: retryProv, + } + if retryProv == ProvenanceMeasured { + // Arithmetic over a measured input: computed, not estimated — and not + // measured either. + row.Provenance = ProvenanceComputed + if out, ok := measured.MeasuredOutcomeForArm(arm); ok { + row.MeasuredRuns = out.Runs + success := out.FirstAttemptSuccessPct + row.FirstAttemptSuccessPct = &success + } + } + return row +} + +// EstimateSessionCostRows produces the full self-describing session-estimate +// table: one row per arm per default calls-per-session point, in the same +// deterministic order as EstimateSessionCosts (arms lexicographic, then calls +// ascending — FR-010), with measured and estimated rows coexisting in the one +// slice. Pass a nil source offline; every row is then identical to what +// EstimateSessionCosts produces, plus its estimated badge. +func EstimateSessionCostRows(proxyMenuTokens int, meanResponseTokensByArm map[string]float64, measured MeasuredOutcomeSource) []SessionCostRow { + armNames := make([]string, 0, len(meanResponseTokensByArm)) + for arm := range meanResponseTokensByArm { + armNames = append(armNames, arm) + } + sort.Strings(armNames) + + rows := make([]SessionCostRow, 0, len(armNames)*len(defaultCallsPerSession)) + for _, arm := range armNames { + for _, calls := range defaultCallsPerSession { + rows = append(rows, EstimateSessionCostRow(arm, proxyMenuTokens, meanResponseTokensByArm[arm], calls, measured)) + } + } + return rows +} + +// SessionCostRowsMixed reports whether a table holds rows of more than one +// provenance. A caller rendering a section badge must check this: a single +// section badge over a mixed table is precisely the misrepresentation FR-013 +// forbids, so a mixed table must be badged per row instead. +func SessionCostRowsMixed(rows []SessionCostRow) bool { + if len(rows) == 0 { + return false + } + first := rows[0].Provenance + for _, r := range rows[1:] { + if r.Provenance != first { + return true + } + } + return false +} diff --git a/bench/session_test.go b/bench/session_test.go index bf7712e42..acb2b652b 100644 --- a/bench/session_test.go +++ b/bench/session_test.go @@ -1,7 +1,11 @@ package bench import ( + "encoding/json" + "math" "reflect" + "strconv" + "strings" "testing" ) @@ -170,3 +174,410 @@ func TestSessionEstimateProvenance(t *testing.T) { t.Errorf("SessionEstimateProvenance = %q, want \"estimated\"", SessionEstimateProvenance) } } + +// --------------------------------------------------------------------------- +// FR-013 — measured figures supersede the literature defaults (T044) +// --------------------------------------------------------------------------- +// +// The hazard these tests exist to prevent: RetryRateForArm returns 0.0 for an +// arm it has never heard of, which is bit-for-bit indistinguishable from a +// retry rate of 0.0 that a live agent loop actually measured. A section-level +// "estimated" badge over the whole session-cost table cannot express the +// difference either, so a reader looking at two 0.0 rows has no way to tell +// which one is evidence and which one is an absence of evidence. +// +// Every test below fails if that distinction collapses back into a bare +// float64. + +// TestRetryRateProvenance_MeasuredZeroIsNotDefaultZero is the load-bearing +// test of FR-013: the SAME numeric rate (0.0) reached two different ways must +// come back carrying two different provenances. +func TestRetryRateProvenance_MeasuredZeroIsNotDefaultZero(t *testing.T) { + const arm = "some_future_arm" // unknown to armRetryRates: default is 0.0 + + defaultRate, defaultProv := ResolveRetryRate(arm, nil) + measuredRate, measuredProv := ResolveRetryRate(arm, MeasuredOutcomes{ + arm: {RetryRate: 0.0, FirstAttemptSuccessPct: 88.0, Runs: 4}, + }) + + if defaultRate != 0.0 || measuredRate != 0.0 { + t.Fatalf("test premise broken: rates should both be 0.0, got default=%v measured=%v", + defaultRate, measuredRate) + } + if defaultProv != ProvenanceEstimated { + t.Errorf("unmeasured arm provenance = %q, want %q", defaultProv, ProvenanceEstimated) + } + if measuredProv != ProvenanceMeasured { + t.Errorf("measured 0.0 provenance = %q, want %q", measuredProv, ProvenanceMeasured) + } + if defaultProv == measuredProv { + t.Fatal("a measured 0.0 and a defaulted 0.0 are indistinguishable — FR-013 collapsed") + } +} + +// TestMeasuredRetryRateSupersedesDefault: where a measurement exists it wins, +// including where it contradicts the literature default, and the superseded +// default never leaks into the arithmetic. +func TestMeasuredRetryRateSupersedesDefault(t *testing.T) { + const arm = "toon_listing" // documented default 0.05 + measured := MeasuredOutcomes{arm: {RetryRate: 0.20, FirstAttemptSuccessPct: 61.5, Runs: 6}} + + rate, prov := ResolveRetryRate(arm, measured) + if rate != 0.20 { + t.Errorf("ResolveRetryRate = %v, want the measured 0.20, not the default %v", + rate, RetryRateForArm(arm)) + } + if prov != ProvenanceMeasured { + t.Errorf("provenance = %q, want %q", prov, ProvenanceMeasured) + } + + // The estimator must price the row with the measured rate. + row := EstimateSessionCostRow(arm, 1000, 100, 5, measured) + if row.RetryRate != 0.20 { + t.Errorf("row.RetryRate = %v, want 0.20 (the measurement)", row.RetryRate) + } + if want := 1000 + 5*100*1.20; float64(row.EstimatedTokens) != want { + t.Errorf("row.EstimatedTokens = %d, want %v (measured rate applied)", row.EstimatedTokens, want) + } + if row.RetryRateProvenance != ProvenanceMeasured { + t.Errorf("row.RetryRateProvenance = %q, want %q", row.RetryRateProvenance, ProvenanceMeasured) + } + // A row whose behavioural input was measured is arithmetic over measured + // inputs — computed — but never "measured": its token magnitudes still + // come from the deterministic tokenizer, not from provider usage. + if row.Provenance != ProvenanceComputed { + t.Errorf("row.Provenance = %q, want %q", row.Provenance, ProvenanceComputed) + } + if row.MeasuredRuns != 6 { + t.Errorf("row.MeasuredRuns = %d, want 6", row.MeasuredRuns) + } + if row.FirstAttemptSuccessPct == nil { + t.Fatal("row.FirstAttemptSuccessPct is nil despite a measurement (FR-013 covers success too)") + } + if *row.FirstAttemptSuccessPct != 61.5 { + t.Errorf("row.FirstAttemptSuccessPct = %v, want 61.5", *row.FirstAttemptSuccessPct) + } +} + +// TestUnmeasuredRowStaysEstimated: with no measurement the row keeps the +// documented default AND stays badged estimated, with no fabricated success +// figure (nil, not 0 — a 0 would read as "measured, and nothing succeeded"). +func TestUnmeasuredRowStaysEstimated(t *testing.T) { + row := EstimateSessionCostRow("toon_listing", 1000, 100, 5, MeasuredOutcomes{}) + if row.RetryRate != 0.05 { + t.Errorf("row.RetryRate = %v, want the documented default 0.05", row.RetryRate) + } + if row.RetryRateProvenance != ProvenanceEstimated { + t.Errorf("row.RetryRateProvenance = %q, want %q", row.RetryRateProvenance, ProvenanceEstimated) + } + if row.Provenance != ProvenanceEstimated { + t.Errorf("row.Provenance = %q, want %q", row.Provenance, ProvenanceEstimated) + } + if row.FirstAttemptSuccessPct != nil { + t.Errorf("row.FirstAttemptSuccessPct = %v, want nil (no measurement exists)", *row.FirstAttemptSuccessPct) + } + if row.MeasuredRuns != 0 { + t.Errorf("row.MeasuredRuns = %d, want 0", row.MeasuredRuns) + } +} + +// TestMeasuredOutcomeWithoutRunsIsNotAMeasurement: a zero-value outcome +// sitting in the map (an arm the loop was configured for but never ran) is an +// absence, not a measurement of zero — the same hazard one level up. +func TestMeasuredOutcomeWithoutRunsIsNotAMeasurement(t *testing.T) { + measured := MeasuredOutcomes{"toon_listing": {}} // Runs == 0 + rate, prov := ResolveRetryRate("toon_listing", measured) + if rate != 0.05 { + t.Errorf("rate = %v, want the default 0.05 — a zero-run entry is not a measurement", rate) + } + if prov != ProvenanceEstimated { + t.Errorf("provenance = %q, want %q for a zero-run entry", prov, ProvenanceEstimated) + } +} + +// TestSessionCostRows_MeasuredAndEstimatedCoexist: the whole point of per-ROW +// provenance — one table, two provenances, each row self-describing, in +// deterministic order. +func TestSessionCostRows_MeasuredAndEstimatedCoexist(t *testing.T) { + means := map[string]float64{"baseline_json": 200, "toon_listing": 100} + measured := MeasuredOutcomes{"toon_listing": {RetryRate: 0.20, FirstAttemptSuccessPct: 61.5, Runs: 6}} + + rows := EstimateSessionCostRows(1000, means, measured) + if want := len(means) * len(DefaultCallsPerSession()); len(rows) != want { + t.Fatalf("rows = %d, want %d (one per arm per calls point)", len(rows), want) + } + + byArm := map[string]string{} + for _, r := range rows { + if prev, seen := byArm[r.Arm]; seen && prev != r.Provenance { + t.Errorf("arm %q rows disagree on provenance: %q vs %q", r.Arm, prev, r.Provenance) + } + byArm[r.Arm] = r.Provenance + if !IsValidProvenance(r.Provenance) { + t.Errorf("row %+v carries provenance %q outside the closed enum", r, r.Provenance) + } + if !IsValidProvenance(r.RetryRateProvenance) { + t.Errorf("row %+v carries retry-rate provenance %q outside the closed enum", r, r.RetryRateProvenance) + } + } + if byArm["baseline_json"] != ProvenanceEstimated { + t.Errorf("unmeasured arm badged %q, want %q", byArm["baseline_json"], ProvenanceEstimated) + } + if byArm["toon_listing"] != ProvenanceComputed { + t.Errorf("measured arm badged %q, want %q", byArm["toon_listing"], ProvenanceComputed) + } + + // Deterministic ordering (FR-010): arms lexicographic, calls ascending. + if rows[0].Arm != "baseline_json" || rows[0].CallsPerSession != 1 { + t.Errorf("first row = %+v, want baseline_json @ 1 call", rows[0]) + } + + // The per-row provenance must survive serialization: this is the field a + // report consumer badges each row with. + data, err := json.Marshal(rows[0]) + if err != nil { + t.Fatalf("marshal row: %v", err) + } + var doc map[string]any + if err := json.Unmarshal(data, &doc); err != nil { + t.Fatalf("unmarshal row: %v", err) + } + for _, key := range []string{"arm", "calls_per_session", "retry_rate", "estimated_tokens", "provenance", "retry_rate_provenance"} { + if _, ok := doc[key]; !ok { + t.Errorf("serialized row missing key %q (got %v)", key, doc) + } + } +} + +// TestEstimateSessionCostsUnchangedWithoutMeasurements guards the existing +// pipeline: with no measurements supplied the legacy rows are untouched, so +// adding FR-013 does not silently move any published figure. +func TestEstimateSessionCostsUnchangedWithoutMeasurements(t *testing.T) { + means := map[string]float64{"baseline_json": 200, "toon_listing": 100} + legacy := EstimateSessionCosts(1000, means) + rows := EstimateSessionCostRows(1000, means, nil) + if len(legacy) != len(rows) { + t.Fatalf("row counts differ: legacy %d, rows %d", len(legacy), len(rows)) + } + for i := range legacy { + if !reflect.DeepEqual(legacy[i], rows[i].SessionCostEstimate) { + t.Errorf("row %d diverges from the legacy estimate:\n got %+v\n want %+v", + i, rows[i].SessionCostEstimate, legacy[i]) + } + } +} + +// --------------------------------------------------------------------------- +// FR-023 — the cost-versus-outcome view (T046) +// --------------------------------------------------------------------------- +// +// Placement note: these exercise bench/report.go, whose companion +// report_test.go is owned by another workstream on this branch. Go does not +// care which _test.go file in a package holds a test, so they live here beside +// the other FR-013/FR-023 work rather than being dropped for want of a file. +// +// What they pin: a reader must be able to see which modes are WORTH their +// savings, not merely which are cheapest. That requires completion outcome +// beside cost at equal prominence (FR-018) and a cell that is cheap-but-worse +// visibly marked a regression rather than quietly winning the cost column. + +// TestCostOutcomeView_PlotsEveryHonestCell: one point per cell that has an +// honest figure, in cell order, on a zero-based cost axis. +func TestCostOutcomeView_PlotsEveryHonestCell(t *testing.T) { + block := sampleAgentLoopBlock() + view := NewCostOutcomeView(block) + + if len(view.Points) != len(block.Cells) { + t.Fatalf("plotted %d points, want %d (one per cell)", len(view.Points), len(block.Cells)) + } + for i, p := range view.Points { + cell := block.Cells[i] + if p.CellID != cell.CellID { + t.Errorf("point %d is %q, want %q — cell order must be preserved", i, p.CellID, cell.CellID) + } + if p.TokensPerCompletedTask != cell.TokensPerCompletedTask { + t.Errorf("point %q cost = %v, want %v", p.CellID, p.TokensPerCompletedTask, cell.TokensPerCompletedTask) + } + if p.CompletionRatePct != cell.CompletionRatePct { + t.Errorf("point %q completion = %v, want %v", p.CellID, p.CompletionRatePct, cell.CompletionRatePct) + } + if p.Regression != cell.Regression { + t.Errorf("point %q regression = %v, want %v", p.CellID, p.Regression, cell.Regression) + } + if p.Provenance != cell.Provenance { + t.Errorf("point %q provenance = %q, want %q", p.CellID, p.Provenance, cell.Provenance) + } + } + + // The cost axis starts at zero: a truncated axis makes a small saving look + // like a large one, which is exactly the misreading this view exists to + // prevent. + if view.CostAxisMaxTokens < 41000 { + t.Errorf("cost axis max = %v, want at least the most expensive cell (41000)", view.CostAxisMaxTokens) + } + // The completion axis is fixed 0..100 so cells stay comparable between runs. + if view.CompletionAxisMaxPct != 100 { + t.Errorf("completion axis max = %v, want a fixed 100", view.CompletionAxisMaxPct) + } + for _, p := range view.Points { + if p.PlotY != view.PlotY(p.CompletionRatePct) { + t.Errorf("point %q PlotY inconsistent with the axis mapping", p.CellID) + } + } +} + +// TestCostOutcomeView_ExcludesWithheldCells: a cell whose figure was withheld +// has no honest position on either axis. It must be named as excluded rather +// than plotted at the origin, where it would read as "free and never +// completes". +func TestCostOutcomeView_ExcludesWithheldCells(t *testing.T) { + block := sampleAgentLoopBlock() + block.Cells = append(block.Cells, AgentLoopCell{ + CellID: "direct_deferred", Provenance: ProvenanceMeasured, Runs: 4, + Withheld: true, WithheldReason: "cross-accounting-source aggregate", + }) + + view := NewCostOutcomeView(block) + for _, p := range view.Points { + if p.CellID == "direct_deferred" { + t.Error("a withheld cell was plotted — it has no honest coordinates") + } + } + if len(view.Excluded) != 1 || view.Excluded[0].CellID != "direct_deferred" { + t.Fatalf("withheld cell not reported as excluded: %+v", view.Excluded) + } + if !strings.Contains(view.Excluded[0].Reason, "cross-accounting-source") { + t.Errorf("exclusion reason = %q, want the withheld reason carried through", view.Excluded[0].Reason) + } +} + +// TestCostOutcomeView_SingleCellDoesNotDivideByZero: one cell (or several with +// identical cost) must not produce NaN positions. +func TestCostOutcomeView_SingleCellDoesNotDivideByZero(t *testing.T) { + view := NewCostOutcomeView(&AgentLoopBlock{Cells: []AgentLoopCell{{ + CellID: "solo", Provenance: ProvenanceMeasured, Runs: 4, + TokensPerCompletedTask: 0, CompletionRatePct: 0, + }}}) + if len(view.Points) != 1 { + t.Fatalf("points = %d, want 1", len(view.Points)) + } + p := view.Points[0] + if math.IsNaN(p.PlotX) || math.IsNaN(p.PlotY) || math.IsInf(p.PlotX, 0) || math.IsInf(p.PlotY, 0) { + t.Errorf("degenerate axis produced non-finite coordinates: %+v", p) + } +} + +// TestCostOutcomeView_NilBlock: an absent agent loop renders nothing rather +// than panicking inside the template. +func TestCostOutcomeView_NilBlock(t *testing.T) { + view := NewCostOutcomeView(nil) + if len(view.Points) != 0 || len(view.Excluded) != 0 { + t.Errorf("nil block produced a view: %+v", view) + } +} + +// TestDashboardV2_CostVersusOutcomeSection renders the real dashboard and +// checks the section carries what FR-018/FR-023 require. +func TestDashboardV2_CostVersusOutcomeSection(t *testing.T) { + rep := sampleReportV2WithBlocks() + html := renderDashboardV2(t, rep) + + // The two headline figures, named, at equal prominence. + for _, want := range []string{"Tokens per completed task", "Completion rate"} { + if !strings.Contains(html, want) { + t.Errorf("cost-versus-outcome section missing %q", want) + } + } + + // Both figures for every cell, plus the per-ROW provenance badge — a cell + // still carrying an assumption must be visible as such beside its + // measured neighbours (FR-013). + for _, cell := range rep.AgentLoop.Cells { + for _, want := range []string{ + cell.CellID, + strconv.FormatFloat(cell.TokensPerCompletedTask, 'f', -1, 64), + strconv.FormatFloat(cell.CompletionRatePct, 'f', 1, 64), + } { + if !strings.Contains(html, want) { + t.Errorf("cell %q: dashboard missing %q", cell.CellID, want) + } + } + // Assert the badge sits in THIS ROW, not merely somewhere on the page. + // A bare Contains passes off the page-wide provenance legend and the + // static "estimated" footnote, so deleting the whole provenance column + // left it green — the assertion could not see its own subject. + row := rowFor(t, html, cell.CellID) + if !strings.Contains(row, `badge badge-`+cell.Provenance) { + t.Errorf("cell %q: its own row carries no provenance badge %q; row was:\n%s", + cell.CellID, cell.Provenance, row) + } + } + + // The accounting source, with the pinned model, and the never-sum rule. + for _, want := range []string{ + rep.AgentLoop.AccountingSource.Identity, + rep.AgentLoop.AccountingSource.Model, + rep.AgentLoop.Suite, + } { + if !strings.Contains(html, want) { + t.Errorf("cost-versus-outcome section missing accounting context %q", want) + } + } + if !strings.Contains(strings.ToLower(html), "never summed") { + t.Error("section does not state that provider-usage figures are never summed with tokenizer figures") + } + + // The cheap-but-worse cell must be marked a regression, not presented as a + // saving (FR-019), and the under-powered cell must not be a headline + // (FR-021). + // Both of these previously matched STATIC TEMPLATE PROSE — the legend + // sentence "Red: a completion regression..." contains "regression", and + // "at least four runs" contains "runs" — so they passed with no regressed + // cell present and with the run-count column deleted entirely. Scope them + // to the row that is supposed to carry the mark. + var regressed, provisional *AgentLoopCell + for i := range rep.AgentLoop.Cells { + c := &rep.AgentLoop.Cells[i] + if c.Regression { + regressed = c + } + if !c.Headline && !c.Withheld && c.Runs > 0 { + provisional = c + } + } + if regressed == nil { + t.Fatal("fixture must contain a regressed cell for this assertion to mean anything") + } + if row := rowFor(t, html, regressed.CellID); !strings.Contains(strings.ToUpper(row), "REGRESSION") { + t.Errorf("the regressed cell %q must be marked in its own row; row was:\n%s", regressed.CellID, row) + } + if provisional != nil { + row := rowFor(t, html, provisional.CellID) + if !strings.Contains(row, strconv.Itoa(provisional.Runs)+" runs") { + t.Errorf("cell %q must show its run count in its own row; row was:\n%s", provisional.CellID, row) + } + } + + // The plot itself: an inline SVG with both axes labelled. + if !strings.Contains(html, " containing the given cell id, so a row-level +// assertion cannot be satisfied by static prose elsewhere on the page. Three +// assertions in this file were vacuous for exactly that reason. +func rowFor(t *testing.T, html, cellID string) string { + t.Helper() + needle := "" + cellID + "" + i := strings.Index(html, needle) + if i < 0 { + t.Fatalf("cell %q does not appear in the rendered dashboard at all", cellID) + } + start := strings.LastIndex(html[:i], "") + end := strings.Index(html[i:], "") + if start < 0 || end < 0 { + t.Fatalf("cell %q is not inside a table row", cellID) + } + return html[start : i+end] +} diff --git a/specs/103-token-bench/blog-draft.md b/specs/103-token-bench/blog-draft.md new file mode 100644 index 000000000..fb0507113 --- /dev/null +++ b/specs/103-token-bench/blog-draft.md @@ -0,0 +1,296 @@ +# DRAFT — "We projected 70% and measured 29.7%" (write-up for mcpproxy.app/blog) + +> **Status: DRAFT, not published.** The blog lives in a separate repository +> (`mcpproxy.app-website`). This file is spec 103 T060: the text, the figures and +> the provenance of every figure, staged for review here so it can be checked +> against the harness that produced it before it becomes a public claim. +> +> **Publish as an extension of the existing 2026-03-19 post, "BM25 vs Embeddings +> vs Lua"** — not as a second post. That post argued the retrieval design; this +> is what the design measured. Duplicating its setup would leave two posts with +> two half-answers. +> +> **Blocking before publication** (FR-031, FR-032, SC-004, SC-012): +> 1. US2 has not run — every tokens-per-*completed*-task figure below is marked +> *not yet measured*, and the post must not imply otherwise. +> 2. The tokenizer↔model divergence tolerance (FR-022) is not established; the +> repo's ~60% and the provider's ~15–20% guidance differ threefold and +> neither is sourced. +> 3. The negative live result in §4 is a single observation on one fleet. It +> stays in the post — it is the most useful thing in it — but it must be +> labelled as one run, not as a curve. +> 4. §6's reproduction line for the payload decomposition points at the +> harness's tests, because the decomposition has no CLI flag today. Re-check +> it against the shipped flags at publication time; a reproduction procedure +> that does not run is worse than none. + +--- + +## 1. We projected ~70% and measured 29.7% + +Spec 102 deferred tool input schemas off the direct surface: the agent gets +names, descriptions and a pointer, and fetches full schemas only for the tools +it actually intends to call. The projection was **~70%** savings on the tool +surface. + +The measurement was **29.7%** on the 45-tool reference corpus and **34.8%** on a +527-tool snapshot. + +That is the interesting result, and this post is about the gap rather than about +the win. Two questions follow from it, and we can answer one of them properly +today: + +* **Where does the payload actually go**, if not into schemas? (§2 — answered) +* **Does a smaller menu make an agent finish more work per token?** (§5 — *not + yet measured*, and no figure in this post claims it) + +**Fleet shapes.** 45 tools: the frozen reference corpus, 7 no-auth servers +(filesystem 14, git 12, memory 9, sqlite 6, time 2, fetch 1, +sequential-thinking 1), every tool carrying a full JSON input schema, frozen +2026-07-14. 527 tools: the LiveMCPTool snapshot, 70 servers. A percentage +without its fleet shape is not a result, so every number below carries one. + +--- + +## 2. Where the payload actually goes + +We decomposed the rendered tool-definition payload of the 45-tool corpus by +attributing every token to a component. Tokens are **not** additive across +concatenation — BPE merges across component boundaries — so the payload is +tokenized once and each token attributed by byte offset, rather than counting +components separately and hoping the shares sum. + +| Component | Share of payload (45-tool corpus) | +|---|---:| +| Input schema | **59.4%** | +| Description | 36.5% | +| Name | 2.4% | +| Structural (separators) | 1.7% | + +*Provenance: measured, deterministic tokenizer (`cl100k_base`), 45-tool frozen +corpus, all 45 tools schema-bearing.* + +**Read this carefully, because it is easy to over-read.** + +* On **this** corpus, schemas dominate — so "descriptions are the real cost" is + not true here. Deferring schemas has room to work. +* It still does **not** explain the 29.7%, and we are not claiming it does. + Spec 102 measured the production **wire** payload, which additionally carries + annotation defaults and wire framing; this decomposition covers the corpus + rendering (name + description + schema). Adjudicating the wire figure with + corpus-rendering evidence would be answering a different question in a + confident voice. The harness says so in the report, next to the verdict. +* The 527-tool snapshot **cannot** answer this question at all: all 527 of its + tools carry zero schemas. A 0% schema share there means the corpus lacks the + thing under test — not that schemas are cheap. Any decomposition that reported + it as a confirmation would be measuring its own dataset gap. + +So: the achievable ceiling is recomputed **per fleet**, never carried forward. +Carrying a ceiling forward is exactly the error that produced the 70% +projection. + +--- + +## 3. What deferring schemas saves on a recorded workload + +We replayed a real recorded workload — the actual sequence of tool calls a real +agent made — and recomputed what it would have cost under each mode, against the +45-tool fleet. + +**`direct_full` → `direct_deferred`: 1296 tokens, 29.5%.** + +*Provenance: measured, deterministic tokenizer, 45-tool frozen corpus. +Counterfactual over recorded traffic.* + +Three things this figure is not: + +1. **It is not observed agent behaviour.** A recording carries the calls, the + arguments and the responses. It does not carry the prompt, the conversation, + the model's state, or any oracle for whether the user's goal was met. Replay + can say what the same call shape would have cost under a different + serialization. It cannot say what the agent would have *done* — which calls + it would have attempted, whether the first attempt would have been right, + whether it would have finished. Those are new decisions by a model that is + not in the recording. Every replay figure is labelled a counterfactual for + this reason. +2. **It is not a historical reconstruction.** The workload is scored against the + fleet you supply, not the fleet as it stood when the session was recorded. + That is internally valid across modes and is not a claim about the past. +3. **It is not a complete workload cost.** By default replay does not read + recorded bodies (they are raw, unmasked user traffic), so absolute + whole-workload cost is reported as *unavailable* — never as zero. The + `direct_full` vs `direct_deferred` delta survives bodies-off precisely + because the call responses are identical between those two modes and cancel. + +29.5% here and 29.7% in spec 102 are the same order of magnitude from two +independent code paths on the same fleet size. That is reassuring. It is not a +confirmation, because they are measured over different payloads (§2). + +--- + +## 4. Where the savings do not materialise + +**On a 45-tool fleet, a live run showed `retrieve_tools` costing MORE than +simply listing every tool.** Negative savings. The proxy lost. + +*Provenance: measured live, single run, 45-tool fleet. One observation, not a +curve — k ≥ 4 with spread is not established for it.* + +The mechanism is not mysterious, and it generalises: + +* A proxy mode has a **fixed floor**. `retrieve_tools`, the `call_tool_*` + variants, `describe_tool`, `read_cache` and the management tools are in + context on every turn, whether or not they are used. That floor is a constant + number of tokens. +* A naive menu has **no floor and a slope**: it costs whatever the fleet costs, + and nothing more. +* Below some fleet size the floor exceeds the menu. At 45 tools — seven small + servers — we are below it. + +So the honest shape of the claim is: **mcpproxy's savings scale with fleet size, +and on a small fleet it is overhead.** If you run one or two servers with a few +dozen tools between them, load them directly. The proxy earns its floor back +when the menu gets long — the regime the 527-tool snapshot represents — and the +quarantine, isolation and observability features may still be worth it to you at +any size, but that is a different argument and should not be smuggled in as a +token saving. + +**The crossover point is not yet measured.** We can tell you the direction and +the mechanism, and we can tell you that 45 tools is on the wrong side of it. We +cannot yet tell you where it is, and we are not going to guess in public. + +--- + +## 5. What we have not measured + +Listing this is the point of the post, not an appendix to it. + +| Question | Status | +|---|---| +| Tokens per **completed** task, per mode | **Not yet measured** — needs a live agent loop under a pinned model | +| Task completion rate per mode | **Not yet measured** | +| First-attempt success rate per mode | **Not yet measured** | +| Corrective vs infrastructure retry rates | **Not yet measured** | +| Cache-read token split | **Not yet measured** — the public suite's per-task output has no cache-read field; it must come from the provider response | +| The fleet size at which the proxy breaks even | **Not yet measured** | +| Tokenizer-vs-model divergence tolerance | **Not yet established** (see the draft header) | + +Everything in §§2–4 is deterministic arithmetic: same inputs, same numbers, no +model involved. Everything in this table needs a model, real spend, and at least +four runs per cell before it is worth printing. + +**Why the distinction is load-bearing:** menu size is not work done. A mode that +shows a smaller menu and causes one extra failed call has saved nothing. Until +the table above is filled, every saving on this page is a statement about +context cost, not about productivity — and we would rather say that plainly than +let a reader infer the stronger claim. + +We also hold ourselves to four rules, which are worth stating because they +constrain what we are allowed to publish next: + +* **Never sum accounting sources.** Deterministic figures come from the tiktoken + tokenizer; live figures come from provider-reported usage. A cross-source + aggregate is withheld with a stated reason, never computed. +* **Never quote a percentage without its fleet shape.** +* **Never report a mode that costs less while completing less as a saving.** It + is a regression, whatever the token figure says. +* **Never let a truncated record contribute silently.** Truncation understates + cost *in our favour*, which is the failure this whole exercise exists to + prevent; excluded records are counted and reported. + +--- + +## 6. Reproducing this + +The deterministic figures reproduce exactly. Nothing here needs an API key or +model spend. + +```bash +git clone https://github.com/smart-mcp-proxy/mcpproxy-go && cd mcpproxy-go + +# The tokenizer downloads its vocabulary once. Setting the cache variable only +# NAMES a cache; it does not fill one. Warm it with network access once: +export TIKTOKEN_CACHE_DIR="$HOME/.cache/tiktoken" +go run ./bench/cmd/bench \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \ + -out /tmp/warm + +# The frozen-corpus mode comparison across every encoding arm: +go run ./bench/cmd/bench \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json \ + -arms all -out bench/results + +# §2, the payload decomposition. NOTE FOR REVIEW: the decomposition has no CLI +# flag yet — today it is exercised from the harness's own tests. Replace this +# with the shipped invocation before publishing, or the procedure is wrong. +go test ./bench/ -run TestPayloadDecomposition -count=1 +``` + +For §3 you supply your own recording — we cannot ship ours, and that is a +limitation, not modesty (see §7): + +```bash +# On a machine that has been using mcpproxy for real work: +mcpproxy activity export --format json > ~/replay-corpus.jsonl + +# A fleet input is MANDATORY: a recording alone has no menu to cost, so a +# recording-only invocation is a hard error, not a degraded run. +CORPUS=specs/083-discovery-profiler/datasets/corpus_v2.tools.json +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-a +go run ./bench/cmd/bench -replay ~/replay-corpus.jsonl -corpus-v2 $CORPUS -out /tmp/run-b + +# Two runs must be byte-identical once the wall-clock stamp is removed: +diff <(jq 'del(.generated_at)' /tmp/run-a/report.json) \ + <(jq 'del(.generated_at)' /tmp/run-b/report.json) + +rm -f ~/replay-corpus.jsonl # it is raw user traffic; delete it when done +``` + +Reports are never committed — only the code, the fixtures and the methodology +are versioned, so the numbers you get are the numbers your machine computed. + +--- + +## 7. Limitations, in one place + +* **The replayed workload is private.** It is real user traffic, so it cannot be + published, which means §3's figure is **not independently reproducible** on + our inputs — only on yours. It is deliberately not the sole support for any + claim here. +* **Frozen corpora are not your fleet.** 45 tools over 7 small servers and a + 527-tool snapshot are two points. Your tool descriptions are probably longer + than ours, which moves §2's split. +* **The 527-tool snapshot carries no schemas**, so it bounds nothing about + schema deferral. +* **§4 is one live run**, on one fleet, with no spread. +* **Nothing here measures success.** See §5. +* **The tokenizer is not the model's tokenizer.** Deterministic figures are + computed with `cl100k_base`; a pinned model will count differently, and the + size of that divergence is not yet established. + +--- + +## 8. So which mode should you run? + +| Your fleet | What the measurements support | +|---|---| +| A handful of servers, tens of tools | **Load them directly.** At 45 tools the proxy's discovery surface cost more than the full menu (§4). Use mcpproxy for quarantine/isolation/observability if you want those, not for token savings. | +| Hundreds of tools, many servers | The regime the design targets. Deferring schemas removed 29.5% of the tool-surface cost on our recorded workload (§3) and schemas are 59.4% of the payload on the schema-bearing corpus (§2) — but we have not measured the crossover, so measure your own fleet with the commands in §6. | +| Anything, and you care about completed work | **We do not yet have a number for you.** §5. | + +--- + +### Provenance table (every figure on this page) + +| Figure | Value | Source | Fleet shape | Provenance | +|---|---|---|---|---| +| Spec 102 projection | ~70% | spec 102 | — | projected, superseded | +| Spec 102 measurement | 29.7% | spec 102 | 45 tools (wire payload) | measured | +| Spec 102 measurement | 34.8% | spec 102 | 527 tools | measured | +| Schema share | 59.4% | US4 decomposition | 45 tools (corpus rendering) | measured, tokenizer | +| Description share | 36.5% | US4 decomposition | 45 tools | measured, tokenizer | +| Name share | 2.4% | US4 decomposition | 45 tools | measured, tokenizer | +| Structural share | 1.7% | US4 decomposition | 45 tools | measured, tokenizer | +| `direct_full` → `direct_deferred` | 1296 tokens / 29.5% | US1 replay | 45 tools | measured, tokenizer, **counterfactual** | +| `retrieve_tools` vs naive menu | negative savings | live run | 45 tools | measured, single run | +| Everything in §5 | — | — | — | **not yet measured** | diff --git a/specs/103-token-bench/findings/webui-savings-vs-bench.md b/specs/103-token-bench/findings/webui-savings-vs-bench.md new file mode 100644 index 000000000..2dfda04f8 --- /dev/null +++ b/specs/103-token-bench/findings/webui-savings-vs-bench.md @@ -0,0 +1,99 @@ +# Finding: the Web UI's savings figure and the bench headline disagree in SIGN + +**Task**: T061 — compare the Web-UI/status savings figure against the bench headline over one live fleet. +**Date**: 2026-09-01 +**Status**: reproduced, mechanism established, not yet fixed. + +Spec 103's own Assumptions say a contradiction between two measurements over the same +data is *a finding to investigate*. This is that contradiction, and it is not a rounding +difference. + +## What was measured + +One isolated mcpproxy instance, the committed 7-server reference config, 45 upstream +tools, both figures read from the same running process: + +| Source | Figure | +|---|---| +| `GET /api/v1/stats/tokens` (Web UI / status) | **+68.4% saved** (4684 → 1478, saved 3206) | +| `bench -live` headline, `retrieve_tools` cell | **−32.5%** (baseline 4364, proxy menu 5783) | + +Same fleet. Opposite sign. One says the proxy saves two thirds of the tool-surface cost; +the other says it costs a third more than not using it. + +## Why they disagree + +`SavingsCalculator.CalculateProxySavings` (`internal/server/tokens/savings.go`) computes: + +``` +SavedTokens = totalTokens - avgQuerySize +``` + +- `totalTokens` — every upstream tool definition (4684 here) +- `avgQuerySize` — one simulated `retrieve_tools` result of `topK` tools (1478 here) + +**The proxy's own menu is never subtracted.** An agent talking to mcpproxy carries +`retrieve_tools`, the three `call_tool_*` variants, `read_cache`, `code_execution` and the +management tools in its context for the whole session — 5783 tokens on this fleet, which is +*larger than the entire upstream baseline it replaces*. That term does not appear in the +formula at all. + +The bench headline includes it, which is why the two disagree, and why they disagree by +roughly the size of the proxy menu. + +## The second problem, which is worse + +```go +metrics.SavedTokens = totalTokens - avgQuerySize +if metrics.SavedTokens < 0 { + metrics.SavedTokens = 0 +} +``` + +Negative savings are **clamped to zero**. So the figure is structurally incapable of +reporting the state bench just measured — that at this fleet size the proxy costs more than +it saves. The clamp guarantees the number is never bad news, which is precisely the property +a savings metric must not have. + +## Why this is not merely a benchmark disagreement + +The bench figure is internal. The Web-UI figure is **shown to users**. It is optimistic by +construction in two independent ways: it omits the dominant cost term at small fleet sizes, +and it cannot express the case where the product does not pay off. + +Neither is a rounding problem, and neither is visible to a reader of the number. + +## What is NOT claimed here + +- **This does not mean mcpproxy does not save tokens.** Savings scale with fleet size: the + proxy menu is a fixed cost while the upstream catalog grows linearly, so the sign flips + once the fleet is large enough. 45 tools is simply below that crossover. Bench's own + break-even analysis exists to locate it. +- **It does not mean the Web-UI number is fabricated.** It answers a coherent question — + "how much smaller is one query result than the whole catalog?" — but that is not the + question its label implies, and it is not what a user pays. +- The `topK` sample is also an arbitrary prefix of the tool list rather than a BM25 ranking + (documented in `estimateQueryResultSize`), so `avgQuerySize` is a stand-in, not a measured + retrieval result. That is a third, smaller inaccuracy in the same figure. + +## Suggested direction + +1. Subtract the proxy menu, so the figure answers what the user pays. `ProxyModeToolDefs` + already supplies the per-mode menu with real schemas, and bench proves the two agree + to the token. +2. Remove the clamp and let the figure go negative, with the UI presenting a small fleet + honestly ("your fleet is below the break-even point") rather than showing zero. +3. Quote the fleet size alongside the percentage, as spec 103 requires of every published + percentage — the number is meaningless without it. + +## Reproduction + +```bash +# isolated instance, scratch data-dir and config, high port +mcpproxy serve --config /mcp_config.json --data-dir --listen 127.0.0.1:18733 + +curl -s -H 'X-API-Key: ' http://127.0.0.1:18733/api/v1/stats/tokens + +go run ./bench/cmd/bench -live -proxy http://127.0.0.1:18733 -api-key \ + -corpus-v2 specs/083-discovery-profiler/datasets/corpus_v2.tools.json -out /bench +``` diff --git a/specs/103-token-bench/tasks.md b/specs/103-token-bench/tasks.md index 94d825a37..923b34ce8 100644 --- a/specs/103-token-bench/tasks.md +++ b/specs/103-token-bench/tasks.md @@ -191,56 +191,56 @@ appear per cell, badged `measured`, averaged over k≥4 runs with spread. ### Tests for User Story 2 ⚠️ -- [ ] T034 [US2] Classification tests in `bench/agentloop_test.go` for the binding Definitions: +- [x] T034 [US2] Classification tests in `bench/agentloop_test.go` for the binding Definitions: first-attempt success, corrective vs infrastructure retry, unit of work, task completion. **The same rule must apply to the baseline arm and the proxy arms**, or the comparison is biased toward whichever carries richer error signal -- [ ] T035 [US2] In `bench/agentloop_test.go`: `completion: no-signal` excludes a record from +- [x] T035 [US2] In `bench/agentloop_test.go`: `completion: no-signal` excludes a record from completion-dependent figures rather than counting it as success or failure. Same file as the task above, so not parallel -- [ ] T036 [US2] In `bench/agentloop_test.go`: a model-dependent figure with `runs < 4` is +- [x] T036 [US2] In `bench/agentloop_test.go`: a model-dependent figure with `runs < 4` is refused as a headline (FR-021) -- [ ] T037 [US2] In `bench/agentloop_test.go`: a mode with lower token cost but lower completion +- [x] T037 [US2] In `bench/agentloop_test.go`: a mode with lower token cost but lower completion is flagged a regression by the completion threshold and is NOT reported as a saving (SC-007) — drive it with a deliberately degraded mode -- [ ] T038 [US2] In `bench/agentloop_test.go`: a cross-accounting-source aggregate is WITHHELD +- [x] T038 [US2] In `bench/agentloop_test.go`: a cross-accounting-source aggregate is WITHHELD with a stated reason, never computed, mirroring the existing `AuthoritativeHeadline`/`withholdHeadline` pattern in `bench/live_report.go` ### Implementation for User Story 2 -- [ ] T039 [US2] Create `bench/agentloop.go` — driver parameterised over a function type so the +- [x] T039 [US2] Create `bench/agentloop.go` — driver parameterised over a function type so the arithmetic is unit-testable with no suite running, following the `bench/flipgate.go` precedent -- [ ] T040 [US2] Implement the FR-020 BASELINE ARM in `bench/agentloop.go`: the same agent +- [x] T040 [US2] Implement the FR-020 BASELINE ARM in `bench/agentloop.go`: the same agent running the same tasks with every upstream tool loaded directly, **bypassing mcpproxy entirely**. This is the denominator every published percentage is measured against, and without it no saving can be quoted. It is NOT the deterministic `baseline` renderer used by the offline arms -- [ ] T041 [US2] Ingest the suite's per-task `meta.json` (`execution_result.success`, +- [x] T041 [US2] Ingest the suite's per-task `meta.json` (`execution_result.success`, `token_usage`, `turn_count`) and `messages.json` trajectories for retry classification -- [ ] T042 [US2] Capture provider-reported `usage` in the driver for the input/output/cache-read +- [x] T042 [US2] Capture provider-reported `usage` in the driver for the input/output/cache-read split. **The suite's own output has no cache-read field**, so that axis must come from the driver or be explicitly declared out of reach (research.md) -- [ ] T043 [US2] Emit the `agent_loop` block from `bench/agentloop.go` with its own populated +- [x] T043 [US2] Emit the `agent_loop` block from `bench/agentloop.go` with its own populated `accounting_source` (provider + pinned model), never summed with tokenizer-sourced figures -- [ ] T044 [US2] Write a FAILING test in `bench/session_test.go`: where a measured retry rate +- [x] T044 [US2] Write a FAILING test in `bench/session_test.go`: where a measured retry rate exists it supersedes the assumed default, and where none exists the row stays badged `estimated`. `RetryRateForArm` returns `0.0` for unknown arms (`bench/session.go:49-62`), indistinguishable from a measured `0.0` — this test is what prevents the silent mix -- [ ] T045 [US2] Implement FR-013 in `bench/session.go`: measured success and retry rates +- [x] T045 [US2] Implement FR-013 in `bench/session.go`: measured success and retry rates supersede the literature-derived `armRetryRates` defaults wherever a measurement exists, and add a per-row provenance field to session-cost rows so measured and estimated rows are distinguishable inside one table -- [ ] T046 [US2] Implement the FR-023 COST-VERSUS-OUTCOME VIEW in `bench/report.go`: cost +- [x] T046 [US2] Implement the FR-023 COST-VERSUS-OUTCOME VIEW in `bench/report.go`: cost plotted against completion outcome per cell, so a reader sees which modes are worth their savings rather than only which are cheapest. Completion rate must sit beside cost at equal prominence (FR-018) -- [ ] T047 [US2] Patch the pinned suite's single MCP factory — `src/agents/mcpmark_agent.py` +- [x] T047 [US2] Patch the pinned suite's single MCP factory — `src/agents/mcpmark_agent.py` `_create_mcp_server()` in the external MCPMark clone, NOT a file in this repo — with an env-gated branch returning an HTTP MCP server at mcpproxy's URL with the API-key header. Pin by commit SHA and record the SHA in `bench/README.md` (FR-028) -- [ ] T048 [US2] In the benchmark's mcpproxy config (a scratch `mcp_config.json`, never +- [x] T048 [US2] In the benchmark's mcpproxy config (a scratch `mcp_config.json`, never committed), configure ALL suite services simultaneously while running one service's tasks — a single server's toolset is too small a fleet to show the asymptote, and the full fleet is also the honest FR-020 baseline @@ -288,24 +288,24 @@ recomputed per corpus and an explicit `confirmed`/`corrected` verdict. **Independent Test**: Someone with no prior context follows the procedure and reproduces the deterministic figures exactly, the model-dependent ones within the stated tolerance. -- [ ] T054 [US3] Write FAILING tests in `bench/reportv2_test.go` for the US3 report behaviours: +- [x] T054 [US3] Write FAILING tests in `bench/reportv2_test.go` for the US3 report behaviours: a private-session-derived figure is marked not-independently-reproducible; a partial run is marked partial and refused for publication - [ ] T055 [US3] Measure the tokenizer's divergence from the pinned model with the provider's token-counting endpoint (no inference spend) and record it in `bench/README.md` as FR-022's numeric tolerance. **The repo currently states ~60% while the provider's guidance says ~15–20%** — neither is sourced, and they differ threefold -- [ ] T056 [US3] In `bench/reportv2.go`, mark figures derived from private recorded sessions as +- [x] T056 [US3] In `bench/reportv2.go`, mark figures derived from private recorded sessions as NOT independently reproducible; they must never be the sole support for a published claim (FR-030) -- [ ] T057 [US3] In `bench/replay.go` and `bench/agentloop.go`, retain raw per-run records under +- [x] T057 [US3] In `bench/replay.go` and `bench/agentloop.go`, retain raw per-run records under the gitignored `bench/results/` and reference them from the report by RUN-LOCAL path, with an explicit note that they are not durable across a results cleanup. A report reference must degrade to "records not retained" rather than dangling (FR-029, and consistent with SC-011's never-committed rule) -- [ ] T058 [US3] In `bench/reportv2.go`, mark partial or interrupted runs as partial and block +- [x] T058 [US3] In `bench/reportv2.go`, mark partial or interrupted runs as partial and block them from publication (FR-032) -- [ ] T059 [US3] Write the reproduction procedure into `bench/README.md`, including warming the +- [x] T059 [US3] Write the reproduction procedure into `bench/README.md`, including warming the tokenizer cache and supplying a fleet input **Checkpoint**: The numbers survive outside scrutiny. @@ -316,7 +316,7 @@ deterministic figures exactly, the model-dependent ones within the stated tolera **Goal**: Results reach developers deciding whether to adopt. -- [ ] T060 [US5] Draft the write-up for `mcpproxy.app/blog` (separate repository): measured +- [x] T060 [US5] Draft the write-up for `mcpproxy.app/blog` (separate repository): measured figures, the fleet shapes they hold for, known limitations, the reproduction procedure, and **plainly where savings do not materialise** (FR-031, SC-012). Extend the existing 2026-03-19 "BM25 vs Embeddings vs Lua" post rather than duplicating it @@ -325,16 +325,16 @@ deterministic figures exactly, the model-dependent ones within the stated tolera ## Phase 8: Polish & Cross-Cutting -- [ ] T061 [P] Compare the Web-UI/status savings figure (`internal/server/tokens/savings.go`, +- [x] T061 [P] Compare the Web-UI/status savings figure (`internal/server/tokens/savings.go`, surfaced via `internal/runtime/runtime.go`) against the bench headline over one live fleet. They are independently implemented over the same data and have never been compared; the spec's own Assumptions say a contradiction is a finding to investigate -- [ ] T062 [P] Verify no generated report became tracked: `git ls-files bench/results` is empty -- [ ] T063 Full gates: `go build ./cmd/mcpproxy` and `-tags server`; +- [x] T062 [P] Verify no generated report became tracked: `git ls-files bench/results` is empty +- [x] T063 Full gates: `go build ./cmd/mcpproxy` and `-tags server`; `go test -race -count=1 ./internal/... ./bench/...`; `/opt/homebrew/bin/golangci-lint run --config .github/.golangci.yml ./...`; `./scripts/test-api-e2e.sh`; `make swagger` diff-clean -- [ ] T064 Cross-model review of the full diff (opencode `gpt-5.6-sol`) staged into +- [x] T064 Cross-model review of the full diff (opencode `gpt-5.6-sol`) staged into `.review-tmp/`, ≤10 rounds per the standing cap; verify each finding against the tree before fixing diff --git a/warmtiktoken b/warmtiktoken new file mode 100755 index 000000000..7b1d22707 Binary files /dev/null and b/warmtiktoken differ