Version 0.1. This document defines what inference-audit records, computes, and may claim.
An inference customer observes an HTTP contract, outputs, timing, and provider-reported accounting. They do not directly observe loaded weights, quantization, GPU state, chat-template expansion, or KV cache residency. A valid audit must keep those two sets separate.
The suite therefore reports boundary evidence. It may say that a request asking for reasoning-off returned billed reasoning tokens. It may not conclude which internal configuration caused that response. Missing evidence is never converted to zero or a pass.
Each JSONL line is one immutable observation with schema_version = 1 and:
- run, target, model, case, repeat, and provider identity;
- prompt SHA-256 and character count, optional stored prompt, expectation, reasoning mode, and temperature;
- start time, status, latency, optional TTFT, and for streamed requests whether the stream
terminated with
[DONE]; - content, reasoning, finish reason, response id, and served provider;
- reported prompt, completion, reasoning, cache-read, cache-write, and cost values;
- aligned logprob material returned by the endpoint;
- a bounded error description when the request or response failed.
Unknown schema versions are rejected. Prompts at most 4,096 characters are stored verbatim. Longer prompts are omitted from each record and remain reproducible from their stable built-in case and suite version; their digest still detects generation drift. Records do not contain request headers or credentials.
A completed tuple is (target, model, case, repeat, requested provider, reasoning mode, temperature). Resume skips only a completed tuple with no error. Changed settings produce new
evidence rather than overwriting prior evidence.
The same model, prompt, temperature, seed, token cap, reasoning setting, and logprob setting are sent to each target in a comparison. OpenRouter pins set an ordered provider list of one and disable fallbacks. A refusal is recorded as a refusal; the runner does not mutate a parameter and retry with a different experiment.
429 and transient 5xx responses and transport failures may be retried with bounded exponential backoff. Other 4xx responses are terminal. A run record contains the terminal attempt.
smoke contains five inexpensive contract cases. standard contains 47:
| Category | Count | Primary evidence |
|---|---|---|
| short | 20 | response agreement and simple instruction following |
| code | 8 | delivery and output behavior |
| JSON | 6 | parseable JSON |
| reasoning | 6 | configuration compliance and output accounting |
| multilingual | 4 | delivery across tokenizer-sensitive inputs |
| long context | 3 | deterministic needle recall at three lengths |
Long prompts are generated from a fixed local PRNG and vocabulary. They contain a deterministic key inserted between 25% and 75% of the filler. Passing means the exact key occurs in output.
The cache suite generates one deterministic prefix and a distinct short suffix per step. Its hit
rate is reported cached prompt tokens / reported prompt tokens when both are present and the
denominator is positive. Absent cache details are absent evidence, not a zero-percent hit.
- Expectation pass: exact equality, required substring, or successful JSON parsing as declared by the case.
- Self agreement: all normalized outputs for one target and case across repeats are identical.
- Consensus agreement: normalized output equals the modal normalized output across targets for the same case and repeat. Computed only for short, reasoning, and multilingual cases.
- Truncation:
finish_reasonislength, or an SSE stream ended without[DONE]. A stream that delivered no finish reason recordstruncated_stream; one that reported a finish reason keeps it as evidence, and the missing[DONE]is still recorded and counted. - Cut: a non-empty output is a strict prefix of another target's output for the same case and repeat.
- Prompt token ratio: reported prompt tokens divided by the cross-target modal count for the same case and repeat. The report uses the median ratio.
- Reasoning-off violation: non-empty reasoning content or a positive reported reasoning-token count when the request mode was off.
- Cache report rate: successful usage-bearing responses with a non-null cached-token count.
- Experimental KL: KL divergence between the first aligned content token's top-k distribution and the leave-one-out probability average of other targets. It requires three reporting targets.
Rates use applicable observations only. A metric with no applicable observations is null and renders as an em dash.
Threshold findings name:
- request errors;
- any reported or observed truncation;
- a median prompt-token ratio outside
[0.95, 1.05]; - any reasoning-off contradiction;
- no cached-token detail across otherwise usage-bearing responses.
A finding is triage, not a verdict of fraud, model substitution, or intentional overbilling.
Receipt prices are non-negative integer micro-dollars per million tokens. Cost is:
(input × input_price
+ output × output_price
+ cache_read × cache_read_price
+ cache_write × cache_write_price) // 1,000,000
The sum is rounded down once. Null cache dimensions contribute zero; the receipt contract must already include unreported cache reads in uncached input. The verifier also requires non-empty request, model, revision, engine-version, and timestamp fields.
Internal arithmetic consistency does not authenticate an issuer or prove a provider's token counts.
Case ids and record fields are stable within a major version. Adding optional evidence is a minor change. Changing a metric's meaning, expectation, record interpretation, or request behavior requires a major version so published results remain interpretable.