Does an inference endpoint deliver and meter the request it claims?
inference-audit tests OpenAI-compatible APIs at the HTTP boundary. It compares pinned providers,
checks parameter compliance and token accounting, measures reported cache reuse, and writes
reproducible JSONL evidence. It does not claim to identify a model from a few outputs.
pip install inference-audit
export OPENAI_API_KEY=...
inference-audit run \
--base-url https://api.example.com/v1 \
--model org/model \
--output run.jsonl
inference-audit report run.jsonl --format markdownThe CLI also loads .env from the current directory. Existing environment variables take
precedence.
The smoke suite sends five requests. The standard suite sends 47:
inference-audit run \
--base-url https://api.example.com/v1 \
--model org/model \
--suite standard \
--repeats 3 \
--max-requests 141 \
--output campaign.jsonl--max-requests is a hard spend guard. Larger runs fail before making network calls.
| Surface | Evidence |
|---|---|
| Delivery | request errors, empty output, truncation, strict-prefix cuts |
| Contract | exact/contains expectations, JSON validity, reasoning-off compliance |
| Context | deterministic needle recall at roughly 6k, 20k, and 45k words |
| Metering | usage presence, prompt-token disagreement, reasoning and cache details |
| Cache | reported hit rate across shared prefixes and time gaps |
| Stability | agreement across repeats and targets |
| Experimental | first aligned content-token KL when three or more targets report logprobs |
Findings describe observations, not causes. A prompt-token mismatch could come from billing, a tokenizer, or a chat template. The report shows the difference and leaves diagnosis to the reader.
export OPENROUTER_API_KEY=...
inference-audit compare \
--model openai/gpt-oss-120b \
--provider Groq \
--provider Cerebras \
--suite standard \
--repeats 2 \
--max-requests 188 \
--output providers.jsonlOmit --provider to discover all listed endpoints. Requests pin one provider and disable
fallbacks. Rejections remain failures; the runner never changes the requested settings to get a
pass.
All 18 discovered providers serving openai/gpt-oss-120b, standard suite, three repeats — 2,538
requests on 2026-09-01 with v0.1.0, $0.66 in provider-reported cost, run with --max-tokens 1000 --workers 6 --retries 4 --retry-delay 3. The table is unedited
inference-audit report --format markdown output:
| target | n | errors | expect | self | consensus | JSON | trunc | tok/modal | cache | latency |
|---|---|---|---|---|---|---|---|---|---|---|
| AkashML | 141 | 3 | 100.0% | 61.7% | 76.4% | 100.0% | 0.7% | 1.000 | 76.2% | 2988 ms |
| Amazon Bedrock | 141 | 0 | 100.0% | 66.0% | 67.8% | 100.0% | 2.1% | 1.000 | 0.0% | 579 ms |
| BaseTen | 141 | 24 | 95.5% | 65.9% | 61.3% | 100.0% | 0.0% | 1.131 | 67.4% | 1162 ms |
| Cerebras | 141 | 0 | 100.0% | 100.0% | 58.9% | 100.0% | 2.1% | 1.000 | 0.0% | 453 ms |
| CoreWeave | 141 | 5 | 100.0% | 59.6% | 74.4% | 100.0% | 0.7% | 1.000 | 75.3% | 4943 ms |
| DeepInfra | 141 | 0 | 100.0% | 68.1% | 64.4% | 100.0% | 0.7% | 0.845 | 0.0% | 5271 ms |
| DigitalOcean | 141 | 2 | 100.0% | 42.6% | 54.4% | 100.0% | 0.0% | 1.000 | 76.2% | 6072 ms |
| 141 | 12 | 71.4% | 34.8% | 47.0% | 87.5% | 0.0% | 1.040 | 0.0% | 3849 ms | |
| Groq | 141 | 0 | 100.0% | 80.9% | 56.7% | 100.0% | 0.0% | 1.048 | 0.0% | 556 ms |
| Mancer 2 | 141 | 8 | 100.0% | 57.4% | 59.5% | 100.0% | 3.0% | 1.000 | 0.0% | 4973 ms |
| Mara | 141 | 8 | 100.0% | 100.0% | 84.3% | 100.0% | 0.0% | 1.000 | 0.0% | 1123 ms |
| Nebius | 141 | 0 | 100.0% | 55.3% | 72.2% | 100.0% | 1.4% | 1.000 | 76.2% | 773 ms |
| Novita | 141 | 2 | 100.0% | 48.9% | 58.4% | 100.0% | 0.7% | 1.020 | 0.0% | 3253 ms |
| Parasail | 141 | 0 | 100.0% | 55.3% | 63.3% | 100.0% | 0.7% | 1.000 | 76.2% | 1658 ms |
| Phala | 141 | 55 | 100.0% | 75.0% | 64.8% | 100.0% | 0.0% | 1.000 | 75.3% | 2418 ms |
| SambaNova | 141 | 0 | 100.0% | 72.3% | 83.3% | 100.0% | 0.0% | 1.000 | 0.0% | 1897 ms |
| SiliconFlow | 141 | 0 | 100.0% | 57.4% | 63.3% | 100.0% | 0.0% | 1.000 | 0.0% | 5524 ms |
| Together | 141 | 0 | 100.0% | 53.2% | 60.0% | 100.0% | 1.4% | 1.000 | 0.0% | 2853 ms |
Reading it the way the tool intends — observations, not verdicts:
- Token accounting disagrees for byte-identical requests. On one shared prompt, 13 providers reported 75 prompt tokens while DeepInfra reported 62, Google and Groq 79, and BaseTen 86. The medians replicated exactly in an independent 940-request campaign the same day (DeepInfra 0.845×, Groq 1.048×, Google 1.040×, BaseTen 1.131×), and held hour-by-hour across an earlier 24-hour campaign collected with a precursor script — in which Google's ratio visibly changed from 1.000 to ~1.04 partway through and stayed there. The records show the disagreement; they cannot say whose tokenizer or template differs, or who is right.
- One endpoint clips output tails. The Google-pinned endpoint repeatedly returned the
long-context needle with its last characters missing (
PINE-447forPINE-4471,ORCA-908forORCA-9083), sometimes returned empty content for "Reply with exactly the word OK", and produced different output at temperature 0 on 27 of 47 cases. - Reported cache behavior is heterogeneous. In a controlled prefix-reuse run
(
inference-audit cache, 8 steps, 4k-word prefix), Nebius and Parasail reported ~99.6% cached tokens once warm, DigitalOcean hit on roughly alternating steps, AkashML almost never hit, and Groq — which reported zero cache across all 2,538 requests above — reported reuse on 2 of 8 steps. Absent or zero cache detail is absent evidence, not proof a provider does not cache. - Errors were terminal HTTP 429 rate limits after four-retry backoff, concentrated in smaller
endpoints (Phala: 55), plus a handful of malformed responses where
choiceswas not an array. That is delivery evidence under pinned, no-fallback routing — not model quality. - Streaming conformance was clean: every successful stream from all 18 providers delivered
[DONE]and final usage. Median time to first token spanned 308 ms (Cerebras) to 4.9 s (Google).
inference-audit cache \
--base-url https://api.example.com/v1 \
--model org/model \
--prefix-words 4000 \
--steps 8 \
--gap 2 \
--stream \
--output cache.jsonlEach step keeps the same deterministic prefix and changes the suffix. --gap models time spent in
a tool call. This checks provider-reported cache counts, not GPU-resident KV state.
--stream exercises SSE without buffering the response. The parser handles split UTF-8 and lines,
records time to first token, captures final usage, and flags streams missing [DONE].
The receipt verifier recomputes four-dimension micro-dollar charges with integer arithmetic:
inference-audit verify-receipt examples/receipt.jsonThe receipt schema ships with the package. A matching total proves arithmetic consistency, not the issuer's identity or token counts.
Runs append versioned JSONL and resume completed cases. Records contain request settings, response data, timing, usage, and errors—never API keys or Authorization headers. Prompts up to 4,096 characters are stored; larger built-in prompts use a SHA-256 digest and character count to avoid duplicating them in every row.
Model responses may be sensitive. Review JSONL before publishing it. A useful public result includes the tool version, raw records, model, provider, region, run time, suite, and request settings.
SPEC.md defines the record and scoring semantics. Key limits:
- Consensus is not truth.
- Token disagreement does not identify the correct tokenizer or template.
- Providers can misreport usage and cache counts.
- One needle test does not prove an entire context-window claim.
- Experimental KL needs larger samples and confidence intervals.
make install
make ciCI runs Ruff, strict mypy, a 95% coverage gate, and package builds.
Apache-2.0