A benchmark is valid only when another maintainer can identify exactly what ran, reproduce the execution envelope, inspect raw evidence, and recompute the published statistics. A formatted report without provenance is not evidence.
The quality policy owns gate thresholds and calibration requirements. The evaluation CLI and adapter protocol define how evidence is produced. This policy owns the conditions under which evidence and performance claims are valid.
Treatment and control use the same prompt variant, executor snapshot, parameters, token budget, tool policy, and run ordering. They differ only in the declared treatment.
- Treatment: mounts the exact release-candidate plugin archive.
- Clean control: runs in a fresh process and temporary home with no project or global Skill, rule file, memory, plugin, MCP server, or reference leakage.
- Token-matched placebo: optional research condition; never a substitute for the clean control.
Run order is randomized from a manifest-recorded seed. A treatment-specific canary must be absent from the control envelope. Any leak invalidates the batch.
Every nightly and RC artifact includes a machine-readable manifest.json with at least:
{
"schema_version": "1.0.0",
"suite": "release",
"version": "0.1.0-rc.1",
"git_sha": "40-character commit SHA",
"status": "passed",
"runs_per_condition": 30,
"environment_id": "immutable runtime/image identifier",
"executor_model": "exact model identifier",
"judge_model": "exact model identifier",
"catalog_sha256": "canonical catalog digest",
"evaluator_sha256": "evaluator source and lock digest",
"started_at": "RFC 3339 timestamp",
"completed_at": "RFC 3339 timestamp",
"summary": {},
"artifacts": []
}The artifact set also records:
- evaluator, catalog, adapter, environment, generation-parameter, and rubric versions or digests;
- full resolved prompt-envelope hashes;
- condition, case, variant, and attempt IDs;
- generation parameters plus unique executor and judge provider request IDs;
- raw outputs, structured grades, raw judge responses, calibration, timing, total executor-plus-judge token usage, and errors;
- hashes for every retained artifact;
- per-case estimates, confidence intervals, and paired deltas.
Unavailable measurements are represented as null plus a reason. They are not represented as zero.
- Nightly: 10 runs per case and condition.
- RC: at least 30 runs per case and condition.
- Nightly routing: one treatment run per trigger query.
- RC routing: at least three treatment runs per trigger query.
- Three-run smoke checks: diagnostic only.
Report sample counts beside every rate. Repeated calls on a fixed public prompt estimate conditional execution reliability, not generalization to unseen tasks. Generalization claims require the access-controlled case source in the quality policy. Unique request IDs and explicit cache-bypass attestations are mandatory; replayed responses are not independent repetitions. Do not describe a single run as a repeated experiment. Do not average assertions with different severities into a single “pass rate.” Report infrastructure failures separately and retain every attempt; retries do not erase the original failure.
CI retains raw outputs and grades as immutable workflow artifacts for at least 90 days. That operational minimum is not sufficient for a longer-lived public performance claim or support decision. Before publishing either, copy the complete recomputable evidence bundle to an authenticated durable evidence store, record its content digest and immutable locator in the release evidence, and retain it for at least the lifetime of the claim or support decision. If the bundle expires or becomes inaccessible, the dependent claim or support decision expires too.
Release assets include the evidence manifest, manifest checksum, evaluated Plugin archive, archive checksum, and aggregate checksum list. Release notes identify the approved RC workflow run and, when a performance claim or support decision is published, the durable raw-evidence locator and digest. Access-controlled evidence may remain in a protected durable store, but authorized maintainers and auditors must be able to inspect and recompute it for the same retention period.
Secrets, credentials, and private user data must not enter public release evidence. If a public scored artifact requires redaction, that batch cannot support a public performance number because reviewers could not reproduce the score from the retained bytes. Run access-controlled promotion cases in a protected durable evidence store and publish only an approved aggregate attestation; never silently rewrite scored content.
The ignored structured-thinking-workspace/ data is exploratory and cannot serve as an accepted baseline because it lacks reliable repetition, model provenance, isolation, and consistent grading. Preserve it only as historical debugging context.
README, release notes, talks, and external documentation may cite a performance number only when they also identify:
- version and commit;
- suite and case count;
- executor and judge model snapshots;
- treatment and baseline sample sizes;
- estimate and 95% confidence interval;
- evidence manifest and checksum;
- known limitations and non-significant comparisons.