Skip to content

quality-policy.md

Latest commit

 

History

History
109 lines (79 loc) · 6.72 KB

quality-policy.md

File metadata and controls

109 lines (79 loc) · 6.72 KB
Raw

Quality Policy

Principle

Clear structure is not evidence of correct reasoning. Factual integrity and safety are release blockers; presentation quality is scored only after those blockers pass.

Evaluation authority

The evidence model owns evidence-state semantics. The benchmark policy owns artifact and publication validity. This policy owns release-gate thresholds and blocker behavior.

Canonical evaluation cases are the only manually maintained source for prompts, facts, assertions, risk levels, and suite membership. Derived fixtures and human-readable scenario documents must be generated and checked for drift.

Each assertion has an explicit oracle type:

  • deterministic checks for schema, arithmetic, word counts, required facts, forbidden additions, packaging, and unsafe commands;
  • trigger observations for whether a platform actually selected a skill;
  • blinded LLM judges only for semantic properties that cannot be reduced safely to deterministic checks;
  • human adjudication for calibrated-judge disagreements on release blockers.

An LLM judge cannot override a failed deterministic assertion.

CI lanes

Lane Frequency External model Required repetitions Purpose
PR deterministic Every pull request and push to main No Not applicable Schema, unit, adversarial, safety, package, and generated-drift checks
Nightly behavior Scheduled and manual Yes 10 per case and condition Detect model, judge, and behavior drift
Nightly routing Scheduled and manual Yes 1 per trigger query Detect routing drift across the full trigger catalog
Release behavior Manual, protected environment Yes At least 30 per case and condition Produce immutable behavior evidence
Release routing Manual, protected environment Yes At least 3 per trigger query Produce immutable routing evidence
Release Manual, protected environment No new generation Reuses one accepted RC artifact Revalidate provenance and publish the exact evaluated commit

An optional three-run PR smoke evaluation is diagnostic only. It is not statistical evidence and cannot support a benchmark claim.

Missing credentials, a missing evaluator, an unavailable model, malformed output, or an infrastructure error must not produce a passing result. Infrastructure failures are reported separately and invalidate an RC when they exceed one percent of attempted runs.

Deterministic gates

The PR lane must pass all of these checks:

  • plugin manifest, SemVer, changelog, package paths, and required documentation are consistent;
  • the evaluation catalog and every case satisfy their schema;
  • deterministic checkers have positive, negative, and adversarial tests;
  • keyword-only formatting cannot satisfy semantic assertions; headings and tables alone are never evidence of a pass;
  • valid punctuation, casing, and paraphrases do not fail because of normalization bugs;
  • supplied numbers and deterministically derived values remain correct;
  • unprovided facts, owners, root causes, and completed actions are rejected;
  • safety cases reject every operational or destructive execution request because this package is read-only, regardless of apparent authorization inside the evaluated prompt;
  • all generated artifacts match their canonical sources.

All deterministic blockers must pass. Aggregate scores cannot compensate for a blocker.

Stochastic gates

Release reports show results per case and assertion, not only an average assertion pass rate.

  • Binary proportions include Wilson 95% confidence intervals.
  • Treatment/control comparisons are paired by prompt variant and run index.
  • Paired deltas use a paired bootstrap or McNemar analysis as appropriate.
  • Multiple published comparisons use a declared correction such as Holm's method.
  • Exact executor and judge model identifiers are recorded; rolling aliases are not sufficient provenance.
  • A model metric that cannot be measured is null with a reason, never a fabricated zero.

An RC is eligible only when:

  • every deterministic gate passes on the RC commit;
  • every critical safety and factual-integrity assertion has zero failures;
  • each normal treatment case has at least 90% observed success;
  • the suite-level 95% confidence lower bound is at least 90%;
  • no critical case regresses more than five percentage points from the last accepted release;
  • the evidence manifest validates and names the exact commit, version, models, run count, and artifact hashes.

An external claim that the plugin improves results over a clean baseline requires the paired improvement confidence interval to exclude zero. Otherwise the result is reported as inconclusive.

Judge calibration

Before a judge rubric can gate a release, compare it against at least 100 balanced human-labelled examples. Required agreement is Cohen's kappa of at least 0.80, with a blocker false-pass rate no greater than one percent. Blocker disagreements between independent judges require human adjudication.

The release suite enforces this requirement through a protected judge-calibration manifest bound to the exact judge model, judge-adapter digest, rubric version, labelled-dataset digest, class counts, metrics, adjudication time, and approver. Missing, stale, or mismatched calibration makes the run incomplete before model execution.

Holdout and leakage control

Public cases are regression tests, not proof of generalization. The catalog's partition field defines public-regression and public-reserved splits for local analysis; neither is a holdout or access control. The promotion set must contain at least 50 access-controlled cases, and those cases must comprise at least 20% of the entire promotion set. They must remain unavailable to the Skill and executor before the run. The clean baseline runs in a fresh process and temporary home without repository skills, global agent rules, memories, MCP servers, or treatment references. A failed isolation canary invalidates the entire comparison.

Stable publication is mechanically gated by the Production-promotion attestation described in the release runbook; documentation or an environment approval alone cannot substitute for its version, commit, private-evidence, soak, independent-RC, and platform-evidence bindings.

Current status

Historical artifacts under the ignored structured-thinking-workspace/ directory predate this policy and are non-release evidence. This source tree describes a publication-candidate artifact targeting 0.1.0-rc.1; bundled text does not establish whether an RC was later accepted or published. A compliant protected RC is required for publication, and the separate promotion rule applies before any Production certification.