Quality Policies

Quality policies turn a suite run into an explicit quality gate. A policy can combine overall pass rate, metadata-group pass rate, evaluator score, total cost, and average latency without treating missing evidence as a normal quality failure.

Create a Policy Version in the Dashboard

Open a suite and choose Manage policy. The page starts with a valid overall-pass-rate policy when the suite has no versions. For an existing suite, it loads the active definition so it can be used as the basis for the next version.

Edit the JSON and resolve every validation message before choosing Create policy version. The page lists versions newest first, keeps each definition expandable, and marks the version that future runs will use as active. Existing versions cannot be edited or deleted.

Return to the suite to run the evaluation. The active-policy badge identifies the contract for the next run. Every historical result that used a policy displays its snapshotted version, aggregate status, and per-rule actual value and requirement.

Create a Policy Version with the Elixir API

{:ok, policy} =
  Aludel.Evals.create_suite_policy(suite, %{
    "schema_version" => 1,
    "rules" => [
      %{"id" => "overall", "type" => "overall_pass_rate", "minimum" => 0.95},
      %{
        "id" => "priority",
        "type" => "metadata_pass_rate",
        "metadata" => %{"priority" => "high"},
        "minimum" => 1.0
      },
      %{
        "id" => "faithfulness",
        "type" => "evaluator_score",
        "metric" => "rubric_judge",
        "minimum" => 85
      },
      %{"id" => "cost", "type" => "total_cost_usd", "maximum" => 0.50},
      %{"id" => "latency", "type" => "average_latency_ms", "maximum" => 1_500}
    ]
  })

Each call creates the next immutable version for that suite. Aludel.Evals.list_suite_policies/1 returns versions newest first, and Aludel.Evals.latest_suite_policy/1 returns the policy that the next suite execution will snapshot.

Policy definitions use JSON-compatible string keys. A version-one policy contains between 1 and 50 rules with unique, non-blank IDs, accepts only the documented fields, and cannot exceed 100,000 encoded bytes.

Rule Reference

Overall pass rate

%{"id" => "overall", "type" => "overall_pass_rate", "minimum" => 0.95}

The minimum is a value from 0.0 through 1.0. An empty suite produces an unavailable result because there is no pass-rate evidence.

Metadata-group pass rate

%{
  "id" => "priority",
  "type" => "metadata_pass_rate",
  "metadata" => %{"priority" => "high", "language" => "en"},
  "minimum" => 1.0
}

A test case belongs to the group when its metadata contains every configured key and value. Additional metadata does not prevent a match. A group with no matching test cases is unavailable.

Evaluator score

%{
  "id" => "faithfulness",
  "type" => "evaluator_score",
  "metric" => "rubric_judge",
  "minimum" => 85
}

The minimum is a score from 0 through 100. Aludel averages completed assertion scores whose type matches metric. For repeated sampling, every retained attempt contributes its matching scores. Errored and unavailable evaluators do not become zero scores; when no completed scores exist, the rule is unavailable.

Total cost budget

%{"id" => "cost", "type" => "total_cost_usd", "maximum" => 0.50}

The rule compares its non-negative maximum with the complete suite-run cost, including every repeated sample. Missing cost data makes the rule unavailable.

Average latency budget

%{"id" => "latency", "type" => "average_latency_ms", "maximum" => 1_500}

The rule uses average latency across all available request samples. Missing latency data makes the rule unavailable.

Read a Policy Result

{
  "schema_version": 1,
  "policy_id": "00000000-0000-0000-0000-000000000000",
  "policy_version": 3,
  "status": "passed",
  "passed": true,
  "rules": [
    {
      "id": "overall",
      "type": "overall_pass_rate",
      "minimum": 0.95,
      "actual": 0.96,
      "sample_count": 25,
      "status": "passed",
      "passed": true
    }
  ]
}

Aggregate status follows these rules:

• passed: every rule passed
• failed: at least one measured rule failed and no rule was unavailable
• unavailable: at least one rule lacked required evidence
• invalid: direct evaluation received a malformed or unsupported definition

Every status other than passed has passed: false and causes the headless task to exit unsuccessfully. Unavailable rules include a stable reason; invalid policies include an errors list. Stored policy versions are validated before persistence, so an invalid outcome is mainly relevant when calling Aludel.Evals.QualityPolicy.evaluate/3 directly.

Execution and Retry Semantics

Suite execution snapshots the latest policy before it starts model requests and stores the policy ID plus the complete result on the suite run. Creating a newer policy while a run is executing does not change that run's contract.

Retrying one test case recalculates the policy result from the updated result set while preserving the original policy version. This keeps a historical run from silently adopting a newer threshold.

Metadata used for group rules is copied into each persisted test case result. Evaluator-score rules read completed assertion evidence, and cost and latency rules use persisted suite aggregates.

CI and Exports

mix aludel.eval uses the stored policy result as its top-level status and process exit gate. Suites without a policy retain the legacy behavior: every test case must pass and an empty suite fails.

mix aludel.eval \
  --suite-id SUITE_ID \
  --prompt-version-id PROMPT_VERSION_ID \
  --provider-id PROVIDER_ID

The JSON envelope includes quality_policy with the policy ID, version, aggregate status, and rule evidence. Suite-run downloads include the immutable definition and its result.

Policy creation is available in the dashboard and Elixir API. The Mix CLI, ExUnit gates, reporters, and exports consume stored policy outcomes; they do not create policy versions.

Related Pages

• Evaluation Suites
• Repeated Sampling
• Evaluator Execution Details
• Exports and CI