ccarvalho-eng · ccarvalho-eng · Sep 4, 2026 · Sep 4, 2026
diff --git a/README.md b/README.md
@@ -16,7 +16,7 @@ Aludel gives teams a clean way to evaluate prompt and model behavior without inv
 - Compare prompt versions and see pass-rate, cost, and latency changes over time.
 - Run evaluation suites with deterministic assertions, model-based judges, repeated sampling, document attachments, and CSV or JSON test case imports.
 - Assert generated output directly in ExUnit, or execute and gate persisted suites from application tests and CI.
-- Execute suites headlessly with console, versioned JSON, JUnit XML, or GitHub annotation reports for CI workflows.
+- Execute suites headlessly from versioned JSON or YAML manifests with console, JSON, JUnit XML, or GitHub annotation reports.
 - Route runs and suites through your app's real LLM workflow with callback execution.
 - Reuse single-turn and multi-turn datasets across suites with provenance and metadata filtering.
 - Find quality, cost, latency, stability, and regression trade-offs with rolling analytics and Pareto analysis.
@@ -33,6 +33,16 @@ Most teams evaluating LLM behavior end up with some combination of scripts, spre
 - **Embedded app callbacks**: evaluate your production-facing workflow without rebuilding it in the dashboard.
 - **Phoenix-native deployment**: mount it in your app or run it as a standalone dashboard.
 
+## UI, CLI, and Library APIs
+
+| Interface | Use it for |
+|---|---|
+| Dashboard UI | Create and version prompts; configure providers; compare models; manage reusable datasets and documents; author assertions, judges, and quality policies; run and retry suites; inspect evaluator evidence; analyze cost, latency, stability, regressions, and Pareto frontiers; review prompt suggestions; export results |
+| Mix CLI | Install Aludel with `mix aludel.install`; create deterministic demo data with `mix aludel.seed`; execute persisted suites by IDs or versioned JSON/YAML manifests with `mix aludel.eval`; emit console, JSON, JUnit, or GitHub Actions reports for local scripts and CI gates |
+| Elixir APIs | Embed the dashboard in a Phoenix router; route execution through host callbacks; create and execute prompts, datasets, suites, policies, and reports; load file-based suites; add custom metrics and reporters; assert evaluations directly from ExUnit |
+
+The dashboard and automation interfaces use the same persisted prompts, providers, datasets, suites, runs, quality policies, and evaluation evidence. A workflow can be authored in the UI, committed as a suite manifest, gated from the CLI, and inspected again in the dashboard without maintaining a second test-case format.
+
 ## Feature Catalog
 
 | Area | Features |
@@ -45,7 +55,7 @@ Most teams evaluating LLM behavior end up with some combination of scripts, spre
 | Assertions | `contains`, `not_contains`, `regex`, `exact_match`, typed `json_field`, scored `json_deep_compare`, custom rubric judges, and seven versioned judge templates |
 | Imports and datasets | CSV and JSON import previews with row-level errors; reusable ordered datasets with variables, messages, assertions, metadata filters, provenance, and idempotent suite population |
 | Prompt evolution | Version and provider trends, version-over-version deltas, suite-scoped Pareto frontiers, failure-grounded prompt suggestions, and explicit accept or dismiss decisions |
-| Automation and exports | Native ExUnit assertions and persisted suite gates, JSON run and suite exports, CSV or JSON evolution exports, a custom reporter behavior, console reports, versioned JSON, JUnit XML, GitHub annotations, and policy-aware `mix aludel.eval` quality gates |
+| Automation and exports | Native ExUnit assertions and persisted suite gates, versioned JSON or YAML suite manifests, JSON run and suite exports, CSV or JSON evolution exports, a custom reporter behavior, console reports, versioned JSON, JUnit XML, GitHub annotations, and policy-aware `mix aludel.eval` quality gates |
 | Execution and extension | Native provider calls, host-app callback execution, pluggable LLM, storage, and document-conversion boundaries, optional callback metadata, and configurable run concurrency |
 | Deployment | Embedded Phoenix dashboard, standalone app, Docker Compose, local/AWS S3/GCS document storage, custom auth/access resolvers, CSP nonce support, theming, and read-only mode |
 | Demo data | Deterministic prompts, providers, datasets, suites, runs, failures, artifacts, and 60 days of comparison history through `mix aludel.seed` |
@@ -305,7 +315,25 @@ Set `run_execution_mode: :sequential` when provider calls must not overlap.
 
 ### Headless suite execution
 
-Run a suite from scripts or CI. JSON schema version 2 is the default:
+Keep the execution target and sampling policy in a versioned manifest while the suite's test cases and dataset provenance remain in Aludel:
+
+```yaml
+schema_version: 1
+suite_id: 9a756a58-eaec-43ca-99e6-f5c016d85d0c
+prompt_version_id: e74cf2e1-94b6-4bcb-9ed9-b259661be906
+provider_id: e1c60ec0-6d55-419b-b958-7d088055254f
+sampling:
+  samples: 5
+  reducer: majority
+```
+
+Run it from scripts or CI:
+
+```bash
+mix aludel.eval --file evals/support-answer.yaml
+```
+
+JSON manifests use the same schema. You can also supply the three identifiers directly:
 
 ```bash
 mix aludel.eval \
@@ -329,6 +357,8 @@ Supported formats are `console`, `json`, `junit`, and `github`; `--pretty` forma
 
 The versioned `aludel_eval` JSON envelope includes suite and provider identifiers, aggregate and total score/cost/latency data, quality-policy evidence, and individual assertion results.
 
+See the [file-based suite guide](https://hexdocs.pm/aludel/file_suites.html) and [wiki examples](https://github.com/ccarvalho-eng/aludel/wiki/File-Based-Suites) for both formats, every sampling reducer, library execution, validation behavior, and CI usage.
+
 ### ExUnit evaluation assertions
 
 Use `Aludel.ExUnit` to keep focused model checks beside application tests:

diff --git a/guides/evaluations.md b/guides/evaluations.md
@@ -282,7 +282,13 @@ When failed suite evidence exists, choose a provider and generate a failure refl
 
 ## 9. Add a CI quality gate
 
-Run the same suite headlessly:
+Store the suite target and sampling configuration in a versioned JSON or YAML manifest, then run it headlessly:
+
+```bash
+mix aludel.eval --file evals/support-answer.yaml
+```
+
+The manifest references persisted records; it does not copy, replace, or delete suite cases or dataset provenance. You can also supply the targets directly:
 
 ```bash
 mix aludel.eval \
@@ -313,7 +319,7 @@ The command emits one schema-version-2 JSON object by default:
 
 The task exits unsuccessfully when arguments or targets are invalid, the prompt version belongs to another prompt, execution cannot be persisted, the suite is empty, or the active quality gate does not pass.
 
-Choose `--format console`, `--format junit`, or `--format github` for a human-readable log, CI test report, or GitHub Actions annotations. Add `--output PATH` to write the report to a file, `--pretty` to pretty-print JSON, or JUnit-only `--include-output` when the artifact is appropriate for generated responses. See the [reporter guide](reporters.html) for complete examples and the custom reporter behavior.
+Choose `--format console`, `--format junit`, or `--format github` for a human-readable log, CI test report, or GitHub Actions annotations. Add `--output PATH` to write the report to a file, `--pretty` to pretty-print JSON, or JUnit-only `--include-output` when the artifact is appropriate for generated responses. See the [file-based suite guide](file_suites.html) for the complete manifest schema and the [reporter guide](reporters.html) for output examples and the custom reporter behavior.
 
 ## 10. Gate evaluations in ExUnit
 

diff --git a/guides/features.md b/guides/features.md
@@ -125,12 +125,13 @@ Aludel provides:
 - JSON exports for suite runs, including assertions, retries, callback metadata, and artifacts
 - JSON and CSV exports for prompt evolution metrics and provider breakdowns
 - `Aludel.ExUnit` assertions for inline generated output, existing suite runs, and execute-and-persist quality gates
+- schema-versioned JSON and YAML execution manifests that reference persisted suites without duplicating dataset ownership
 - `Aludel.Evals.Reporter` with console, schema-version-2 JSON, JUnit XML, GitHub annotation, and custom reporter support
-- `mix aludel.eval` for headless suite execution and optional report file output
+- `mix aludel.eval` for identifier-based or manifest-based headless suite execution and optional report file output
 
-The Mix task emits JSON by default and accepts `--format console|json|junit|github`, `--output PATH`, JSON-only `--pretty`, and JUnit-only `--include-output`. It exits unsuccessfully for invalid targets, execution errors, empty suites, or a non-passing active quality gate. The normalized report model keeps each output format independent from suite-run persistence.
+The Mix task accepts either `--file PATH` or the three target ID flags. It emits JSON by default and accepts `--format console|json|junit|github`, `--output PATH`, JSON-only `--pretty`, and JUnit-only `--include-output`. It exits unsuccessfully for invalid manifests or targets, execution errors, empty suites, or a non-passing active quality gate. The normalized report model keeps each output format independent from suite-run persistence.
 
-See the [ExUnit evaluation guide](ex_unit.html) for application-test examples and the [reporter guide](reporters.html) for output formats, CI configuration, and custom reporter modules.
+See the [file-based suite guide](file_suites.html) for manifest examples, the [ExUnit evaluation guide](ex_unit.html) for application-test examples, and the [reporter guide](reporters.html) for output formats, CI configuration, and custom reporter modules.
 
 ## Execution modes
 

diff --git a/guides/file_suites.md b/guides/file_suites.md
@@ -0,0 +1,139 @@
+# File-Based Evaluation Suites
+
+Versioned suite manifests make a persisted Aludel evaluation reproducible from a repository, script, or CI job. A manifest selects the suite, prompt version, provider, and optional repeated-sampling policy. The database remains authoritative for test cases, document associations, dataset provenance, and quality policies.
+
+## YAML manifest
+
+Create `evals/support-answer.yaml`:
+
+```yaml
+schema_version: 1
+suite_id: 9a756a58-eaec-43ca-99e6-f5c016d85d0c
+prompt_version_id: e74cf2e1-94b6-4bcb-9ed9-b259661be906
+provider_id: e1c60ec0-6d55-419b-b958-7d088055254f
+sampling:
+  samples: 5
+  reducer: majority
+```
+
+Replace the example UUIDs with IDs from your Aludel installation, then run:
+
+```bash
+mix aludel.eval --file evals/support-answer.yaml
+```
+
+`.yaml` and `.yml` extensions are supported.
+
+## JSON manifest
+
+JSON uses the same schema:
+
+```json
+{
+  "schema_version": 1,
+  "suite_id": "9a756a58-eaec-43ca-99e6-f5c016d85d0c",
+  "prompt_version_id": "e74cf2e1-94b6-4bcb-9ed9-b259661be906",
+  "provider_id": "e1c60ec0-6d55-419b-b958-7d088055254f",
+  "sampling": {
+    "samples": 5,
+    "reducer": "minimum_pass_rate",
+    "minimum_pass_rate": 0.8
+  }
+}
+```
+
+Run it the same way:
+
+```bash
+mix aludel.eval --file evals/support-answer.json
+```
+
+## Sampling options
+
+`sampling` is optional. Without it, each test case runs once and must pass.
+
+| Reducer | Meaning | Additional field |
+|---|---|---|
+| `all` | Every attempt must pass | None |
+| `any` | At least one attempt must pass | None |
+| `majority` | More than half of the attempts must pass | None |
+| `minimum_pass_rate` | The pass rate must meet a chosen threshold | `minimum_pass_rate`, from `0.0` through `1.0` |
+
+`samples` accepts integers from 1 through 20. The limit prevents an accidental manifest change from creating unbounded provider traffic.
+
+For example, require four of five attempts to pass:
+
+```yaml
+sampling:
+  samples: 5
+  reducer: minimum_pass_rate
+  minimum_pass_rate: 0.8
+```
+
+## Reports and quality gates
+
+Reporter flags stay on the command line, so the same manifest can serve local output and different CI systems:
+
+```bash
+mix aludel.eval \
+  --file evals/support-answer.yaml \
+  --format junit \
+  --output aludel-junit.xml
+```
+
+```bash
+mix aludel.eval \
+  --file evals/support-answer.yaml \
+  --format github
+```
+
+The task exits unsuccessfully when the manifest or target records are invalid, execution fails, the suite is empty, or the effective quality decision does not pass. The suite's latest immutable quality policy is snapshotted at execution time. Without a policy, every persisted test case must pass.
+
+`--file` is mutually exclusive with `--suite-id`, `--prompt-version-id`, and `--provider-id`. Report flags such as `--format`, `--output`, `--pretty`, and `--include-output` work with either target style.
+
+## Load and execute in Elixir
+
+Use `Aludel.Evals.FileSuite` from releases, scripts, or application code:
+
+```elixir
+alias Aludel.Evals.FileSuite
+
+with {:ok, file_suite} <- FileSuite.load("evals/support-answer.yaml"),
+     {:ok, suite_run} <- FileSuite.execute(file_suite) do
+  {:ok, suite_run}
+end
+```
+
+For the common one-step form:
+
+```elixir
+{:ok, suite_run} =
+  Aludel.Evals.FileSuite.load_and_execute("evals/support-answer.json")
+```
+
+In-memory callers can use `load_string/2` with `:json` or `:yaml` before calling `execute/1`.
+
+## Validation and ownership
+
+Manifest validation happens before database lookup or model execution. Aludel rejects:
+
+- files larger than 256 KiB or content that is not valid UTF-8
+- extensions other than `.json`, `.yaml`, and `.yml`
+- malformed content, duplicate mapping keys, YAML aliases, explicit YAML tags, and multiple YAML documents
+- missing or unknown fields and unsupported schema versions
+- non-UUID target identifiers
+- invalid sample counts, reducers, or minimum pass rates
+- prompt versions that do not belong to the suite's prompt
+
+Errors return a stable map with `code` and `message` fields. The Mix task wraps that error in its schema-version-2 JSON error envelope.
+
+A manifest deliberately does not contain test cases, provider credentials, output paths, or quality-policy definitions. Editing or executing one cannot repopulate a suite or alter its dataset links. Manage those records in Aludel, then commit only their stable identifiers and execution settings.
+
+## CI example
+
+```yaml
+- name: Run evaluation gate
+  run: mix aludel.eval --file evals/support-answer.yaml --format github
+```
+
+Provider configuration and credentials should come from the application's normal runtime configuration. Do not commit secrets to a suite manifest.