feat(evals): add file-based suites

[ccarvalho-eng]

Motivation

Persisted evaluation suites could be run headlessly only by repeating three target flags, which left sampling configuration outside version control. Teams need one reviewable file that selects an existing suite without introducing a second owner for test cases or dataset provenance.

Changes

• add Aludel.Evals.FileSuite for schema-version-1 JSON and YAML manifests
• keep test cases, documents, dataset provenance, and quality policies database-owned
• support all existing bounded sampling reducers in manifests
• add mix aludel.eval --file PATH while preserving direct target flags and every reporter option
• reject oversized input, invalid UTF-8, duplicate keys, YAML aliases, explicit tags, multiple documents, unknown fields, invalid UUIDs, and invalid sampling before execution
• promote yaml_elixir to a direct runtime dependency and lock its latest compatible patch
• distinguish dashboard UI, Mix CLI, and Elixir API capabilities in the README
• add focused README, HexDocs, and wiki examples for JSON, YAML, reducers, CI, validation, and ownership

Test Plan

• Added focused loader tests for format parity, sampling reducers, strict schema validation, duplicate keys, YAML parser extensions, bounded input, file loading, target ownership, execution, and non-mutating dataset behavior.
• Added Mix task tests for manifest execution, repeated sampling, mutually exclusive target options, and stable invalid-manifest errors.
• Full repository verification, type analysis, dependency advisories, production compilation, documentation generation, package construction, and security diff review pass.

Next Steps

• Add curated red-team dataset templates in the next focused feature.

[coderabbitai]

Warning

Review limit reached

Next included review available in 25 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: eeaac6ce-a443-482e-baf8-4367f713fd9e

📥 Commits

Reviewing files that changed from the base of the PR and between 81dcb05 and 051874c.

⛔ Files ignored due to path filters (1)

• mix.lock is excluded by !**/*.lock

📒 Files selected for processing (9)

• README.md
• guides/evaluations.md
• guides/features.md
• guides/file_suites.md
• lib/aludel/evals/file_suite.ex
• lib/mix/tasks/aludel.eval.ex
• mix.exs
• test/aludel/evals/file_suite_test.exs
• test/mix/tasks/aludel.eval_test.exs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}