Skip to content

feat(evaluators): Add example for LLMAJ evaluator - #286

Open
wrisa wants to merge 1 commit into
mainfrom
feature/SAO-16858-add-LLM-evaluator-example
Open

wrisa wants to merge 1 commit into
mainfrom
feature/SAO-16858-add-LLM-evaluator-example

Conversation

@wrisa

@wrisa wrisa commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Added a runnable example for the galileo.llm (LLM-as-judge) evaluator, mirroring the existing examples/galileo_luna/ example. Demonstrates three control patterns: pre-execution prompt injection detection, composite post-execution toxic output scoring, and regex-based API key leak prevention.

Scope

  • User-facing/API changes: None — example only.
  • Internal changes: New examples/galileo_llm/ directory with setup_controls.py, demo_agent.py, pyproject.toml, and README.md.
  • Out of scope: No evaluator logic changes; no SDK or server changes.

Risk and Rollout

  • Risk level: Low — example code only, no production path affected.
  • Rollback plan: Delete the examples/galileo_llm/ directory.

Testing

  • Added or updated automated tests - N/A, example code.
  • Ran make check (or explained why not) - lint passes (ruff check clean).
  • Manually verified behavior - ran setup_controls.py and demo_agent.py against local server with PyCharm debugger; breakpoints hit through LlmEvaluator._evaluate and server POST /api/v1/evaluation endpoint.

Checklist

  • Linked issue/spec (if applicable) - SAO-16858
  • Updated docs/examples for user-facing changes - README.md documents all env vars, scorer configuration, boolean threshold pattern (operator=eq, threshold=True), and CA file setup.
  • Included any required follow-up tasks - example will need updating after feature/SAO-17516-request-payload-contract merges if the control payload contract changes affect setup_controls.py.

@codecov

codecov Bot commented Oct 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@josjeon
josjeon self-requested a review October 7, 2026 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants