feat(examples): add TypeSafe Jev evaluator example

[mikebranc-cisco]

Requested by the Agent Control team (Namrata) and tracked as SAO-17570.

Summary

Adds a worked example of integrating a third-party evaluation API as an Agent Control evaluator, plus an agent guarded by it.

TypeSafe Jev is a System One model: it does not generate text, it answers typed questions about some state and returns calibrated values. Every question in a request is answered in one pass over the state, so the evaluator's job is to turn a vector of answers into the single boolean a control needs. decide.combine resolves them with any or all without giving up that batching.

The demo is a support assistant for a fictional bank with a synthetic customer table and one real tool. Controls screen three different things, and each fails differently:

• the user's message, so a jailbreak is stopped before the model is called
• the tool call the model chose, so a broad query for identifying fields is stopped before it reaches the data
• the model's reply, so one that reads an SSN back is withheld

Asking for your own SSN exercises two at once: the lookup is legitimate and allowed, the reply is not and is withheld. A fourth control steers rather than blocks, routing a request for personal financial advice to a licensed advisor.

Scope

User-facing/API changes: none. No core change, no new dependency for existing users. The evaluator registers through an entry point in the example's own pyproject.toml:

[project.entry-points."agent_control.evaluators"]
"typesafe.jev" = "jev:TypeSafeJevEvaluator"

Internal changes: one row added to examples/README.md. Nothing else outside examples/typesafe_jev/.

Out of scope: CONTRIBUTING.md points new evaluators at evaluators/contrib/template/. This one lives in the example instead, per Namrata's call, so it can be promoted to a contrib package later without blocking the example now. It follows examples/deepeval, which already ships an evaluator this way.

Risk and Rollout

Risk level: low. Additive only — 19 files, all new except one line in the examples index. Nothing imports it, nothing runs it unless you install it. The evaluator is inert until a control names typesafe.jev.

Rollback plan: revert the commit. There is no migration, no config change, and no effect on an existing deployment.

Testing

• Added or updated automated tests — 22 unit tests over the decision logic, no network calls
• Ran make check (or explained why not) — ruff and mypy clean on the example; pytest green. The example is outside the uv workspace by design (evaluators/contrib/* is excluded the same way), so its tests run from its own directory
• Manually verified behavior — full clean-room run below

Verified from an empty database and a fresh virtualenv, following the README as a new user:

Step	Result
Fresh venv + repo packages	registry: json, list, regex, sql
pip install -e . in the example	registry: + typesafe.jev
setup_controls.py against an empty DB	created the agent and 5 controls
demo_agent.py	7 message cases and 3 tool cases, all correct
chat_app.py with no model credentials	all 6 outcomes correct via the offline stand-in
chat_app.py with a real model	correct

The no-credentials path matters most: a fresh clone runs the whole guardrail flow with no API keys for the model, which is what the README promises.

Checklist

• Linked issue/spec (if applicable) — SAO-17570
• Updated docs/examples for user-facing changes — the example carries its own README; one row added to the examples index
• Included any required follow-up tasks — see below

Follow-ups, not blocking:

• Promote jev/ to evaluators/contrib/typesafe/ when the team wants it released as a distribution. It was built that way first; moving it back is mostly a directory move plus a semantic_release entry.
• The README links only to https://typesafe.ai. Worth deep-linking their System One docs once we settle which URLs to cite.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!