Skip to content

README.md

Latest commit

 

History

History
149 lines (116 loc) · 7.01 KB

README.md

File metadata and controls

149 lines (116 loc) · 7.01 KB
Raw

Governed Enterprise Talent Operations Agent

A production-grade, governed agentic system with a first-class evaluation and observability plane. A LangGraph orchestrator runs enterprise talent-operations tasks (policy RAG, read-only NL2SQL analytics, assistive candidate matching, human-approved actions) under deterministic control: the model classifies, retrieves, and proposes, while deterministic code owns identity, routing, tenant scope, tool authorization, budgets, approvals, and terminal state.

The system is built to not fail silently: adversarial canaries, an async sampled LLM judge, and PSI/CUSUM drift detection produce a degradation verdict that opens incidents and executes a response (freeze data-egress tools + record a rollback target), backed by SLO burn-rate and canary-stalled alerting.

See docs/architecture/flagship-system.md for the full architecture, request-lifecycle, detect→act loop, security model, and dashboard/metric placement — it is the single source of truth and is kept in lock-step with the code.

Layout

Path What
src/talent_ops/ The flagship system: orchestrator, MCP tool gateway, policy/security, durability, LLM plane, eval plane
src/project_00…04/, src/capstone/, src/shared/ Staged reference implementations that build up the reusable controls (typed graphs → resumable approval → governed RAG → safe SQL → multi-agent investigation → capstone)
infra/ Docker Compose stack, Dockerfile, OPA policy, Keycloak realm, LiteLLM config, k8s/terraform stubs
observability/ Prometheus config + SLO/burn-rate/canary-stalled rules, Alertmanager, Grafana (11 dashboards), OTel/Loki/Tempo
evals/ Golden release-gate set + rubrics + load reports
docs/, runbooks/ Architecture SSOT, production-readiness scorecard, open gaps, threat model, metric dictionary, ADRs, incident/rollback runbooks

Run locally

uv sync --extra dev
uv run pytest
uv run ruff check .
uv run mypy src
uv run uvicorn src.capstone.api:app --reload

No model key is needed for deterministic tests: an explicitly labelled deterministic fixture stands in for the LLM. LLM_REQUIRED=true makes production fail rather than silently degrade.

Run the full stack

docker compose -f infra/docker/compose.yaml up --build -d

If Docker has no Buildx, build once with the legacy builder and start without rebuilding:

DOCKER_BUILDKIT=0 docker build -t docker-agent -f infra/docker/Dockerfile .
docker compose -f infra/docker/compose.yaml up -d --no-build

Open:

Observability: open, drive, and verify the dashboards

Grafana renders only what Prometheus scrapes from the running agent container's /metrics. So the flow is: bring the stack up → generate traffic → verify samples landed → open Grafana. Two scripts make this reproducible.

1. Confirm the app is up and being scraped

docker compose -f infra/docker/compose.yaml ps agent          # STATUS must be "healthy"

Then check http://localhost:9090/targets — the talent-agent job must be UP (not DOWN).

2. Drive real traffic through the live agent

./scripts/drive_traffic.sh 4        # 4 loops of 5 authenticated requests + a canary sweep

The full stack enforces OIDC, so this script mints per-role tokens from Keycloak inside the container network (the token issuer is keycloak:8080, so host-minted tokens are rejected) and posts authenticated /api/runs. Realm talent-ops ships demo users employee.acme, hr.acme, recruiter.acme, analyst.acme, admin.acme (password == role name).

3. Verify the metrics actually landed (objective check — exits non-zero if not)

./scripts/verify_metrics.sh
talent-agent scrape target: up
runs total                             291
denied (security)                      254
canary runs                            244
tool calls                             37
degradation status (1=degraded)        1
dataset rows (candidates)              750
OK: metrics are flowing.

4. Open Grafana — http://localhost:3001 (admin / admin), then set the time range to "Last 15 minutes" and refresh. The 11 provisioned dashboards:

Dashboard Shows
Production Overview run volume, route mix, success rate, denials at a glance
Service SLOs latency SLOs + multi-window burn-rate
Security and Governance tool authz denials, ACL-leak signal, injection/PII blocks
Evaluation and Drift eval dimensions, PSI/CUSUM drift, degradation verdict
Agent Trajectory route accuracy, tool precision, terminal correctness
RAG and Evidence retrieval hit-ratio, citation/evidence coverage
NL2SQL analytics-route query behavior
Business Outcomes task success framed as business KPIs
Model and Economics tokens, cost per route
Reliability and Persistence checkpoint/outbox/approval durability
Observability Health scrape health, trace/log completeness

Troubleshooting "No data"

Symptom Cause → fix
All panels empty Time range too old — set to "Last 15 minutes" (most common).
rate() panels empty Needs ≥2 scrapes — wait ~15s after driving traffic, refresh.
talent-agent target DOWN at /targets Agent crashed — docker compose ... logs agent; check it's healthy.
verify_metrics.sh says target UP but 0 runs You haven't driven traffic — run ./scripts/drive_traffic.sh 4.
Ran a Python demo but nothing shows An in-process client has its own registry and is never scraped — traffic must hit the running container (that's what drive_traffic.sh does).

Production configuration

Dev defaults are intentionally fail-open for several controls. Production deploys must load .env.production.example, which flips OIDC-required identity, OPA, egress allowlist, LLM_REQUIRED, and the Postgres checkpointer to their fail-closed values. See docs/production-readiness-scorecard.md → "Production profile".

Claim boundary

This repository demonstrates production patterns with local, automated evidence. It does not claim that local stubs prove real cloud availability, production latency, legal compliance, or managed-service operation. Those require the deployment and live-evaluation gates listed in docs/open-gaps.md. Current label: course-complete deterministic reference with an executed detect→act loop and a fail-closed production profile; not production-certified.