A deployable multi-agent platform, built failure-first — and verified against the running deployment rather than the source tree.
Most agent codebases are demos wearing production clothing. They pass their tests because the tests mock the thing that would have failed. A dictionary stands in for PostgreSQL, so tenant isolation is never tested; an in-process lock stands in for Redis, so nothing ever races; a missing sandbox binary degrades to running untrusted code on the host, and the suite still reports green.
The failure mode is not that these systems are unfinished. It is that their evidence
is unfalsifiable. FORCE ROW LEVEL SECURITY cannot be verified against a mock — a
mock returns whatever the test author believed. An agent platform makes claims about
authority, isolation, durability, and spend; every one of those is a claim about
infrastructure behaviour, and infrastructure is precisely what gets mocked away.
AgentFoundry is the counter-position. Every control is tested against the real component it claims to use, and the top tier of the suite speaks HTTP to a deployed container and imports none of the application.
A senior production engineer's answer to "which agent mechanisms earn their complexity, and how would you prove it?" — built as an executable argument rather than a slide deck.
It is a working service (FastAPI + LangGraph over PostgreSQL, Redis, and MinIO) that runs governed agent runs for authenticated tenants, plus 50 modules covering 35 capabilities behind substitutable contracts. Each capability exists because a specific failure made it necessary, and each one carries the evidence for its own claim — see the capability matrix, which names the failure, the invariant, and the implementing module for every row.
The method is the point:
BASELINE → BREAK → MINIMUM FIX → BREAK AGAIN → INJECT → MEASURE → DECIDE
Every capability answers six questions: what breaks without it, which invariant it
creates, what complexity it adds, how to break it, which metric proves it, and when
not to use it. The Git history is the architecture narrative — git log --reverse
reads as the order in which the failures were discovered.
cp deploy/.env.example deploy/.env # fill in CCR_ZAI_TOKEN and APP_DB_PASSWORD
bash deploy/generate-dev-keys.sh # local JWT signing keypair (development only)
docker compose -f deploy/docker-compose.yaml --env-file deploy/.env up -d --wait
uv run pytest tests/smoke -q # verifies the running container, not the source--wait returns only once every healthcheck is green and the migration job has exited
zero — the difference between "the containers started" and "the system is serving", and
what makes the smoke run deterministic rather than racing the schema.
See deploy/README.md for the topology and the decisions behind it.
flowchart TB
Client([Tenant client]) -->|"Bearer JWT"| API
subgraph Deployment["docker compose · agentfoundry:local (832 MB, uid 10001, non-root)"]
MIG[["migrate<br/>one-shot schema job<br/>must exit 0 before api starts"]]
API["<b>api</b> · FastAPI/uvicorn<br/>POST /v1/runs · GET /v1/runs/id<br/>POST /v1/runs/id/resume<br/>/health/live · /health/ready"]
end
MIG -.->|"gate"| API
API -->|"role aria_app<br/>NOSUPERUSER NOBYPASSRLS"| PG[("postgres · pgvector 16<br/>LangGraph checkpoints<br/>tenant rows under FORCE RLS<br/>audit hash-chain · vectors")]
API --> RD[("redis<br/>leases · admission buckets<br/>budget reservations")]
API --> MO[("minio<br/>artifacts · WORM object lock")]
API -->|"LiteLLM"| LLM{{"provider<br/>GLM-5.2 via Z.AI<br/>or NVIDIA NIM"}}
style API fill:#1f6feb,color:#fff
style MIG fill:#6e7681,color:#fff
The service never connects as the database owner. It connects as aria_app, which
is NOSUPERUSER NOBYPASSRLS. This is not defence in depth — it is the entire mechanism.
A BYPASSRLS role makes every row-level security policy in the schema decorative, so
connecting as the owner would silently disable tenant isolation while pg_policies
continued to look correct.
flowchart LR
R([POST /v1/runs]) --> AZ{"verify JWT<br/>scope + tenant claim"}
AZ -->|"no token"| E401[401]
AZ -->|"wrong scope /<br/>no tenant claim"| E403[403]
AZ -->|"admitted"| G
subgraph G["LangGraph · checkpointed per tenant:run_id"]
direction LR
AGENT["agent<br/>model call<br/>token + step budget"]
GATE{"gate<br/>policy decision<br/>per tool call"}
TOOLS["tools<br/>typed dispatch<br/>sandbox + effect ledger"]
FB["tool_feedback<br/>denials returned<br/>to the model"]
VER["verify"]
AGENT -->|"tool calls"| GATE
AGENT -->|"final answer"| VER
GATE -->|"allow"| TOOLS
GATE -->|"deny"| FB
GATE -->|"require_approval<br/>action digest pending"| STOP((halt))
TOOLS --> FB
FB --> AGENT
end
VER --> OK[201 RunResponse]
G -.->|"deadline"| E504[504]
G -.->|"ambiguous effect"| E409[409]
G -.->|"provider failed"| E503[503]
style GATE fill:#d29922,color:#000
style AZ fill:#d29922,color:#000
The gate runs before dispatch and per call, not as a post-hoc audit. A denial
becomes a ToolMessage fed back to the model, so the agent learns it was refused
instead of silently losing a turn. require_approval halts the run against an action
digest — approving a proposal binds the exact arguments, so mutating them after
approval invalidates it.
CAPABILITY → CONTRACT → IMPLEMENTATION
Every capability is a frozen contract in agentfoundry/contracts/ with at least a
deterministic reference implementation and, where it matters, a production adapter
behind the same seam. Absence of an adapter is reported as absence — never as a green
production control.
Four CI tiers, because "the tests pass" was answering the wrong question. Each tier can catch a class of defect the tier below it structurally cannot.
| Tier | Tests | Needs | Catches what the tier below cannot |
|---|---|---|---|
| Lint | — | nothing | dead imports, unresolvable annotations, shared mutable defaults |
| Hermetic | 339 | nothing | contract violations, state-machine defects, fail-closed logic |
| Integration | 616 | PostgreSQL · Redis · MinIO · gVisor · provider key | RLS that only looks correct, races, real kernel isolation, real provider behaviour |
| Smoke | 10 | a deployed container | wrong entrypoint, a dependency that existed only in the editable install, an env var present on one laptop, a migration job that did not run |
| Total | 965 |
Last full run against the compose substrate, a live provider key, and a rootless
runsc — all 965 in one process, no tier isolation:
964 passed, 1 skipped, 14 warnings in 103.01s
The single skip is honest and specific: test_live_memory_governance needs real
vectors, and the configured provider (GLM via Z.AI) serves chat only — its embedding
models answer 1211 Unknown Model. It skips because the provider lacks a capability,
not because a key is missing, and the skip reason says exactly that. Set
NVIDIA_API_KEY and it runs.
Integration does not skip for a missing substrate. It connects to real PostgreSQL, Redis and MinIO and fails without them. A connection error is a missing environment, not a passing test, and reporting it as green is how a substrate claim quietly stops being true. The only skips permitted are capability statements like the one above.
uv run pytest --ignore=tests/integration --ignore=tests/smoke -q # 339, no services
# integration — same environment the CI `integration` job uses
export DATABASE_URL="postgresql://aria:aria@127.0.0.1:5433/aria"
export DATA_PLANE_DB_URL="postgresql://aria_app:${APP_DB_PASSWORD}@127.0.0.1:5433/aria"
export REDIS_URL="redis://127.0.0.1:6379/0" MINIO_ENDPOINT="127.0.0.1:9000"
uv run --extra dev --extra deepagents pytest tests/integration -qDATA_PLANE_DB_URL is not optional. It carries the unprivileged role — pointing the
suite at the owner would make every RLS assertion pass for the wrong reason. See the
integration job in .github/workflows/ci.yml.
Ruff's ruleset is pinned in pyproject.toml rather than inherited from whatever the
installed version defaults to, so uv run ruff check . answers the same question on
every machine. Adopting it surfaced three defects worth having: a return annotation
naming a type nothing at module scope could resolve (hidden by from __future__ import annotations, so it never failed), two raise statements inside except blocks that
discarded the causing exception, and two shared mutable model instances used as default
arguments. Rules that fire on pre-existing style are listed individually in the config
as tracked debt, so adopting each one is a reviewable commit rather than silent drift.
ruff format is deliberately not enforced: it would reflow 428 files, and a
formatting diff that size buries every real change made alongside it.
Seven decisions that are the actual content of this repository.
Tenant isolation is a database guarantee, not a WHERE clause. Rows are protected
by FORCE ROW LEVEL SECURITY under a NOBYPASSRLS role. The tests assert cross-tenant
denial against real PostgreSQL, because an application-level filter is one forgotten
predicate away from a breach and a mock can never notice.
A 503 is a promise, not a status code. It tells the caller to retry and tells the
on-call nobody needs paging. Only the provider gets to make that promise. A bare
except Exception → 503 had turned every defect in our own graph into a reassuring
"upstream is down" — so the handler now re-raises anything that is not a genuine
provider failure, and lets it become a 500 with a traceback.
(agentfoundry/models/glm.py, agentfoundry/api/app.py:386)
Migration is a job, not startup work. Startup migration is fine with one replica and broken with three — replicas race the same DDL, and a failed migration leaves a container serving against a half-applied schema. As a job, a failure stops the rollout and nothing serves.
Liveness and readiness are different probes, treated differently. The container's
HEALTHCHECK uses /health/live only. /health/ready opens a Postgres connection and
pings Redis — correct for a load balancer deciding where to route, wrong for an
orchestrator deciding whether to kill the process. Wiring readiness into the restart
policy converts a five-second database blip into a restart storm.
The default identity provider is unroutable on purpose. The service boots, health
checks pass, and every authenticated route answers 401, because no token verifies
against https://issuer.invalid. A deployment with no IdP is inert rather than
open. Same instinct as fail-closed config: in production the settings validator
rejects a Postgres DSN without sslmode=require, a non-rediss:// Redis URL, and
non-HTTPS OIDC endpoints — at construction, so a misconfigured deployment fails to
start instead of quietly transmitting credentials in the clear.
Untrusted code runs under a real kernel, or it does not run. The sandbox seam has a
rootless gVisor adapter, and 9 tests exercise the live Sentry kernel — resource
limits, filesystem confinement, egress denial, and the timeout-vs-kill discrimination
ladder. Where runsc is absent the provider refuses execution and the tests skip
with a reason. Unavailable isolation is never a pass.
And one that is smaller but says the most about the standard: the evaluation judge
records the model that actually produced the score. A handle claiming llama-3.1-70b
over a score GLM generated would make two incomparable evaluation runs look comparable
— which is the single thing a model snapshot exists to prevent.
| Path | What lives there |
|---|---|
agentfoundry/contracts/ |
frozen seams — every capability's typed contract |
agentfoundry/api/, graph/ |
the deployed service and its LangGraph agent loop |
agentfoundry/runtime/ |
T0 and governed single-agent execution |
agentfoundry/policy/, approval/, sandbox/ |
deterministic authority: PDP, action digests, isolation |
agentfoundry/durability/, effects/ |
checkpoints, leases, outbox; idempotency and UNKNOWN reconciliation |
agentfoundry/tenancy/, db/ |
tenant-scoped stores and the RLS schema |
agentfoundry/memory/, retrieval/, context/ |
trust-gated memory, provenance-aware evidence, context budgets |
agentfoundry/multiagent/, patterns/, workspace/ |
coordination labs and topology experiments |
agentfoundry/eval/, evaluation/, observability/, chaos/ |
the evidence planes |
agentfoundry/reference_apps/aria/ |
ARIA deep-research composition over the shared core |
deploy/ |
compose substrate, migrations, Kubernetes manifests |
docs/adr/ |
55 architecture decision records |
- Architecture and capability matrix — planes, ownership, and the failure each capability answers
- Labs and scenario packs — ten scenarios with their break, fix, and metric
- Limitations and rejected complexity — what this does not prove, and what was deliberately not built
- Deployment topology and decisions
- Deterministic chaos report
- Original ARIA audit and the reconstruction manifest
- Production hardening plan — the audit that drove the substrate work
This is a development and CI substrate demonstrating production controls, not a
production deployment. It does not claim hosted availability, throughput, regional
recovery, or SLOs; the local WORM emulator is not cloud Object Lock; and a compose file
is not a topology. Production additionally needs TLS termination, managed Postgres with
restores that are actually drilled, a real secret store rather than an .env file, and
gVisor available on the node.
Where a claim is not backed by evidence in this repository, it is recorded as not backed. See limitations.