AgentFoundry → Production

Audit date: 2026-08-08. Scope: all 26 modules, 3,029 LOC, 125 tests.

Verified facts

Repo state. 3,029 lines of source across 26 modules, 1,347 lines of tests, 125 tests passing in 1.08s at 94% branch coverage. Sole third-party runtime dependency: cryptography==48.0.0. No AI library, no database driver, no HTTP client, no telemetry SDK anywhere in uv.lock.

Provider. CCR_ZAI_TOKEN in the environment is a valid Z.AI credential.

Endpoint	Result
`https://api.z.ai/api/paas/v4/chat/completions`	`HTTP 429` — code 1113, insufficient balance
`https://api.z.ai/api/coding/paas/v4/chat/completions`	`HTTP 200` — OpenAI-compatible, works
`https://api.z.ai/api/anthropic/v1/messages`	`HTTP 200` — Anthropic-compatible, works

The Anthropic-compatible endpoint returned a well-formed tool_use block with stop_reason: "tool_use" and real token accounting. This means the official anthropic Python SDK drives GLM-5.2 unmodified via a base_url override — typed responses, streaming, and the tool-use loop all work without writing an HTTP shim.

Model id: glm-5.2. It is a reasoning model — the OpenAI-compatible response spent all 20 output tokens on reasoning_content before emitting any text. Budget max_tokens generously (≥4096) or responses truncate inside the reasoning phase.

LangGraph stack. Verified working end-to-end against GLM-5.2 on 2026-08-08: langgraph 1.2.10, langchain-anthropic 1.5.4, langgraph-checkpoint-postgres 3.1.2, anthropic 0.121.0. A StateGraph with an agent ⇄ ToolNode cycle driven by ChatAnthropic(model="glm-5.2", base_url="https://api.z.ai/api/anthropic") completed a full tool round-trip: AIMessage(tool_calls=[get_weather]) → ToolMessage → AIMessage("It's currently 18°C and raining in Paris.").

Architecture: LangGraph StateGraph

Agent orchestration is LangGraph. The repo already anticipated this — multiagent/specialists.py:58 ships langgraph_send_payloads() documented as mapping "directly to LangGraph Send without importing it in core."

Graph the control flow, not the implementations. Governance that decides what happens next becomes graph structure; governance that happens inside a step stays a library call.

Concern	Construct
Agent loop	`StateGraph`, `agent ⇄ tools` cycle
Durable resumable state	`AsyncPostgresSaver` checkpointer
Human approval	`interrupt()` + `Command(resume=...)`
Policy authorization	conditional edge / pre-tool gate node
Budget exhaustion	gate node → conditional edge to `END`
Tool dispatch	`ToolNode` over `@tool`-decorated functions
Specialist fan-out	`Send`
Context assembly	pre-model state-reduction node

Not graphs — these are concerns inside a node, and modelling them as topology would be over-engineering: gVisor sandboxing (a subprocess a tool node calls), the effect ledger, the audit hash chain, memory signing, retrieval, tenancy.

Two modules become redundant and are deleted, not integrated:

durability/checkpoint.py — InMemoryCheckpointStore and the never-executed PostgresCheckpointStore. LangGraph's checkpointer is the same state machine, already battle-tested, with thread-scoped resume built in.
The persistence half of approval/store.py. The exact-action binding policy survives; the pause/resume plumbing is interrupt().

The core finding

There is no agent loop in this repository.

SingleAgentRuntime.run() (agentfoundry/runtime/single.py:70) makes one model call and returns. ToolRegistry.dispatch() (agentfoundry/tools/registry.py:83) executes one tool by name. Nothing connects them. No code anywhere reads a tool_use block, executes the tool, appends a tool_result, and calls the model again.

Every governance mechanism in the repo — policy, approval, budget, effect ledger, sandbox — is designed to sit between a model proposing an action and that action executing. That junction does not exist in the code. The mechanisms are gates on an empty corridor.

This is the first thing to build, and until it exists nothing else can be exercised.

Tier map — what each module needs

Module	Now	Needs
`runtime/single.py`	one model call, `FakeModel`	keep as T0 baseline; graph supersedes
`models/gateway.py`	`Callable` type alias, no adapter	`ChatAnthropic` factory + routing/fallback
`tools/registry.py`	`Mapping[str, type]` schema	`@tool` + Pydantic args; feeds `ToolNode`
`retrieval/index.py`	token-set overlap over a dict	pgvector + real embeddings
`memory/governed.py`	dict + Ed25519 (crypto is real)	Postgres persistence
`durability/checkpoint.py`	dict; Postgres class never run	delete → `AsyncPostgresSaver`
`approval/store.py`	in-memory pause records	policy kept; plumbing → `interrupt()`
`durability/lease.py`, `outbox.py`	in-process	Redis lease; Postgres outbox
`jobs/queue.py`	in-process	Redis Streams
`tenancy/store.py`	dict	Postgres RLS (SQL already written)
`finops/budget.py`	dict + `threading.Lock`	Redis atomic counters
`observability/trace.py`	dict + `perf_counter`	OpenTelemetry SDK
`sandbox/provider.py`	`runsc` code never executed	install runsc; run it
`policy/engine.py`	callables; `CedarAdapter` is a stub	keep PDP; Cedar or delete
`cache/safe.py`	dict	Redis
`reference_apps/aria/app.py`	injected `specialist` callable	real agent
`contracts.py`	dataclasses + `__post_init__`	Pydantic
—	—	new: agent loop, HTTP service, config, migrations

Dependencies to add

dependencies = [
  "langgraph>=1.2",                        # agent orchestration (StateGraph)
  "langchain-anthropic>=1.5",              # ChatAnthropic -> Z.AI -> glm-5.2
  "langgraph-checkpoint-postgres>=3.1",    # AsyncPostgresSaver
  "anthropic>=0.121",                      # transitive, pinned deliberately
  "pydantic>=2.9",                         # config + tool schemas
  "pydantic-settings>=2.6",
  "psycopg[binary,pool]>=3.2",             # checkpoints, outbox, tenancy RLS, memory
  "redis>=5.2",                            # leases, budgets, queue, cache
  "pgvector>=0.3",                         # evidence index
  "opentelemetry-sdk>=1.29",
  "opentelemetry-exporter-otlp-proto-grpc>=1.29",
  "opentelemetry-instrumentation-httpx>=0.50",
  "tenacity>=9.0",                         # retry/backoff around the provider
  "fastapi>=0.115",                        # service surface
  "uvicorn[standard]>=0.32",
  "cryptography==48.0.0",                  # already present, already real
]

[project.optional-dependencies]
dev = [
  "pytest>=8.3,<10", "pytest-cov>=6,<8", "pytest-asyncio>=0.24",
  "testcontainers[postgres,redis]>=4.9",   # real PG + Redis in tests
  "respx>=0.22",                            # HTTP-level provider stubs
  "ruff>=0.8", "mypy>=1.13",
]

Phase 0 — Agent loop + real provider

Nothing else matters until this exists.

0.1 Configuration (`agentfoundry/config.py` — new)

pydantic-settings model reading AGENTFOUNDRY_* env vars: provider base URL, API key (SecretStr), model id, Postgres DSN, Redis URL, OTLP endpoint, budget defaults. Fails loudly at startup on a missing required value — no silent defaults for credentials.

0.2 GLM provider (`agentfoundry/models/glm.py` — new)

class GLMProvider:
    """Drives GLM-5.2 through Z.AI's Anthropic-compatible Messages API."""

    def __init__(self, settings: Settings) -> None:
        self._client = anthropic.AsyncAnthropic(
            api_key=settings.zai_api_key.get_secret_value(),
            base_url="https://api.z.ai/api/anthropic",   # NOT api/paas/v4 — no balance there
            max_retries=0,                                # tenacity owns retry
        )

Responsibilities: translate ModelRequest → Messages API call; map stop_reason; surface real usage (input/output/cache-read) into TokenUsage; raise typed errors (ProviderRateLimited, ProviderUnavailable) that ResilientExecutor.retryable already knows how to classify. Wrap in tenacity with exponential backoff honoring retry-after.

FakeModel stays — it becomes the deterministic test double, which is legitimate. What changes is that it is no longer the only implementation.

0.3 The agent graph (`agentfoundry/graph/` — new)

The missing centre of the system, as a StateGraph rather than a hand-rolled loop.

State (graph/state.py) — messages: Annotated[list, add_messages] plus TenantContext, run id, cumulative token/cost counters, policy version, and the pending tool-approval record.

Nodes (graph/nodes.py):

Node	Does
`context`	`ContextBuilder` reduces `messages` to a budgeted view
`agent`	`ChatAnthropic.bind_tools(...)` invoke
`gate`	`DeterministicPDP.decide()` per proposed `tool_call`; budget reserve; raises `interrupt()` when approval is required
`tools`	`ToolNode` — dispatch, wrapped in the effect ledger for idempotency
`verify`	`IndependentVerifier` over the draft answer

Edges (graph/build.py): START → context → agent; conditional from agent via tools_condition → gate (not straight to tools); gate → tools on allow, → agent with an is_error ToolMessage on deny, → END on budget exhaustion; tools → agent.

Compiled with AsyncPostgresSaver and interrupt_before on the gate. thread_id is the run id, which gives resume, replay, and time-travel for free — and is what versioning/resume.py was approximating.

Every existing governance module attaches here. This is what converts them from documentation into enforcement.

Explicitly not graphs: gVisor (a subprocess the run_python tool calls), the effect ledger (a decorator around tool execution), the audit chain, memory signing, retrieval, tenancy. Graph the control flow, not the implementations.

0.4 Async conversion

LangGraph is async-native (ainvoke, astream, AsyncPostgresSaver), so this follows from the architecture rather than being a separate choice. ToolRegistry currently blocks a ThreadPoolExecutor thread per call and SpecialistRunner spawns real threads; both become async node functions with asyncio.wait_for for timeouts. Pure-logic modules (policy, planning, context, audit, learning, evaluation) stay synchronous — they do no I/O and are called from inside nodes.

0.5 Real tools (`agentfoundry/tools/builtin/` — new)

At minimum: web_search (httpx + a search API), http_fetch (behind the existing EgressGuard), read_file/write_file (path-confined), run_python (routed to the sandbox). These give the agent something to actually do, and give the effect ledger and sandbox real traffic.

Schema change. ToolSpec.schema: Mapping[str, type] cannot express JSON Schema, so it cannot be sent to a model. Replace with a Pydantic model per tool; derive both the JSON Schema for the API and the validation for dispatch from it.

Deliverable: an agent that answers a question requiring three tool calls, with policy denials and budget exhaustion observable in the trace.

Phase 1 — Real state

1.1 Postgres

Run the SQL that already exists. deploy/postgres/001_tenant_rls.sql is correct and has never been executed. Checkpoint tables come from AsyncPostgresSaver.setup() — do not hand-write them; delete PostgresCheckpointStore.SCHEMA. Add migrations for af_outbox, af_memory, af_effects, af_evidence. Port TenantStore onto af_resources with SET LOCAL agentfoundry.tenant_id per transaction, and connect as af_app, not the table owner — the existing SQL uses FORCE ROW LEVEL SECURITY, which is silently bypassed for owners.

1.2 Redis

DistributedLease via SET NX PX + a Lua compare-and-delete for fenced release (the current lease.py fencing-token logic is correct — it just needs a shared store). RedisBudgetLedger via a Lua script making reserve/commit atomic; the current threading.Lock is per-process and cannot bound fleet spend. RedisJobQueue on Streams with consumer groups, replacing the in-process jobs/queue.py.

1.3 Outbox relay

A process that reads af_outbox in the same transaction as the state change and publishes. Without it the outbox is a table nobody drains.

Phase 2 — Real retrieval

InMemoryEvidenceIndex.search scores with len(query_terms & content_terms) over a dict. Replace with pgvector:

Embeddings from GLM (embedding-3 on the Z.AI endpoint) or local sentence-transformers for offline determinism.
af_evidence table with a vector column, HNSW index, and RLS on tenant_id so tenant isolation is enforced by the database rather than the _allowed() Python check.
Hybrid scoring: normalize lexical (Postgres ts_rank) and vector similarity to [0,1] before combining. Current bug: score = lexical + vector adds a [0,1] ratio to an unnormalized cosine, so scores are not comparable — and benchmark() reports recall_at_k computed off them.
Keep Evidence immutability, evidence_id, and ACLs. That design is sound.

Phase 3 — Observability

Replace InMemoryTraceRecorder's perf_counter + dict with the OpenTelemetry SDK. Keep the span(category, name) context-manager signature so no call site changes. Emit spans for each loop iteration, model call, tool dispatch, policy decision, and retrieval query; record token counts and cost as span attributes. Export OTLP to Jaeger or Langfuse in docker-compose. OTelExporterAdapter.__init__(self, emit) — an untyped callback — is deleted, not adapted.

Phase 4 — Sandbox

GVisorSandbox is written and has never run. Install rootless runsc in the container image, then:

Add except subprocess.TimeoutExpired → SandboxResult(137, ...). Currently the one enforcement path that matters raises a raw exception while SimulatedSandbox returns a structured result for the same condition.
Implement the egress_hosts allowlist that currently raises NotImplementedError, or document it as deliberately unsupported.
Mark the real tests @pytest.mark.gvisor and run them in CI. The marker is declared in pyproject.toml and used by zero tests.

Phase 5 — Tests that can fail

The current suite is 125 deterministic in-memory round-trips in 1.08s. It proves the dataclasses match the dataclasses.

Add three tiers:

Unit (existing, keep) — pure logic, no I/O. Fast.
Integration (@pytest.mark.integration) — testcontainers spins real Postgres and Redis. Assert RLS actually blocks a cross-tenant read as af_app; assert two processes contending for a lease produce one winner; assert concurrent budget reservations cannot oversubscribe; assert outbox survives a killed transaction.
Live (@pytest.mark.live) — real GLM calls. Assert the loop completes a multi-tool task; assert malformed tool args produce an is_error result the model recovers from; assert budget exhaustion halts mid-run; assert a checkpointed run resumes on a second process.

CI runs unit + integration on every push; live nightly with the key in a secret.

Also fix, with regression tests:

Defect	Location
`KeyError` when a provider has no price entry, instead of failover	`models/gateway.py:52`
`SignedEvidencePack.verify()` takes the private key — unusable for third-party verification	`compliance/audit.py:98`
`subprocess.TimeoutExpired` uncaught	`sandbox/provider.py:88`
Unnormalized score mixing	`retrieval/index.py:88`
`inject_for_test` on a production class	`memory/governed.py:148`
`CedarAdapter` reaches into `self._delegate._rules["*"]` and reimplements `decide()` — contains no Cedar	`policy/engine.py:70`

Phase 6 — Make it solve a problem

A library nobody calls is still not production. Add:

agentfoundry/service/app.py — FastAPI: POST /runs (start), GET /runs/{id} (status + trace), POST /runs/{id}/approve (resolve a pending approval), GET /healthz. Auth middleware builds the TenantContext from a verified token rather than trusting a caller-supplied tenant id.
ARIA, for real — AriaResearchApp currently takes an injected specialist: Callable. Replace with a coordinator StateGraph that fans out to specialist subgraphs via Send (the payload shape langgraph_send_payloads() already emits), each running the same agent ⇄ gate ⇄ tools cycle over real retrieval and web search, with results merged by an operator.add reducer on the state. Keep IndependentVerifier as the verify node — a separate verifier pass over citations is a genuinely good design and becomes meaningful once the claims are model-generated.
docker-compose.yml — Postgres, Redis, Jaeger, the API. make up && make demo answers a research question end to end.
Ops — structured JSON logging with run/tenant/trace ids, /metrics, graceful shutdown draining in-flight runs, and secrets from env only.

Sequencing

Phase	Work	Unblocks
0	Loop + GLM + async + real tools	everything
1	Postgres + Redis	durability, multi-process
2	pgvector retrieval	ARIA quality
3	OpenTelemetry	debugging phases 4–6
4	gVisor	`run_python` tool
5	Integration + live tests	trusting any of it
6	Service + ARIA + compose	usable by someone

Phases 1–3 are parallelizable once 0 lands. Phase 5 is written alongside each phase, not after.

What stays

Not everything here is scaffolding. These are correct and carry forward unchanged:

effects/ledger.py + reconcile.py — idempotency keys and UNKNOWN reconciliation. This is the hard part of exactly-once and it is right.
durability/lease.py fencing-token logic — needs a shared store, not a redesign.
compliance/audit.py hash chain — canonical JSON, correct chaining.
memory/governed.py Ed25519 signing, supersession, expiry — real crypto, correct.
policy/engine.py DeterministicPDP — deny-by-default with a fail-closed exception boundary.
contracts.py, Evidence identity/versioning, context/builder.py budget policies.
The seams themselves. Every Protocol in this repo is the right shape; the work is implementing behind them, not redrawing them.

FilesExpand file tree

PRODUCTION_PLAN.md

Latest commit

History