Skip to content

CLAUDE.md

Latest commit

 

History

History
247 lines (182 loc) · 21.8 KB

CLAUDE.md

File metadata and controls

247 lines (182 loc) · 21.8 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Build & Run

# Install dependencies
uv sync --dev

# Run checks (do all three before every commit)
uv run mypy src tests
uv run ruff check src tests && uv run ruff format --check src tests
uv run pytest tests/unit -q                    # hundreds of test files, fast, no Docker needed (CI runs -n 4)

# E2E tests (need Docker for testcontainer Postgres + sandbox)
DOCKER_HOST=unix:///Users/tom/.docker/run/docker.sock uv run pytest tests/e2e -q

# Run a single test
uv run pytest tests/unit/test_context.py::TestBuildMessages::test_simple_conversation -xvs

# Migrations (aios migrate also applies the procrastinate schema +
# lock-release trigger; bare `alembic upgrade head` does not)
set -a && source .env && set +a
uv run aios migrate

# Start the system (two processes)
uv run python -m aios api      # API server on :8080 (AIOS_API_PORT)
uv run python -m aios worker   # procrastinate worker

Worktree smoke / dev isolation

Every git worktree gets its own isolated DB on the shared local Postgres via aios dev bootstrap. The first step of any worktree smoke is to bootstrap, not to source .env. Sourcing the main checkout's .env from a worktree silently points you at the shared dev DB (localhost:5433/aios) — real connection records and live traffic land in a shared/prod-shadow DB (#349).

Sanctioned worktree-smoke sequence:

uv run aios dev bootstrap       # provisions aios_dev_<id> + writes worktree-local .env
# open a FRESH shell (no stale AIOS_* exports)
set -a && source .env && set +a # the WORKTREE-LOCAL .env, written by bootstrap
uv run aios dev status          # sanity check: expect `mode: isolated`
uv run python -m aios api       # / aios worker

aios dev status prints a mode: line — isolated (bootstrapped, on its own aios_dev_* DB), shared (linked worktree pointed at the shared dev DB — fix before proceeding), or unbootstrapped (run aios dev bootstrap). As a backstop, aios api / aios worker hard-fail when started from a linked worktree against the shared DB; override with AIOS_ALLOW_SHARED_DB=1 only if you really mean it.

Client CLI

aios is a typer-based client CLI that talks to a running API. Config is read from env (AIOS_URL, AIOS_API_KEY) or .env; every command accepts --url / --api-key overrides and a global --format {table,json}.

# Reachability + auth check
uv run aios status

# Resource inspection
uv run aios agents list
uv run aios agents get <agent_id>
uv run aios sessions list --status running
uv run aios sessions events <session_id> --kind message

# Interactive chat (creates a session, streams the reply)
uv run aios chat --agent <agent_id> --environment-id <env_id>

# One-shot: send a message and stream until the turn ends
uv run aios chat --agent <agent_id> --environment-id <env_id> -m "list /workspace"

# Tail a running session from another terminal
uv run aios sessions stream <session_id>

# Post a user message without entering the REPL
uv run aios sessions send <session_id> "hello"

# Create resources from JSON (thin wire — server validates)
uv run aios agents create --file agent.json
uv run aios skills create --dir path/to/my-skill --title "My Skill"

Every resource has CRUD subcommands. See aios <resource> --help for specifics. Operator commands (aios api, aios worker, aios migrate) are also typer-backed but keep their original behavior.

Code generation invariants

Two pre-commit invariants are checked in CI; both fail if the committed artifact drifts from its source-of-truth:

  • openapi.json is regenerated from FastAPI introspection. Fix on drift: ./scripts/regen-openapi.sh && git add openapi.json.
  • packages/aios-sdk/aios_sdk/_generated/ is regenerated from openapi.json. Fix on drift: ./scripts/regen-client.sh && git add packages/aios-sdk (also regenerates openapi.json as a prerequisite).

Trigger: any API route, response model, error envelope, or pydantic schema change that flows through FastAPI introspection. Run both before pushing when you've touched API-layer code.

Database migrations

  • Migration failure fails API candidate admission. The API image starts its candidate through aios-deploy-entrypoint --candidate aios api, using the new container's baked-in migration scripts. aios migrate retries bounded lock contention and exits non-zero when attempts are exhausted; the entrypoint propagates that status without starting the API, so the candidate cannot become healthy or be promoted. Ordinary aios api startup does not migrate: this keeps application-image rollback startable against a forward-migrated database. Automated rollback must never run migration downgrades.

  • There is no new-code/old-schema serving window. The candidate API does not start until its migrations complete. Migrations must nevertheless remain compatible with the old application image because that image can continue serving during admission and can be selected for application rollback.

  • Destructive migrations need expand/contract. For drop column, drop table, change type, etc., the old image may still query a removed shape while the candidate is being admitted or after application rollback. A PR that introduces a destructive migration MUST either:

    • Split into three deploys: deploy A (code handles both old and new schema) → deploy B (apply the migration through candidate admission) → deploy C (code requires the new schema), OR
    • Ship together with code that handles both schemas across the rollout: read-tolerant of either form, write-tolerant of either form.

  • Only aios-api owns migrations. aios-worker, aios-signal, aios-telegram, and aios-whatsapp do not run migrations. The eumemic-ops coolify-flags audit enforces this constellation-wide: any non-aios-api app with migrate in either its pre- or post-deployment command fails the audit.

Operator-facing background lives in eumemic-ops at skills/coolify-api/references/db-direct.md §"Run migrations against the new code (use post_deployment_command)".

Architecture

aios is an event-driven agent runtime. The headline property: every tool is implicitly async — the model stays responsive to user messages even while tools are running.

The step model

There is no loop. A procrastinate wake_session job runs run_session_step() which calls the model once, kicks off tool calls as fire-and-forget asyncio tasks, and returns. Tool completion appends a result event and defers a new wake. The "loop" is the job queue re-entering the step function.

Key flow: wake_session → run_session_step (loop.py) → stream_litellm (completion.py) → launch_tool_calls (tool_dispatch.py) → tool tasks append results → defer_wake → next step.

Process split

  • API process (aios api): FastAPI server. Handles HTTP requests, appends user messages, defers wake jobs, serves SSE streams. Does NOT run tools or call models.
  • Worker process (aios worker): Runs procrastinate jobs. Calls the model, dispatches tools, manages sandbox containers. Holds worker-scoped globals on harness/runtime.py.

Both share Postgres: same database, same LISTEN/NOTIFY, same procrastinate job queue.

Context builder

harness/context.py builds the chat-completions message list from the event log. It's the most critical file in the harness:

  • Monotonicity invariant: appending events only appends to the context, never rewrites earlier messages (prompt cache stability).
  • Pending-result synthesis: in-flight tool calls get synthetic "pending" results so the message structure is valid.
  • Blind-spot injection: tool results that arrived during inference are injected as user messages at the tail.
  • reacting_to watermark: each assistant message records the max seq of events it saw, so find_sessions_needing_inference knows what's "new."

Externally-executed tools (custom + always_ask)

Tool calls the harness doesn't execute itself — type="custom" (client-executed) and always_ask-gated built-ins/MCP awaiting operator approval — sit unresolved in the event log alongside any async in-flight built-in. The session does NOT park. End of step flips status="idle" with stop_reason={"type":"end_turn"} regardless of what the model emitted; pending tool calls are observable via the derived Session.awaiting view.

The same wake mechanism drives all three cases:

  • In-flight async built-in: task's _tool_lifecycle.finally appends the result and triggers a sweep.
  • Custom tool: client POSTs to /sessions/:id/tool-results (operator) or /connectors/runtime/tool-results (connector runtime) — both call defer_wake after appending.
  • always_ask awaiting confirm: operator POSTs to /sessions/:id/tool-confirmations; allow appends a lifecycle event that the next step's _dispatch_confirmed_tools consumes.

A user message arriving during any of these wakes the session via the same find_sessions_needing_inference short-circuit (sweep.py — if any(evt.role == "user")). The model sees pending tool calls via the _PENDING_BACKGROUND / _PENDING_EXTERNAL synthesized placeholders in build_messages and can interleave responses freely.

Built-in and externally-executed tools can be called in the same response. The inference gate waits for all tool_call_ids before re-firing under the no-user-stimulus path; a user message lifts that wait.

Two pools in one process

The worker runs two Postgres connection pools:

  • asyncpg (ours): event log, session state, all application queries. The pool sets statement_timeout/idle_in_transaction_session_timeout and TCP keepalive GUCs via server_settings (see db/pool.py); the psycopg3/migrate path deliberately does not.
  • psycopg3 (procrastinate's): job queue. Separate connector, separate connections.

They don't share connections; they share Postgres state.

Workflows

The durable workflow runtime runs deterministic, replayable scripts as an append-only journal with replay-from-memo — a different shape from the session step. A workflow run is a row plus its wf_run_events journal; append_run_event (db/queries/workflows.py) is the single writer to that journal, allocating a gapless seq and idempotent on (run_id, call_key, type). run_workflow_step (workflows/step.py) rebuilds the memo from the call_result journal events and hands it to the script host, which fast-forwards already-resolved capabilities by .send()-ing results into the script coroutine. No model is called in a step — it is pure deterministic replay.

  • Credential-free out-of-process host: run_script_host (workflows/host_launcher.py) spawns python -m aios.workflows.wf_script_host as a separate subprocess under a deny-by-default env allowlist (_CHILD_ENV_ALLOWLIST). The child imports only stdlib + aios.workflows.determinism/._protocol — never aios.crypto/aios.db/aios.harness/aios.services — so the master CryptoBox and the all-accounts pool are never inherited. The parent enforces a wall-clock SIGKILL deadline.
  • Per-run compute ceiling (#780): _scaled_seconds(base, init_len) (host_launcher.py) scales the subprocess deadline by INIT-frame size (DEFAULT_DEADLINE_SECONDS=30, DEADLINE_SECONDS_PER_INIT_MIB=30, capped at MAX_SCALED_SECONDS=600) — a run replaying a large accumulated memo gets proportionally more time.
  • Attenuation clamp (#794): a step computes child_surface = attenuation_service.clamp(surface_of(agent), surface_of(run)) (step.py); clamp (services/attenuation.py) is the lattice meet, so a child session's authority never exceeds the run's snapshotted surface. The result is written immutably to the child session.surface.
  • Journaled log()/phase() (#783): script log()/phase() → ANNOTATION frames (workflows/_protocol.py) → journaled by step.py as type="annotation" events (WfRunEventType in models/workflows.py), keyed by call_key so the (run_id, call_key, type) unique constraint dedups them to emit-once across replays.

Where it lives: step = workflows/step.py (run_workflow_step); sweep = workflows/sweep.py (wake_runs_needing_step → list_run_ids_needing_step in db/queries/workflows.py); queries = db/queries/workflows.py; job = wake_workflow (harness/tasks.py).

Shares vs. diverges from session wake: it shares the procrastinate job queue, lock=run_id / queueing_lock=run_id, LISTEN/NOTIFY, and commit-before-NOTIFY. It diverges: workflow wake is defer_run_wake (services/wake.py) which appends no journal span (wf_run_events is single-writer), supports batch=True coalescing (#780, workflow_wake_batch_seconds), and runs at _BACKGROUND_PRIORITY=-10 so it yields to interactive sessions; and the step is deterministic replay with no model call plus the attenuation clamp — unlike the session step.

Conventions

  • Python 3.13+ with PEP 695 type parameter syntax: class ListResponse[T](BaseModel), def _list_scoped[T](...). Do NOT use TypeVar + Generic[T].
  • from __future__ import annotations at the top of every source file.
  • Raw SQL + asyncpg, no ORM. Every query lives in the db/queries package — one module per resource (db/queries/sessions.py, events.py, connections.py, vaults.py, triggers.py, …). Shared tenant-scoping helpers (_get_scoped, _list_scoped, _archive_scoped, _build_set_assignments, parse_jsonb, _escape_like) live in db/queries/__init__.py and are imported by the per-resource modules. Every public query is re-exported from the package root, so from aios.db import queries; queries.foo() and from aios.db.queries import foo both work and stay patchable (patch.object(queries, "foo", ...)).
  • pg_notify($1, $2) function form, never literal NOTIFY. Postgres case-folds unquoted identifiers; our ULIDs have uppercase.
  • structlog with structlog.stdlib.LoggerFactory(). NOT PrintLoggerFactory.
  • Extreme simplicity. No defensive guards for model mistakes, no fuzzy matching, no model-specific shims. The model sees raw errors and retries through the session log. Ask: "can the model handle this failure itself?" If yes, don't add complexity.
  • Events are OpenAI chat-completions format, not Anthropic Messages format. LiteLLM translates at the provider boundary.
  • Conventional commits: feat:, fix:, refactor:, etc. Substantial commit bodies.
  • Co-Authored-By trailer on AI-authored commits.
  • Ask non-trivial design decisions before writing code.

How to approach changes

  • Investigate before fixing — when something behaves oddly (unexpected errors, stuck jobs, state inconsistencies), find the root cause first. Don't kill processes, delete rows, or reset state as a shortcut.
  • Fail hard, no fallbacks — no silent skipping, no dummy values, no error suppression. The model sees raw errors and retries through the session log; that IS the design.
  • Correct-by-construction — the Key invariants below (gapless seq, monotonic context, tool-always-appends-result) are examples of this stance. Prefer designs that produce valid state in one pass over multi-stage corrective pipelines.
  • Unify toward minimal primitives; compose, don't accrete — favor a small set of orthogonal primitives whose compositions express the full space of desired behavior, over a growing catalog of ad-hoc features. When a new capability is asked for, first ask: is it a composition of existing primitives, or a small generalization of one? Add a new mechanism only when it genuinely isn't. Encode variation as a kind (a discriminated arm), never a boolean flag, so illegal combinations are unrepresentable rather than runtime-guarded — the triggers action union is the model. This is the architecture-altitude complement of "premature abstractions are still bad," not its contradiction: unify only once you've found the right primitive — one that dissolves special cases and makes invalid states impossible — never to pre-emptively DRY three similar lines.
  • Push back on bad requests — if a request conflicts with existing architecture, or has an obvious root-cause fix the user missed, flag it before implementing. You have context the user may not in the moment.
  • Don't deprecate, delete — remove old code paths rather than leaving shims. Git history preserves them.
  • After refactors, grep exhaustively — for any rename or move, search the whole tree (including tests) for the old name before declaring done.
  • Seek the perfected design, not the smallest diff — when the "minimal change" is shaped around avoiding some adjacent system (branch-protection rules, schema migrations, default settings, fixture organization, an upstream config someone else owns), that adjacent system is usually in scope. Update it. The right configuration at the time of writing compounds; the workaround accumulates and obscures intent. Constraints that feel like "I don't want to mess with X" are almost always negotiable — interrogate them, don't accept them. Proactive complement to broken windows: not just "fix what's flagged" but "build it as if from scratch every time." Scope discipline still applies — pick the right shape for the change, but its breadth should still match the task.
  • Broken windows policy — when a review (or your own pass) flags a clear quality issue, fix it now even if it's pre-existing. Don't pile new code on top of broken patterns or label them "out of scope." This complements rather than contradicts "don't add features beyond the task": the rule is reactive, not proactive — you don't go hunting for cleanup, but you don't ignore what you've already seen. Premature abstractions are still bad; three similar lines is still better than one over-engineered helper. The policy targets flagged issues, not aesthetic preferences.

Key invariants

  1. Gapless seq per session — every append_event locks the session row, increments last_event_seq, inserts. No gaps.
  2. Monotonic context — the context is a monotonic function of the log. Prompt cache stays stable within a windowing chunk.
  3. reacting_to watermark — every assistant message carries the seq of the latest user/tool event in its context.
  4. Tool-always-appends-result — every async tool task MUST append exactly one tool_result event and defer a wake (try/except/finally).
  5. Procrastinate lock — lock="{session_id}" for mutual exclusion, queueing_lock="{session_id}" for wake deduplication.
  6. NOTIFY after commit — pg_notify fires outside the transaction so subscribers don't see uncommitted rows.
  7. Request recoverability — every request a session composes for its own inference is recoverable from durable state: the log, plus the request record on the span that opened the send (model_request_start, or the model_workflow_park record), plus its per-account request_blobs (harness/request_capture.py). services/requests.rebuild_request recomposes it with the same build_messages → finalize_messages the step uses. At the same RENDER_VERSION the rebuild is byte-for-byte, and payload_sha says whether it was. Any rendering change must bump RENDER_VERSION (the golden test tests/unit/test_render_golden.py enforces it). payload_sha covers the composed request ({messages, tools, params}, without an inline api_key). The wire bytes are different: cache breakpoints, provider kwargs and body-budget eviction come after capture. context_admission_payload_digest on model_request_end is the separate wire-level digest, and it includes auth.

Environment variables

All aios settings use the AIOS_ prefix (Pydantic settings with env_prefix="AIOS_"):

  • AIOS_API_KEY — bearer auth key. Must hash to a row in account_keys; auth is no longer an env-var direct compare. On a fresh DB, mint one by POSTing to /v1/accounts/bootstrap (gated by AIOS_BOOTSTRAP_TOKEN) and store the returned plaintext_key as AIOS_API_KEY for both the API service and clients. A placeholder value (e.g. test-aios-key-do-not-deploy) will silently 401 against every request.
  • AIOS_BOOTSTRAP_TOKEN — bearer token that gates POST /v1/accounts/bootstrap; required to mint the root account's first API key on a fresh DB.
  • AIOS_VAULT_KEY — base64-encoded 32-byte libsodium key for encrypted database rows. Rotate with AIOS_VAULT_KEY_PREVIOUS plus aios rekey; do NOT regenerate without the rekey run if Postgres has encrypted data.
  • AIOS_VAULT_KEY_PREVIOUS — optional decrypt-only previous vault key used during aios rekey; unset after rotation completes.
  • AIOS_EGRESS_CA_KEY — required base64-encoded 32-byte key for the sandbox egress CA (sandbox/egress_ca.py). Separate from AIOS_VAULT_KEY so vault-key holders cannot mint sandbox-trusted TLS certificates.
  • AIOS_DB_URL — Postgres connection string
  • AIOS_API_PORT — default 8080
  • AIOS_TAVILY_API_KEY — for web_fetch/web_search tools
  • AIOS_AUTO_REVIEW_MODEL / AIOS_AUTO_REVIEW_TIMEOUT_S / AIOS_AUTO_REVIEW_STRANDED_AFTER_S — the auto_review MCP checker: model (LiteLLM string, no default — unset means every auto_review call fails closed to a card, so set it when enabling the policy), total verdict budget (5s, one retry inside), and the sweep's stranded-review fail-closed bound (60s).

Model provider keys use standard LiteLLM env vars (no AIOS_ prefix): OPENROUTER_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, etc. These are the root-owned fallback: per-account keys and proxy base URLs live in the model_providers resource (/v1/model-providers), resolved nearest-ancestor-wins up the account tree at model-call time and encrypted per-account via the same subkey scheme as vaults. A model_providers row is looked up whole (key + api_base together, never field-merged across accounts) — see aios.services.model_providers and the conflict guard it runs before every model call.