Skip to content

architecture.md

Latest commit

 

History

History
160 lines (111 loc) · 49.8 KB

architecture.md

File metadata and controls

160 lines (111 loc) · 49.8 KB
Raw

Trinity - Autonomous Agent Orchestration Platform - Architecture

Purpose: The always-loaded core — system shape, the full invariant list, and a map to every area file. Deep detail lives in docs/memory/architecture/, read on demand.

Editorial rules: (1) One home per feature — each subsystem is described exactly once, in exactly one area file; every other mention is a pointer. (2) Catalogs are catalogs — router/service/endpoint entries are ≤2 lines; deeper behavior lives in the owning area file or docs/memory/feature-flows/. (3) No changelog narration — git history records what was replaced and when; issue tags (#526) are kept as lookup keys. (4) Write path — architecture updates go to the owning area file named in the Architecture Map; this core file changes only when the map itself changes (a new area, a new invariant, a topology change). (5) Size budget — this core stays under 500 lines. It is @-imported by CLAUDE.md, so every line is paid by every session; area files are referenced by path and are never imported.

System Overview

Trinity is an autonomous agent orchestration and infrastructure platform — sovereign infrastructure for deploying, orchestrating, and governing fleets of autonomous AI agents on your own hardware. Each agent runs as an isolated Docker container with standardized interfaces for credentials, tools, and MCP server integrations.

┌─────────────────────────────────────────────────────────────────────────────┐
│                           Trinity Agent Platform                             │
├─────────────────────────────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐    │
│  │   Frontend   │  │   Backend    │  │  MCP Server  │  │    Vector    │    │
│  │   (Vue.js)   │  │  (FastAPI)   │  │  (FastMCP)   │  │   (Logs)     │    │
│  │   :80        │  │   :8000      │  │   :8080      │  │   :8686      │    │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘    │
│         │                 │                 │                 │             │
│         └─────────────────┼─────────────────┼─────────────────┘             │
│                           │                 │                               │
│                    ┌──────┴──────┐   ┌──────┴──────┐                       │
│                    │    Redis    │   │   Docker    │                       │
│                    │    :6379    │   │   Engine    │                       │
│                    └─────────────┘   └──────┬──────┘                       │
│                                             │                               │
│         ┌───────────────────────────────────┼───────────────────────────┐  │
│         │                                   │                           │  │
│    ┌────┴────┐    ┌─────────┐    ┌─────────┴┐    ┌─────────┐           │  │
│    │ Agent 1 │    │ Agent 2 │    │ Agent 3  │    │ Agent N │           │  │
│    │ :8000   │    │ :8000   │    │ :8000    │    │ :8000   │           │  │
│    └─────────┘    └─────────┘    └──────────┘    └─────────┘           │  │
│         Agent Network (172.28.0.0/16)                                   │  │
└─────────────────────────────────────────────────────────────────────────────┘

Technology Stack

Layer Technologies
Frontend Vue.js 3 (Composition API), Tailwind CSS 3, Pinia 2, Vite 5
Backend FastAPI 0.100+, Python 3.13, Docker SDK 7.x, SQLite 3, Redis 7, httpx 0.24+
Agent runtime Python 3.13, Node.js 20, Go 1.21, Claude Code (latest)
Infrastructure Docker, nginx (prod reverse proxy), Cloudflare Tunnel (public endpoints), Tailscale (private VPN), GCP, Vertex AI Search (docs Q&A)

Python version is a single declaration, guarded (#1891). The three image Dockerfiles (docker/{backend,scheduler,base-image}/Dockerfile) are the source of truth; every python-version: in .github/workflows/ must equal that pin, enforced by tests/unit/test_1891_python_version_parity.py (scans all workflows, so a new one with a stale pin fails too; publish-cli.yml is allowlisted — PyPI packaging is governed by CLI consumers, not the image runtime). CI previously ran 3.11 against 3.13 images, which by construction cannot catch the stdlib-removal class that had already shipped twice (crypt → #1615, audioop → the audioop-lts VoIP pin). The same principle covers the tz database one layer down (#1823): the backend and scheduler images must ship the IANA backward-compatibility links (tzdata-legacy + the tzdata wheel — zoneinfo reads the system database first and falls back to the wheel only for keys the system lacks), because the floating python:3.13-slim tag silently migrated bookworm→trixie, where those links were split out of tzdata, and every schedule stored under a legacy alias (Europe/Kiev) stopped resolving — 500 on create, silently non-firing in the scheduler — while CI stayed green because tests/requirements-test.txt already declared tzdata (#1771), i.e. the test environment was more capable than production; guarded by tests/unit/test_1823_tz_capability_parity.py, so a future image slim-down that drops either source fails CI instead of re-shipping it.

Architecture Map

Every area below is read on demand, not auto-loaded. The third column is the point: it names the regression that reading the file prevents, because a topic label gives an agent no reason to spend the tool call. If you are about to change a file matched by the middle column, open the area file first.

Loading is a deliberate step, not an ambient one (#2642). Before changing a path in the "Owns" column, read the named area file first — as an explicit action, at the point the reasoning needs it, not after the first edit. Nothing injects it for you: no hook fires, and a hook would fire too late anyway (an edit comes after the reasoning that needed the file). Tooling that resolves a path to its owner parses this table via scripts/docs/architecture_map.py; there is no second copy of the map to drift.

Area file Owns Read it first, because
backend.md src/backend/routers/**
src/backend/services/**
src/backend/db/**
src/backend/main.py
src/backend/models.py
config/vector.yaml		 docker_service.agent_container_states() is deliberately tri-state; the older helpers collapse no container and Docker unreadable into one falsy value, so a naive reconcile empties a customer's roster on any daemon fault (#2196).
api-endpoints.md src/backend/routers/**
src/backend/main.py		 Static routes must be registered before the /{name} catch-all (Invariant #4). Declared after it, GET /api/systems/manifests answers "system 'manifests' not found" instead of listing manifests.
execution.md src/backend/services/task_execution_service.py
src/backend/services/chat_execution_service.py
src/backend/services/capacity_manager.py
src/backend/services/slot_service.py
src/backend/services/backlog_service.py
src/backend/services/dispatch_breaker.py
src/backend/services/dispatch_admission_service.py
src/backend/services/session_turn_service.py
src/backend/services/loop_service.py
src/backend/services/reminder_service.py
src/backend/services/idempotency_service.py
src/backend/services/event_dispatch_service.py		 Every terminal side effect is gated on winning the status CAS, and each CAS winner must close its own dispatch activity. A terminal writer that skips the close leaves the row started until the 120-minute sweep fabricates duration_ms = now - started_at, permanently recording a 15-minute run as a ~120-minute failure (#1804, third occurrence of that class after #45 and #767).
reliability.md src/backend/services/cleanup_service.py
src/backend/services/retention_guard.py
src/backend/services/canary_service.py
src/backend/services/canary_alerts.py
src/backend/canary/**
src/backend/services/heartbeat_service.py
src/backend/services/monitoring_service.py
src/backend/services/db_backup_service.py
src/backend/db/backup_primitives.py
src/backend/services/redelivery_governor.py		 OPS_SETTINGS_DEFAULTS is a safety floor, not a policy knob. It is read at prune time and is the fallback for every install that never wrote a row, so lowering one retroactively hard-DELETEs those installs' history seconds after their next boot. #1638 cost a live instance ~3 months of data and shipped green because a test asserted the destructive default.
agent-lifecycle.md src/backend/services/agent_service/**
src/backend/services/git_service/**
src/backend/services/template_service.py
src/backend/services/template_schedules.py
src/backend/services/template_plugins.py
src/backend/services/skill_service.py
src/backend/services/skill_packaging.py
src/backend/services/compatibility/**
src/backend/services/subscription_auto_switch.py		 lifecycle._apply_git_env_from_db takes its PAT gate as a required per-call-site parameter, never a default. Sharing one gate bakes the global platform PAT into a previously tokenless container, configure_push_remote then clears the push blackhole, and a private knowledge base can reach the shared public upstream (the ent#162 class). An AST guard pins the exact writer set, so a third writer on any container-seeded path fails CI.
agent-runtime.md docker/base-image/** .env is re-read at every spawn (#1999), but it must NOT be authoritative for ANTHROPIC_API_KEY while subscription auth is active: Claude Code prefers that key over the OAuth token, no recreate ever cleans it off the workspace volume, and SUB-003 then mis-attributes the failure to each healthy subscription in turn (#2114). Four policy modules here are byte-identical vendored mirrors of backend files (Invariant #5) and are parity-tested.
integrations.md src/backend/adapters/**
src/backend/services/event_bus.py
src/backend/services/ws_ticket_service.py
src/backend/services/voip_service.py
src/backend/services/slack_service.py
src/backend/services/tts_service.py
src/backend/services/voice_reply_service.py
src/backend/routers/agent_brain_orb.py
src/backend/services/agent_shared_files_service.py		 /ws is SCOPE_ALL and unfiltered, so a broadcast that carries a payload discloses it to every connected client. Broadcasts must be thin triggers carrying identifiers only, with listeners refetching through the access-controlled REST route (#918, pinned by test_918_report_broadcast.py).
observability.md src/backend/routers/reports.py
src/backend/routers/evaluations.py
src/backend/services/report_service.py
src/backend/services/report_export.py
src/backend/services/activity_service.py
src/frontend/src/components/reports/**		 Report renderer drift is silent: the write succeeds and the report degrades to a raw JSON dump. test_1535_report_prompt_guidance.py pins the prompt shapes against the display_hint enum and the renderer keys, and shapeOk must stay inside ReportRenderer.vue or that guard's matched set empties (#1535, #2162).
workspace.md src/backend/client_portal/**
src/frontend/src/views/Portal.vue
src/frontend/src/stores/clientPortal.js
src/frontend/src/components/portal/**		 The roster payload is the only capability channel a portal principal has. GET /api/settings/feature-flags is get_current_user-gated and its store returns an empty list for every external client, so a UI gate written against it is dead for exactly the audience it targets (#2128). Per-agent fields on that payload also fail in deliberately opposite directions and must not be normalised to match.
frontend.md src/frontend/src/**
src/frontend/scripts/**		 The dashboard renders three modes over one store seam. A filter added outside stores/network.js::visibleAgents silently applies to one mode only (ent#261). Any change here is also bound by docs/memory/design-system-contract.md and the raw-color ratchet, whose per-file counts may only shrink.
mcp-server.md src/mcp-server/**
src/helper-mcp/**		 Tool visibility is an allowlist (OPERATOR_SCOPES), never a deny-check. The previous !== "connector" shape advertised every operator tool to any scope it had not heard of, including a null auth context where canAccess filtering is skipped entirely (#848).
database.md src/backend/db/schema.py
src/backend/db/migrations.py
src/backend/db/tables.py
src/backend/migrations/versions/**
src/backend/db/alembic_runner.py		 Every schema change needs BOTH tracks: a versioned SQLite entry and a new Alembic revision (Invariant #9). Two revisions sharing a down_revision are two heads, and upgrade head resolves its target before applying anything, so it applies ZERO revisions while git reports no conflict (#2068).
security.md src/backend/dependencies.py
src/backend/error_handlers.py
src/backend/services/credential_encryption.py
src/backend/services/secret_settings.py
src/backend/services/credential_paths.py
src/backend/utils/url_validation.py
src/backend/utils/safe_yaml.py
docker-compose.yml
docker-compose.prod.yml		 require_admin and assert_admin reject agent principals themselves. An agent-scoped MCP key resolves to its owner CARRYING the owner's role, so on a default admin-owned install any agent's injected key satisfied every admin gate. That was five separate incidents (trinity-ops-agent#232, #1644, #1816, ent#236, ent#293) before the gate was fixed rather than the endpoints; never write require_role("admin"), which is a third spelling that lets agent keys through.
background-services.md src/backend/main.py Every loop here starts in EVERY uvicorn worker. A cycle without a Redis leader lease double-probes the fleet, double-persists its rows, and gives each cross-cycle marker two writers (#1464, #1632, #1881). Whether the lease fails open or closed is decided per loop and stated in each entry.

Architectural Invariants

These are structural patterns that must be preserved. Breaking them causes cascading issues.

  1. Three-Layer Backend: Router → Service → DB — Every feature follows routers/X.py → services/X_service.py → db/X.py. Routers hold no business logic, services hold no SQL, db modules hold no HTTP concerns. A layer that outgrows one file becomes a package, never a second module beside it (#1028) — routers/settings/ (ten domain modules), services/git_service/ (six) and services/agent_client/ (circuit / http_pool / client), each composing onto the same public name so every existing from routers.settings import … / from services.git_service import … resolves unchanged. Two rules make the split a refactor rather than a rewrite. (a) The package re-exports the PUBLIC surface only. A private collaborator mirrored on both the package and its owning module gives one function two monkeypatch targets, and a patch applied to the wrong one silently detaches — so a stale patch must raise instead, and a test reaches a private through the module that owns it (agent_client.circuit._CIRCUIT_HASH_PREFIX). (b) Callers reach a sibling through the module object, not a from-import alias (public_chat_service.agent_requires_email(...)), for the same reason one level up. Route-registration ORDER is the one behaviour a regrouping can change silently (Invariant #4), so routers/settings/__init__.py includes the /{key} catch-all last and tests/unit/test_1028_settings_package.py pins the fork-point route set as a frozen literal — read out of git, it skipped on every CI run, since every checkout is fetch-depth: 1. Guards that walk a layer must use rglob: routers/settings/ was the first subdirectory under routers/, and the Invariant #8 and #14 scanners were still globbing one level deep, going blind on ten modules while staying green.

  2. DB Layer: Class-per-domain with Mixin Composition — Each db/ file defines an XOperations class. Agent-specific settings use mixins (db/agent_settings/) composed into AgentOperations. New agent settings → new mixin, not a bigger class. A larger domain follows the same shape when a single file grows unreviewable: ScheduleOperations is composed in db/schedules/__init__.py (#1481) from ten concern-scoped mixins — ScheduleCommonMixin (MRO-shared _generate_id + the _norm_ts leaf helper) / ScheduleCrudMixin / ScheduleWebhooksMixin / ScheduleExecutionsMixin / ScheduleQueueMixin / ScheduleCleanupMixin / ScheduleAnalyticsMixin / ScheduleStatsMixin / ScheduleGitConfigMixin / ScheduleRetentionMixin. The facade import path from db.schedules import ScheduleOperations is preserved by the package re-export, and cross-slice references resolve at runtime via the composed class's MRO (self.<method>()) with no import edges between the mixin files — the only intra-package import is the bare _norm_ts module-global.

  3. Schema in db/schema.py, Migrations in db/migrations.py — All OSS table DDL lives in schema.py. Schema changes require a versioned migration in migrations.py (tracked in the schema_migrations table). Never create tables ad-hoc in service code. Runner safety (#1160): init_database() wraps both migration passes + init_schema in a cross-process flock (db/migration_lock.py) so workers + scheduler can't race; table-rebuild migrations use _atomic_rebuild (rename-swap inside BEGIN/COMMIT) so a crash mid-rebuild rolls back; a failed migration is named via add_note and surfaced as first_pending in the /health 503. Backend split (#1183): the db/migrations.py runner (PRAGMA + INSERT OR IGNORE) is SQLite-only; PostgreSQL is owned by Alembic — init_database()'s non-SQLite branch calls db/alembic_runner.upgrade_to_head() (src/backend/migrations/ + alembic.ini; env.py targets db/tables.py MetaData). Fresh PG built by the 0001_baseline revision (reuses init_schema_postgres DDL); pre-Alembic PG stamped at baseline. Multi-worker serialisation (#1425): the flock above only guards the SQLite runner — on PG, upgrade_to_head() wraps its stamp+upgrade in a PostgreSQL session advisory lock (pg_advisory_lock, fixed key) so concurrent worker/scheduler boots can't both enter command.upgrade() and deadlock inside a revision. Revision-id width (#1420): alembic_version.version_num is VARCHAR(255) (env.py version_table_column_type + 0001_baseline DDL; the 0008a_widen_alembic_version migration widens existing 32-wide DBs) because Trinity's descriptive NNNN_<table>_<change> ids exceed Alembic's 32-char default and PostgreSQL enforces the width (SQLite doesn't — so the truncation only breaks PG boot). Keep revision ids ≤255; the pg-migrations CI job runs a real alembic upgrade head and tests/unit/test_alembic_revision_id_length.py lints the ids. One head per version-line (#2068): scripts/ci/check_alembic_heads.py (pure stdlib, parses revision/down_revision without importing anything; wired unconditionally into schema-parity, so unlike path-filtered-and-advisory pg-migrations it is a required, always-evaluated gate) fails any version-line it is pointed at that does not resolve to exactly one head. Two revisions sharing a down_revision are two heads, and since upgrade head is singular and resolves its target before applying anything, such a graph applies zero revisions — every revision merged since the fork stops arriving, not only the one that forked — while git reports no conflict, because each file is individually valid and the defect exists only in the relationship. A merge revision's tuple down_revision counts all its members, so alembic merge correctly collapses a fork; the guard skips (loudly) a directory that is absent or holds no revisions, and fails closed when revision files are present but none parses. The private track's version-line is asserted the same way at its own branch point — that arm skips on public CI, which never checks the submodule out — and deploy-dev greps the registration boot log, so a version-line that merely degrades rather than crash-loops cannot ship green. Fresh only at PR-event time (#2533): that gate runs on the pull_request event, and GitHub recomputes refs/pull/N/merge when the base advances but does not re-trigger workflows — so a PR can read mergeable, all checks green against a base that no longer exists, which is how #2526's fork merged 75 minutes after its last green run. alembic-head-watch.yml closes that window for this class by re-running the same script, unmodified, over an in-memory git merge-tree of every open migration PR against the live dev tip on each push to dev (cron is only a dropped-run backstop, and fires solely from main); it is advisory — never a required context — and covers the Alembic single-head class only, since every other check is equally stale. It evaluates through an unrelated conflict (#2828): --write-tree writes the merged tree on exit 1 too, and the guard reads only the two version directories, so a PR that conflicts with dev in learnings.md — four of nine open PRs on 2026-09-15, two of them carrying a live fork — still gets a real verdict, with the unrelated paths named on the status and comment. Only a revision file edited on both sides is conflict, and that is published as a visible error status, never silence — "not evaluated" reading like "nothing to say" is how two forks reached the train green. Both coexist during transition, so a schema change lands in both migrations.py (SQLite) and a new Alembic revision (Postgres) — both tracks CI-guarded by the schema-parity job (SQLite parity pytest + the scripts/ci/check_alembic_parity.py cross-track guard that fails a DDL change missing its Alembic revision, #1342) — until SQLite end-of-support September 1, 2026 (#1278; guide docs/migrations/SQLITE_TO_POSTGRES.md) — after which the goal (#746) is tables.py MetaData as single source with autogenerated revisions. Two-track (open-core): enterprise owns only enterprise_* tables via a separate runner (enterprise/backend/_migrations.py, tracked in enterprise_schema_migrations, never OSS schema_migrations); one file per migration (NNNN_slug.py with NAME + upgrade(cursor, conn), filename order). Enterprise migrations may FK-into OSS tables but must never ALTER one — OSS enforcement goes through an OSS migration as an edition-agnostic primitive (e.g. users.suspended_at, #995). The enterprise runner runs from register_enterprise after OSS init_database.

  4. Router Registration Order Matters — In main.py, static routes like /api/agents/context-stats must come before /{name} catch-all. New collection-level agent endpoints must be registered before parameterized routes.

  5. Agent Server Mirrors Backend (Subset) — docker/base-image/agent_server/routers/ has routers that mirror a subset of backend routers (chat, credentials, files, git, skills, dashboard). The backend proxies to the agent server. Changes to agent-internal APIs must update both sides. Byte-identical vendored mirrors (the agent server ships as its own image and structurally cannot import src/backend, so a shared policy is copied rather than imported, each with a parity test): credential_paths.py (#11), model_context.py (#1521), safe_yaml.py (#1965 — the ent#314 hardened YAML loader), and mcp_validator.py (#2007). A guard that walks only one of the two trees is not a guard: ent#314's AST scan had an empty allowlist over the whole backend and still missed six bare yaml.safe_load calls in the agent server on the same author-controlled documents (template.yaml, skill frontmatter, dashboard.yaml, .trinity/persistent-state.yaml), because it never looked there. It now walks docker/base-image/agent_server/ too — extend the scan, not just the fix, whenever a policy gains a second home.

  6. Frontend: Store = Domain, View = Page — Pinia stores (stores/agents.js) are domain-scoped, not view-scoped. Views compose from multiple stores. Composables (composables/use*.js) extract reusable logic. API calls go through stores, not views directly.

  7. Single API Client (api.js) — One Axios instance with auth interceptor. Stores call api.get()/api.post(). No raw fetch() or duplicate Axios instances.

  8. Auth Pattern: Depends(get_current_user) + AuthorizedAgent — Every authenticated endpoint uses FastAPI Depends() for auth. Agent-scoped endpoints use AuthorizedAgent or OwnedAgentByName for access control. Role-gated endpoints use require_role("creator") or require_admin (ROLE-001). internal.py is the only exception (no auth, for agent-to-backend calls). Enumeration-safety (self-uniform, #186): an agent-access handler must be self-uniform — it must never return an existence-404 followed by an access-403 in the same function (that differential lets any caller enumerate which agents exist). The four dependency helpers (get_authorized_agent/get_owned_agent/…_by_name, dependencies.py) now return a uniform 404 for both a non-existent AND an inaccessible/unowned agent — they evaluate existence and access before branching (equal query-count → equal timing) and run _enforce_connector_scope first. Access-first inline handlers (can_user_access/share → 403 before any existence lookup) are already self-uniform and stay 403. When adding an agent endpoint: route through a dependency, or if checking inline, check access first — do not reintroduce a 404-then-403 split. The MCP third surface mirrors this (Invariant #13): chat.ts checkAgentAccess returns one uniform reason and never discloses the owner username. Guarded by tests/unit/test_186_enumeration_uniformity.py. Shared imperative-guard family (#1310): the inline access/owner/admin checks are consolidated behind five leaf helpers in dependencies.py — assert_admin (403; _reject_connector_principal + reject_agent_principal + role != "admin"), assert_agent_access (403; _enforce_connector_scope + can_user_access_agent), assert_agent_owner (403; _enforce_connector_scope(owner_op=True) + can_user_share_agent), assert_owns_or_admin (403; id != owner AND role != "admin"), assert_owns (403; id != owner, no admin bypass). All raise 403 (access-first → self-uniform, above). An admin gate is never agent-callable (ent#293/ent#297): get_current_user resolves an agent-scoped MCP key to its owner carrying the owner's role, so on a default admin-owned install any non-ephemeral agent's injected TRINITY_MCP_API_KEY satisfied every admin gate. Five occurrences of that one class — trinity-ops-agent#232, #1644 (retention acknowledge), #1816 (system-agent restart), ent#236 and ent#293 (skills-library repointing) — were each closed by bolting reject_agent_principal onto one more endpoint, 18 of them against 114 admin-gated call sites. Five occurrences meant the gate was wrong, not the endpoints, so require_admin and assert_admin now reject agent principals themselves (#1890): a route added tomorrow inherits the protection without anyone remembering to ask, and the per-endpoint reject_agent_principal calls are belt-and-braces rather than the mechanism. Safe by construction and verified, not assumed — the agent-key flows that must keep working (heartbeat, structured reports, the #1083 result callback) authorize on current_user.agent_name self-checks and never touch an admin gate, and User.agent_name is populated only for scope == "agent", so trinity-system still passes. This is the grant-vs-use line: the endpoint that uses a capability may be agent-callable; the endpoint that grants one is human-only. Two admin-gated reads that were uses, not grants, were re-opened rather than left half-working — POST /api/monitoring/agents/{name}/check drops to get_current_user (the admin gate was redundant with AuthorizedAgentByName) and GET /api/subscriptions is owner-scoped, not un-gated (un-gating would hand any role=user a fleet-wide owner_email + agent-name enumeration oracle — the Invariant #8 disclosure class, reintroduced by a fix for a different disclosure). Never write require_role("admin") — it is a third spelling that rejects connector but not agent principals, and require_role stays deliberately permissive because agent-spawned agent creation runs through require_role("creator") (ent#69 Part 2), so a blanket rejection there would break ghost spawning. Use require_admin (equivalent — admin is last in ROLE_HIERARCHY — and it rejects agent principals); an AST guard in tests/unit/test_293_admin_gate_rejects_agent_keys.py fails the build if the spelling reappears. The trigger to revisit an existing gate is a change in what the endpoint does — escalating a handler's destructiveness silently re-prices every principal that could already reach it. Imperative vs path-dependency: agent name in the path → prefer the path-dependency (uniform-404); agent name derived from a resolved resource (session/notification/subscription/execution row) or a composite gate → use the imperative helper; both run _enforce_connector_scope first so the connector boundary is enforced identically. assert_agent_owner ≠ delete-authorization — it wraps the owner-or-admin can_user_share_agent, NOT the is_system-guarded can_user_delete_agent; delete paths keep the delete predicate. Permanent exceptions (own intentional-404 helpers, enumeration-safe by construction): nevermined._require_read_access/_require_write_access, reports.get_report, sessions._session_or_404 (compound 404). The admin gate is an ALLOWLIST, not two named enemies (#2323). ent#293/#297 closed it against agent and connector by name, which is a denylist over a free-text column: mcp_api_keys.scope carries no CHECK constraint, and any scope setting neither agent_name nor connector_agent walks both rejections and inherits the owner's role across ~163 admin-gated sites. models.User.mcp_scope's own docstring predicted it — "fail-closed against a sixth scope a future PR invents" — and #2323's ops scope was that sixth. Both gates now additionally require mcp_scope ∈ ADMIN_GATE_SCOPES = {None, "user", "system"}: None is the JWT human the role check was written for, user legitimately drives admin endpoints today (ops dashboards, trinity-ops-agent) so removing it would be a behaviour change, and system is trinity-system. Zero behaviour change over the live domain except portal_delegate, which passed both named guards and was contained only by its route fence — it now has a second layer, unreachable in normal operation and firing only if that fence is holed. A principal object lacking mcp_scope entirely fails closed via a dedicated sentinel — never getattr(..., None), which would make the absent attribute the privileged JWT value (the documented getattr-discriminator trap); models.User always declares the field, so the only objects this rejects are stand-ins that do not match the real principal. assert_admin takes allow_scopes= so an individual endpoint can opt a bounded scope in (allow_scopes={"ops"} on the ops reads) — the grant that makes a machine credential authorized by what it is rather than by who owns it, and which flips the failure direction so a new ops route is inaccessible until opted in rather than silently reachable. require_role is deliberately not consulted (agent-spawned creation through require_role("creator") is supported ent#69 behaviour). Static guard: tests/unit/test_1310_auth_wiring.py; behavioral proof: tests/unit/test_1310_auth_consolidation.py; allowlist + future-scope proof: tests/unit/test_293_admin_gate_rejects_agent_keys.py.

  9. Channel Adapter ABC — External messaging (Slack, Telegram, WhatsApp/Twilio) follows adapters/base.py → ChannelAdapter ABC with NormalizedMessage and ChannelResponse. New channels must implement this interface.

  10. WebSocket Events for Real-Time — All real-time updates go through WebSocket broadcast (agent_activity, agent_collaboration). Frontend subscribes via utils/websocket.js. Don't poll for state that should be pushed. Transport is the Redis Streams event bus in services/event_bus.py (RELIABILITY-003, #306) — ConnectionManager / FilteredWebSocketManager are thin shims that XADD to trinity:events; the StreamDispatcher runs one XREAD BLOCK per backend process and fans out to registered clients. New broadcast sites should continue calling the existing manager.broadcast(...) / filtered_manager.broadcast_filtered(...) API — do not bypass it to publish directly.

  11. Docker as Source of Truth — Agent container state comes from Docker labels (trinity.*), not from an in-memory registry. docker_service.py is the single point of Docker interaction.

  12. Credentials: File Injection, Never Stored in DB as Plaintext — Credentials use .env files injected into containers (CRED-002). Encrypted exports use AES-256-GCM (.credentials.enc). Redis holds transient secrets. Exception with mandatory encryption: channel bot/auth tokens (Slack, Telegram, WhatsApp) and subscription/Nevermined OAuth tokens are persisted in SQLite because they drive long-lived background processes (webhook receivers, scheduled bots) that can't depend on container env vars. These MUST be wrapped in AES-256-GCM JSON envelopes via services/credential_encryption.py — plaintext persistence is forbidden. Tables under this rule: subscription_credentials.encrypted_credentials, nevermined_agent_config.encrypted_credentials, telegram_bindings.bot_token_encrypted, whatsapp_bindings.auth_token_encrypted, agent_git_config.github_pat_encrypted, users.github_pat_encrypted (per-user GitHub PAT — ent#162; resolved live by owner at agent creation via settings_service.resolve_github_pat), slack_workspaces.bot_token (TEXT column, JSON-envelope content), slack_link_connections.slack_bot_token (TEXT column, JSON-envelope content — encrypted by #453, 2026-05-05), system_settings['elevenlabs_api_key_encrypted'] (platform ElevenLabs key, JSON-envelope value — ent#117; runtime-resolved by settings_service.get_elevenlabs_api_key()), system_settings['a2a_outbound_endpoints_encrypted'] (the outbound A2A endpoint list — one envelope over a JSON document whose entries each carry a peer credential; #736. Chosen over a new table precisely because this location is already blessed: it is why #736 ships with no migration and no Alembic revision), and — since ent#435 — the six credential-bearing settings system_settings['{anthropic_api_key,github_pat,google_api_key,slack_app_token,slack_client_secret,slack_signing_secret}_encrypted'] — plus resend_api_key_encrypted since ent#582 (the email-provider key, env-only before, so no cleartext row predates it; membership is SECRET_SETTING_KEYS, which the rotation script derives from). ent#435 closes the invariant's own loophole: those six rows held live third-party credentials in CLEARTEXT while this list read as though everything was covered, so the platform's encryption-at-rest posture was contradicted by a single SELECT (CWE-312) — every dump, backup, replica and snapshot carried usable tokens, readable without CREDENTIAL_ENCRYPTION_KEY. The fix is a policy leaf (services/secret_settings.py) plus a sink guard: db.set_setting raises SecretSettingWriteError (→ 422) for any key in SECRET_SETTING_KEYS or merely credential-shaped (*_api_key/*_token/*_secret/*_pat/*_password/*_credentials), because the generic PUT /api/settings/{key} catch-all can address ANY key — the same door #506, #1609, ent#12, #1644, ent#14 and ent#346 each found open. Three properties are load-bearing. (1) The key NAME moves (<key> → <key>_encrypted, cleartext row DELETED), not just the value: a same-named key that may hold either form leaves "is this install encrypted?" unanswerable by inspection, which IS the reported defect; with the rename the reporter's own query (SELECT key … WHERE key IN (…) returning nothing) is the proof and the guard keeps it true. (2) The read path lazily migrates — settings_service._resolve_secret_setting resolves encrypted → legacy-cleartext-encrypted-and-deleted-on-sight → env → '', so a restored pre-fix backup or a direct DB write makes cleartext transient rather than merely absent right now; steady state pays nothing (the legacy key is never read while the encrypted row exists). It is fail-OPEN on read (an unreadable envelope degrades to env, never a 500 on the agent-start path) but deliberately does not fall back to a stale legacy row, which would resurrect a replaced credential; and fail-CLOSED on write. (3) slack_client_id is a documented exemption, not an omission — an OAuth client_id is emitted verbatim in the browser-visible authorize URL (the whatsapp_bindings.account_sid "(public)" precedent) — recorded with its reason in PUBLIC_CREDENTIAL_SHAPED_KEYS so a later reader can tell reviewed from overlooked. Dual-track sweep (secret_settings_encryption + Alembic 0041) sharing ONE decision function (plan_migration) since the two drivers cannot share SQL; hard-fails on a missing key only when there is something to encrypt (the #453 choice, so a fresh install still boots); downgrade() is a deliberate no-op, because the honest inverse is "write these credentials back in cleartext". scripts/deploy/rotate-credential-key.py gains a row-keyed system_settings pass, which also closes the pre-existing gap that left elevenlabs_api_key_encrypted/a2a_outbound_endpoints_encrypted out of every rotation (envelope-in-a-row is invisible to a column sweep). Encryption protects the DB going forward only — historical backups still hold the plaintext, so the affected tokens must be rotated: runbook docs/migrations/SECRET_SETTINGS_ENCRYPTION_2026-08.md. Guards: tests/unit/test_ent435_settings_sink_guard.py (AST — every system_settings writer is gated or listed with a reason; OSS tree only, the private submodule owns its twin per the #1677 convention).

  13. MCP Server = Third Surface in Sync — The MCP server (src/mcp-server/src/tools/*.ts) is a TypeScript proxy over the backend API. When adding a backend endpoint for external access, the MCP tool module needs updating too. Three surfaces must stay in sync: backend router, agent server (if internal), MCP tool (if external). Every router outside the by-design exclusions (internal, setup, auth, public, paid) declares its MCP surface in a first-line # mcp: header — a pointer to the covering tool module, or # mcp: none — <reason> for a deliberately unexposed surface (admin, grant-vs-use human-only, UI-only) — so /validate-architecture can tell "unexposed on purpose" from "forgotten".

  14. Pydantic Models Centralized in models.py — Request/response models live in models.py, not scattered across routers (#654). Keeps the API contract in one place. Scope: this invariant governs router models — a class X(BaseModel) must not be defined under routers/ (enforced by the static guard tests/unit/test_models_centralized.py). Two model homes are intentionally separate and out of scope: db_models.py (DB-row / persistence models — a distinct layer) and adapters/base.py (the ChannelAdapter ABC's NormalizedMessage/ChannelResponse). One documented exception, allowlisted in the guard: routers/canary.py::RunCycleRequest evaluates INVARIANTS (from the canary library) in a Field(description=…) at class-definition time, and the canary library imports TaskExecutionStatus back from models — relocating it would force models.py to from canary import …, inverting the dependency direction of a module meant to be a low-level leaf everything imports from.

  15. API URL Nesting Convention — Agent-scoped resources nest under /api/agents/{name}/.... Platform-wide resources get top-level prefixes (/api/executions, /api/operator-queue).

  16. Time-Window SQL uses iso_cutoff(), not datetime('now', ...) — Columns written via utc_now_iso() are ISO-Z strings (T separator, Z suffix); SQLite's datetime('now', ...) emits a different format (space separator, no suffix), making lexicographic comparison silently incorrect (#476). For rolling-window filters on ISO-Z TEXT columns, compute the cutoff in Python via iso_cutoff(hours) from utils/helpers.py and pass it as a bound parameter. Write-side/read-boundary cousin (#1474): the standalone scheduler is a separate package that can't import utils/helpers.py, so its execution/schedule timestamps (started_at/completed_at/validated_at/last_run_at/retry_scheduled_at + process-schedule variants) now serialize via a vendored behavioural mirror src/scheduler/utils.py (utc_now_iso/to_utc_iso, Z-suffixed) — functionally identical to the backend copy but textually divergent (to_utc_iso uses an early return vs the backend's if/else), so unlike failure_classifier.py a source diff CANNOT verify it; the contract is agreement on output, enforced by tests/unit/test_1713_scheduler_utils_parity.py (#1713) and the property/edge suites tests/unit/test_1771b_timestamp_helpers_{properties,edges}.py (#1771) — previously naive strings that JS new Date(...) mis-parsed as local time. parse_scheduler_ts reads tolerantly and returns naive UTC so the historical model type + aware − naive duration math are preserved (write + read are one atomic change). The leaking backend read boundaries (db/schedules.py get_agent_executions_summary/get_fleet_executions/get_agent_schedules_summary last_run_at; db/activities.py _row_to_activity/_mapping_to_activity) normalize naive stored strings via parse_iso_timestamp+to_utc_iso, fixing historical rows for all consumers; the 5 execution panels also parse via parseUTC as render-layer defense (covers WS-pushed timestamps). Honest caveat: agent_schedules.next_run_at stays mixed-format across the scheduler (Z) and the backend's own next_run_at.isoformat() writers (db/schedules.py, tz-aware +00:00/offset) — safe because next_run_at is only ever parse-compared in Python, never lexicographically in SQL (Invariant #16's trap doesn't apply). main.py SchedulerStatus.last_check (health endpoint, not a DB row / not a relative-time surface) is out of scope.

  17. Non-root containers — every Trinity-built image MUST end with a USER directive switching to a non-root user; the backend additionally requires group_add: ${DOCKER_GID:-999} in compose for Docker socket access on Linux. New Dockerfiles failing this are rejected at review (#874). CI guards in .github/workflows/container-security.yml (path-filtered on docker/**, docker-compose*.yml, scripts/deploy/start.sh, src/mcp-server/Dockerfile, independent of the ui-gated e2e workflow): verify-non-root execs the backend/scheduler/mcp-server containers, asserts UID 1000, and proves group_add works by running docker.from_env().ping() from inside the backend (not a /api/agents probe — list_all_agents_fast swallows Docker errors, a false-positive trap); verify-prod-frontend-uid builds the prod frontend out-of-band and asserts UID 101 (nginxinc/nginx-unprivileged). Dev-only docker/frontend/Dockerfile is exempt. Upgrading deployments must re-own their data path and agent-configs volume per docs/migrations/NON_ROOT_CONTAINERS_2026-05.md.

  18. Trigger boundaries accept Idempotency-Key (RELIABILITY-006, #525) — every producer boundary that creates an execution accepts an optional Idempotency-Key header and routes it through services/idempotency_service.py (begin/complete/fail) backed by the idempotency_keys table. The same (scope, key) within 24h yields one execution; duplicates short-circuit with the original result + X-Idempotent-Replay: true (in-flight duplicate → 409). Enforcement lives at the router layer, not solely in TaskExecutionService, because sync /chat runs an inline path and /api/webhooks/{token} creates no execution. Wired boundaries: /chat, /task, /api/internal/execute-task, /api/webhooks/{token} (auto-derives (token, body_hash)), /api/paid/{name}/chat (#1018 — always keys on (payment-signature ∥ message) via derive_payment_key, a divergent client header never forks execution; a completed-unsettled snapshot re-drives settle then upgrade_snapshots), /api/agents/{name}/fan-out, and the scheduler (Idempotency-Key: sched:{execution_id}) + MCP chat_with_agent/fan_out (deterministic key over call args). Any new trigger type must accept an idempotency key before merge — the dedup layer is fail-open (a key never blocks a real execution), so the cost of adding it is one begin/complete/fail triple.

Network Topology (#589)

Two Docker bridge networks, by design — agents physically cannot route to Redis.

Network Subnet Members
trinity-platform-network 172.29.0.0/16 redis, scheduler, vector
trinity-agent-network 172.28.0.0/16 agents, frontend

Bridges (members of both networks): backend (primary HTTP API — Redis on platform side, agents on agent side), mcp-server (agents reach http://mcp-server:8080/mcp via Docker DNS), otel-collector (agents push metrics), cloudflared (prod only — proxies to backend and public agents).

Rule: agents are never on trinity-platform-network. Any new service that mounts the agent network must NOT connect to Redis — full stop. The three agent-container-create sites hard-code the network name trinity-agent-network (cited by function so the reference survives line drift): crud.py::_create_agent_container (#1484), lifecycle.py::_provision_folders_and_run_agent_container, and system_agent_service.py::SystemAgentService._create_system_agent.

Redis ACL users:

User Auth Purpose
default REDIS_PASSWORD Admin / recovery / ad-hoc ops; +@all
backend REDIS_BACKEND_PASSWORD Backend runtime; data ops only, -@dangerous
scheduler REDIS_BACKEND_PASSWORD Scheduler runtime; same access pattern

backend/scheduler cannot run FLUSHALL, CONFIG, SHUTDOWN, DEBUG, MIGRATE, REPLICAOF, MONITOR, or other @dangerous categories. Both passwords are mandatory in .env; compose refuses to render without them, and src/backend/config.py / src/scheduler/config.py raise on import if REDIS_URL lacks credentials. Upgrade path: docs/migrations/REDIS_AUTH.md.

Development Environment

Local and production use the same ports. Local URLs, auth, and admin credentials: see CLAUDE.md / CLAUDE.local.md.

Port Service
80 Frontend (nginx/Vite) — prod: https://your-domain.com
8000 Backend (FastAPI) — /docs for OpenAPI
8080 MCP Server (/mcp)
8686 Vector health
2222–2262 Agent SSH

Data Persistence

  • Bind mount (survives docker-compose down -v): ~/trinity-data/ → /data — contains trinity.db (SQLite), agent-files/ (FILES-001), agent-data-tmp/ (transient export staging, #1169), agent-import-tmp/ (transient copy-intent snapshot staging, ent#15 — free-space preflighted, AGENT_IMPORT_MAX_BYTES-capped, swept after 24h), and backups/ (#2216 — the automatic database recovery points: trinity-backup-YYYYMMDD.{db,dump} nightly + pre-migration-YYYYMMDD-HHMMSS.db at boot; dir 0700; same-disk scope — protects against corruption/slips, not disk loss; retention backup_retention_days + MIN_KEEP=3).
  • Docker volumes: redis-data (Redis AOF), agent-configs, audit-data, audit-logs, per-agent agent-{name}-workspace (the durable home volume — declared data_paths runtime data lives under /home/developer/data here, #1169), agent-{name}-public (FILES-001), and shared-folder volumes.