Trinity Architecture — Backend catalog

Part of the Trinity architecture set. Core map, invariants and topology: architecture.md. This file is not auto-loaded.

Owns: src/backend/routers/**, src/backend/services/**, src/backend/db/**, src/backend/main.py, src/backend/models.py, config/vector.yaml

Read this before changing the paths above: docker_service.agent_container_states() is deliberately tri-state; the older helpers collapse no container and Docker unreadable into one falsy value, so a naive reconcile empties a customer's roster on any daemon fault (#2196).

Write path: changes to this area land here, not in the core (core editorial rule 4). Keep the core's map row in step if the owned paths change.

Backend (`src/backend/`)

Core modules:

Module	Purpose
`main.py`	FastAPI app initialization, WebSocket manager, router mounting. `lifespan` is an orchestrator, not a procedure (#1028) — it was 580 lines at cyclomatic complexity 109, the longest function in the backend, and is now a flat list of `await _phase()` calls over sixteen helpers whose bodies were moved verbatim. The ORDER is the contract, and several pairs are load-bearing in ways invisible at the call site: logging first, so a later hang cannot swallow the boot log (#858); the event bus before any WebSocket client needs a live dispatcher to register with (#306); Docker/system-agent before the services that sweep the fleet; startup recovery before the channel transports, so inbound traffic cannot create an execution that races the reconcile; and the event-bus drain LAST on shutdown, so late-lifecycle broadcasts emitted while the other services stop still land on the stream. Each phase keeps its own `try/except` — a failing phase must not take the boot down. New startup work goes in a phase helper, never inline: `tests/unit/test_1028_lifespan_phases.py` pins the sequence (with the reason for each constrained pair recorded beside it), the `yield` split, the per-helper thresholds, orphan/double calls, and that `@asynccontextmanager` sits on `lifespan` and nowhere else — the extraction moved it one definition down onto the first helper, leaving `lifespan` a bare async generator FastAPI cannot use, and the entire unit suite stayed green because nothing imports `main` to ask what shape it is
`config.py`	Centralized configuration constants
`models.py`	All Pydantic request/response models (Invariant #14)
`dependencies.py`	FastAPI dependencies (auth, token validation, role hierarchy, agent access control)
`database.py`	SQLite persistence facade — orchestrates 27 domain operation classes from `db/`
`db/write_params.py`	Parameter objects for the four widest writers (#1482): `update_execution_status(execution_id, status, result: ExecutionResult, *, claim_token=)`, `create_task_execution(..., fields: TaskExecutionFields)`, `create_schedule_execution(..., source: ExecutionSource)`, `add_chat_message(..., fields: ChatMessageFields)`. Frozen + `kw_only`; field names = column names. The facade passes the object straight through, so a new column is ONE field here instead of a parameter added (and forwarded) in both layers — the EXEC-023 class, where the facade silently dropped `claude_session_id`. In `ExecutionResult`, `None` NULLs a column except `retry_count`/`turn_integrity` (None = leave as is); `claim_token` stays a keyword because it is a CAS precondition, not a value. Tests: `tests/unit/_write_params.py::flat_kwargs` reads a mocked call's fields
`logging_config.py`	Structured JSON logging (captured by Vector); OTel trace ID in log entries for log-trace correlation (RELIABILITY-002)
`error_handlers.py`	App-level exception handlers, registered in `main.py`. Owns the 422 shape: Pydantic's `input` (the rejected value) is stripped from every validation-error entry, so a failed `SecretStr` validation cannot return the secret (ent#109)
`utils/safe_yaml.py`	The hardened parser for every author-controlled YAML — size cap + alias policy + duplicate-key rejection (ent#314). Two policies, both required arguments (no default): `AliasPolicy.BUDGET` bounds expansion cost (manifests, `template.yaml` catalog — a legitimate anchor still parses), `AliasPolicy.REJECT` refuses any alias (skills frontmatter, the live agent-writable `template.yaml`, whose consumer walks every field). Consolidates #1884's manifest loader and two `_NoAliasSafeLoader` copies; the catalog path had no guard, which is the ent#314 hole. Bare `safe_load` leaves alias amplification that only blows up at SERIALIZATION (measured 416 B → 110 MB via `json.dumps`; the parse is 0.001 s, so an input cap cannot close it) and silent last-wins duplicate keys. Rejects, never truncates.
`utils/addressee.py`	The ONE addressee-email rule, `normalize_addressee_email` (strip + lower-case; not email-shaped → `None`) — a pure leaf importing only `typing`, so `models.py` uses it without importing a service; re-exported by `services/turn_audience.py`. Its boundary form (`models._validate_audience_email`, blank → absent, else refuse by name) serves both `audience_email` validators (#2955)
`utils/url_validation.py`	The SSRF module. `_validate_public_https_url(url, , label, …)` (#736) is the ONE public-HTTPS-destination gate — HTTPS-only, no userinfo, IDNA/A-label normalised once, every resolved address vetted, DNS failure fatal — shared by `validate_template_registry_url` and the new `validate_a2a_endpoint_url`. Extracted rather than cloned a third time: a third ~90%-identical copy inside the module whose job is centralising this policy would be the Invariant #5 failure happening in the file that exists to prevent it. `validate_skills_library_url` stays separate by design* (github.com-allowlisted, DNS-failure-tolerant — pinned by a test that explains why unifying it would be wrong). The A2A form returns the validated addresses, so the caller can pin the connection to one this function approved
`redis_breaker_util.py`	Shared Redis plumbing (fail-open client, Lua `ScriptCache`, decode helpers) used by both circuit breakers. Also home to the `SingleFlightLock` primitive (#1920) — the one shared fail-open, ownership-checked single-flight lock (`acquire` / `refresh_if_owned` → `LeaseState{OWNED,REACQUIRED,LOST,DEGRADED}` / `release_if_owned`) reusing `lock_token_matches` as the ownership predicate; mints a unique per-acquire `uuid` token and releases via GET-then-DELETE compare-and-delete. That release is deliberately non-atomic (accepted GET→DELETE TOCTOU window, the same one `ops.py` shipped), NOT atomic-Lua, so it stays fakeredis-unit-testable; the async `ResumeLock`'s Lua compare-and-delete (`session_turn_service.py`) is the stronger, deliberately-separate primitive — do NOT merge `ResumeLock` onto this weaker release. Consumers: `system_seed_service`, `agent_service/ephemeral`, `routers/ops` (adds `refresh_if_owned`), `skill_service` ×2 (injected client), `cornelius_agent_service`, `compatibility/fixes` (the last two were verbatim twins of system_seed's constant-"1" + unconditional-delete bug, fixed with it), `agent_service/deploy` (`agent:deploy_op:{base_name}`, #2060 — adopted at ship, never hand-rolled on dev). Deliberate non-consumers: `ResumeLock` (async + Lua + blocking-poll), the `monitoring`/`operator_queue` leader leases (stable cross-cycle worker id), and canary's Lua-CAD leader lease

OpenTelemetry tracing (RELIABILITY-002): auto-instrumentation for FastAPI/httpx/Redis; traceparent propagated through inter-agent calls; OTLP/gRPC export to trinity-otel-collector:4317; OTEL_ENABLED=1, OTEL_SAMPLE_RATE (default 10%).

Routers (routers/) — 72 router modules:

Core Agent:

agents.py - Core CRUD, start/stop, logs, stats, queue, activities, terminal (1795 lines)
agent_config.py - Per-agent settings: autonomy, read-only, resources, capabilities, capacity, timeout, api-key
agent_files.py - Files, info, playbooks, permissions, metrics, shared folders, file-sharing toggle + list/revoke (FILES-001) + the declared-metric registry read/refresh (ent#477: GET /metrics/definitions, POST /metrics/definitions/refresh). GET /playbooks reads through services/agent_skills_listing.py (?last_known=true: a stopped agent's last-known list, ent#754)
metric_points.py - The metric WRITE path (ent#478): POST /api/agents/{name}/metrics/points. Its own module rather than more of agent_files.py because it carries a limiter, a size guard, batch idempotency and an audit path — router → services/metric_points_service.py → db/metric_points.py (Invariant #1). Gate order is the design: self-gate (access-first) → rate → size → claim → definitions → validate → cap → insert, with ONE _reject choke point releasing the claim on every non-2xx exit past the claim
agent_data.py - Runtime-data export/import (data_paths) over the durable home volume (#1169)
agent_brain_orb.py - Brain Orb proxy: /brain-orb/data + /scopes + /tool (read) + /scope (owner mutation) + /voice-token (Phase 3 ephemeral Gemini mint) — see Brain Orb (#58, #60)
loops.py - Sequential agent loops: start/get/stop + agent-scoped list (#740)
reminders.py - Agent self-reminders: create/list/cancel (self-gated) (#1296) — see Agent Self-Reminders
files.py - Public download endpoint for outbound agent file sharing (FILES-001)
agent_rename.py - Rename endpoint (RENAME-001)
a2a.py - A2A protocol: the authenticated per-agent card (#737) plus the inbound server on a separate prefix-less a2a_server_router — public GET /a2a/{name}/.well-known/agent-card.json (per-IP rate limited) + POST /a2a/{name} JSON-RPC (message/send, message/stream SSE, tasks/get, tasks/cancel). Exposure is opt-in per agent (agent_ownership.a2a_exposed, default OFF); non-exposed/inaccessible → uniform 404 (Invariant #8). messageId dedup is scoped per (agent, caller principal) — the field is peer-controlled and only unique per-client (ent#157). Also hosts the OUTBOUND client (#736): POST /{name}/a2a/call + POST /{name}/a2a/task, a Trinity agent tasking an EXTERNAL A2A agent. The target is never caller-supplied — it is a name resolved through services/a2a_outbound.py — and the routes are thin (auth + HTTP error map + audit); orchestration lives in services/a2a_outbound_service.py. Default OFF (A2A_OUTBOUND_ENABLED), both routes 404 when off. The inbound door also takes payment (ent#679): its principal is OPTIONAL (get_user_or_anonymous, which degrades on a 401 but RE-RAISES a 403 so a fenced key cannot become a payer), and the handler branches once — a resolved principal gets today's free path byte-identically, an anonymous caller gets today's 401 unless the agent is BOTH exposed and Nevermined-enabled, in which case it gets the paid door's own 402 with resource.url on THIS door. The card producer _card_with_exposed_skills also declares the price (a2a_card_service.with_payment_extension), so both card surfaces state it and an unpriced agent's card is byte-identical. #3215: _base_url_from_request now delegates to utils/public_url.py::public_base_url — one module, one X-Forwarded-Proto upgrade rule — and the anonymous JSON-RPC door uses it too, so neither 402 door can advertise an origin the caller did not call; it also honours the Settings public_chat_url row, not just the env var. The two CARD routes take _card_base_url (configured-first, review I3), which is the card's pre-#3215 behaviour and not a second copy of the logic: a card is republished to buyers elsewhere and the get_agent_a2a_card MCP tool reads it from backend:8000, so a same-host card advertises an internal host to every external buyer. A buyer that follows the card lands on the configured host, where the door's own same-host rule mints the 402 for that very origin. _task_object MIRRORS the x402 metadata onto the Task's top-level metadata from the same assignment as status.message.metadata (the spec location, which the provider SDK reads) — without it a typed client renders a settled, charged turn as metadata: null; the free path emits no metadata key, so unpaid Task bytes and the idempotency snapshots over them are unchanged
aauth.py - AAuth discovery documents (ent#623; behind the graduated aauth_enabled switch since ent#862): GET /.well-known/aauth-{agent,jwks,resource}.json, per-IP limited, uniform 404 unless AAuth is live. The AAuth path on POST /a2a/{name} lives in a2a.py (_A2AServerRoute routes a Signature-Key request around the bearer dependency) — see requirements mcp.md §32.8
aauth_person.py - AAuth person token (ent#878, prototype; require_live needs BOTH aauth_enabled and aauth_prototype_enabled since ent#862, and is the only reader of the prototype switch): POST /api/agents/{name}/aauth/person-token — the flag gate as a Depends (uniform 404 before body validation), the agent's OWN key only, two per-agent rate budgets + a fleet budget, the audit row (aauth_person_token_{issued,pending,denied,refused}, token_jti never the token), and the HTTP mapping of the service outcome — see requirements mcp.md §32.9
aauth_resource.py - AAuth resource call (ent#879, prototype, both switches): POST /api/agents/{name}/aauth/resource-call — ent#878's gates reused (aauth_person.require_live / assert_own_key, public since ent#879) before the allow-list, three rate budgets, the discriminated call-XOR-resume body, the audit row (aauth_resource_call after egress / aauth_resource_refused before it; never a token, body, justification, handle, url or code) and the HTTP mapping of the ladder's outcome (protocol outcomes 200 with outcome, refusals 4xx/5xx with possibly_executed on a POST timeout) — see requirements mcp.md §32.10
agent_ssh.py - SSH access endpoint
credentials.py - Credential injection/export/import (CRED-002)
chat.py / chat/ - Agent chat/activity monitoring
internal.py - Internal endpoints for agent startup, scheduler task execution (gated by the router-level verify_internal_secret dependency — X-Internal-Secret shared secret, C-003, not a user principal; see Container Security). Also hosts the dark pull/work-stealing seams GET /api/internal/next-task (atomic claim + lease; the turn limit is clamped to the agent timeout the lease is sized from, #2846) + POST /api/internal/tasks/{id}/result (CAS) on a separate pull_router with dual-auth (internal secret OR the calling agent's own scoped MCP key — the system agent's own system-scoped key qualifies for trinity-system only, #3814) — inert until PULL_MODE_PILOT_AGENTS is set (#1081 Phase 1)
templates.py - Template listing and GitHub repo fetching; local: ids resolve through template_service.contained_template_dir (name-allowlist + resolve()/is_relative_to), the same barrier shape the create path has used since #950 (#1900)
sharing.py - Agent sharing between users
git.py - Git sync endpoints (status, sync, log, pull)

Auth & Security:

auth.py - Admin login (username OR registered email + password, #82), email auth, token validation. POST /api/auth/email/verify re-checks the allow-list before redeeming a code (#2381). Codes live in one shared table keyed on (email, code) with nothing binding a code to the channel that minted it: /email/request checks the allow-list before minting, but telegram_adapter / whatsapp_adapter /login <email> mint after a shape check only, and mcp_auth_service mints for any address users already knows. Their own redeemers grant channel- or connector-scope; this redeemer issues a full platform JWT carrying the matched account's role, and get_or_create_email_user resolves by the email column alone — so an attacker-controlled address bound to the admin row turned any of those mints into an admin session that no password reset or restart revokes. The refusal shares the bad-code branch verbatim (status, detail, audit action, rate-limit accounting) so it is not a membership oracle, and fails closed on a read error. Narrows nothing legitimate: web codes already passed this check at mint, channel users redeem in their own channel, and every account that can email-login is allow-listed by construction (sharing, access-request approval, the admin allow-list UI). The deeper question — whether an email-column lookup may return the admin row at all — is deliberately not closed here
users.py - User management, roles (ROLE-001); PUT /me/email self-service sign-in email (#82 transition); GET|PUT|DELETE /me/preferences[/{key}] — the caller's own per-user UI preferences (trinity-enterprise#413, OSS-core): services/user_preferences_service.py (key allowlist → 404, JSON-object-only → 422, 256 KiB cap → 413, base_updated_at insert-only/compare-and-set → 409 carrying the live record) over db/user_preferences.py (single-statement conditional write: INSERT refused by the PK, UPDATE … WHERE updated_at = :base decided by rowcount — never the single-writer read-then-write of db/canvas.py). All three routes reject_non_interactive_principal (JWT humans only): an agent-scoped key resolves to its owner and must not read or rewrite the operator's board
mcp_keys.py - MCP API key management
connector.py - Per-agent MCP connector: config + scoped-key mint/regenerate/revoke + exposed-playbook allow-list (/api/agents/{name}/connector*). OSS-core (ent#46 → #118); see mcp-connector.md
mcp_auth.py - #848 inline email auth internal surface (/api/internal/mcp-auth/{request,verify,playbooks,chat}): lets a keyless MCP client sign in with the 6-digit email code and then reach the connector playbooks of agents shared with that address. Dual-gated — X-Internal-Secret AND MCP_INLINE_AUTH_ENABLED (404s the whole surface when off, so a disabled deploy doesn't advertise it). The secret authenticates the caller, never the action: every data call re-gates on assert_email_may_reach_agent → db.email_has_agent_access(agent, email) + connector-enabled, so a compromised MCP server cannot reach an agent the asserted email cannot. See mcp-connector.md
agent_mcp_key.py - The agent's OWN scope='agent' Trinity MCP key: read + container config-truth probe + rotation (/api/agents/{name}/mcp-key*); see Agent MCP Key and agent-mcp-key.md (#1854)
setup.py - First-time setup wizard; required admin email (sign-in identity) + opt-in hosted intake (trinity-enterprise#38, #82). Setup token removed — no token, no Redis dependency for setup (trinity-enterprise#49). The endpoint's precondition is now its own, not a derived flag (#2381): POST /admin-password refuses whenever a usable admin account already exists, so it can only ever provision the FIRST account. ent#49 accepted the tokenless first-run window on the premise there is no admin yet; that premise is false for every install booting with ADMIN_PASSWORD set (mandatory in docker-compose.prod.yml, always present after start.sh), where _ensure_admin_user has already created one — and setup_completed said false anyway, because the setup_completed_backfill migration asks "does an admin exist?" during run_all_migrations, i.e. before _ensure_admin_user populates users, then records itself as applied so it never asks again (PostgreSQL had no writer for the key at all). So an unauthenticated caller could overwrite a live admin's password hash and bind their own email as its sign-in identity. The check is fail-closed (a DB read error refuses) and runs above the bcrypt hash, on a route with no auth and no rate limit. Username comes from utils/admin_identity.admin_username(), not a literal — _ensure_admin_user honours ADMIN_USERNAME, and against the old hardcoded "admin" an ADMIN_USERNAME=root install missed the real admin entirely and update_user_password (an upsert) INSERTed a second role='admin' account for the caller. The residual window is now exactly the installs the wizard exists for: no admin, no way in, wizard is the only door — see First-Run Provisioning

Scheduling & Execution:

schedules.py - Schedule CRUD and control
executions.py - Fleet execution list/stats (EXEC-022)
analytics.py - Agent-scoped Overview analytics (#1107)
compatibility.py - Agent compatibility validation: report + auto-fix (#668) — see Agent Compatibility Validation

Organization & Tags:

tags.py - Agent tagging
system_views.py - Saved system views
systems.py - System manifest deployment; also the read-only bundled-manifest catalog GET /manifests + /manifests/{id} backing the ent#126 UI install surface — both require_role("creator") and both declared above the parameterized routes (Invariant #4 twice: /{system_name} and /{system_name}/manifest)

Monitoring & Operations:

monitoring.py - Fleet health monitoring (MON-001)
telemetry.py - Host telemetry (CPU/memory/disk)
activities.py - Activity timeline
agent_dashboard.py - Agent-defined dashboard (dashboard.yaml). /{agent_name}/exists answers BOTH tab gates in one DB-only request — {has_dashboard, has_declared_metrics} (ent#479) — and is AuthorizedAgentByName-gated, which it was not: a bare get_current_user made it a fleet-wide existence oracle for any logged-in principal. The path parameter is spelled agent_name because that is the name the dependency's Path(...) binds; a route spelling it {name} leaves the dependency nothing to bind and answers 422 to every caller including the owner
alerts.py - Cost threshold alerts
notifications.py - Agent notifications
reports.py - Agent-published structured reports (#918) — see Agent Reports
evaluations.py - Behavioral-evaluation referee surface (ent#206): write is human-admin-only (require_admin + reject_agent_principal), read is access-scoped — see Agent Evaluations
operator_queue.py - Operating Room queue (OPS-001); its list/stats leave mention rows out (ent#631)
queue_mentions.py - Person tags on the queue-item ledger (trinity-enterprise#631): create (idempotent on (agent_name, request_id)), list for an addressee, read-CAS, one conversation's tags by id prefix
ops.py - Operating Room sync routes; the fleet-restart handler moved to services/fleet_ops_service.py (#1028)
logs.py - Container log endpoints
audit.py - Platform audit log (SEC-001)

Public Access & Monetization:

public_links.py - Public agent link management
public.py - Public chat routes; the turn orchestration moved to services/public_chat_service.py, which also owns the per-IP/per-token chat caps and the #311 access-gate predicates (#1028). Also Caddy's unauthenticated on-demand-TLS ask gate, GET /api/public/tls-allowed (#2380)
paid.py - x402 payment-gated chat (NVM-001). The 402/403/verify/settle ORCHESTRATION moved to services/paid_turn_service.py (ent#679) so the A2A inbound door runs the identical money logic; this router is now the HTTP shape over it (header read, outcome → response map) and its behaviour is unchanged. #3215: both the 402 and GET /{name}/info resolve the plan's scheme first, and the origin for resource.url comes from utils/public_url.py rather than request.base_url — a token is minted and verified against that URL, and behind the standard proxy topology request.base_url is http. /info therefore loads the config WITH the key (resolving a scheme needs a Payments instance) and emits an absolute URL; the key never reaches the body
nevermined.py - Nevermined payment config (NVM-001)
slack.py - Slack integration: OAuth, events, multi-agent channel routing, per-agent binding (SLACK-001/002)
telegram.py - Telegram bot integration: webhook receiver, bot binding, group config (TELEGRAM-001)
whatsapp.py - WhatsApp via Twilio: webhook receiver, binding CRUD + test (WHATSAPP-001)
voip.py - VoIP telephony: binding CRUD, outbound call trigger, Media Streams WS entrypoint — see VoIP
webhooks.py - Public webhook trigger endpoint + JWT-auth webhook management (WEBHOOK-001)
messages.py - Proactive agent-to-user messaging (#321)
public_memory.py - Per-user memory write endpoint for channel sessions (MEM-001, #888); since ent#637 also for a scheduled run addressed to a seat (services/schedule_seat_memory.seat_for_execution reads the ent#498 stamp), and the ONE writer of agent_notes — every write is recorded in public_user_memory_writes for the Workspace's see/undo surface
seat_decisions.py - The companion's half of the seat decision record (trinity-enterprise#638, R25): record / list for the seat an execution serves — the MEM-001 seat rule (schedule_seat_memory.seat_for_execution → verified channel email), Idempotency-Key through idempotency_service, decided_by_role from assignment_provider.resolve_assignment (never a container read), and an email-free answer shape. Policy in services/seat_decision_service.py; the person's half is the Workspace route

Subscriptions & Skills:

subscriptions.py - Subscription management (SUB-002). #2572: POST now also runs the credential-less adoption sweep (subscription_service.adopt_for_credentialless_agents) beside the #1089 rollover fan-out, in its own swallow-everything try/except — the sweep must never fail a registration whose credential is already stored, which is why the body param is payload and the injected one http_request (two things called request is how a log line inside an except raises AttributeError into a 500). The manual assign/clear routes now write the subscription_assign / subscription_clear audit rows this router had never carried (#2421 owns the remaining three actions); id and name only, never a token. #471 usage observability — extended GET /{id}/usage (failure counters + one-gate rate_limited_now + headroom block, source: anthropic|observed), GET /{id}/usage/breakdown (per-agent, cost-ranked), POST /{id}/usage/refresh (click probe), GET/PUT /settings/headroom-auto-refresh — all assert_admin; ent#433 adds GET /{id}/headroom/history?window=24h|7d|30d (bounded last-per-bucket series, bucket_start + real fetched_at, coverage_pct; 422 on an unknown window, id-OR-name + 404 parity with /usage; read-only, never probes); ent#434 adds PUT /settings/headroom-alert-threshold (0 disables — the operator_queue_retention_days idiom — else 50–99 with a named 422; the escalation tier is DERIVED max(threshold, 90), never a second knob, because two settable thresholds are an oscillator and validate_ops_setting is per-key so it cannot express the cross-field invariant) and extends the auto-refresh GET with a weekly_alert block carrying active + an inactive_reason of no_subscriptions / threshold_disabled / auto_refresh_off / redis_unavailable / count_unavailable, so "no alerts" is distinguishable from "not checking" (#2217) — an unreadable subscription list is its OWN reason rather than falling through to active: true, which it did while None == 0 decided the first arm. Rendered by the threshold control in SubscriptionsPanel.vue, whose decidable rules live in utils/headroomAlertSettings.js (vitest runs environment: 'node' with no mount harness, so a rule inside the SFC is untestable — the ent#392 precedent). The key is 422-blocked on the generic PUT /api/settings/{key}
skills.py - Skill CRUD and assignment. #2703: assignment DELIVERS — POST/PUT call skill_service.deliver_assigned (the start-path injection, all assigned + force=False; a subset would rewrite CLAUDE.md's section) and return a delivery block (injected|partial|pending_start|in_progress|not_delivered+reason) on a 200 the row never rolls back; bounded by SKILL_DELIVERY_BUDGET_SECONDS with the work kept alive past it. ent#672: delivery.skills[name].warnings names a deprecated skill on every outcome (read from the library entry, so a stopped agent is told too); GET /skills/library carries deprecated / superseded_by. All four routes fire the thin agent_skills_changed WS trigger (skill_service.broadcast_skills_changed, identifiers only — #918 rule) — see skill-injection.md
skill_gate.py - The in-container skill-gate hook's check, POST /api/skill-gate/check (trinity-enterprise#752): the agent from its key (get_self_agent), every verdict a 200; and the per-agent gate map, /api/agents/{agent_name}/skill-gates (agent_router, trinity-enterprise#753) — see skill-gate.md
settings/ (package, #1028 — twelve domain modules, provider_keys carrying the #2715 Resend/Gemini routes split out of credentials, aauth carrying ent#862's GET/PUT /aauth — the AAuth switches + issuer, admin AND require_interactive on write because turning it on publishes the instance key and accepts signed callers; the generic catch-all refuses the three keys on PUT and DELETE; generic.py's /{key} catch-all is included LAST, Invariant #4) - Platform admin settings (incl. Slack transport connect/disconnect/install). #2572: both Anthropic-key clear paths fire the credential-less adoption sweep, each gated on the route's existing deleted truthiness — the dedicated DELETE /api-keys/anthropic and the generic DELETE /{key} keyed on the two key aliases, because db.delete_setting has no delete-side twin of ent#435's db.set_setting sink guard and so reaches the key without ever touching clear_secret_setting. The hook sits on the ROUTES, not that leaf, which also serves github_pat and the Slack keys. Hosts the Tier-2 usage-sharing family (ent#12/ent#437): GET /telemetry-sharing (admin + human-only; ?preview=0 skips the aggregate build), PUT /telemetry-sharing (consent; mints/discards the share id), POST /telemetry-sharing/ask/dismiss (the once-per-install "don't ask again" marker) — see Opt-in Instance Telemetry

Content & Files:

image_generation.py - Image generation REST endpoints (IMG-001)
avatar.py - Agent avatar generation and serving (AVATAR-001)
docs.py - Documentation endpoints

System:

system_agent.py - System agent management
sessions.py - Session endpoints (rows + turn API; the Agent Detail surface retired in ent#358) — see Resumable Turns

Services (services/) — 127 service modules (155 counting subpackages):

Core:

docker_service.py - Docker container management (single point of Docker interaction, Invariant #11). Also owns the tri-state container-state pair agent_container_states() (batch, one sparse=True containers.list()) / agent_container_state(name) (single) added by #2196: None means Docker could not be asked, distinct from {} / "missing" meaning Docker answered: no container. Every older helper collapses those two into one falsy value, which is why a naive "drop agents with no container" reconcile empties a customer's roster on any Docker fault. sparse=True is load-bearing and constrains the code: the SDK default full-inspects every container, and under sparse .name returns None while .labels raises — so the key comes from attrs["Names"][0] and the status is classified explicitly to running/stopped rather than reusing list_all_agents_fast's raw-status fall-through (which would pass paused/removing through to a consumer's Literal). list_all_agents_fast's []-on-fault contract is unchanged. ent#403: agent_container_runtimes() is the batch sibling of get_agent_runtime(name) — one sparse list for the whole fleet, so the Workspace roster's model control does not pay an inspect per agent. Under sparse=True docker-py's .labels raises, so it reads attrs["Labels"], the key the /containers/json summary carries; tri-state like the pair above, and its consumer fails OPEN to claude-code. #2215: get_next_available_port(exclude=None) is atomic-with-Redis-up — a per-port SETNX reservation (port_alloc:{port}, TTL 600s, no release: the container's trinity.ssh-port label becomes the durable truth) is the LAST allocation gate, and the label scan is strict (a Docker listing fault raises instead of degrading to the empty set, which would allocate 2222 over an existing fleet; docker_client is None demo mode keeps the empty set) — deliberately the OPPOSITE resolution to #2196's tri-state readers above, because a roster read must degrade to "unchanged" while an allocator must never degrade to "allocate over the fleet". Redis down ⇒ fail-open to today's racy allocation, converged by crud's bounded bind-conflict retry (D2) — never build on "atomic, unconditionally". reserve_port_for_recreate (SET no-NX, fail-open) re-asserts a recreating agent's own port across recreate_container_with_updated_config's remove→create gap. is_port_available is netns-blind in production (documented weak filter)
docker_utils.py - Docker utility helpers
template_service.py - GitHub template cloning and processing; owns the tolerant credentials: readers (credential_shape_errors / credential_mcp_server_names / credential_env_file_names / credential_mcp_env_vars / declared_credential_names) — read paths degrade + surface credential_errors, the .env writer raises CredentialDeclarationError → 400 (ent#128) — see template-processing.md. ent#89: both builders also surface the declared schedules: (normalized) + schedule_errors via services/template_schedules.py, the same tolerant-reader shape; both GitHub catalog list paths are now fenced per-template (_safe_build_github_template) — they were bare list comprehensions, so a raise in the untrusted GitHub builder 500'd the whole GitHub half of GET /api/templates (ent#128 PR-A fenced only the local path). fetch_template_metadata_for_create(repo, pat, ref) is the creation-path metadata read: resolved PAT + parsed ref, cache-bypassed, loud on any failure — the catalog cache uses the global platform PAT off the default branch and would silently return {} for a private repo read with a per-user token. ent#14: the GitHub half of the catalog resolves admin override → remote registry (template_registry_service) → DEFAULT_GITHUB_TEMPLATE_REPOS in both get_all_templates() and get_github_template() (the second feeds GET /api/templates/{id} and agent creation, so one ladder or the display fields diverge by surface); registry entries arrive as admin_override dicts and are fenced so any failure degrades to the bundled floor. The per-repo fetch reason is now cached beside the metadata and surfaced as metadata_unavailable, because "declares no template.yaml" and "could not read it" were the same value — and fork_to_own: required silently read as absent under a rate limit. fetch_template_metadata_result_for_create is the reason-preserving form of the creation read (the plain wrapper drops the reason, which is fine for schedules and wrong for a security gate): the fork_to_own gate decides from IT, not from gh_template, because the catalog read answers 404 for a private repo the platform PAT cannot see and would classify a required template as absent
"credential declaration standard" (ent#128): credentials: is FROZEN names-only; the sibling credential_setup: enriches it per variable and normalize_credential_requirements(data, *, source_trust) emits base-set-plus-overlay records as credential_requirements (never raises, never mutates — _build_template runs unfenced in get_all_templates()). Contract: docs/schemas/trinity-agent-credentials.schema.json
template_registry_service.py - Remote template registry (TMPL-002, ent#14): byte-capped streaming fetch (the ceiling counts WIRE bytes via iter_raw() — Content-Length is absent on chunked responses and trivially lied about, and iter_bytes() yields decoded chunks, so with httpx's default Accept-Encoding: gzip, deflate a legal-looking 199 KiB body inflated ~1030:1 to a 458 MB event-loop-thread allocation before the running total was consulted; any Content-Encoding is now refused before the body is read) + follow_redirects=False → utils/safe_yaml.load_template_registry_yaml (AliasPolicy.REJECT, the ent#314 rule pinned at the utils/ layer) → allowlisted four-field parse → own TTL cache (3600 s + jitter, deliberately unaligned with the 600 s per-repo TTL to avoid a correlated herd; 7-day serve-stale cap; 60 s negative cache; cross-worker generation counter because a per-process invalidate half-applies under --workers 2; durable last-known-good as sanitized parsed JSON). Never raises — every failure returns []. See platform-settings.md
template_schedules.py - Tolerant schedules: reader (ent#89): schedule_shape_errors / normalize_declared_schedules over one private _parse, consumed by both template_service builders, the crud materializer, and compatibility check T-018. Total by contract — a raise would empty the catalog, enter the creation rollback fence, or fail-open T-018. Bounds (MAX_DECLARED_SCHEDULES=20, name/description/message limits), intra-block name dedupe, and strict cron/timezone via schedule_validation.validate_cron_expression. Stdlib + schedule_validation only — template_service imports it, so it must not import back. See template-processing.md
template_metrics.py - Tolerant metrics: reader (ent#477): metric_shape_errors / normalize_declared_metrics over one private _parse, plus parse_cadence and definition_hash. Third instance of the ent#89 / #1704 leaf idiom, consumed by metric_registry, the crud resolvers and compatibility check D-009. Total by contract — a raise would enter the creation rollback fence or fail-open D-009. An entry with ANY error is dropped AND named, never partially materialized: ent#478 validates recorded points against the registry rows, so a row assembled from the half that parsed would accept points its author never declared. Bounds (50 metrics, 50 status values, 10 dimensions, 20 x- keys / 1 KB; label/description/unit limits), ^[a-z][a-z0-9_]{0,63}$ names (a case variant is invalid, never a second series), closed enums for type/direction/aggregation, and ONE cadence parser (<n>(s|m|h|d|w) or ISO 8601 PnW/P[nD][T[nH][nM][nS]], 60s–366d, years/months rejected as non-fixed durations) normalizing to cadence_seconds — ent#479's stale rule reads that column and must never re-parse. Errors echo indices, key names, type names and closed-enum values ONLY (the ent#89 _safe_echo rule — the list is persisted into checks_json and rendered). Stdlib only, so it cannot import template_service (which imports the sibling leaves) — hence the triplicated _type_name/_safe_echo/_did_you_mean, registered debt
metric_points_service.py - The point VALIDATOR (ent#478): a pure leaf over the ent#477 definitions — no DB, no settings, no HTTP, so every reason code is testable without a database and Invariant #1's middle layer stays free of both SQL and HTTP. Points validate against the STORED type, never type_conflict (a refused type change is informational: prior points under that name were interpreted as the stored type). Imports {services.template_metrics, utils.helpers} only — it reuses the declaration reader's constants so the point rules and the declaration rules agree by construction rather than by review. ts is gated on the RFC 3339 SHAPE before fromisoformat sees it (which accepts week dates and year 0999, and 0999 normalises to 999-01-01T… — lexicographically ABOVE every real timestamp, so one such row would head the ts DESC read index forever) and range-checked as a datetime for the same three-digit-year reason. Messages echo codes, indices and dimension KEYS only — never a value, never a dimension value, which may be a customer name or an email; this output lands in an LLM's context
metric_read_service.py - The metric READ (ent#479) and the platform's ONE staleness rule. freshness(cadence_seconds, last_point_at, now) is pure, clock-injected and exported — the read route, the get_agent_health block, the dashboard binding import it and the Workspace role card reaches it through the objective join (ent#676), because a second implementation is a second rule the first time either is edited. stale ⟺ cadence declared AND now − last_point_at > 2 × cadence, strict > (exactly 2× is not stale), and no cadence → stale: null, never false: "not stale" and "unanswerable" are different claims. Composition (read_agent_metrics) is store-only — definitions + newest points + freshness + bucketed series — so a STOPPED agent answers identically; database.db is resolved at CALL time so freshness stays importable without a store. Bucketed by default and raw only on the single-metric= path, because 50 metrics × 2000 points is a ten-megabyte "read". bind_dashboard_widgets fills a metric:-carrying widget from the registry and must write history in the SAME {values, trend, …} shape _enrich_widgets_with_history writes — DashboardPanel.vue reads history.values, so a bare list silently loses the sparkline with every backend assertion green. is_bound is the one spelling the snapshot writer and the history enrichment both skip on: a bound number in agent_dashboard_values would be a second source for a value the registry owns. latest_by_metric (ent#666) is the cheap half — {name: tile} with no series, bucket or stats work — and both it and read_agent_metrics fold through ONE _latest_entry, so the objective join's actual and the tile's number are the same computation rather than two that happen to agree (pinned by a parity test on a dimensioned sum metric, the case a re-implemented fold would break); it takes definitions=/names= so the join pays one registry read for the names an objective references, and names=[] performs no store read at all. A widget's dims: selector (ent#730) is resolved inside bind_dashboard_widgets only: parse_dims_selector (public, pure, total) validates it with the write path's metric_points_service.validate_dims and is shared with compat X-009, so "is this selector valid" has one answer on both surfaces; the match is by canonical_dims against the selected series' OWN bounded store read (db.metric_series_points_for_dims: its newest points inside the tile's window, else its single newest point back to SELECTED_SERIES_LOOKBACK_HOURS), never against the metric's shared 200-point read, which a busy series can fill on its own (#3293), and the widget is filled through the SAME single fill block as the fold ("pick a source, fill once"), so value, colour and history cannot drift between the two paths. It never changes _latest_entry, whose fold the objective join and the role card share, and a failed selector is one of three named refusals that never falls back to the fold. _threshold_verdict is the one threshold rule behind both color and the typed threshold_verdict; a truthy non-text metric: (number, boolean, list or mapping) is refused per widget (metric_name_invalid); before ent#730 a list or mapping raised out of the loop, which runs outside the store try, and a number or boolean read metric_undeclared; a falsy metric: (0, false, [], {}) stays unbound, as before
metric_access_service.py - Who may read another agent's declared metrics (ent#727): can_read_agent_metrics(reader, target) — the ONE predicate, self or an agent_permissions grant (the chat_with_agent edge); ent#80 replaces its body, never adds a second gate — and audit_cross_agent_read, one authorization row per (reader, target, route, actor) per hour, marker set only after the row is written.
objective_join_service.py - The objective ↔ metric join (ent#666) — the ONE place a target and an actual are put side by side, so the role card (ent#527), the project view (ent#661 v3) and proactivity (ent#605) consume it instead of each deriving a gap. Two halves in one module: the PURE half (canon_root, parse_objective, objective_concerns, objective_is_active, select_objectives, resolve_direction, gap, join_objectives) imports at module top with no store or Docker behind it, and the async half (read_objective_files, read_objective_join) goes through the agent door, because objective files live in the agent's own canon and nothing is copied platform-side (E7/E13) — a stopped agent answers unavailable: agent_stopped, never a cached number. gap.status is position, never pace: behind says the number is on the wrong side of the target now, not that the agent is late against by; by and horizon ride every row so a consumer can judge pace itself, and hold has its own arm (on_target within tolerance, else off_target) because a value that should be held has no good side to be on. Direction is the registry's first, falling through to the objective file's up/down/hold when the registry says neutral — that is the column DEFAULT, indistinguishable from silence, and no bundled template declares one, so registry-only would ship a feature where every gap is not_computable on day one; direction_source is what tells a declared hold apart from a default. Stale is orthogonal to gap (a stale metric keeps its gap and carries stale: true) and every failure is a NAMED finding with the fix attached — except a supporting-only agent's undeclared metric, which gets metric_not_declared_here rather than "declare it and refresh", advice it cannot take (the registry is per-agent; cross-agent reads are ent#80). The fan-out is deliberately narrow: ≤ 2 reads in flight, aborting to agent_unreachable on the first typed transport failure, because AgentClient's circuit trips at three and it is the breaker chat rides on — and it does its own transport rather than AgentClient.read_file, which flattens every AgentClientError into {"success": False} and so cannot tell "this file is unreadable" from "this agent is gone". Scans 100 files, filters, then caps the OUTPUT at 20: capping the listing first hides an agent's own objective behind twenty foreign ones in a shared fleet canon
objectives_read_budget.py - The per-agent budget on the objective fan-out, spelled once (ent#676): enforce (429) for GET /api/agents/{name}/objectives, admit (never raises) for the Workspace role card — two doors, one bucket. Detail: feature-flows/workspace-role-card.md.
canon_roles_service.py - read_canon_roles(agent_name) lists <canon>/roles/*.yaml with each file's id, title and updated stamp through the objective join's agent door (trinity-enterprise#817). Never raises and never blank: a stopped/missing/unreachable agent is unavailable, and no canon / no roles/ / unlisted / timeout is a named reason. Backs GET /api/agents/{name}/canon/roles; the gated assignments section reads it for its seat picker and for drift checked on read (a row's recorded stamp vs the file's current one).
metric_registry.py - Declared-metric reconcile service (ent#477). Two entry points: reconcile_metric_definitions(name, template_dict, source=…) for the creation path (the resolver already parsed the template — re-reading would be a second source that can disagree) and refresh_from_running_agent(name, source=…) for every live trigger, plus spawn_refresh_from_running_agent (the fire-and-forget container-start hook, spawn_gitignore_merge_after_clone's shape). The live read is docker_service.execute_command_in_container with a fixed argv (head -c 256K /home/developer/template.yaml, bounded by the primitive's enforced timeout, #2969) — the compatibility collector's own door, capped in-container so an oversized file never crosses the socket — then the backend's hardened loader, i.e. the SAME parse creation used. /api/template-info is not a door for this (it does not return metrics) and neither is /api/metrics (the superseded path). Never retires on absence of evidence (#2196): exec failure, empty output, oversize, unparseable YAML and a stopped container all raise RefreshUnavailable with a named reason and leave the registry untouched — only a template that PARSED and carries no block retires rows. Imports {template_metrics, database, docker_service, utils.safe_yaml}; never services.agent_service (#1991)
agent_client/ (package, #1028 — circuit / http_pool / client) - HTTP client for agent container communication (chat, session, injection); hosts the transport circuit breaker — see Circuit Breakers
settings_service.py - Centralized settings retrieval (API keys, ops config, agent quotas). ent#582 adds the per-call resolvers for the first-run keys: get_gemini_api_key (encrypted google_api_key → config.GEMINI_API_KEY, i.e. GEMINI_API_KEY → GOOGLE_API_KEY), get_resend_api_key, get_email_provider (a Resend key saved in Settings selects Resend over EMAIL_PROVIDER) and get/set/clear_email_from_address (email_from_address → SMTP_FROM). Every platform Gemini consumer (voice, portal voice, VoIP, Brain Orb voice, Telegram transcription, image/avatar generation, feature flags, /api/version) calls the resolver instead of the import-frozen config.GEMINI_API_KEY
a2a_gate.py - Open-core seam for the A2A inbound allow-list: OSS registers no provider → any authenticated owner/shared caller is allowed; a private module can register one to further restrict caller identities. Fails open (a provider error never blocks an authenticated caller), so it is a restriction layered on auth, not a security boundary. A seam file — its comments describe the mechanism only and are grepped by enterprise-docs-guard.yml (#1461 class) (ent#157)
login_policy_gate.py - Open-core seam for email-code sign-in policy (ent#849): OSS registers no provider → emailed codes work as before; a registered provider can turn them off. email_code_allowed() answers platform sign-in (web /api/auth/email/*, MCP inline auth); email_code_allowed_for(email) answers surfaces that also serve outsiders (Workspace guest sign-in, Telegram/WhatsApp /login, Slack require_email, public-link email verification) and refuses only platform users and whitelisted addresses. While codes are refused those request steps mint and send in a detached task for every address, and the public-link per-address 429 counts every request, so neither reply, status nor timing reveals membership. Fails open on a provider or lookup error. /token never consults it (admin break-glass). Every function that calls create_login_code/verify_login_code/create_verification/verify_code/send_verification_code must ask it (or is exempted with a reason) — tests/unit/test_ent849_email_code_policy.py fails per function on a new caller that does not. A seam file — grepped by enterprise-docs-guard.yml
a2a_outbound.py - Open-core seam for the OUTBOUND A2A target registry, and the deliberate inverse of a2a_gate: it FAILS CLOSED — no provider, a provider that raises, or a provider returning a malformed object all refuse the call, because this seam decides where a credential is sent whereas a2a_gate only restricts an already-authenticated caller. The isinstance(ResolvedEndpoint) check on the return value is load-bearing rather than defensive: under a stubbed sys.modules a MagicMock module returns a truthy endpoint with a mock .url, silently inverting fail-closed inside the suite that proves it closed. Ships the OSS provider — admin-managed named endpoints in system_settings as one AES-256-GCM envelope (Invariant #12's elevenlabs_api_key_encrypted shape), so OSS is functional with no new table, no migration, no Alembic revision. Shipping a working source rather than only the seam is deliberate: a seam with no registered provider resolves nothing, so the tool would answer "no targets configured" on every install. A private per-agent provider may register and take precedence (#736). Each record's credential carries a kind (api_key default — i.e. every pre-#3185 record — or payment_token), a LABEL on the one slot rather than a second secret: it rides that slot's existing three write paths, is inferred from the value when omitted using the SAME predicate the client sends on (a2a_protocol.decode_payment_token, so the label can never disagree with the transport), and api_key is persisted as the key's ABSENCE so a relabel leaves a pre-#3185-shaped record (#3185)
ask_service.py - The ask sink (trinity-enterprise#611): the one path that raises an ask (raise_ask, the native create behind ask_operator; the gate path calls it with raised_by="gate"); a gate raise may pass addressee= to reach exactly one person, bypassing role resolution — an agent's raise that names one is a programming error, so an agent never chooses who is asked (ent#661) and the one path that ends it (answer / cancel / bulk_cancel / expire: CAS writer → audit → thin trigger → ending observers); since #3247 raise_ask(replaces=) also ends one of the agent's OWN pending asks (disposed_by='agent', reason replaced) in the same locked transaction as the create (_replace_target, _refuse_already_pending, request_id_of). See operating-room.md → Endings / Raising an ask.
portal_capabilities.py - Open-core seam for per-person Workspace capabilities that depend on data the core does not hold (whether an email was invited to something a module owns): a provider registry keyed by capability name, asked by the roster as has(name, email). OSS registers nothing → False; fails closed (a raising provider, or any answer but True, is False) because a bit that errs toward True advertises routes that would refuse the caller. A seam file — grepped by enterprise-docs-guard.yml (ent#661)
turn_context.py - Open-core seam for lines added to a Workspace turn: a list-based provider registry (register_provider / collect(TurnContext)) that both composers call — the 1:1 chat (portal_chat, on BOTH the resumed and the cold arm, beside the open-canvas line) and the room wake (_wake_agent, ahead of the file manifest). OSS registers one provider — the discussed-ask line (client_portal/asks/service.py::_register_turn_context, registered at import; trinity-enterprise#747), which answers only for a chat that is an ask's discussion chat; every other turn gets "" and composes byte-identically. A private module may register its own. The context is the PLATFORM's: chat_id is the resolved chat row or room, and internal_audience is the caller's is_platform for a chat and not room_is_user_facing(...) for a room, so an internal-only line never reaches a turn an outside client can read. Fails open: a provider that raises is skipped. A seam file — its comments describe the mechanism only and are grepped by enterprise-docs-guard.yml (#1461 class) (ent#661)
assignment_provider.py - Open-core seam for role assignments — which human fills which business role for an agent, and which of them the agent primarily serves. OSS registers no provider → resolve_assignment(agent_name, triggered_by) returns None and the execution-context block renders byte-identically to a build without the seam; a registered module answers with a display name, a role id, the stakeholder list, and whether proactive contact is on file. Deliberately SYNC: compose_system_prompt is a plain def called from async handlers, so a provider that blocks on I/O here stalls the worker — providers answer from memory, never over HTTP. The seam owns the failure handling rather than delegating it (the mfa_gate position, not a2a_outbound's): compose_system_prompt has NO exception handler, so an escape costs all three of its callers the execution-context block and costs one of them the platform prompt entirely. Shape validation sits beside the try because try/except cannot see the defect it catches — a str where a list was promised iterates into single characters and renders the WRONG prompt without raising, which is harder to notice than a missing line. triggered_by is passed through so a provider can suppress the answer for an audience that must not see staff identities; the seam carries the label and never decides that policy. A seam file — its comments describe the mechanism only and are grepped by enterprise-docs-guard.yml (#1461 class) (trinity-enterprise#500) A provider may also answer people_for(agent_name, role) (optional; trinity-enterprise#611): resolve_role_people returns the people who fill a role for the ask sink's to addressing, and None on any failure, so the core defaults hold. Two trigger-free resolvers (trinity-enterprise#811/#814) read the provider's optional seat_for / served_seat_for through getattr and ALWAYS return a dict, never raise: resolve_seat(agent) answers the seat the agent holds itself or serves through its primary ({case: holds|serves|none, role_id, reason}) for the readiness gate (ent#813), the objective join (ent#812) and the role card; resolve_served_seat(agent, email) answers the seat a run serves for one person — holds wins, the person's own seat next, the primary's last — for the prompt's Seat this run serves line and the seat-decision stamp. Every degraded case reads as no seat with a named reason.
role_addressing.py - The ONE rule that turns an outbound object's to: role (primary | approver | viewer | operator) into people (trinity-enterprise#606): a provider's people_for answer first, else primary → owner, operator → nobody, approver/viewer refused (RoleRefused, named codes). Shared by asks (ask_service._address — a role several people fill delivers the one ask to each of them, trinity-enterprise#816), reports (routers/reports._report_audience) and messages (routers/messages._recipient); agent-supplied emails are still honoured and logged as deprecated (log_email_addressing).
a2a_outbound_service.py - Outbound orchestration: kill switch → rate bounds (per-agent and a fleet Redis key — a per-agent limit bounds one agent, the fleet is the exhaustion path) → resolve → validate → effect_guard → call → agent_activities row. Ordering is load-bearing: bounds before resolution so a flood can't become a DNS amplifier, and validation before the guard so a refused URL never burns an effect claim (#736)
a2a_client.py - The protocol client, FastAPI-free (raises A2ACallError, mapped 1:1 at the router). One resolution → one validated IP → pinned for both hops, with the registered hostname carried for Host + SNI + cert verification, so DNS rebinding is closed rather than accepted as a residual (the template-registry validator can accept it; this request carries a credential). trust_env=False — every other control reasons about the target IP and a proxy makes the target irrelevant — with the CA context rebuilt explicitly, since that flag also disables httpx's SSL_CERT_FILE/SSL_CERT_DIR. Wire-byte ceilings over aiter_raw(); any Content-Encoding refused, not decoded; a 3xx is a failure on both hops; a wall-clock deadline over cancellable awaits, because httpx's read timeout is per-read and a trickling tarpit resets it forever. The card is a hint: its url must be same-origin (default-port-equivalent — Trinity's own card emits no port) and securitySchemes never selects the credential, which is why #736 ships while ent#159 is blocked. Errors ride the body on HTTP 200, so the body is parsed for error even on 200 (#736). 402/403 on the RPC hop are classified BEFORE the encoding and length guards (#3185): a CDN-gzipped or oversized "pay me" previously reported rpc_encoding/rpc_too_large — an outage, for an endpoint working perfectly — so the header is read first and the body only when identity-encoded and under a ceiling 16× tighter than the answer cap, with the outcome surviving from the status alone when it cannot be read. The in-band x402.payment.status check runs before _parse_task on both the send and the poll path, or a priced input-required reaches the agent as a prompt it polls forever; the payment block handed back is allowlisted, leaf-capped and scrubbed against the token and its base64 forms and the decoded payload's long leaves
aauth/ (package, ent#623; Flows 1–4 behind aauth_enabled since ent#862, Flows 5–6 additionally behind aauth_prototype_enabled, both OFF) - AAuth agent-identity-only mode, self-hosted bootstrap: config (the two switches + issuer: active_issuer() for Flows 1–4, prototype_active_issuer() read only at require_live), settings (the admin read/write rules behind /api/settings/aauth: sources, live, diff-only writes, invalid_issuer, the readiness checklist), jose (Ed25519 JWS — PyJWT only knows EdDSA), httpsig (RFC 9421 profile + strict RFC 8941 subset), keys (instance signing key, AES-256-GCM in system_settings, insert-if-absent), signer (per-agent aa-agent+jwt + request signing), discovery (issuer metadata/JWKS through the pinned a2a_client egress, same-origin jwks_uri, 8 s deadline), verifier (FastAPI-free; trusted-issuer pre-gate before any fetch; replay fail-closed), inbound (audit/activity attribution + identity-scoped tasks/*), whoami (operator CLI); ent#878 adds person_server (PS metadata: four required fields, token endpoints same-origin, jwks_uri may be off-host, 60 s negative cache), requirement (the strict AAuth-Requirement / Retry-After grammar), egress (every status with headers + capped body under the a2a_client guards), person_token (the binding state machine: worker-memory held token stamped with the key thumbprint, Redis pending records aauth:pt:*, fail-closed state_unavailable), and makes signer's per-agent key HKDF-derived from the instance seed per hour window — both workers hold the same cnf, and a cache hit honours min_remaining against the window end so a whole challenge chain signs with one key. ent#879 adds resource_allowlist (the SINK between an LLM-supplied URL and a signed request: aauth_resource_allowlist, origin equality, EMPTY on any unreadable state), resource_meta (resource metadata + control paths + tolerant AAuth-Budget, cached per (issuer, document)), resource_token (the aa-resource+jwt verification sink — one red test per check), auth_token (inspection-only aa-auth+jwt, T5), ps_client (the auth-token exchange, the Redis auth-token cache aauth:at:* with TTL exp − 300, pending records in ent#878's handle namespace with kinds ps_interaction / ps_approval / resource_polling), resource_client (the bounded challenge ladder: ≤ 6 resource sends, ≤ 3 PS exchanges, one 25 s budget, no inline polling, every held token scrubbed from bodies and errors, resume re-sends the STORED request) and resource_call (the route's seam: the ladder under effect_guard("aauth_resource") — only result/denied complete, every other outcome releases — plus the audit + activity rows). a2a_gate.explicit_inbound_identities is its fail-closed allow-list accessor
a2a_protocol.py - Shared JSON-RPC/A2A vocabulary — error codes, the dialect table, envelope helpers — imported by both routers/a2a.py (inbound) and a2a_client.py (outbound) so the two cannot drift. Dialect defaults to v0.3 (an absent version is the spec's back-compat rule); 1.x is defined and deliberately refused, since no peer exists to verify it against. "Target v1.0 only" was rejected on evidence: Trinity's own card pins 0.3.0 and its server dispatches slash names, so a v1.0-only client cannot talk to Trinity and #738 federation would be dead on arrival (#736). Also owns the x402 payment vocabulary (#3185) — the dotted metadata keys, the status values, the two HTTP header names, and the stdlib base64/JSON token codec decode_payment_token, which lives here rather than in the client because the endpoint store reads it too (one answer to "is this credential a payment token?") and the inbound side will read the same names
operator_intake_service.py - Fire-and-forget, once-per-install opt-in operator intake POST at first-run; owns installation_id (trinity-enterprise#38). ent#463 adds a Settings-home surface (GET/PUT /api/settings/operator-intake) so an admin can opt in / opt out after first-run — necessary since #2385 stops rendering the welcome form on any install with a pre-provisioned admin. Both surfaces converge on submit_operator_intake and preserve the at-most-once marker (no second intake client, no forked payload); opt-out is a durable decline and does NOT roll back the marker. Two accessors over the id (ent#545): get_or_create_installation_id is the writers' accessor (the consent POST, the product-event emit, the #1987 canary label) and get_installation_id is the non-minting read for every GET-shaped path — a get_or_create_* on a read path is a durable write with a race (learnings 2026-08-05); the enterprise funnel read was the last such caller. The mint is a write-once claim through insert_setting_if_absent (a losing worker reads the winner's id back), closing the SELECT-then-upsert race #1987 had recorded as pre-existing
mcp_auth_service.py - #848 inline-login issue/verify + the per-call access gate behind routers/mcp_auth.py. Owns the enumeration-safety contract (#186): /request answers ONE constant 202 on every path — known, unknown, malformed, rate-limited, backend-threw — with no audit row, and does no branch-dependent work on the request path at all (the known-check, the cap read and the code INSERT all run in a Starlette BackgroundTasks task after the response flushes, because a committing write on only one branch measured ~1.9× even on in-process SQLite). _email_is_known fails closed, so a lookup error reads as "unknown" and mails nothing. Rate limits are account-scoped, never per-IP — client_ip is always the MCP server, so a per-IP bucket collapses every user into one and 30 wrong codes would lock inline login out fleet-wide (the #591 DoS); a global ceiling checked before the known-branch bounds the unknown side without becoming its own differential. assert_email_may_reach_agent is the load-bearing gate: the internal secret authenticates the caller, the asserted email's own standing authorizes the action
agent_mcp_key_service.py - Agent MCP-key detection / self-heal / rotation (#1854): the in-container digest probe + verdict interpretation, heal_agent_mcp_key_env for the start-time drift path, and the rotation orchestration (fail-closed lock, capture-before-mint, spawn-id reconcile before delivery, captured-id DELETE, no plaintext returned) — see agent-mcp-key.md
agent_runtime_state.py - Single enumeration point for every name-keyed per-agent Redis keyspace; clear_agent_breakers (safe on a live container) vs clear_agent_runtime_state (adds slots and the agent:skills_list: copy; teardown only) — see Circuit Breakers (#1560)
agent_skills_listing.py - The one proxy of an agent's GET /api/skills for the agent page, the public link and the MCP connector (the Workspace's own skill read in client_portal/service.py still calls the agent directly — a known gap), keeping each successful listing in Redis (agent:skills_list:{name}, no TTL, 256 KB cap) for a stopped agent's Skills tab; cleared on teardown and create, never on start (ent#754) — see skills-tab.md
instance_identity.py - get_instance_label(): the short label naming WHICH instance an outbound alert came from (TRINITY_INSTANCE_NAME → FRONTEND_URL host's first DNS label → installation_id[:8] → None). Stdlib-only leaf, never raises — consumed by the canary Slack sink today, reusable by any future webhook sink (#1987) — see Canary Harness

Execution & Scheduling:

task_execution_service.py - Unified task execution lifecycle: slot mgmt, activity tracking, sanitization (EXEC-024); #678 reader-race auto-retry + #792 SUB-003 switch-retry (pre-raise 429/auth interception → single retry with the same execution_id after a successful subscription switch, so one-shot triggers recover instead of FAILED; the two retries have different budgets — #678 keeps its flat _AUTO_RETRY_MAX_TIMEOUT_S 300s ceiling because it re-dispatches a turn that never started, while the SUB-003 retry re-runs the user's whole turn and therefore gets the turn's REMAINING budget (#2789: sharing the ceiling capped every 3600s agent at 300s and the agent server killed its own process group mid tool-use, discarding a billed turn). state.applied_timeout_seconds carries whichever ceiling was in force so a post-retry timeout is attributed against it rather than the untouched configured limit — _handle_timeout measures the retry, since start_time is reset before it; see task-execution-service.md); records dispatch-breaker outcomes (see Circuit Breakers); hosts apply_result + the 202 dispatch path (see Fire-and-Forget Dispatch). Single terminal applier — the chat split (#1483) delegates /task sync/async to execute_task/apply_result and never introduces a second applier. #1853: the FAILED branch of apply_result now mirrors the SUCCESS branch's telemetry, persisting execution_log (the sanitized stream-json transcript, via the same sanitize_execution_log), a tool_calls summary (#1741), and claude_session_id (UUID-validated agent-side) alongside the already-salvaged cost/context — so an error_during_execution (502) / timeout (504) FAILED row is diagnosable via the same API as a SUCCESS row, surviving agent stop+delete under the 30-day execution_log_retention_days window. The columns are added to the existing db.update_execution_status call above the if won gate — no new CAS writer, _write_terminal_and_gate untouched, and every side-effect stays gated on won (#1578/#1804). _extract_agent_error returns the transcript (3-tuple) that the agent's structured 502/504 body now carries (_execution_error_502_detail / _timeout_504_detail gained a validated session_id + execution_log). Residual: _write_terminal_and_gate terminals (backend timeout/budget/crash) and standalone-scheduler RETRY-001 FAILED writes still land bare. #2467: the SUCCESS branch also derives turn-integrity flags from the transcript at write time via services/execution_integrity.py — background tasks killed at CLI exit (keyed on the stream's kill lifecycle events, never the ledger, which drains before exit) + the #2127 waited-path pending count — into new nullable schedule_executions.turn_integrity (conditional kwarg, retry_count pattern) with a visible notice prepended to the stored response for kills; backend-side deliberately so it reaches every deployed agent image with no rebuild (the #1741 precedent); NULL ≡ "no evidence", never "verified healthy" — see parallel-headless-execution.md
chat_execution_service.py - Chat execution lifecycle behind routers/chat.py (#1483, Invariant #1): /chat setup + the transitional sync-chat run_chat_turn applier (the ONE divergent terminal writer — sync-chat does its own chat_sessions persistence + mode="chat" prompt execute_task doesn't; convergence onto execute_task is a tracked follow-up), plus the /task dispatch orchestration (derive/spoof, file upload, row+activity creation, async/sync fork delegating to execute_task) and terminate_execution. HTTP-free — raises ChatDispatchError, mapped 1:1 at the thin router
dispatch_admission_service.py - Request admission for both /chat and /task (#1483): idempotency begin/replay + audit, the /chat pure-state breaker read (#526 F1), and CapacityManager.acquire for /chat. HTTP-free — returns ChatAdmission/ChatAdmissionReplay, raises the already-domain CapacityFull/CircuitOpen/EphemeralBudgetExhausted
chat_persistence_service.py - Authenticated /task chat-session persistence (#1444): persist_chat_session (SUCCESS-guard + IDOR owner-check + fail-loud no-user-content ERROR log) + persist_and_broadcast_chat_session (chat_response_ready WS)
chat_signals.py - Dependency-free leaf: the chat domain signals (ChatAdmission/ChatExecutionContext NamedTuples, ChatAdmissionReplay, ChatDispatchError) the chat services return/raise and the router maps to HTTP (#1483)
execution_integrity.py - Pure leaf (#2467): derives turn-integrity flags (killed background tasks + waited-path pending) from a transcript's task lifecycle events for apply_result's terminal write; structural fields only, charset-validated, capped — see parallel-headless-execution.md
capacity_manager.py - Unified capacity facade for admit/release/status — see Capacity & Backlog
slot_service.py - Internal to CapacityManager: atomic N-ary capacity counter (Redis ZSET, dynamic per-agent TTL) (CAPACITY-001)
backlog_service.py - Internal to CapacityManager: persistent SQLite FIFO overflow store with drain-on-release (BACKLOG-001)
dispatch_breaker.py - Per-agent dispatch circuit breaker (RELIABILITY-007, #526) — see Circuit Breakers
scheduler_service.py - APScheduler-based scheduling
validation_service.py - Post-run business validation (VALIDATE-001): frames the parent run for a clean-context auditor and maps its verdict onto business_status; an empty response is decided platform-side as validation_unavailable before any validator runs, and a validator cannot award that status to itself (#2959) — see Business Validation
cleanup_service.py - Watchdog reconciliation + retention sweeps — see Soft Delete & Retention
idempotency_service.py - Trigger-boundary dedup (begin/complete/fail) (RELIABILITY-006, #525; Invariant #18); effect_guard per-execution effects (#1084); intent_guard caller-declared cross-execution keys on human-facing sends (trinity-enterprise#665)
rate_limiter.py - Shared sliding-window request limiter (#1023; see Container Security)

Real-time delivery:

event_bus.py - Redis Streams transport for WebSocket delivery — see Real-time Delivery
ws_ticket_service.py - Single-use WebSocket auth tickets (C-002, #550) — see Real-time Delivery
event_dispatch_service.py - EVT-001 subscription dispatch primitives (trigger_subscription, extracted from the router) + the shared system-emit helper emit_task_terminal_event/spawn_task_terminal_event — see Task Completion Events (#1578)

Monitoring & Activities:

activity_service.py - Activity tracking and timeline; owns close_execution_activity / spawn_close_execution_activity, the single closer every CAS-won terminal writer calls (#1804) — see Terminal-Activity Close Contract
monitoring_service.py - Fleet-wide health monitoring, 30s loop — authoritative for aggregate status; lifespan-resumed from persisted monitoring_config, default OFF (MON-001, #1121)
monitoring_alerts.py - Alert threshold configuration
heartbeat_service.py - Agent push-heartbeat liveness layer — see Heartbeat Liveness
platform_alerts.py - The platform-alert seam (#3246, OPS-001-PLATFORM-ALERTS, requirements §26.14). Stdlib-only leaf (DB imported lazily): the Kind registry (prefix, lifetime class, clear hook, material_keys), observe(agent, kind, key, …) → at most one pending row per subject <kind>:<key> updated in place (last_seen_at, context.seen_count), clear / reconcile → ended cancelled by the platform (condition_cleared), lifetime (OPERATOR_PLATFORM_ALERT_LIFETIME_DAYS, 14) and snooze (OPERATOR_PLATFORM_ALERT_SNOOZE_DAYS, 7) readers, and the pure plan_sweep the upgrade migration runs. Never raises into an emitter. Only four emitter families use it in PR 1 (headroom, skills adoption, system agent, circuit breaker); the rest are ratcheted in PREFIXES_NOT_YET_ON_SEAM until PR 2.
operator_queue_service.py - Operating Room sync with agent containers (OPS-001); the agent-authored ingestion boundary enforces the per-agent depth/rate/size caps + reserved-id guard + leader lock (#1632); platform-minted alerts (is_platform_minted) have no agent audience — never written to the agent file, and an operator answer lands them acknowledged (#2372)
person_mention_service.py - Tagging a person in a chat or a room (trinity-enterprise#631): who may tag (a platform person), who may be tagged (accounts that can reach the conversation's agents — the ent#450 shape), named refusals before any write, idempotent delivery of ONE operator_queue row (type mention, delivered → read, never pending) per (conversation, message, person), and the tagged person's door (can_see decides whether any content crosses). See workspace.md → Person tags.
operator_queue_choices.py - The one rule for what an approval's response may be (#2376, #3242): an offered option (exact match, else ResponseNotOfferedError with the list) or the reserved SOMETHING_ELSE = "(something else)" with an instruction in response_text (InstructionRequiredError / ReservedValueError / NotOffMenuError, all ReservedAnswerError); import-free leaf the ask sink calls before any write; also holds OPTIONS_DROPPED_MARKER. Mirrored in utils/operatorQueue.js + MCP types.ts (parity-tested)
sync_health_service.py - Git sync health polling — see Git Sync Health
sync_freeze_policy.py - The one sync-health rule (state / reason / recommendation / freeze), stdlib leaf, vendored byte-identically to src/scheduler/ (ent#706) — see Git Sync Health
sync_health_view.py - sync_view(row, config): the backend's single call into the policy, used by the poller and every sync-health read (ent#706)
canary_service.py - Orchestration-invariant watcher — see Canary Harness
compatibility/ - Agent compatibility validation package (spec/collector/static_checks/ai_checks/fixes) — see Agent Compatibility Validation

Auth & Credentials:

credential_requirements_service.py - Per-agent credential checklist (ent#127): a bounded docker exec probe for live .env key names joined against normalize_credential_requirements; degraded dominates the empty state — see guided-credential-setup.md
setup_url_display.py - UTS-46 nontransitional canonical host + eTLD+1 for an author-supplied setup_url, failing closed to inert text (ent#127). Leaf; zero deps on template_service
credential_encryption.py - AES-256-GCM encryption for .credentials.enc and DB-persisted tokens (CRED-002, Invariant #12). Supports online key rotation (#267): an optional decrypt-only CREDENTIAL_ENCRYPTION_KEY_SECONDARY (the previous key) keeps old-key ciphertext readable while new writes use the primary; rewrap() + scripts/deploy/rotate-credential-key.py re-encrypt persisted DB secrets onto the new key (runbook docs/migrations/CREDENTIAL_KEY_ROTATION.md)
runtime_secret_scrub.py - Runtime secret-scrub seam (ent#279): identity-based redaction of a producer-STAGED value out of persisted output, catching arbitrary values the pattern-based utils/credential_sanitizer (prefix regexes + KEY=value) cannot. A generic OSS mechanism — mechanism-only by design, names no consumer. Sync API: stage_secret(agent_name, value) (fail CLOSED — any failure, Redis down or hard cap, raises StagingUnavailable so a producer that cannot stage refuses delivery), get_staged_values(), and scrub_text/scrub_obj for the execution-terminal persistence chokepoints to redact staged values (marker ***REDACTED***; raw / once-JSON-escaped / base64 renditions; longest-first; falsy-passthrough) before the write. Store = ONE global Redis HASH secret_scrub:staged (field sha256(value) → AES-256-GCM envelope via the credential-encryption singleton) + a secret_scrub:staged_at ZSET giving 24h per-member expiry (Redis holds only envelopes; values ≥8 chars only). The scrub side fails OPEN (a Redis/decrypt error → no identity scrub + one throttled ERROR; the pattern pass still runs and the terminal still persists — no NEW secret is delivered during the outage because the stage side is closed). Behaviour-neutral for OSS (no staged values ⇒ every scrub is a no-op) — see runtime-secret-scrub.md
subscription_service.py - Subscription management (SUB-002); owns derive_auth_mode — the ONE auth-mode enum derivation shared by AgentAuthStatus and the #471 pressure batch endpoint. ent#582: is_claude_auth_configured (the claude_auth_configured flag's one definition) and connect_agents_to_first_credential — when a write gives the install its FIRST Claude credential, every agent created without one gets the new subscription assigned, and agents with a running container are recreated in the background via the SUB-003 _restart_agent under the #799 switch lock. Candidates come from the DB rows (db.list_agents_awaiting_first_credential: not ephemeral, no subscription, use_platform_api_key on, never a successful execution) filtered to Claude runtime, never from list_all_agents_fast (which reads [] on a Docker fault). The restart re-checks each agent under the lock and skips one with a running execution (db.agent_has_running_execution) or an already-current env, so the call is idempotent; returns an int count (connected_agents on both save responses). system_seed_service._connect_seeded_agents re-runs it after a seed pass that created agents (a create can straddle the save). Scope is deliberately "could not authenticate anyway": an established fleet, or an agent on its own .env key, is never touched. #2572 adds the credential-less predicate and the adoption sweep: instance_has_api_key (which MUST stay get_anthropic_api_key() — env fallback included — never has_secret_setting(), a DB-only presence check that would adopt a fleet off a working env key), credentialless_agent_names (one get_agent_subscription_map read, [] whenever the instance has a key) and adopt_for_credentialless_agents (awaited decide+persist, backgrounded container apply). Its runtime gate is LABEL-STRICT via docker_service.agent_container_runtime_labels and fails CLOSED on an unreadable Docker — the opposite of ent#403's fail-open, because a UI affordance and a persisted credential assignment are not the same bet
subscription_headroom_service.py - Live subscription headroom (#471): actual 5h/7d utilization % + reset times read from the anthropic-ratelimit-unified-* headers of a max_tokens=1 micro-ping probe (the /api/oauth/usage endpoint is 403-scope-dead for stored sk-ant-oat01- setup tokens — the closed-PR #2170 mechanism, established 2026-08-19). Click-to-refresh (floored ≥60s/subscription) + default-ON ambient refresh (subscription_headroom_auto_refresh setting, 15-min floor, demand-driven). Fail-CLOSED without Redis — _read_snapshot is tri-state ((redis_ok, snapshot); a client object existing ≠ a reachable server), so a Redis outage yields observed-only, never a probe storm; the dashboard batch path uses wait=False (stale-while-revalidate via strong-ref background task) so a hung provider can't wedge the 60s poll. Snapshot: Redis subscription:headroom:{id} (7d TTL); probes single-flighted (#1920); a probe 429 updates snapshot status only, never subscription_rate_limit_events. services/subscription_headroom_service.resolve_rate_limited_now is the ONE rate_limited_now derivation (#2157 one-gate rule), consumed by both decorate_usage and pressure_states. It is three-state, not an OR (#447): fresh provider verdict says limited → limited; fresh verdict says allowed → NOT limited; no usable verdict → the 2h event predicate. It shipped as db_predicate OR fresh_verdict, which made a fresh, authoritative "allowed, 32% used, resets 19:10" structurally powerless to clear the badge — because nothing clears a failure row on success: clear_rate_limit_events has had zero production callers since #444 removed the one call (clearing was destroying auto-switch's detection signal), so the event half only ever decays with the clock. Observed live: two subscriptions wearing LIMIT while every agent on them answered normally. The db predicate is an inference from past failures; a probe is ground truth about now, so it wins in both directions. _headroom_indicates_healthy is deliberately not the negation of _headroom_indicates_limited — "not limited" is also true for a stale snapshot, a rejected token and a transport error, none of which are evidence of headroom, so all three fall through to the predicate rather than clearing it (#2353's rule, preserved). Both predicates judge a window against ONE named allowlist, NON_BLOCKING_WINDOW_STATUSES (#2396) — an ALLOWLIST of statuses meaning "requests are being served", never a blocklist of blockers, so an unrecognised status still reads as limited (the inverse mistake is the #848 deny-check lesson). It shipped as the bare literal ("allowed",) in one predicate and (None, "allowed") in the other, which made the provider's own near-the-limit tier allowed_warning read as a hard rate limit on every surface — a healthy subscription approaching its weekly window wore LIMIT while the provider was still serving it (observed live: a snapshot recording seven_day: 90%, allowed_warning beside 47 successful executions and zero subscription_rate_limit_events). allowed_warning now also counts as positive proof of headroom — the quota was reached and the answer was yes — so it clears a stale 2h db predicate; excluding it would be #447 returning in a narrower window. Note the ordering that bounds the blast radius: a real HTTP 429 sets the top-level snapshot status and is checked BEFORE any window, and the db predicate is a third independent detector, so the window arm is the weakest of the three. This is the DISPLAY predicate only: auto-switch candidate filtering reads the kind-blind has_recent_subscription_failures, so a just-recovered subscription is still skipped as a switch target and #444's ping-pong cannot return. The event half of that predicate counts failure_kind = 'rate_limit' ONLY (#2352) — it was kind-blind, so an auth failure (401/403: a dead, expired, or .env-shadowed token) set the flag and every surface reported a credential problem as quota exhaustion, sending the operator to wait out a window that was never full; the display layer had honoured the split since #471 (rateLimitEventCount reads the rate_limit kind alone) and the predicate was the layer that had not caught up. NULL failure_kind (pre-#471) is excluded — unknown is never promoted to "429". This is a split, not a filter: db.has_recent_subscription_failures preserves the kind-BLIND semantics for the two candidate-selection listings (list_viable_alternative_subscriptions, list_assignable_subscriptions — since #2409 the db only FILTERS; the services rank, see subscription_auto_switch.py), which must keep skipping a subscription that failed for ANY reason — narrowing them in place would have auto-switched agents onto subscriptions whose token had just been rejected (the #444 class). Two predicates, two meanings; re-merging them is how the bug happened. pressure_states therefore also emits auth_failures_24h beside the total, since after the split a dead-token subscription is no longer rate_limited_now and a bare total cannot tell the badge which word to use. The observed (DB-derived) arm is always populated; SUB-004's windows are deduped since #471 (executions = sole cost/context/turn source; chat_messages contributes output_tokens only — a /chat/persisted-/task turn writes cost into BOTH tables). ent#433 adds the durable half — _probe_and_store now also writes each probe to subscription_headroom_history, so utilization TRENDS survive a snapshot that overwrites itself every probe. Three properties are load-bearing: the history write runs after _store_snapshot and off the event loop (asyncio.to_thread; a sync SQLAlchemy write on the loop stalls everything for up to the 30s busy timeout when it lands during the 03:30 backup or 04:30 VACUUM, and try/except handles errors but not blocking) — order pinned by test; it catches Exception, never BaseException, so shutdown's CancelledError still propagates; and it adds no probe, inheriting #471's entire rate-bounding envelope (60s floor, single-flight, fail-closed ambient) by sitting inside _probe_and_store. get_history(subscription_id, window) is the read — pure DB, never probes, so viewing a trend costs no quota. ent#434 adds a THIRD predicate beside the two binary ones — classify_headroom → saturated | has_headroom | unassessable — because the alert path needs a state the binary pair cannot express. _headroom_indicates_healthy returns False for a stale snapshot, a rejected token, a transport error AND a genuinely saturated subscription, so it cannot tell no evidence from no headroom; #2396's docstring named ent#434 as its consumer and was wrong, since the fleet escalation would then be blocked by exactly the condition it reports. The docstring is corrected and the BODY left untouched (resolve_rate_limited_now consumes it — a behavioural edit moves every LIMIT badge). The classifier keys on utilization, not window status: allowed_warning is deliberately non-blocking, so a status-driven classifier would file a live 90% warning-tier reading as has_headroom — the exact reading the alert exists for. Status remains the stronger signal (a blocking 7d status or a probe 429 is saturated with or without a number, counting toward the fleet claim but raising no percentage-crossing alert). Every could-not-tell path returns unassessable, including the easily-missed 5h-only snapshot (parse_unified_headers's top guard passes on 5h-utilization alone, so a gate reading "ok status + non-blocking windows" would claim WEEKLY headroom on zero weekly evidence). ensure_reading(sid, max_age) is the ONE probe decision per subscription per cycle — two consumers each deciding for themselves means the second is floored out by MIN_PROBE_INTERVAL_SECONDS and reads "no data" on precisely the subscriptions the first just refreshed. SAMPLE_INTERVAL_SECONDS (3600, floored at REFRESH_SECONDS) is a constant, not a knob (#1644) — and literally so: it reads no env var, because neither compose uses env_file and an unforwarded read would be inert while still reading as configurable. The one real lever is SUBSCRIPTION_SWEEP_CONCURRENCY (probes per sweep chunk), forwarded in both composes #2409 adds the auto-switch ranker beside the classifier, over ONE gate. headroom_reading(headroom, max_age_seconds) is the usability gate both consume — fresh? probe answered (ok/rate_limited/invalid_token)? window shape? — and classify_headroom became six lines of policy over it with byte-identical verdicts (pinned by a differential test against a frozen copy of the pre-#2409 function over the full age × status × window × threshold product; the ent#434 suite is unchanged). cached_headroom_readings(ids) is the selector's read: ONE MGET over subscription:headroom:{id}, tri-state on Redis (client unbuildable or a raise ⇒ every candidate unknown in one attempt; a malformed snapshot blinds only itself), and it never calls get_headroom/_locked_probe. rank_subscriptions / selection_verdict / selection_sort_key are threshold-free — the alert threshold is an operator knob that must not steer where agents land, and it is 0 when alerts are off: tier measured (fresh, serving, weekly figure) sorted by the FULLER window (primary = nearest wall) in HEADROOM_BAND_PCT=10 bands with agent_count as the in-band tiebreak, then the other window, then name; unknown (no/stale/error/5h-only/non-finite, or a STALE refusal) = exactly today's agent_count ASC, name ASC; refused (a FRESH probe 429 / blocking window / rejected token) is dropped. Two bounds: MAX_READING_AGE_SECONDS (2h — now OWNED here, = max(2h, 2 × SAMPLE_INTERVAL_SECONDS) so it always covers two sampler intervals, re-exported by subscription_headroom_alerts) bounds the weekly figure, which is a lower bound within the provider's fixed window; FRESHNESS_SECONDS (30 min, the LIMIT-badge bound) bounds the 5h figure (it can fully reset inside 2h) and any refusal (a point-in-time verdict). describe_reading is the notification/activity shape — tiers and figures, never a token. ent#582: check_token(token) sends that same one-message probe (_post_probe) for a raw token BEFORE it is registered — ok/rate_limited/invalid_token/error, backing POST /api/subscriptions/test.
subscription_headroom_alerts.py - The weekly-window alert (ent#434): pure projection + tier decision + deterministic emitter, no durable state. Since #3246 it reports through platform_alerts.observe (one row per subscription, updated in place, reconcile ends cleared ones, 14-day lifetime) — the id-as-state-machine below is history. The alert id WAS the state machine — the 7d window was measured fixed-with-reset (live history: seven_day_resets_at constant across five days of probes, then a +7d step), so utilization is monotonic within a window, a hysteresis floor is dead code, and sub-headroom-{sid}-{reset-day}-{tier} gives edge-trigger, cross-worker dedup and re-arm for free via create_item's UNIQUE(agent_name, request_id) ON CONFLICT DO NOTHING (the id is quantised to the DAY as a belt: if some plan did behave as rolling it degrades to one alert/day, not one/probe). The threshold fires; the projection ranks — projected_end = utilization_pct / fraction_of_window_elapsed, so urgency is the operator's own burn rate rather than a configured time gap that is wrong for everyone; the alert is never withheld at the threshold, priority is low under 100% projected and high over. Fleet escalation needs ≥2 subscriptions (with one, "this is full" and "every one is full" are the same fact) and every member assessable — one unassessable blocks the claim and is named (ent#100). The denominator is built from the subscription roster, never from the sweep's results: a member whose sampling raised carries classification: None by design (the swallow protects #447) and a mid-cycle lease yield breaks with a short result list, so a truthiness filter dropped exactly the members that should have blocked the claim — a 3-subscription instance emitted "All 2 registered subscriptions are at or past 75%" with the per-subscription alerts suppressed by the early return. fleet_verdict was always correct; the caller had narrowed its input. It states what was MEASURED and deliberately does NOT say "auto-switch has nowhere to go": since #2409 the selector ranks over the SAME cached readings this sweep samples, but its candidate set is also narrowed by the 2h failure filter the sweep never sees, so the body describes the measurement and never promises what the selector will do. Platform-only emitter on sentinel _sub-headroom (_PLATFORM_ALARM_SENTINELS + _RESERVED_ID_PREFIXES + the test_1677 allowlist; expires_at None). Residual: create_item has no UPDATE path, so a 75% row still reads 75% at 92% — the escalation is a separate, self-contained item
subscription_auto_switch.py - SUB-003 auto-switch orchestration (handle_subscription_failure → _perform_auto_switch, hot-reload #1089). Since #2409 it owns alternative SELECTION — it used to be a first-match pick inside the db layer (ORDER BY agent_count ASC, first survivor of the 2h skip-list), which read no headroom, moved agents onto subscriptions at 99% of their weekly window, and sorted an unused dead-token subscription FIRST (no agents ⇒ no failure rows). select_best_alternative_subscription(current) = filter → rank → first: the db lists the survivors (list_viable_alternative_subscriptions, kind-blind #444/#2352, agent_count ASC, name ASC — filter only, the name tiebreak makes the fallback order deterministic), the headroom service ranks them (rank_subscriptions over cached_headroom_readings, one MGET, never a probe — a candidate the filter dropped is never even read), and the pick returns with a why. Runs under the per-agent switch lock via asyncio.to_thread (both reads are blocking). Fail-open on the ranking half only, and LOUDLY: Redis down, or a lazy import that resolved to the wrong module (importlib.import_module answers from sys.modules, not from a package attribute a previous test left behind — learnings 2026-08-12), degrades to the db's order with a WARNING; an all-unknown set with ambient refresh OFF logs that the ranker is inert. The only case that now yields no target where it previously did is every survivor is currently refused by the provider (an approved deviation from the issue's literal AC #3, recorded on it). _perform_auto_switch(..., destination_headroom=why) surfaces the pick on the activity details, the notification metadata and the result, plus one clause in the notification text (on a two-subscription install the ranking cannot change the pick, so the explanation IS the value). subscription_service.select_subscription_for_new_agent() (#74 new-agent auto-assign) rides the same ranker — db.list_assignable_subscriptions() → rank → first candidate whose token decrypts (#340), one decrypt in the common case; database resolved at call time for the creation harnesses. get_least_used_subscription / the db select_best_alternative_subscription are gone. See subscription-auto-switch.md #2638 adds three things the reactive switch could not do. (1) The 2h skip-list is OVERRIDABLE per candidate: _readmit_recovered reads the filter's complement (db.list_recently_failed_alternatives) and readmits one only on positive evidence — subscription_headroom_service.recovery_verdict = a FRESH not-refusing reading (serving_now), or a blocked window's own reset instant ELAPSED and predating the failure (window_reset; without that ordering a subscription that 429'd after its rollover is readmitted on a reset it already consumed). Absence of evidence readmits nothing, so #444's ping-pong — caused by FORGETTING a failure — stays closed, and the fail-open ranking branch readmits nobody by construction because it is precisely where the evidence could not be read. A window_reset candidate is handed to the ranker as UNKNOWN: its blocked flag describes the window that just rolled over, and rank_subscriptions would drop it as refused — the readmission would be inert in exactly the case it exists for. (2) ensure_serviceable_subscription switches BEFORE the first dispatch when the assigned subscription is already known-refused (fresh provider refusal, or a 429 in the 2h window via the 429-only DISPLAY predicate — an auth failure is a credential problem another subscription may share). It never raises, records NO failure event (nothing failed; a synthetic one would poison the skip-list it feeds), dispatches anyway when there is no alternative, and performs the SAME _perform_auto_switch so one activity/notification/hot-reload happens whichever path fired — pre_dispatch=True changes only the wording. (3) fallback_to_api_key is the last resort when no subscription can serve: clear the assignment, set use_platform_api_key, RESTART (the reload endpoint pushes an OAuth token; this needs the opposite change, which lifecycle's auth block already derives from DB state). Setting subscription_api_key_fallback (default ON, fail-OPEN on a read error) with key_configured on the GET, because a toggle reading only "on" with no key stored describes a remedy that cannot run. earliest_known_reset feeds the portal's honest "quota resets at …" refusal. #2572 adds a FOURTH caller of agent_switch_lock / _restart_agent — subscription_service's credential-less adoption sweep — and changes nothing in this module: SUB-003 stays failure-driven and its precondition 2 (the agent already has a subscription) is deliberately NOT relaxed, because that precondition is where the #2572 gap was written down, not a defect. The sweep also deliberately does not become a fourth _hot_reload_subscription_token producer: credential-less → subscription is an auth-MODE change, and that helper's docstring invariant is that all its producers are sub→sub by construction. #3470 — the walk. SubscriptionWalk / Rung / advance_subscription_walk are the shared decision core both dispatch loops (task_execution_service._call_agent_with_retries, chat_execution_service._walk_chat_dispatch) drive: each refused attempt names the next credential (another subscription, then the platform key) and the turn is re-issued with it as a per-spawn auth_override; nothing moves until a trial serves, then commit_subscription_walk commits that subscription once. fallback_to_api_key no longer clears/flags/restarts — the key is a rung (api_key_rung, kind-aware refusal marker platform_api_key:refused). See subscription-auto-switch.md §#3470.
ssh_service.py - Ephemeral SSH credential generation
email_service.py - Email sending for verification codes. Provider, Resend key and sender resolve per send through settings_service (ent#582), not the import-frozen config values
platform_keys_service.py - Format rules + live checks for the keys the first-run flow configures (ent#582): the Anthropic sk-ant-oat-in-the-API-key-tab refusal, Resend (GET /domains — key valid AND sender domain verified; a sending-only key passes with a warning), Gemini (model list, key in a header). Messages are operator copy, twinned by frontend/…/onboarding/steps/credentialSteps.js

Git & GitHub:

git_service/ (package, #1028 — sync / remotes / token_scrub (the ent#615 credential-helper install + embedded-token sweep) / provisioning / conflicts / gitignore / trinity_files) - Git sync operations for GitHub-native agents; persistent-state allowlist primitive (S4, #383); rebind_origin_and_push + inspect_container_git — the in-container half of the ent#109 repo binding (push committed history by explicit URL, THEN repoint origin, so a push failure leaves the agent untouched; reads origin back to prove the rewire took)
github_service.py - GitHub API client (repo creation, validation, org detection, branch listing)
agent_service/repo_binding.py - Post-creation repo binding orchestration (ent#109) — see Post-Creation Repo Binding. HTTP-free: raises BindError, mapped 1:1 at the thin routers/git.py endpoint (Invariant #1)
agent_service/fork_to_own.py - Fork-to-own template copy (trinity-enterprise#93): a template declaring fork_to_own: required is copied at creation into a user-owned repo (private by default; the user's PAT creates + pushes, then persists as the per-agent PAT #347 so recreates never fall back to the platform PAT). Origin = the user's repo; GIT_UPSTREAM_REPO env + a credential-less upstream remote (startup.sh) keep template updates one git pull upstream away. Destination collisions: empty or template-tip SHA-match → reuse; already bound to a live agent or holding other data → 409. ent#109 extracted the destination half into inspect_or_create_destination_repo() (created | empty | branches) + validate_destination_pat(), shared with the post-creation rebind; the reuse/refuse POLICY stays in each caller, because this path's reuse branch IS the template-tip SHA comparison
agent_service/crud.py - Agent create/delete. create_agent_internal is a thin orchestrator over fenced phase-helpers (#1484): CC-trivial gates + the if docker_client: try/except/else stay inline (so what is caught is byte-identical), while each fat phase (ephemeral pre-gate, template/fork resolution, config staging, env build, volume mounts, container create, register, materialize) is a private _* helper returning into the orchestrator's locals. The except/else read a single orchestrator-populated _RollbackHandles dataclass (agent MCP key + git-config reservation + ephemeral slot + the ent#313 container_floor_ts). Container reclaim on failure (ent#313): _rollback_failed_creation stays DB/quota-only, but the except-path now also awaits _reclaim_failed_creation_container, which removes the container and clears the name-keyed Redis keyspace (clear_agent_runtime_state, #1560). The old docstring deferred both to "the cleanup watchdog" — none exists for a non-ephemeral agent (_sweep_ephemeral_agents is gated on the trinity.ephemeral label), so a post-containers.run failure leaked a running, ownerless container that also pinned its workspace volume, blocking #1581's unattached-strike counter forever (a phantom in the Docker-as-truth listing, Invariant #11). When the create returned a handle the removal is unambiguous; when it did not (the observed 60s Docker read timeout — the daemon created it, the client never got the handle) the container is re-derived by name behind three fail-closed gates: not a 409 name conflict, no agent_ownership row, and a trinity.created label at or after container_floor_ts. Anything unprovable refuses — a false negative costs one manual docker rm -f, a false positive deletes a running agent (possibly another install's, on a shared daemon). Do not re-monolith it and keep every helper < 100 SLOC / CC < 20. The github:+fork phase stays OUTSIDE the try (so FORK_* 4xx don't flatten to 500), and the network is hard-coded in _create_agent_container (AC #5). Module split to creation_phases.py is the deferred #1028 follow-up. Tokenless public templates (ent#123): a github:owner/repo create with NO resolvable PAT is allowed when the repo is public — source-mode only (_gate_tokenless_request: non-source-mode → named 400; also normalizes the resolver's "" to None), validated via a credential-less git ls-remote probe (git_service.probe_anonymous_repo_access — same transport as the clone, immune to the anon REST cap; definitive not-found/auth-challenge → combined "not found or private" 400, transient → fail-closed 502), env carries repo+sync flags but no token vars (_apply_github_env; the rebuild seams gate on repo the same way — see lifecycle._apply_git_env_from_db below), startup.sh clones anonymously (GIT_TERMINAL_PROMPT=0, blackholed push remote, workspace-.env PAT fallback on restart), and backend push paths (sync_to_github/reset) pre-check baked env OR the per-agent PAT row → no_write_credentials 409 — whose message now points at Bind to your own repo on the agent's own Git tab (ent#109) rather than the retired create-a-new-agent-and-import workaround. Requires a rebuilt base image (old images silently skip the tokenless clone — release-note ordering requirement). Template-declared schedules (ent#89): _TemplateResolution.declared_schedules is a normalized carrier populated by both resolver branches — the github: branch reads its own fresh template.yaml via fetch_template_metadata_for_create (resolved PAT + parsed @branch ref, cache-bypassed), the local: branch normalizes tr.template_data. It is deliberately NOT folded into template_data, which is raw YAML on the local: path and {} on the github: path (which has never populated it — hence #383's persistent_state and #1169's data_paths being effectively local:-only) and whose truthiness gates _stage_config_files' credential-file generation. reconcile_declared_schedules(agent_name, declared, owner_username) runs inside _materialize_agent_files beside the persistent_state/data_paths steps — non-fatal, ghost-skipped at the caller, name-match idempotent, and it CHECKS db.create_schedule's falsy return (it returns None on three paths and never raises).
agent_service/lifecycle.py - Agent start/stop + the two container-rebuild paths. _apply_git_env_from_db is the single owner of the GitHub-sync env block across both (ent#109): recreate_container_with_updated_config (config drift — production callers: start_agent_internal, fired by nine predicates and base-image drift at cold start, so a base-image rebuild reaches the whole fleet; plus ent#109's bind path, which calls it directly and so carries its own clear_agent_breakers — the #1560 clear lives at the call site, not in the helper, because start_agent_internal gates it on needs_recreation or not was_already_running and moving it in would clear breakers on paths that today do not) and _apply_persisted_auth_env → recreate_missing_container (rebuild from nothing). Before this the block lived only on the second path and the first re-derived just GITHUB_PAT, replaying whatever GITHUB_REPO/GIT_SYNC_* the old container carried. Three properties, each load-bearing: (1) the PAT gate is a required per-call-site parameter, never a default — per_agent_only (config drift) preserves #211 verbatim so a global-only platform PAT is never injected into a previously-tokenless container, while effective (rebuild-from-nothing, no old container to inherit from) uses the 2-tier per-agent→global resolver; sharing one gate would bake the platform PAT into tokenless agents, configure_push_remote would clear the push blackhole, and a private KB could reach the shared public upstream (the ent#162 class). An AST guard in tests/unit/test_ent109_git_env_seam.py pins the writer set — {recreate_container_with_updated_config: per_agent_only, _apply_persisted_auth_env: effective} exactly — so CI fails both on a flipped gate and on a third writer appearing on any container-seeded path (the bug ent#109 fixes was two writers with one wrong). (2) Set-or-clear over _GIT_ENV_KEYS, since the config-drift path writes into a dict seeded from the old container — a deleted agent_git_config row pops the whole set (incl. an orphaned GITHUB_PAT: the per-agent token is a column on that row, so no row ⇒ no per-agent credential and no repo), a source_mode flip clears the mode/branch pair; GITHUB_PAT stays set-only while a repo is bound. (3) GIT_SYNC_AUTO = DB auto_sync_enabled alone (#3010), derive-only: creation writes the flag from the same _git_auto_sync_baked predicate that bakes the env (ghosts included), and the agent loop reads the flag live every cycle, so the env is only its fallback. No write-back — PUT .../auto-sync is OwnedAgentByName while POST .../start (the recreate trigger) is AuthorizedAgentByName. (4) Correct, never introduce on the config-drift path: the block is written only when the old container already carried GITHUB_REPO or a PAT resolves. Repo-gating the block while per-agent-gating the PAT would otherwise hand startup.sh GIT_SYNC_ENABLED=true with no token for an agent bound via POST /{agent}/git/initialize on the global platform PAT — whose only credential lives in the container's .git/config — and the restart branch would rewrite origin to the credential-less URL and blackhole its push remote. effective is exempt (no old container; not introducing the block there is the #843/#1439 silently-empty-agent bug). GIT_WORKING_BRANCH and GIT_UPSTREAM_REPO are deliberately not owned here (only read by startup.sh's clone branch / no DB column). Making the #389 toggle authoritative is a tracked follow-up.

Integrations:

slack_service.py - Slack API client (OAuth, messaging, verification) (SLACK-001)
nevermined_payment_service.py - x402 payment verification and settlement (NVM-001). Facilitator calls are bounded fleet-wide by NEVERMINED_MAX_INFLIGHT (8) with a NEVERMINED_FACILITATOR_WAIT_SECONDS (5.0) wait budget, one semaphore per event loop — verify/settle run on the default thread executor, so an unbounded set of them starves the whole fleet's threads, and the caller needs no credential to trigger one. Knobs documented in nevermined-payments.md. The plan's x402 SCHEME is resolved, never assumed (#3215): the SDK's build_payment_required defaults to nvm:erc4337 and the facilitator is POSTed the requirements document verbatim, so accepts[0].scheme is the only channel it travels through — a fiat (card) plan advertised as crypto is unpayable, which is the defect. Verify and settle read the PRESENTED token (scheme_from_token over accepted.{scheme,network,planId}, allow-listed against the SDK's vocabularies and checked against this agent's plan id): zero network calls on the money path, and a settle re-driven hours after its verify is byte-stable with it, which a TTL'd cache cannot promise. The 402 and /info use resolve_plan_scheme — Trinity's own parse of plans.get_plan (parity-tested against the SDK's resolve_scheme, which swallows every failure at DEBUG and silently returns crypto) behind a (environment, plan_id) cache, single-flight futures, its OWN 2-slot gate (never a facilitator slot: wait_for does not cancel the worker thread, and an anonymous 402 flood must not starve paying traffic), stale-while-revalidate and a WARN once per negative window. plan_scheme=None is the pre-#3215 document, which is what keeps crypto bytes identical
paid_turn_service.py - The ONE x402 paid-turn orchestrator, shared by routers/paid.py and the A2A inbound door (ent#679 T3): verify → dedup gate → execute → settle, with the #1018 branches (honest success_unsettled on a delivered-but-unsettled turn, a replayed unsettled snapshot re-settling and converging, no settle on a failed/cancelled turn) existing exactly once. Takes every collaborator as a PARAMETER and imports none of them — a service that imported the payment service and the execution bridge would be a second router, and the parameter list is what lets the unit suite drive all ten outcomes with plain stand-ins
a2a_payment_gate.py - The A2A-shaped adapter around paid_turn_service (ent#679): is_priced (exposure ∧ enabled config ∧ key — deliberately NOT the SDK check, since "takes payment" and "can process one now" are different facts owing a 401 and a 501 respectively), metadata-first token extraction with the deprecated payment-signature header as fallback, the 402/403 bodies, and the Task a payer gets back with its x402 metadata. Consults the a2a_gate allow-list after verify as x402:{payer} and fails CLOSED there — the inverse of that seam's authenticated-caller bias, because on this path the payment IS the authorization. Nothing here is edition-aware; the path is reachable only with the entitled exposure flag on
proactive_message_service.py - Agent-to-user proactive messaging with rate limiting and audit (#321)
channel_completion_report.py - Reports a delegated/background execution's terminal back to its originating channel chat/thread (ent#224 Slack, ent#265 Telegram, ent#457 Workspace): inherited-context-only (never inline turns), binding-agent consent + delivery, effect-guarded at-most-once. The Workspace leg's consent is by construction, not by flag — a portal session belongs to exactly one client, so there is no third party for an allow_proactive bit to protect, which is why the recipient is read from the SESSION ROW (the platform's own record of whose chat this is) rather than from the execution's inherited stamp alone; delivery is a persisted assistant message and the sidebar's last_message_at is touched like any other writer's. Durable, not immediately visible — the Workspace does NOT poll its threads (refreshThreads() is event-driven; the only interval is the 20s asks poll on a different surface), so a client sitting on the thread sees the report at their next reload or thread switch; an idle history poll is a tracked follow-up. A report landing mid-turn can also be read AS that turn's answer, since the client detects a reply by an assistant-row count delta and this is a second writer of those rows — the honest fix needs a per-row discriminator the table does not carry. INLINE_CHANNEL_TRIGGERS gains "public" so a Workspace turn's OWN execution is never reported twice — public links and x402 share that trigger and are unaffected, since they stamp no source_channel_chat_id and never reach the gate — see channel-completion-report.md
channel_history.py - Persists a delivered proactive group/channel broadcast into the channel session (#1649), so the agent has a record of its own outreach. Session keys are derived by driving the channel adapter's own get_session_identifier() (never re-implemented — that drifts). Slack = real recall: channel sessions are thread-scoped, so a broadcast filed at its own ts IS the session an in-thread reply resolves to (needs slack_service.send_message_detailed() to return the ts). Telegram = real recall since ent#600: group sessions are per chat, so a broadcast lands in the session a participant's reply reads — unless the group's context is off, which records nothing; purge_telegram_group_history deletes a group's sessions (chat + forum topics) when its context is switched off. See integrations.md → Telegram group conversation context. #903 shared-thread attribution (sender_email=None); persist on confirmed delivery only; fail-soft
telegram_group_context.py - Telegram group conversation context policy (ent#600): history bounds + env overrides, the delimited/sanitised history block, the reply-quote line, and the per-group context_status. See integrations.md
tts_service.py - Shared outbound-voice TTS layer (epic #24): ElevenLabs synth → ffmpeg OGG/Opus transcode; shared char cost-cap; fail-soft (any error → text fallback). Key resolved at call time via settings_service.get_elevenlabs_api_key() (stored setting → env, ent#117), not the frozen config value. Consumed by voice_reply_service (ent#117) and the STT path. Also owns the one voice gate every surface shares (#2157): resolve_voice_id(agent) / the pure resolve_voice_from_config(...) = platform key AND agent-level tts_voice_replies_enabled AND (own tts_voice_id else platform default) — read by the channel path, the Workspace roster card's voice_available, the portal /tts endpoint, and the narrated-surface prompt, so the surfaces can no longer disagree about whether an agent may be spoken aloud. Voice INPUT is a separate bit (#2212): dictation needs the platform key only — nothing is spoken back — so the Workspace card also carries stt_available = client_portal.service._stt_ready — key present AND the key's speech-to-text capability not refused — exactly the /stt gate. Presence is not capability (#2695) — stt_available is the key's speech-to-text VERDICT from stt_capability_service.py (below), not is_available(). The two cannot be collapsed: on a key-but-no-effective-voice agent voice_available is false while transcription works. The client prefers the server path (MediaRecorder → POST /stt, which answers with real statuses and messages) over the browser Web Speech API whenever stt_available is set — Web Speech is a browser-hosted service that reports no event at all in Chromium (measured) and ends at the first pause; it stays the no-key fallback, and with neither path available the mic is not rendered
stt_capability_service.py - Does the configured ElevenLabs key actually TRANSCRIBE? (#2695). ElevenLabs permissions are per endpoint, so a TTS-only key passed the presence check and rendered a mic that 401'd on every press. One-byte non-audio probe of /v1/speech-to-text (authorises before it validates) → refused (401/403, provider status word as detail) / capable (any other definitive answer) / unknown (transport, 5xx); fail-soft — only refused withholds server-side dictation, and resolveMicMode then falls back to the browser's own Web Speech engine where one exists, so the mic is hidden only where there is neither. Cached in Redis under a digest of the key (stt:capability:<sha256[:16]>, 6h decided / 2min unknown) so a key change is a miss by construction; readers are bounded (WAIT_BUDGET_SECONDS 4s answers unknown now, the 8s probe fills the cache behind it) and single-flighted per process. Redis is the one authority: the per-process _local is read only when Redis cannot be asked (no client / read raised), and a Redis MISS evicts it — consulting it on a miss let worker B serve a refused that worker A's invalidate (re-saving the key) had already deleted, for the rest of the 6h (the cross-worker-staleness AC). A real /stt 401 stores refused (record_live_refusal); GET/PUT /api/settings/elevenlabs carry stt_capability/stt_detail/stt_checked_at and re-saving invalidates the row; the Settings → Voice panel renders it via utils/sttCapability.js. The truncated SHA-256 is a row NAME, never a verifier (same shape as db/mcp_keys.py) — do not HMAC it over SECRET_KEY, which is per-process random when unset and would defeat the shared cache The provider-error branch says why (#2696): classify_stt_failure(status, body) maps every non-200 onto a named category — permission / auth / quota (401/402/403 by the provider's status word) → 503 with an operator-actionable sentence, rate_limit (429) → 429 with the existing retry wording, audio (400/413/415/422) → 422 "the recording could not be read", provider (5xx) → 502, anything else unknown → 502 — no arm returns the old opaque "Could not transcribe the audio", and a test sweeps 300–599 to pin that. The client sentence never carries the provider body; record_live_failure keeps the status word + category under stt:last_failure:<digest> (24h) for the admin panel only. GET/PUT /api/settings/elevenlabs carry stt_capability + stt_detail + stt_checked_at + stt_last_failure beside key_configured, and re-saving a key invalidates its row; the Settings → Voice panel renders the distinction (utils/sttCapability.js).
voice_reply_service.py - Per-message voice-reply delivery (ent#117): backs the send_voice_reply MCP tool. Given an agent + resolved channel destination (channel/chat id/thread from the execution) + text, gates on TTS availability + agent-level enable + the per-channel flag, wraps delivery in effect_guard("voice_reply", …) (#1084), synthesizes, and delivers via each channel's send primitive (Telegram _send_voice, Slack slack_service.upload_file, WhatsApp create_share_from_bytes + Twilio MediaUrl). The route refuses a delegated child (source_channel_agent set) with delegated_turn before any channel branch (#3232): inherited context exists for the consent-gated completion report only. Fail-soft → not-delivered so the agent falls back to text. Replaces the old always-voice adapter path (_maybe_send_voice removed) — replies are TEXT by default, voice is a per-message agent choice
agent_shared_files_service.py - Outbound file sharing — see Outbound File Sharing
turn_audience.py - Who a turn was for (ent#549): pure audience_of(execution) + evidence-based resolve_turn_audience — same section; a leaf, no HTTP, no SQL; re-exports normalize_addressee_email from utils/addressee.py (#2955)
loop_service.py - Sequential agent loop runner — see Sequential Agent Loops
reminder_service.py - Agent self-reminder create (bounds + timeout clamp + provenance + relative→absolute) — see Agent Self-Reminders (#1296)
client_roster_service.py - Aggregates external channel clients (Telegram + WhatsApp) into the Sharing-tab roster; cross-channel sort + per-channel failure degradation (#20)
voip_service.py - VoIP outbound-call orchestration — see VoIP

Content & Media:

image_generation_service.py / image_generation_prompts.py - Platform image generation via Gemini (IMG-001)

Skills & System:

skill_service.py - Skills library sync + full-directory package injection (ent#183): git archive-sourced tars via the existing agent-server restore primitive, tree-SHA versioning, manifest prune, declaration-only dep check — see skill-injection.md. ent#236 adds the removal half — remove_skills (manifest-driven, takes the same per-agent inject lock) and reconcile_agent_skills (start-path diff of the agent's platform-managed skill dirs against the assignment set, so a removal reaches an agent that was stopped when it happened), plus durable sync status in system_settings (the in-process _last_sync is invisible to the other uvicorn worker). ent#237 makes it multi-source: sync_library iterates every enabled skill_sources row under ONE library-wide lock, one failing source never blinds the others (per-source outcomes, aggregate succeeds if any synced), and commit_changed — the gate ent#236's fleet re-inject fires on — is computed per source against that source's own durable last_commit_sha and OR'd. services/skill_source_clone.py owns one checkout's git lifecycle (clone/fetch/reset/checkout, tag-pin refusal on both the update and clone paths — the update path compares the tag peeled (^{commit}), because every trinity-skills release tag is annotated and a bare rev-parse of one is the tag object, not the recorded commit (#2550) — non-repo quarantine); skill_service orchestrates N of them. Editing a source has to reach disk, which took three fixes of one shape (row moved, checkout didn't): (1) _update_* fetches origin, written at clone time, so a repointed url kept pulling the OLD repo forever while reporting success with a moving commit that fed the fleet re-inject — sync now compares origin against the configured remote (canonical_remote, credential-stripped so a PAT-bearing origin isn't a false repoint) and discards + re-clones on a genuine mismatch, since remote set-url would leave the old repo's tag refs behind and a tag name present in both at different commits then reads as a moved pin forever; an unreadable origin counts as a match, because the action gated on it is an rmtree. (2) update_source clears the sync bookkeeping on a url/ref/ref_type change — without it the documented tag-bump path (v1 → v2) compared v2 against v1's recorded SHA and was refused as moved_tag, and on the fresh-clone path that refusal rmtrees first, so bumping a tag emptied the source. Non-identity edits (name/enabled/priority) must NOT clear it: a cleared baseline on disable would let a tag moved during the disabled window be adopted silently on re-enable. (3) discard_source_checkout reclaims <source_id>/ + <source_id>.broken on delete, with _reclaim_orphan_checkouts (full sweeps only, fail-closed on the row read, id-shape-scoped so the legacy checkout and operator dirs are out of scope, logs what it reclaims per #1644) as the backstop for the crash window and for installs predating the fix. ent#332 makes each source's layout root resolvable — catalog.yaml skills_root: (ent#314 hardened parse, segment-wise validation, realpath containment) → evidence-gated skills/ probe (dual-layout keeps legacy + flags layout_conflict) → .claude/skills/ fallback; every invalid tier falls through, and skill_packaging.filter_skill_archive(source_root=…) rewrites tar arcnames to the canonical agent-side .claude/skills/ destination so manifests/prune/removal stay destination-canonical with zero migration (requirements §21.1.4)
skill_packaging.py - Pure skill-package primitives (ent#183): hardened frontmatter contract parse, archive member vetting, injection-tar assembly, prune diff; compute_removal (ent#236) is compute_prune against an empty new manifest, so confinement + cap live in one place. ent#672: the contract carries the library's lifecycle keys (deprecated, superseded-by → superseded_by, the author's text) and deprecation_warning is the one producer of the deprecated[:<successor>] code — see skill-injection.md
skill_gate_service.py - The gated-skill check (trinity-enterprise#751): enforce at every dispatch road (raises SkillApprovalRequired / SkillGateRefused, skill_gate_errors.py), exactly-once approved dispatch from skill_gate_requests, the in-container hook's check_invocation (#752) and the fail-closed marker sync; list_skill_gates is the one read of the gate map — see skill-gate.md
skill_gate_map_service.py - The per-agent gate map's only writer (trinity-enterprise#753): set / clear / the state-driven library-default reconcile / a person's unassign, each under the per-agent marker lock; called by routers/skill_gate.py, routers/skills.py::_sync_gates, the start path and the library sweep. Also the Skills tab's reads (ent#754): approver_status (viewer_fills via enforce's own functions), hook_status (the manager-only /health probe) and the self-approved run flags for executions and Workspace history
skills_sync_service.py - Scheduled skills-library auto-sync + fleet-wide re-inject (ent#236). Leader-locked (skills:sync:leader), both flags default OFF; sweeps only when the library commit actually changed, over running non-ghost agents at bounded concurrency with skip-and-report on inject-lock contention. Backend-hosted rather than in the standalone scheduler because the sweep must reach agent containers and the scheduler is platform-network-only
skill_sets.py / skill_set_service.py - Skill sets (trinity-enterprise#530): the pure catalog parser + plan_member_rows (fail-closed), and resolution (skill precedence, fingerprint-cached), per-agent honest status, assign/unassign/replace. Every inject path (start, manual Sync, fleet re-inject) runs reconcile_agent BEFORE reading the names, then prunes. See database.md → agent_skill_sets
system_agent_service.py - System agent lifecycle. #1816: ensure_deployed is read-only when the container is running (3-state check_base_image_state → base_image_state, WARNING + an edge-triggered base-image-stale- operator alarm on stale only, never on unknown) and delegates to start_agent_internal when it is stopped, so the cold boundary adopts a rebuilt base image through the shared lifecycle instead of a bare container_start. Creation converges on the recreate path's contract (TRINITY_AGENT_AUTH_TOKEN, trinity.full-capabilities) so no predicate is permanently false. The generic recreate_missing_container rebuild refuses trinity-system (409, ADOPT-006) — it reconstructs a regular agent and would irreversibly downgrade the orchestrator (system-scoped MCP key deactivated for an agent-scoped one); ensure_deployed's create branch is the only supported rebuild. #3814: creation applies apply_pull_mode_env like every other env assembler, so a piloted system agent is born with its pull worker; the running branch also reports pull state (_report_pull_mode: WARNING while piloted, a system_agent_pull_drift platform alert while the running container lacks the worker, since it is never recreated while running)
cornelius_agent_service.py - First-run auto-seed of the default "Cornelius" second-brain agent (public github:Abilityai/cornelius, cloned anonymously on the ent#123 tokenless path, Brain Orb enabled) — see Brain Orb (trinity-enterprise#107, source-seeded #1656). Invoked via the ent#124 first-run orchestrator; accepts an optional precomputed fresh verdict. #2215: a create_failed/create_blocked result (or a belt-except raise) is no longer log-only — the orchestrator raises a system-seed-cornelius-failed operator-queue alert; the seed flag stays unset, so the next boot retries (#1790 asymmetry untouched)
system_seed_service.py - First-run seed of the default system manifest (trinity-enterprise#124): ensure_first_run_seeded() (both call sites: setup-completion bg task + lifespan safety-net) resolves a persisted first-run verdict (first_run_fresh — computed once BEFORE Cornelius provisions, so sibling-seeded agents can't poison later passes), runs the Cornelius seeder, then deploys the bundled config/manifests/default-system.yaml (env override/disable via TRINITY_DEFAULT_SYSTEM_MANIFEST) through system_service.deploy_manifest — durable default_system_seeded flag + system_seed:provision lock (via the shared SingleFlightLock #1920: ownership-checked acquire with a unique per-acquire token + compare-and-delete release, the lock kept LOCAL to the seeding pass — never on the module-level singleton — so two overlapping in-process passes can't clobber each other's ownership state and leak the loser's lock for the TTL) + reserved-name existence backstop (the deploy path suffixes collisions instead of 409ing); partial/failed seeds raise an operator-queue alert; fail-open, never blocks boot. #2215: the whole pass additionally runs under ONE pass-level lock (first_run_seed:provision, TTL 900s, via the shared SingleFlightLock #1920 — unique per-acquire token + compare-and-delete release in finally; fail-open) — the two inner seeder locks serialise each seeder only against itself, so worker A's slow Cornelius clone used to run concurrently with worker B's fleet deploy (the port-collision / concurrent-transient burst that latches partial); the loser skips the ENTIRE pass and writes no flags, and a failed Cornelius seed raises system-seed-cornelius-failed. The #1920 lock fix is class hygiene, NOT the double-seed guard: the real double-seed window is a >600s deploy outliving the 600s TTL — a double-acquire that a token + compare-and-delete does not touch (system_seed adopts no lease refresh, so its lock sits near-expiry for the whole deploy, unlike ops which refreshes every iteration), and the reserved-name existence backstop remains the actual convergence guard. See requirements §16.5.1 (roadmap.md)
system_service.py - System manifest operations; owns the full deploy orchestration deploy_manifest() (moved out of routers/systems.py in ent#124 — Invariant #1; the router is a thin HTTP wrapper, and the default create_agent_fn lazily resolves the routers/agents ws-broadcasting facade). ent#126 adds the pure resolvers resolve_permission_edges/resolve_schedule_previews that BOTH the dry-run preview and the writers (configure_permissions/create_schedules) consume, so the preview cannot drift from the deploy on the fields it models (ent#89's duplicate-name skip is a deploy-side read the pure resolver does not make — see resolve_schedule_previews); _preflight_template also validates merged resources through the create path's own normalize_cpu/normalize_memory; plus the TRINITY_MANIFESTS_DIR-rooted catalog readers list_bundled_manifests/read_bundled_manifest (layered path confinement — incl. an is_symlink() check on the unresolved entry, for parity with the listing that skips symlinks outright — open-once + fstat + O_NOFOLLOW, fail-soft per file with every reason exiting through the deploy report's own _failure_reason sanitizer). ent#454: system_member_names (#2373's ONE membership predicate) keeps its signature and becomes a thin wrapper over system_membership, which returns SystemMembership{members, tags_readable, evidence} — the body MOVED, not copied. The two extra facts exist for a DESTRUCTIVE consumer: this body deliberately degrades to the raw name prefix when the tag read fails, which is the right ranking for restart (over-capture restarts one extra container and logs it; under-capture restarts a subset and reports success) and the wrong one for a verb that deletes, so such a caller REFUSES on tags_readable == False at its own call site rather than forking the predicate — and evidence (tag/prefix/both, derived from the same two sets that decided membership) is what lets it show "matched by name only" and take an operator's opt-out. Also SYSTEM_VIEW_AUTO_DESCRIPTION, the auto-created view's description as a constant shared by create_system_view and any reader that must find that view again — nothing persists a system↔view link, so the description IS the identifier
log_archive_service.py / archive_storage.py - Log archival + storage backend
session_cleanup_service.py - Session JSONL reaper — see Resumable Turns
db_vacuum_service.py / audit_retention_service.py - Daily maintenance jobs (see Background Services)
db_backup_service.py - Daily database backup for BOTH backends (#2216) — see Automatic Database Backups; its stdlib-only copy/verify/prune/boot-hook leaf is db/backup_primitives.py (import-free of database/services — it runs inside init_database() at import time)

Channel Adapters (adapters/) — pluggable external messaging (SLACK-002, Invariant #9):

base.py - ChannelAdapter ABC, NormalizedMessage, ChannelResponse models. Default-no-op record_inbound_activity(message, agent_name) hook (#1533) — the router calls it once per delivered DM at step 5c (after the access gate, skipped for groups, best-effort) so Telegram/WhatsApp can upsert their chat link and bump the message_count/last_active the Sharing-tab roster reads; Slack/VoIP inherit the no-op. In-flight progress-indicator seam (ent#264): default-no-op indicate_progress(message, elapsed_seconds) hook + progress_threshold_seconds/progress_interval_seconds capability attrs (None threshold ⇒ the router never arms a driver — Slack/WhatsApp/VoIP today); all per-turn indicator state rides NormalizedMessage.metadata (adapters are long-lived singletons handling concurrent turns)
message_router.py - ChannelMessageRouter: rate limiting, agent resolution, execution pipeline; injects MEM-001 per-user memory into execute_task(system_prompt=…) gated on verified_email and not is_group (#895), and MEM-001 summarization is sender-filtered — get_recent_public_chat_messages(session_id, sender_email=user_email) so a thread-scoped multi-participant session never feeds one user's turns into another's durable memory (#903); calls the adapter's async enrich_message hook then prepends a [Channel: #x]\n[From: …] identity prefix for the current turn (#350) while history is attributed per stored sender_label (persisted with each channel user turn, #903 — replayed as Alice:/Bob: in build_public_chat_context); passes the agent's public avatar URL as agent_avatar_url so channels with a per-message bot icon render it (Slack icon_url, best-effort — #292); owns the per-turn progress driver (ent#264) — armed after the step-8 indicate_processing (both wrapped so a raising hook never aborts the turn), ticks adapter.indicate_progress past the adapter's threshold, and resolves at all three terminals via _resolve_indicator (cancels AND awaits the driver dead, settles the shielded in-flight placeholder send, THEN indicate_done) with an idempotent finally backstop — inline-sync coupling: bound to channel turns executing inline (execute_task awaited); if #1081/#1083 ever move channel triggers off the inline path, the successor is the durable seam ent#265 rides
slack_adapter.py - DMs, @mentions, thread replies, agent identity via chat:write.customize; inbound file downloads SSRF-gated two-tier like WhatsApp (#1951, the #1932 shape): credentialed hop 1 *.slack.com only (the bot token's blast radius), validated 30x targets also *.slack-edge.com / *.slack-files.com, follow_redirects=False with every hop re-validated before it is issued and a bounded budget; https-only; refusals log at ERROR (a quiet fail-closed media gate is how #1932 hid a three-month outage); enrich_message resolves sender display name + channel name via users.info/conversations.info so the agent sees who/where (best-effort, #350); outbound voice replies are per-message (ent#117) via the send_voice_reply tool → voice_reply_service (Slack path = inline MP3 through slack_service.upload_file); the adapter no longer speaks replies unconditionally
transports/slack_socket.py - Socket Mode: N concurrent WebSockets per SLACK_SOCKET_CONNECTION_COUNT (default 2, range 1–10), per-client watchdog, envelope-ID dedup ring (#244)
transports/slack_webhook.py - HTTP webhook transport (production fallback)
telegram_adapter.py - DMs, group chats (@mention/observe modes), voice transcription, /login flow; in-flight status indicator (ent#264): 👀 reaction ack at dispatch (setMessageReaction — ✅/⚙️ are NOT in the allowed bot reaction set, so the Slack ⏳→✅ pattern doesn't port; reaction CLEARED at every terminal, no success-👍 swap) + 30s-threshold elapsed-time placeholder maintained via editMessageText at 60s cadence (disable_notification, static template text only, deleted at terminal with neutral edit-to-done fallback), gated by a default-ON per-binding toggle (telegram_bindings.progress_indicator_enabled, PUT /api/agents/{name}/telegram/progress-indicator, human-only) and in groups on @mention/reply triggers or all trigger mode (observe stays typing-only), all fail-soft with a 2-consecutive-failure degraded quiesce; outbound voice replies are per-message (ent#117) via the send_voice_reply tool → voice_reply_service (sendVoice); the adapter no longer speaks replies unconditionally; group sessions keyed per chat, un-tagged messages parsed as observe_only (ent#600 — see integrations.md)
transports/telegram_webhook.py - Telegram Bot API webhook (inbound POST + setWebhook registration)
whatsapp_adapter.py - DMs via Twilio (WHATSAPP-001); media downloads SSRF-gated two-tier: source/credentialed hop *.twilio.com only, validated 30x targets also *.twiliocdn.com (the media CDN, #1932); /login//logout//whoami commands + markdown→WhatsApp syntax conversion (#467); outbound voice replies are per-message (ent#117) via the send_voice_reply tool → voice_reply_service (audio/ogg note hosted via create_share_from_bytes(require_sharing_enabled=False) → Twilio MediaUrl); the adapter no longer speaks replies unconditionally
transports/twilio_webhook.py - Twilio webhook: HMAC-SHA1 signature validation, MessageSid dedup, form-encoded body
transports/twilio_media_stream.py + transports/voip_audio.py - VoIP Media Streams bridge (a voice transport, NOT a text ChannelAdapter) — see VoIP

Agent cascade registry (db/agent_cleanup.py) — one registry answers both "what dies with an agent" and "what follows it through a rename". AGENT_REFS carries the OSS tables (Table objects resolved from db/tables.py MetaData); EXTRA_AGENT_REFS carries tables owned by a registered private module, keyed by NAME through raw text() because the OSS MetaData cannot see them — appending one to AGENT_REFS would KeyError and break agent deletion. Both lists are swept by BOTH paths (trinity-enterprise#500). cascade_rename used to iterate AGENT_REFS alone while register_agent_owned_table's docstring promised both — survivable only because the one production caller, db/agent_settings/metadata.py::rename_agent, carried its OWN copy of the sweep. That is precisely what made it long-lived and what made it dangerous: the behaviour lived in a caller, so the shared function disagreed with its own contract, a cross-repo module author would have believed the contract, and a second caller would have silently dropped it. A registered table left on the OLD name is not merely orphaned — the freed name can be taken by a new agent, which then INHERITS those rows, the recycled-name cross-wire delete_reports_to_refs exists to prevent one function below; for a table recording which human an agent serves that is one agent inheriting another's people. The caller's duplicate loop was deleted when the shared sweep landed: one question, one place (the #1819 lesson). test_agent_cleanup_parity.py guards schema ↔ registry; test_1819_rename_cascade_parity.py guards registry ↔ rename behaviour; test_ent500_cascade_rename.py guards the private-table half, using a table deliberately absent from the OSS MetaData (a test against an OSS table passes on the broken code).

Channel DB modules: db/slack_channels.py (workspace connections, channel-agent bindings, active threads), db/telegram_channels.py (bindings, group configs incl. the per-group allow_proactive completion-report consent ent#265, chat links), db/whatsapp_channels.py (bindings, chat links, verified-email lookup), db/voip.py (voice bindings, call logs, daily-cap window). All persisted tokens AES-256-GCM encrypted (Invariant #12).

Opt-in Instance Telemetry (ent#437)

The second cut of the Tier-2 sharing channel (ent#12 — services/telemetry_sharing_service.py, requirements §45.1/§45.2). Four things changed, and each closes a specific hole. The ask is reachable and snooze-first: the wizard ask only renders on a zero-agent install (seeding made that permanently false) and #2385 removed the welcome form on every pre-provisioned-admin install, so consent now lives in the first-run overlay's sharing step (components/onboarding/FirstRunOverlay.vue over the firstRunSteps.js registry, ent#581; admin + profileVerified-gated, with the #2381 sign-in-email nudge as the sibling email step). The overlay's Skip is a per-browser skip (a pre-ent#581 14-day snooze is honoured while it lasts); "Don't ask again" and consent write the server marker telemetry_sharing_dismissed_at; the warm copy is chosen once per browser after the install's first successful autonomous run, derived on read and memoised (telemetry_sharing_first_value_at) rather than hooked into the dispatch hot path (#2314 in progress). The step gates on four booleans from feature-flags and calls the admin status route only when it will render; the preview loads on expand (?preview=0). The share identity is not the install identity: installation_id rides with the operator's email and company in the ent#38 intake POST, so the aggregate carries a separate sharing_id — minted with insert_setting_if_absent (atomic across workers) on the off→on transition, deleted on revoke, re-minted on re-consent; the validator bans installation_id outright, the id is logged as an 8-char prefix, and instance.trinity_version is the release version (utils/app_version.py), never a commit SHA that adoption timing could re-join to other streams. The schema is enforced (PAYLOAD_SCHEMA_V2, telemetry-owned wire enums for trigger buckets and statuses, funnel steps derived from _FUNNEL_STEPS): share_now refuses to send on a violation — fail-closed egress — and the last 5 attempts, successes and failures, are kept in telemetry_sharing_recent_sends and rendered under Recent sends — each entry and the last-shared stamp recording the origin they went to, and the receiver sentence decided from that record rather than from the URL configured at read time (#2571) — with a 404 recorded against the default origin worded as a 404 at the default address (the ent#190 receiver has been live since 2026-09-04; the entitlement-gated benchmark view's read of it is the private module's one outbound call, under the same two gates as the heartbeat). Delivery survives a missing receiver: the consent-time backfill is retried at every due wake until the first 2xx (telemetry_sharing_backfill_delivered_at), windows are cumulative from last_shared_at in whole days, and a Redis tick marker (telemetry_share:tick, TTL half the interval — the send cadence, never the wake — a fresh lock per claim, released only when the receiver did not acknowledge) makes one worker send per interval. The cadence is anchored on the persisted stamp, not on process age (#2618): the loop wakes every 10 minutes (+ ≤10 min jitter, sleep-first) and sends when last_shared_at is empty, unparseable, in the future, or older than the interval, so a restart never resets it — before, the loop slept the whole interval from process start and a daily-restarting install shared once and never again. share_now reports an acknowledged send as True even if the local stamp write fails (with the stamp unwritable, the marker is what stops a sibling re-sending an accepted snapshot); after five consecutive failures attempts fall to one per half-interval, measured from the persisted send log; the heartbeat is stopped in lifespan shutdown so a cancellation mid-send releases the marker. The payload adds an outcome mix (by_trigger projected from db.get_fleet_execution_timeline + shape_execution_timeline, cost dropped; by_status from the new db.count_terminal_executions_by_status; provider rate_limit/auth counts, whose reader has an unconditional cutoff and so gets an explicit wide window for an all-time backfill) and the #2380 install lane. Every reader is fenced and coerced (a stubbed db or services.* degrades a field, never the payload) and the builder runs off the event loop. Generic DELETE /api/settings/{key} stays open for the telemetry_sharing_ prefix by design: it is the reset path and every deletion moves toward off / ask again / re-mint. OSS-core by decision, consistent with §45.1.

Vector Log Aggregator (`config/vector.yaml`)

Vector 0.43.1 (timberio/vector:0.43.1-alpine). Captures all container stdout/stderr via Docker socket; routes platform logs to /data/logs/platform.json and agent logs to /data/logs/agents.json; enriches with container metadata; parses JSON logs. Health: http://localhost:8686/health. Query: docker exec trinity-vector sh -c "tail -50 /data/logs/platform.json" | jq . (same for agents.json).

Docker Desktop local override (#1432): the docker_logs source busy-loops on Docker Desktop / VM-based runtimes (the virtualized log relay closes each follow stream after backlog flush; docker_logs reconnects with no backoff → a storm that pegs the Docker VM). Native Linux dockerd is unaffected, so prod is unchanged. For local dev, config/vector.local.yaml swaps to a file source tailing /var/lib/docker/containers/*/*-json.log (immune to the follow-close bug), applied via a gitignored docker-compose.override.yml (template docker-compose.override.example.yml) that start.sh auto-creates when it detects Docker Desktop (TRINITY_LOCAL_LOG_SOURCE=docker|file overrides). The file source keys by container ID → a single /data/logs/local-*.json instead of the name-split files. See docs/QUERYING_LOGS.md.

Navigation Menu

Repository navigation

backend.md

backend.md

Trinity Architecture — Backend catalog

Backend (`src/backend/`)

Opt-in Instance Telemetry (ent#437)

Vector Log Aggregator (`config/vector.yaml`)

FilesExpand file tree

backend.md

Latest commit

History

backend.md

File metadata and controls

Trinity Architecture — Backend catalog

Backend (src/backend/)

Opt-in Instance Telemetry (ent#437)

Vector Log Aggregator (config/vector.yaml)

Backend (`src/backend/`)

Vector Log Aggregator (`config/vector.yaml`)