perf: scale agent execution with typed execution state, a claim dispatcher, and owned continuation

[ksarangmath]

Summary

Under sustained load, agent execution was limited by the harness rather than by model latency or Postgres capacity. A load test running the real API, worker and maintenance binaries against Postgres 18, with a fake streaming model, showed four costs:

• Claim contention. Every execution slot polled the global wakeup queue on its own schedule, so hundreds of concurrent claim transactions fought over the same rows.
• A queue round trip on every step. After each model call or tool batch, the agent went back through the global queue, even when its next step was already ready.
• About 29 connection-pool acquisitions per model call. State written in one transaction was reloaded in the next.
• Recomputing "what next" from history. Large SQL frontier functions (about 150 ms to plan, re-planned after routine autovacuum) ran several times per step.

This PR fixes all four. It replaces history-derived scheduling with an agent execution engine that keeps typed per-agent state and updates it through explicit transitions.

Worker

1. Claim dispatcher. Each worker runs a small number of claim loops (OMNARA_WORKER_CLAIM_CONCURRENCY, default 4). A loop reserves an execution slot before claiming, so a worker never leases more agents than it can run.
2. Owned continuation. When an agent's next step is ready, the worker keeps the agent under its existing lease instead of returning it to the queue.
   • This is bounded by OMNARA_WORKER_CONTINUATION_MAX_MODEL_STARTS (default 2 additional model calls) and OMNARA_WORKER_CONTINUATION_MAX_DURATION (default 30s).
   • The budget is checked only before preparing the next step. A step that has been prepared always runs.
3. Fused transitions. Accepting a model output, choosing the next step and preparing the next model call happen in one transaction.
   • If choosing or preparing the next step fails, the transaction rolls back and the output is accepted alone in a fresh transaction. The agent returns to the queue and the worker logs the cause.

Agent execution engine (internal/storage/internal/agentexecution)

• Typed execution state. agent_execution_state holds one row per agent:

  • the current turn;
  • a monotonic stop boundary;
  • context watermarks;
  • pointers to the governing model context and to the pending tool batch, output-limit continuation, config change and checkpoint;
  • summaries recomputed when each transaction commits.

  It has no counters and no JSON. A model context, config change or checkpoint records the inputs that opened its turn once, as an immutable typed array. Database triggers check that every member is an admitted input of the same agent, in event order.

• One selection policy. A pure SelectNext function decides the next work. Global claim, owned continuation, release and recovery all use it. It loads state in two round trips: the agent head, then a pipelined batch of typed queries for the current turn and the rows its pointers reference.

• Semantic transitions. Each transition is an explicit operation:

  • input receipt, admission and backlog changes;
  • config activation;
  • preparation, retry and compaction;
  • output and checkpoint acceptance;
  • tool, interaction and process completion;
  • cancellation, archive, release, reaping and parent/child effects.

• Unit of work. Every execution-writing transaction runs through one unit. A unit can only be opened inside a retry boundary. The unit:

  • locks the agents in its plan in sorted order;
  • if a transition finds it must write another agent (for example, a subagent's result reaching its parent), rolls back and restarts with the larger plan, up to eight attempts;
  • commits through a single lifecycle: per changed agent, one fact reload and one fused head-and-queue statement, sent in two pipelined batches even for wide transactions; then durable webhook enqueue; then post-commit notifications.

  Ordinary transitions lock only their own agent, so sibling subagents don't serialize on their parent.

• Backlog. Moving, ranking and admitting queued inputs use indexed lookups on one partial index over pending content inputs. A move updates one row. When a rank gap runs out, one statement renumbers that agent's pending inputs in that mode (74 ms for 10,000 pending inputs).

• Enforced boundary. Execution writes live in a private query package. SQL rules and a Go analyzer reject execution DML anywhere else.

• Removed. The 14 history frontier functions, the inferred-turn views, the scattered wakeup reconciliation, and wakeup metadata columns.

Fixes found along the way

• A continuation after a turn that carried unanswered inputs from an earlier turn was rejected at preparation on every retry, leaving the agent stuck. Preparation now uses the selected context's immutable opening lineage.
• Model preparation, compaction and terminal-failure paths locked a child agent before its parent. Stopping a subagent and launching one used inconsistent lock orders against machine teardown. All of them now go through sorted lock plans.
• A subagent launch could reuse a machine grant that was deleted while it waited for locks. Launch now revalidates its bindings and restarts with a fresh plan.
• The global claim query anti-joined every runtime lease and sorted every ready agent, so each claim got slower as load grew (10 ms at 3,000 busy agents). It now walks the ready index and stops at the first agent it can lock.
• Releasing a runtime lease scanned every tool call. The only index on tool_calls.runtime_lock_id was partial on running calls, so the foreign-key check on each lease delete was a sequential scan of all tool-call history: 5.8 ms at 100,000 tool calls. A partial index on the reference makes it 0.1 ms regardless of history.
• Every model-context insert took a key-share lock on the shared org and project rows through redundant foreign keys, creating a MultiXact per model call in a busy org. Contexts, processes and process actions now reference the agent row with an org-qualified key, which the writer already holds locked. The redundant project reference on agent inputs is gone. Config and model-revision references stay.
• The execution backlog query read every input the agent had ever received. A partial index on received content inputs reads only the pending ones (1,182 buffers down to 6).

Results

Measured at e2ab1ef, before the schema changes in 3ad8a45 (measured with the plan tests above, since the load test starts from a fresh database every run), on a fanless MacBook Air (Apple M4, 4 performance and 6 efficiency cores, 24 GB) running every component:

• Postgres 18 with every commit flushed to disk;
• Redis;
• the API;
• four workers with 16 database connections each;
• a fake model that streams tokens at realistic speed.

Before each run, the Go build cache, database and Redis are reset, and the machine idles back to its idle temperature. Runs alternate between main and this branch.

Scenario	main	This PR
2,000 agents, no idle time: model calls/s (3 runs each)	292 / 184 / 293	410 / 403 / 407
Same: tool result to next model request, p50 / p95	2.5 / 4.5 s, 7.6 / 10.1 s, 2.3 / 4.7 s	76 / 277 ms, 83 / 593 ms, 72 / 265 ms
Same: message to first model request, p50 / p95	1.0 / 2.0 s, 3.4 / 4.9 s, 0.9 / 2.2 s	195 / 379 ms, 218 / 627 ms, 196 / 368 ms
Same: Postgres CPU per model call	21.6 / 40.0 / 21.6 ms	9.4 / 10.3 / 9.4 ms
Same: pool acquisitions per model call	29.2 / 30.5 / 29.3	10.1 / 10.1 / 10.1
5,000 agents, 2–30 s model calls, 0–120 s idle: model calls/s (2 runs each)	130 / 130	139 / 142
Same: tool result to next model request, p50 / p95	1.05 / 5.7 s, 0.45 / 6.7 s	36 / 99 ms, 36 / 93 ms
Same: message to first model request, p50 / p95	0.63 / 2.9 s, 0.35 / 3.0 s	113 / 214 ms, 108 / 199 ms
Same: turn duration, p50	29.0 / 28.0 s	25.5 / 26.0 s
Same: Postgres CPU per model call	26.0 / 30.4 ms	5.5 / 6.1 ms

With no idle time, main keeps Postgres busy on every core and its latency swings from run to run. This branch keeps about 1,900 of the 2,000 agents in a model call, so its throughput is close to what the agent count allows. With realistic idle time, this branch uses about a fifth of main's Postgres CPU per call, and every latency improves.

With realistic idle time, this branch scales linearly to about 10,000 agents (271 model calls/s, tool result to next request 371 ms p50). Past that, the laptop runs out of CPU and memory for the workers, the fake model and the kernel, while Postgres stays under two cores. Raising claim loops to 16, or connections per worker to 32, doesn't raise that ceiling.

Behavior notes

• Fairness. An agent can keep its worker for up to three model calls or 30 seconds. An agent that stays continuously ready keeps its queue position instead of moving to the back after unrelated updates.
• Queue rows. An agent holding a runtime lease has no queue row. Release and reaping restore it atomically.
• Lineage. Continuations, retries and compactions carry the opening inputs of the context they continue.
• Archive. Archiving stops in-flight and admitted work before marking the agent archived.
• Webhooks. Deleting a project or organization sends no event webhooks for it, including events produced by the deletion. This is now documented.

Pending input cap

An agent holds at most 50 pending content inputs, counting queued and steering together.

• Rejection. A new submission past the cap gets agent_backlog_full (HTTP 409). The check runs under the agent lock after the idempotency lookup, so an identical retry still succeeds, and it reads at most 50 index entries.
• Freeing capacity. Admission and cancellation free capacity. Reordering and delivery-mode changes don't use any.
• Internal producers:
  • A child's results, failures and questions to its parent are always accepted, and still count toward the cap.
  • Integration inbox deliveries stay pending and retry under the inbox's existing policy. If retries run out, the sender is told the agent's queue was full.
  • A cron occurrence that hits a full agent is skipped and recorded.
  • A parent's message to a full subagent fails with a tool error the parent can read.
• Attachments. Attachments on a submission are uploaded as prepared blobs, and their artifact rows commit in the same transaction as the input. A rejected submission leaves no files, and its blobs are deleted unless the commit outcome is unknown.
• Composer. The web composer shows the error and restores the exact unsent draft.

Deployment

Migration 052 requires a writer-stop maintenance window:

1. Stop the API, workers, maintenance and daemon-report handlers.
2. Run migrations.
3. Start the new binaries.

The migration builds every existing agent's execution state from its history, validates every agent, and then drops the old frontier functions. If it finds malformed or ambiguous history, it fails and rolls back with the offending agent IDs. Runtime leases and in-flight external work are preserved; normal recovery handles interrupted work after startup. The details are in docs/self-hosting/deployment.mdx.

Testing

• Build, vet, make golangci-lint, make verify-static, and make sqlc-check sql-rules sqlc-vet sqlc-vet-db migration-check migration-compat-check.
• make test-integration-storage, test-integration-runtime and test-integration-httpapi.
• The core, worker, kernel and tools packages with -race.
• All deterministic service E2Es (35 tests), with none skipped.
• New coverage:
  • precedence and boundary tests for the selector;
  • command tests for every transition family, including custody, duplicate and out-of-order reports, late old-turn results, and parent/child effects in both lock orders;
  • rollback and lock-plan expansion tests, including bounded expansion and a claim that keeps its original agent across a restart;
  • a test that fused acceptance creates no MultiXacts, and a claim-plan test that reads a bounded number of buffers with 5,000 ready and 5,000 leased agents;
  • lease-delete and backlog plan tests over long histories, a test that context inserts leave the org and project rows unlocked, and scope tests that reject cross-org and cross-project rows;
  • a cutover rollback test that injects a failure after all DDL and checks the released schema is restored;
  • backlog cost tests with 10,000 pending and 100,000 historical inputs, plus concurrent-move, stale-anchor and exhausted-gap tests;
  • pending-cap tests: the 50th input is accepted and the 51st rejected, queued and steering count together, concurrent submissions race for the last slot, idempotent replay at capacity, capacity reopening, each internal producer, the API error contract and the composer draft;
  • attachment tests: rejected, racing, validation-failed and deferred-rejected submissions leave no artifacts; an idempotent replay at capacity returns the original; an unknown commit outcome keeps the blobs;
  • a test that a unit cannot be opened outside a retry boundary;
  • worker tests with the continuation budget expiring before, during and after preparation, and with the next step's preparation failing immediately or at commit;
  • a test that eight sibling subagents prepare while their parent row is locked elsewhere;
  • statement-count tests that pin round trips and statements per transition, and constant round trips for 1,000-agent transactions;
  • generic-plan tests;
  • 10,000-agent linearity tests;
  • lock-order race tests;
  • upgrade tests from the released schema, including malformed-history rejection and rollback;
  • a catalog test proving the removed functions are absent;
  • regression tests for each fix above.

Excluding the one-time migration, handwritten production code is about 200 lines smaller than on main, and generated code is about 2,850 lines smaller. Most of this PR's added lines are tests.

🤖 Generated with Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 170dead. Configure here.}

[cursor]

sqlc-vet-disable lacks rationale comments

Low Severity

New @sqlc-vet-disable annotations on ExecutionModelRevision, ListExecutionIntegrationTargetAgents, and ClearExecutionIntegrationTargets omit an adjacent rationale for which vet rule is disabled and why that is safe. Existing disables on main always include that comment.

Additional Locations (2)

• internal/storage/internal/agentexecution/queries/commands_agents.sql#L26-L28
• internal/storage/internal/agentexecution/queries/commands_agents.sql#L31-L33

 

^{Reviewed by Cursor Bugbot for commit 170dead. Configure here.}

[cursor]

This PR is too large for Bugbot to review. It changes 59,580 lines and 3,060,629 characters. Split the change into smaller pull requests to get a review.

[github-actions]

React Doctor found no new issues. 🎉

_{Reviewed by React Doctor for commit 6e5a8b1.}