Skip to content

perf: index lease references and scope hot inserts to the agent row - #646

Closed
ksarangmath wants to merge 1 commit into
perf/agent-execution-scalingfrom
perf/hot-path-schema
Closed

ksarangmath wants to merge 1 commit into
perf/agent-execution-scalingfrom
perf/hot-path-schema

Conversation

@ksarangmath

Copy link
Copy Markdown
Contributor

Summary

Stacked on #631. Migration 054 removes two per-step costs found by the agent-execution load test, and drops redundant foreign keys on hot insert paths.

  • Lease release scanned all tool-call history. tool_calls.runtime_lock_id references agent_runtime_locks(id), but its only index was partial (state = 'running') and led with agent_id. So the foreign-key check on every lease delete was a sequential scan of tool_calls, and every release got slower as history grew.
    • At 100,000 tool calls, the check read 3,543 buffers and the delete took 5.8 ms.
    • With tool_calls(runtime_lock_id) WHERE runtime_lock_id IS NOT NULL, it reads 2 buffers and takes 0.1 ms, regardless of history.
    • This has been present since migration 5.
  • The execution backlog query read every input the agent ever had. A partial index on received content inputs takes it from 1,182 buffers to 6.
  • Hot inserts locked shared rows. Every model-context insert took a key-share lock on the org row and the project row. Those rows are shared by every agent in them, so busy orgs created about one MultiXact per model call.
    • The org and project references on model_call_contexts, processes and process_actions become one org-qualified reference to the agent row: (org_id, project_id, agent_id) → agents(org_id, project_id, id). The writer already holds that row locked.
    • agent_inputs.project_id → projects is dropped, because (project_id, agent_id) → agents already guarantees it.
    • Config and configured-model-revision references stay; they protect real integrity.

Cross-org and cross-project rows are still rejected by the database for every replaced constraint. No Go code or queries change.

Migration

The migration is non-transactional so its indexes can be built concurrently. In order:

  1. Build the indexes.
  2. Add each replacement foreign key NOT VALID.
  3. Validate each replacement.
  4. Drop the old constraints, only once every replacement has validated.

Validation holds no exclusive table lock; adding and dropping constraints take brief metadata locks. The migration is not atomic. If it is interrupted mid-way, an invalid concurrent index may need dropping before a rerun.

Testing

  • New tests, each failing before the migration and passing after:
    • TestExecutionLeaseDeleteHistoryBound: lease deletes use the index, and their cost stays flat from 1,000 to 100,000 tool calls. A live reference still blocks the delete.
    • TestExecutionHotPathPlans: the backlog uses the new index within a bounded number of buffers, including under a generic plan.
    • TestContextInsertAvoidsSharedScopeLocks: overlapping context inserts create no MultiXact on the org or project row.
    • TestExecutionHotPathScopeConstraints: wrong-org and wrong-project rows are rejected for contexts, processes, process actions and inputs.
    • TestHotPathMigrationConcurrentBuildAndValidation: on 50,000 agents and contexts, validation lets concurrent writers proceed.
  • Build, vet, make golangci-lint, make verify-static, and the sqlc, migration and migration-compat targets.
  • Storage, runtime and httpapi integration suites; race suites; all 35 service E2Es with none skipped.

The end-to-end load benchmark is still to run, on a quiet machine. The lease fix matters most on long histories, which the load test doesn't build up, because it starts from a fresh database every run.

🤖 Generated with Claude Code

- Index tool_calls.runtime_lock_id so the foreign-key check on every lease
  delete no longer scans all tool-call history.
- Index received content inputs per agent for the execution backlog.
- Replace the org and project references on model contexts, processes and
  process actions with one org-qualified reference to the agent row, which
  the writer already holds locked, and drop the redundant project reference
  on agent inputs. Inserts stop key-share locking the shared org and project
  rows. Config and model-revision references stay.
- Build indexes concurrently and validate every replacement before dropping
  the old constraints.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the perf PRs of type perf (auto-applied from title) label Oct 12, 2026
@ksarangmath

Copy link
Copy Markdown
Contributor Author

Folding this into #631: the changes move into its cutover migration 052.

@ksarangmath
ksarangmath deleted the perf/hot-path-schema branch October 12, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

perf PRs of type perf (auto-applied from title)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant