feat(engine): capture per-control execution duration [SAO-18023]

[josjeon]

Summary

• Implements SAO-18023: Capture per-control execution duration in observability events.
• Measures each completed control's condition evaluation with a monotonic clock and carries its duration through the evaluation response and Python SDK observability event.

Both DevStacks ran the same control test. On the left, metrics.duration_ns is 0 because the previous version did not provide a per-control duration; it does not mean the control took zero time. On the right, PR #283 supplies the measured duration through the SDK and Agent Observability: 96,576,040 ns, displayed by Tiffany’s UI as 96.58 ms on the control row.

Scope

• User-facing/API changes: optional top-level ControlMatch.execution_duration_ms in matches, non_matches, and errors; TypeScript clients expose executionDurationMs.
• Internal changes: engine timing, Python SDK event mapping, decorator serialization/debug logging, contract documentation, and regression coverage.
• Timing includes condition traversal, visited leaves, evaluator work, and waiting for an evaluator concurrency slot. Control loading and request/response overhead are outside the measurement.
• Completed results, including handled evaluation errors/timeouts, have nonnegative durations. Precheck errors retain null; cancelled evaluations without results stay omitted. Older responses without the field remain valid, and measured zero stays distinct from unknown timing.
• Out of scope: the FE work tracked in SAO-18028, Orbit ingestion changes, production deployment, and load/capacity benchmarks.

Risk and Rollout

• Risk level: low. The response field is optional and evaluation decisions are unchanged.
• Rollback plan: revert this commit; existing SDK/model consumers continue to accept responses without the duration field.
• Follow-up: Python Orbit OTEL v2 ingestion currently drops the explicit agent_control.execution_duration_ms attribute. Its passthrough needs a separate change for that AO path. This PR does not establish end-to-end AO UI behavior.

Testing

• Added or updated automated tests: per-control outcomes/errors/timeouts, cancellation/prechecks, legacy compatibility, HTTP serialization/error redaction, SDK event/decorator propagation, OTEL conversion, and TypeScript response mapping.
• Ran the repository's make prepush gate locally and full make check in Python CI; both passed. Local focused HTTP tests require no database; CI also exercised the full database-backed suite with Postgres.
• Python checks: 76 model tests, 138 engine tests, 151 relevant Python SDK tests, and 3 HTTP contract tests passed (368 total).
• TypeScript checks: make sdk-ts-lint sdk-ts-typecheck sdk-ts-test sdk-ts-build passed; 15 tests passed.
• Ruff lint/format checks passed for all four new Python test files; git diff --check passed.
• Local OpenAPI generation completed; local Speakeasy generation required an account login. CI subsequently regenerated the SDK successfully and confirmed the committed generated model matches exact generator output. The complete TypeScript SDK CI job, including make sdk-ts-generate-check and naming checks, passed.
• Manually verified behavior in an isolated DevStack; see evidence below.

DevStack validation — 2026-10-05

• Isolated GCP DevStack: josjeon-sao18023-pr283, UUID c94170b6-7a6b-4f7e-945c-8351693d41bc, standard / capacity_s.
• BE source: 3690373c3c6a182a2647bd3b894a648bb6fc66bd; all six Agent Control package refs in the Orbit integration branch pin this immutable commit.
• Verified Orbit checkout in the successful image build: 6b8b3a8aa98b96386a7738f06b0ded75093e4b31.
• Image tag: gcr.io/rungalileo-dev/agent-control:codex-sao-18023-devstack-pr283; pushed manifest digest: sha256:8974bf0f30e206b547e824731d3995b5d311233a8d3c2a3d39283dd9d125ece2.
• Provision and custom update workflows succeeded. Aegis subsequently reported running, the custom Agent Control version, and zero failed/pending pods. The digest is build/push evidence; an in-pod digest/source-hash inspection was not performed.

Requirement	Deployed test evidence
Runtime auth / readiness	API, PostgreSQL and AC health returned 200. Missing auth returned 401 requiring Authorization; Bearer runtime-token exchange and evaluation succeeded.
Match, non-match, handled error timing	Two successful evaluation responses: one raw HTTP call and one public exact-head Python SDK call. Each had one entry per category with finite, nonnegative top-level execution_duration_ms (six completed entries total).
Safe errors / scope skip	Internal evaluator error was sanitized, deny error failed closed, and post-only control was absent during the pre stage.
SDK propagation	All three typed results and three recorded events preserved the exact values captured from that same SDK HTTP response. Timing stayed outside evaluator result/event metadata.
OTEL mapping	Local conversion preserved the duration attribute and exact nanosecond span delta. Events were recorded in a local sink; AO ingestion was not exercised.
Legacy compatibility	Local omitted/null/zero replays passed; these are compatibility checks, not live timing measurements.
Orbit integration	Dependency compilation, module-lock generation, 24 launcher tests and mypy passed.

Observed functional samples (separate requests, milliseconds):

Category	Raw API	SDK
Match	0.130080	0.114650
Non-match	0.250990	0.293050
Handled error	90.031554	88.694064

The probe completed with exit 0 and status=passed. Successful SDK responses were counted; total HTTP attempts were not counted because its authentication path can retry. This is bounded functional validation, not a load test or latency/capacity acceptance result. Deployed TypeScript/FE, Luna/external evaluators, and AO ingestion remain outside this validation. Timeout/precheck/cancellation coverage is automated, not live here.

The first signup attempt rejected a + email alias before creating an account. Correcting only the probe's address format allowed bootstrap and validation to pass. The probe received successful deletion acknowledgements for its own bindings/controls, project and temporary API key; no absence GET readback was performed.

The dedicated stack was destroyed after testing with retain_data=false. Final Aegis readback is destroyed, with no last error; teardown workflow.

Checklist

• Linked issue/spec.
• Updated docs for the new public field and timing boundaries.
• Included required follow-up tasks and completed SDK generation verification.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[namrataghadi-galileo]

Regenerate the UI OpenAPI types — controls.py (line 960) adds execution_duration_ms to the public ControlMatch response, but the checked-in UI type at api-types.ts (line 1649) still omits it.

[namrataghadi-galileo]

Lets also make sure the ts-sdk is also updated.

[josjeon]

Fixed in bf3ace8. I updated the checked-in UI OpenAPI type so ControlMatch now includes the optional nullable execution_duration_ms field, matching the public API response schema. Prettier and the generated OpenAPI contract validation both pass.