Repository navigation
Commit d1b9d21
authored
feat(engine): capture per-control execution duration [SAO-18023] (#283)
## Summary
- Implements [SAO-18023: Capture per-control execution duration in
observability events](https://splunk.atlassian.net/browse/SAO-18023).
- Measures each completed control's condition evaluation with a
monotonic clock and carries its duration through the evaluation response
and Python SDK observability event.
<img width="1920" height="1008" alt="image"
src="https://github.com/user-attachments/assets/a75344d1-6849-48be-b62a-d80c4d829d2b"
/>
Both DevStacks ran the same control test.
On the left, metrics.duration_ns is 0 because the previous version did
not provide a per-control duration; it does not mean the control took
zero time.
On the right, PR #283 supplies the measured duration through the SDK and
Agent Observability: 96,576,040 ns, displayed by Tiffany’s UI as 96.58
ms on the control row.
## Scope
- User-facing/API changes: optional top-level
`ControlMatch.execution_duration_ms` in `matches`, `non_matches`, and
`errors`; TypeScript clients expose `executionDurationMs`.
- Internal changes: engine timing, Python SDK event mapping, decorator
serialization/debug logging, contract documentation, and regression
coverage.
- Timing includes condition traversal, visited leaves, evaluator work,
and waiting for an evaluator concurrency slot. Control loading and
request/response overhead are outside the measurement.
- Completed results, including handled evaluation errors/timeouts, have
nonnegative durations. Precheck errors retain `null`; cancelled
evaluations without results stay omitted. Older responses without the
field remain valid, and measured zero stays distinct from unknown
timing.
- Out of scope: the FE work tracked in
[SAO-18028](https://splunk.atlassian.net/browse/SAO-18028), Orbit
ingestion changes, production deployment, and load/capacity benchmarks.
## Risk and Rollout
- Risk level: low. The response field is optional and evaluation
decisions are unchanged.
- Rollback plan: revert this commit; existing SDK/model consumers
continue to accept responses without the duration field.
- Follow-up: Python Orbit OTEL v2 ingestion currently drops the explicit
`agent_control.execution_duration_ms` attribute. Its passthrough needs a
separate change for that AO path. This PR does not establish end-to-end
AO UI behavior.
## Testing
- [x] Added or updated automated tests: per-control
outcomes/errors/timeouts, cancellation/prechecks, legacy compatibility,
HTTP serialization/error redaction, SDK event/decorator propagation,
OTEL conversion, and TypeScript response mapping.
- [x] Ran the repository's `make prepush` gate locally and full `make
check` in [Python
CI](https://github.com/agentcontrol/agent-control/actions/runs/37391050258/job/112035944219);
both passed. Local focused HTTP tests require no database; CI also
exercised the full database-backed suite with Postgres.
- Python checks: 76 model tests, 138 engine tests, 151 relevant Python
SDK tests, and 3 HTTP contract tests passed (368 total).
- TypeScript checks: `make sdk-ts-lint sdk-ts-typecheck sdk-ts-test
sdk-ts-build` passed; 15 tests passed.
- Ruff lint/format checks passed for all four new Python test files;
`git diff --check` passed.
- Local OpenAPI generation completed; local Speakeasy generation
required an account login. CI subsequently regenerated the SDK
successfully and confirmed the committed generated model matches exact
generator output. The complete [TypeScript SDK CI
job](https://github.com/agentcontrol/agent-control/actions/runs/37391050258/job/112035944315),
including `make sdk-ts-generate-check` and naming checks, passed.
- [x] Manually verified behavior in an isolated DevStack; see evidence
below.
## DevStack validation — 2026-10-05
- Isolated GCP DevStack: `josjeon-sao18023-pr283`, UUID
`c94170b6-7a6b-4f7e-945c-8351693d41bc`, standard / capacity_s.
- BE source: `3690373c3c6a182a2647bd3b894a648bb6fc66bd`; all six Agent
Control package refs in the Orbit integration branch pin this immutable
commit.
- Verified Orbit checkout in the [successful image
build](https://github.com/rungalileo/galileo-infra-automation/actions/runs/37394261717/job/112046360923):
`6b8b3a8aa98b96386a7738f06b0ded75093e4b31`.
- Image tag:
`gcr.io/rungalileo-dev/agent-control:codex-sao-18023-devstack-pr283`;
pushed manifest digest:
`sha256:8974bf0f30e206b547e824731d3995b5d311233a8d3c2a3d39283dd9d125ece2`.
-
[Provision](https://github.com/rungalileo/galileo-infra-automation/actions/runs/37392938858)
and [custom
update](https://github.com/rungalileo/galileo-infra-automation/actions/runs/37394261717)
workflows succeeded. Aegis subsequently reported `running`, the custom
Agent Control version, and zero failed/pending pods. The digest is
build/push evidence; an in-pod digest/source-hash inspection was not
performed.
| Requirement | Deployed test evidence |
| --- | --- |
| Runtime auth / readiness | API, PostgreSQL and AC health returned 200.
Missing auth returned 401 requiring `Authorization`; Bearer
runtime-token exchange and evaluation succeeded. |
| Match, non-match, handled error timing | Two successful evaluation
responses: one raw HTTP call and one public exact-head Python SDK call.
Each had one entry per category with finite, nonnegative top-level
`execution_duration_ms` (six completed entries total). |
| Safe errors / scope skip | Internal evaluator error was sanitized,
deny error failed closed, and post-only control was absent during the
pre stage. |
| SDK propagation | All three typed results and three recorded events
preserved the exact values captured from that same SDK HTTP response.
Timing stayed outside evaluator result/event metadata. |
| OTEL mapping | Local conversion preserved the duration attribute and
exact nanosecond span delta. Events were recorded in a local sink; AO
ingestion was not exercised. |
| Legacy compatibility | Local omitted/null/zero replays passed; these
are compatibility checks, not live timing measurements. |
| Orbit integration | Dependency compilation, module-lock generation, 24
launcher tests and mypy passed. |
Observed functional samples (separate requests, milliseconds):
| Category | Raw API | SDK |
| --- | ---: | ---: |
| Match | 0.130080 | 0.114650 |
| Non-match | 0.250990 | 0.293050 |
| Handled error | 90.031554 | 88.694064 |
The probe completed with exit 0 and `status=passed`. Successful SDK
responses were counted; total HTTP attempts were not counted because its
authentication path can retry. This is bounded functional validation,
not a load test or latency/capacity acceptance result. Deployed
TypeScript/FE, Luna/external evaluators, and AO ingestion remain outside
this validation. Timeout/precheck/cancellation coverage is automated,
not live here.
The first signup attempt rejected a `+` email alias before creating an
account. Correcting only the probe's address format allowed bootstrap
and validation to pass. The probe received successful deletion
acknowledgements for its own bindings/controls, project and temporary
API key; no absence GET readback was performed.
The dedicated stack was destroyed after testing with
`retain_data=false`. Final Aegis readback is `destroyed`, with no last
error; [teardown
workflow](https://github.com/rungalileo/galileo-infra-automation/actions/runs/37395484152).
## Checklist
- [x] Linked issue/spec.
- [x] Updated docs for the new public field and timing boundaries.
- [x] Included required follow-up tasks and completed SDK generation
verification.1 parent db9c667 commit d1b9d21
12 files changed
Lines changed: 725 additions & 1 deletion
File tree
- engine
- src/agent_control_engine
- tests
- models
- src/agent_control_models
- tests
- sdks
- python
- src/agent_control
- tests
- typescript
- src/generated/models
- tests
- server/tests
- ui/src/core/api/generated
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
201 | 202 | | |
202 | 203 | | |
203 | 204 | | |
| 205 | + | |
204 | 206 | | |
205 | 207 | | |
206 | 208 | | |
| |||
673 | 675 | | |
674 | 676 | | |
675 | 677 | | |
| 678 | + | |
676 | 679 | | |
677 | 680 | | |
678 | 681 | | |
| |||
701 | 704 | | |
702 | 705 | | |
703 | 706 | | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
704 | 710 | | |
705 | 711 | | |
706 | 712 | | |
| |||
753 | 759 | | |
754 | 760 | | |
755 | 761 | | |
| 762 | + | |
756 | 763 | | |
757 | 764 | | |
758 | 765 | | |
| |||
776 | 783 | | |
777 | 784 | | |
778 | 785 | | |
| 786 | + | |
779 | 787 | | |
780 | 788 | | |
781 | 789 | | |
| |||
792 | 800 | | |
793 | 801 | | |
794 | 802 | | |
| 803 | + | |
795 | 804 | | |
796 | 805 | | |
797 | 806 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
20 | 34 | | |
21 | 35 | | |
22 | 36 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
957 | 957 | | |
958 | 958 | | |
959 | 959 | | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
960 | 965 | | |
961 | 966 | | |
962 | 967 | | |
| |||
0 commit comments