Skip to content

test: make the durable approval timeout test deterministic - #2029

Merged
4pmtong merged 3 commits into
mainfrom
fix/deterministic-approval-budget-test
Oct 9, 2026
Merged

4pmtong merged 3 commits into
mainfrom
fix/deterministic-approval-budget-test

Conversation

@4pmtong

@4pmtong 4pmtong commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

test_durable_approval_wait_outlives_agent_execution_timeout sometimes fails on CI. It raises CancelledError at await waiter, which surfaces as TimeoutError from async with ActiveExecutionTimeout(0.02).

The test gave the agent a 20 ms budget on the real event-loop clock. Before the approval wait can pause that budget, authorize_tool_checkpoint evaluates the permission policy and writes the approval to SQLite. On a slow runner, that work can take longer than 20 ms. The pause then saves a remaining budget of zero. When the wait resumes after the decision, the timeout fires right away and cancels the test task.

Product behavior is correct here. Active work before the wait should count against the budget. Only the test's timing assumption is wrong.

Changes

  • During the test, the event-loop clock is driven by hand: it is patched with monkeypatch.context() and restored afterwards. Only the simulated human wait moves the clock, so SQLite writes and task scheduling no longer decide whether the budget runs out.
  • The test uses a 30-minute agent budget and a one-hour decision delay. The delay stays below the approval expiry, which the test asserts from the ASK payload.
  • The test now asserts directly what it is meant to prove:
    • the budget is paused when the ASK is published;
    • the full budget is still left after the hour of waiting;
    • the approved decision is returned to the tool loop;
    • the budget is restored on resume, and the timeout never expires.

No product code changes.

Testing

  • uv run pytest tests/app/permission_policy: 152 passed.
  • The new test run 200 times in a row: 200 passed.
  • Under CPU load (28 busy loops on a 14-core machine, with 6 parallel copies running 200 times each), the previous version failed 3 of 1,200 runs with the same signature as CI. The new version passed 1,200 of 1,200.
  • With a 30 ms or a 2 s delay injected into policy evaluation, the previous version failed every run and the new version passed.
  • The new test fails in each of these regression scenarios:
    • the approval wait no longer pauses the budget;
    • the timeout ignores pause and resume;
    • resuming re-arms the original deadline and charges the wait.
  • ruff check, ruff format --check, and git diff --check pass.

@4pmtong
4pmtong marked this pull request as ready for review October 9, 2026 04:07
@4pmtong
4pmtong merged commit 61abc66 into main Oct 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant