Skip to content

Commit a17d980

Browse files
mchwangclaude
andcommitted
Address F1 contract review round 26
- Releasing an unknown preparation requires a verified no-open-files check, not quiescence. - D start must return at once and run Docker setup asynchronously (new prerequisite). - F2 extracts a completed attempt's work before the terminal write and storage removal. - Suggestion cancel after a restart relies on D recovery stopping the provider. - The task budget starts at the first attempt's admission, so it covers preparation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 621dfb8 commit a17d980

1 file changed

Lines changed: 13 additions & 5 deletions

File tree

‎docs/implementation/runner-lifecycle.md‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -124,9 +124,9 @@ A `pending` attempt has no `settled` promise, so the slot rules need a separate
124124

125125
- Preparation receives an `AbortSignal` from the job. Stopping a pending attempt aborts that signal with the first reason.
126126
- **Preparation files have a durable owner.** The attempt ID is used as a directory name only after it passes the UUID check under "Terms used", so it cannot contain `/` or `..`. Before preparation creates anything, F uses the fixed attempt directory `<runner root>/<runner owner token>/attempts/<attemptId>/`, and passes it as the parent to the clone helper. The helper names its clone with `mkdtemp`, so the fixed parent is what makes the clone findable after a crash. **Today's `createTaskClone` (`git/clone.ts`) runs git with synchronous `execFileSync`.** It cannot be aborted and would block the event loop, so F1 cannot use it for preparation as it is. D, which owns `git/clone.ts`, must add an asynchronous variant that takes an `AbortSignal`, spawns git in its own process group, and exposes that group's ID (third prerequisite under "Shutdown").
127-
- **Preparation processes have a durable owner.** Right after spawning a preparation child, in the same synchronous turn, F saves the process group ID and the child's start time on the attempt row (`preparation_pgid`, `preparation_started_at`). A process group outlives the runner if the runner crashes, so startup recovery uses these fields to stop it (step 4). To close the gap between spawn and that write, F first saves `preparation_started_at` with a null `preparation_pgid` (**preparation starting**), then spawns, then saves the group ID. If recovery finds an attempt that is marked as starting but has no group ID, a child may exist that nobody can identify. Recovery then **fails closed**: it does not open admission, it leaves that attempt directory in place, and the CLI names the attempt and its directory. The user must confirm that no process is using it, with `--release-preparation <attemptId>`. That flag removes the directory only if its contents stay unchanged over a 10-second check. The path comes only from IDs that are already saved, so a crash cannot leave an unknown path. The job removes the directory when the attempt ends. Startup recovery handles the rest (step 4).
127+
- **Preparation processes have a durable owner.** Right after spawning a preparation child, in the same synchronous turn, F saves the process group ID and the child's start time on the attempt row (`preparation_pgid`, `preparation_started_at`). A process group outlives the runner if the runner crashes, so startup recovery uses these fields to stop it (step 4). To close the gap between spawn and that write, F first saves `preparation_started_at` with a null `preparation_pgid` (**preparation starting**), then spawns, then saves the group ID. If recovery finds an attempt that is marked as starting but has no group ID, a child may exist that nobody can identify. Recovery then **fails closed**: it does not open admission, it leaves that attempt directory in place, and the CLI names the attempt and its directory. The user must stop the unknown process and then confirm with `--release-preparation <attemptId>`. A quiet directory proves nothing, because a blocked process may resume later, so codeboost does not rely on it. Before removing anything, it checks that no process has a file open or a working directory inside the attempt directory, using `lsof +D` on macOS or `/proc/*/fd` and `/proc/*/cwd` on Linux. If a process is found, or the check cannot run, it refuses and keeps admission closed. The path comes only from IDs that are already saved, so a crash cannot leave an unknown path. The job removes the directory when the attempt ends. Startup recovery handles the rest (step 4).
128128
- An `AbortSignal` does not stop a child process by itself. Every preparation subprocess (for example `git clone`) is started in its own process group, owned by the job. On abort, F sends `SIGTERM` to the group, waits a fixed grace period (5 seconds), then sends `SIGKILL`, and awaits the `close` event. Preparation settles only after that. A child that ignores `SIGTERM` is therefore still bounded by the kill. Preparation never leaves work running after it settles.
129-
- Preparation counts against two limits, so it cannot hold a slot forever. If the **task time budget** runs out first, the job records the first reason `time-limit`, then aborts the signal; the pending row ends `cancelled` and the task goes to `needs human`, as for a running attempt. If the **attempt deadline** passes first, the job aborts the signal with `timeout` and records no first reason; the row ends `failed` "Timed out while preparing".
129+
- Preparation counts against two limits, so it cannot hold a slot forever. The task budget is already running during preparation: the admission transaction of the task's first attempt moves the task to `running` and sets `budget_deadline` (see "Proposed storage additions"), before preparation starts. If the **task time budget** runs out first, the job records the first reason `time-limit`, then aborts the signal; the pending row ends `cancelled` and the task goes to `needs human`, as for a running attempt. If the **attempt deadline** passes first, the job aborts the signal with `timeout` and records no first reason; the row ends `failed` "Timed out while preparing".
130130
- Shutdown aborts every preparation signal in step 4.
131131

132132
| Case | What F does | When the slot is freed |
@@ -141,6 +141,8 @@ A `pending` attempt has no `settled` promise, so the slot rules need a separate
141141

142142
In every case, a failed terminal write leaves an unresolved marker (see "Slots and concurrency", rule 5).
143143

144+
**Launch setup must not block the server.** The D4 adapters do their Docker setup synchronously inside the start call: `createVendorNetwork`, profile creation and `startProfileInvocation` use `execFileSync` and `spawnSync` (`agents/network/network.ts`, `agents/container/`). While that runs, the event loop cannot set `closing`, cancel the launch, or begin the HTTP drain. **Prerequisite:** D's start call must return its handle at once and do the Docker setup asynchronously inside that handle's lifecycle. Then `cancel()` works during setup, and `settled` resolves only after setup cleanup. It is listed with the other D prerequisites under "Shutdown".
145+
144146
**Rule for D.** D's start call must keep this shape: it either throws with nothing left running, or it returns a handle that settles only after everything it started has stopped. An adapter that needs an asynchronous start must still follow this rule. Changing it is a change to D's contract.
145147

146148
### Rules for the running state
@@ -203,7 +205,7 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
203205
2. Validate the output (schema, size, file-scope audit for writable attempts). Do not await anything between the last check and the transaction.
204206
3. In one `Store` transaction: confirm that the attempt ID is the task's current attempt, the state is `running`, there is no first reason, the captured context is still current, the task is not closed, and `tasks.cancel_requested` is null (the same guard as the `running → completed` row). Do not compare the task's state version here; the attempt's own transitions have increased it. Then write `completed` and the result, and increase the state version.
205207
4. If step 3 refuses, reread durable state and settle the row by the rules above. Keep the original diagnostic.
206-
5. For a writable attempt, call `removeTaskFilesystems` after the terminal write, whatever the outcome. Free the slot only after it succeeds. If it fails, keep an unresolved marker, and startup recovery removes the storage.
208+
5. For a completed writable attempt, F2's result extraction (committing the change and recording its ledger entry) is part of step 2 above. It runs from the still-present task storage before the step-3 transaction. If step 3 then refuses, F2's rules discard the extracted commit. Then, for every writable attempt, call `removeTaskFilesystems` after the terminal write, whatever the outcome. Free the slot only after it succeeds. If it fails, keep an unresolved marker, and startup recovery removes the storage.
207209

208210
**Irreversible actions.** Before each commit, push, PR open or merge, re-read the task's state version, the plan's `review_version` (which `saveReview` and `addReviewNote` advance) and the coordinator's `closing` flag **after the final await**. Stop if any of them changed. For a merge this means an approval or choice edit made during the final GitHub check blocks the merge. A check made before an await does not count. (This follows the AGENTS.md rules on guarded external actions.)
209211

@@ -247,6 +249,8 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
247249

248250
This belongs to D (D5 or a D follow-up), not F. It changes `agents/contract.ts`, so it goes through D's contract tests.
249251

252+
**Also before F1 merges: asynchronous launch.** D's start call must return a handle immediately and run its Docker setup asynchronously and abortably (see "Launch setup must not block the server").
253+
250254
**Third prerequisite before F1 merges: abortable preparation helpers.** Both preparation helpers are synchronous today. `createTaskClone` runs git with `execFileSync`, and `prepareTaskFilesystems` runs `docker` with `execFileSync` and `spawnSync` (`agents/container/storage.ts`). Either would block the event loop, and cancel or shutdown could not abort it. D must add asynchronous variants of both that take an `AbortSignal`, spawn their subprocesses in their own process group, expose the group ID, and settle only after those processes have exited. Storage allocation must also label its volumes with the runner owner and attempt ID before it returns (see "Recovered storage handles"), so that storage allocated just before a crash is found by recovery.
251255

252256
**Prerequisite before F2's writable attempts: a bounded task-volume export.** D must add a bounded diff export over the task volumes (see "Partial output and interrupted rebases"). F1 does not need it, because F1 has no writable attempts yet.
@@ -299,7 +303,7 @@ F1 owns the production planning endpoints that G4 needs. They wrap E3's coordina
299303
| `POST /api/plan/import` | `Store.importRevision` | expected revision |
300304
| `POST /api/plan/suggestions` | E3 coordinator start (`beginSuggestions`) | expected revision and snapshot |
301305
| `GET /api/plan/suggestions/:id` | `Store.getSuggestions` | reads only the request row |
302-
| `POST /api/plan/suggestions/:id/cancel` | `pending`: `handle.cancel(reason)` on the `SuggestionHandle` that `SuggestionCoordinator.start()` returned. E3's coordinator has no cancel-by-ID method, so F keeps a map from request ID to handle, adds each handle when it starts the request, and removes it only after the handle's outcome settles. If the request is `pending` but F holds no handle (for example after a restart), F calls `Store.cancelSuggestions`: no provider runs in this process, so there is nothing to stop. `ready`: `Store.cancelSuggestions` (dismiss), because no invocation is running. | request is `pending` or `ready`, as `Store.cancelSuggestions` already allows |
306+
| `POST /api/plan/suggestions/:id/cancel` | `pending`: `handle.cancel(reason)` on the `SuggestionHandle` that `SuggestionCoordinator.start()` returned. E3's coordinator has no cancel-by-ID method, so F keeps a map from request ID to handle, adds each handle when it starts the request, and removes it only after the handle's outcome settles. If the request is `pending` but F holds no handle (for example after a restart), F calls `Store.cancelSuggestions`. It is safe to do so because startup recovery has already stopped the crashed process's provider. A live planning provider runs only through D (G4 after D5), in a container labelled with this database's runner owner, and D's `recoverLeftovers` (startup step 2) stops it before admission opens. Until D5 there is no live planning provider at all ("Planning agent not available yet"). If a planning provider ever runs outside D, this endpoint must refuse such a request instead, until startup has reconciled it. `ready`: `Store.cancelSuggestions` (dismiss), because no invocation is running. | request is `pending` or `ready`, as `Store.cancelSuggestions` already allows |
303307
| `POST /api/plan/suggestions/:id/apply` | `Store.applySuggestion` | request `ready` and bound to the current revision and snapshot |
304308

305309
Live planning invocation waits for D5. Until then, the server returns "Planning agent not available yet" instead of using a fake provider.
@@ -373,7 +377,7 @@ A merged v5 task also gets its `task-closed` event, keyed by the merge attempt I
373377

374378
| Table | Key columns |
375379
|---|---|
376-
| `tasks` | `plan_key` (primary key, references `plans(key)`), `status`, `requeue_pending` (boolean), `rebase_in_progress` (JSON or null; see "Partial output and interrupted rebases"), `budget_deadline` (absolute time the task budget ends; null until the budget starts, which is the task's first move to `running`, when it is set to that time plus the configured budget), `cancel_requested` (action ID or null), `state_version`, `context_generation`, `assignment_id`, `referenced_code_hash`, `current_attempt_id` (nullable; the composite foreign key `(plan_key, current_attempt_id)` references `attempts(plan_key, id)`, so a task can only point at its own attempt), `created_at`, `updated_at` |
380+
| `tasks` | `plan_key` (primary key, references `plans(key)`), `status`, `requeue_pending` (boolean), `rebase_in_progress` (JSON or null; see "Partial output and interrupted rebases"), `budget_deadline` (absolute time the task budget ends; null until the budget starts: the admission transaction of the task's first attempt moves the task to `running` and sets it to that time plus the configured budget, so the budget covers preparation), `cancel_requested` (action ID or null), `state_version`, `context_generation`, `assignment_id`, `referenced_code_hash`, `current_attempt_id` (nullable; the composite foreign key `(plan_key, current_attempt_id)` references `attempts(plan_key, id)`, so a task can only point at its own attempt), `created_at`, `updated_at` |
377381
| `attempts` | `id` (**primary key**; never reused), `deadline` (the attempt's absolute deadline, saved at admission), `plan_key` (references `tasks(plan_key)`; `(plan_key, id)` is also unique, for the composite key), `kind`, `phase`, `item`, `state`, `context` (JSON), `first_reason`, `stop_reason`, `exit_code`, `signal`, `result` (JSON, only for `completed`, 1 MiB or less), `diagnostic` (bounded), `diagnostic_ref` (partial-output file, or null), `preparation_pgid` and `preparation_started_at` (or null), `allocation_id` (task storage, or null), `created_at`, `started_at`, `settled_at` |
378382
| `user_actions` | `plan_key`, `action_id` (together the primary key), `kind`, `request_hash`, `response` (JSON, bounded), `created_at` |
379383
| `feedback_events` | the fields listed above, with `id` as primary key, with a unique index on `(plan_key, kind, action_id)` |
@@ -488,6 +492,10 @@ Each case needs a test that fails before the fix and passes after it. Each test
488492
| Two starters after a crash (review round 23) | Runner killed (the OS releases its lock) → two processes start at once → exactly one gets the exclusive lock and the other exits; the lock file is never deleted |
489493
| Retry versus recovery requeue (review round 24) | Restart lists the task and sets `requeue_pending` → user retry → refused → "Resume" and I3 requeue race → exactly one admits an attempt |
490494
| Crash during a rebase (review round 25) | F3 sets `rebase_in_progress` → runner killed → restart → step 4 aborts that rebase and clears the field by compare-and-swap on its `attemptId` → only then is the task listed for I3 |
495+
| Release an unknown preparation (review round 26) | A process still has a file open in the attempt directory → `--release-preparation` refuses and admission stays closed; after the process exits → released |
496+
| Shutdown during launch setup (review round 26) | D start returns its handle → Docker setup still running → shutdown → `/api/runner` keeps answering, `cancel` reaches setup → setup cleanup → `settled` |
497+
| Completed writable attempt keeps its work (review round 26) | `execute` exits 0 → F2 extracts the commit from task storage → `completed` → only then `removeTaskFilesystems` |
498+
| Suggestion cancel after a restart (review round 26) | Provider container from the crashed process → startup step 2 removes it → pending request → cancel endpoint → `cancelled` |
491499
| Old attempt settles after a retry (D/F contract) | Attempt A cancelled and settled → retry B admitted → a late publish from A is refused → B's row and the visible status are unchanged |
492500

493501
## Decisions (approved 2026-09-25)

0 commit comments

Comments
 (0)