You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a17d980
Browse filesBrowse the repository at this point in the historyBrowse files
- Releasing an unknown preparation requires a verified no-open-files
check, not quiescence.
- D start must return at once and run Docker setup asynchronously
(new prerequisite).
- F2 extracts a completed attempt's work before the terminal write and
storage removal.
- Suggestion cancel after a restart relies on D recovery stopping the provider.
- The task budget starts at the first attempt's admission, so it covers
preparation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/implementation/runner-lifecycle.md
+13-5Lines changed: 13 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -124,9 +124,9 @@ A `pending` attempt has no `settled` promise, so the slot rules need a separate
124
124
125
125
- Preparation receives an `AbortSignal` from the job. Stopping a pending attempt aborts that signal with the first reason.
126
126
-**Preparation files have a durable owner.** The attempt ID is used as a directory name only after it passes the UUID check under "Terms used", so it cannot contain `/` or `..`. Before preparation creates anything, F uses the fixed attempt directory `<runner root>/<runner owner token>/attempts/<attemptId>/`, and passes it as the parent to the clone helper. The helper names its clone with `mkdtemp`, so the fixed parent is what makes the clone findable after a crash. **Today's `createTaskClone` (`git/clone.ts`) runs git with synchronous `execFileSync`.** It cannot be aborted and would block the event loop, so F1 cannot use it for preparation as it is. D, which owns `git/clone.ts`, must add an asynchronous variant that takes an `AbortSignal`, spawns git in its own process group, and exposes that group's ID (third prerequisite under "Shutdown").
127
-
- **Preparation processes have a durable owner.** Right after spawning a preparation child, in the same synchronous turn, F saves the process group ID and the child's start time on the attempt row (`preparation_pgid`, `preparation_started_at`). A process group outlives the runner if the runner crashes, so startup recovery uses these fields to stop it (step 4). To close the gap between spawn and that write, F first saves `preparation_started_at` with a null `preparation_pgid` (**preparation starting**), then spawns, then saves the group ID. If recovery finds an attempt that is marked as starting but has no group ID, a child may exist that nobody can identify. Recovery then **fails closed**: it does not open admission, it leaves that attempt directory in place, and the CLI names the attempt and its directory. The user must confirm that no process is using it, with `--release-preparation <attemptId>`. That flag removes the directory only if its contents stay unchanged over a 10-second check. The path comes only from IDs that are already saved, so a crash cannot leave an unknown path. The job removes the directory when the attempt ends. Startup recovery handles the rest (step 4).
127
+
- **Preparation processes have a durable owner.** Right after spawning a preparation child, in the same synchronous turn, F saves the process group ID and the child's start time on the attempt row (`preparation_pgid`, `preparation_started_at`). A process group outlives the runner if the runner crashes, so startup recovery uses these fields to stop it (step 4). To close the gap between spawn and that write, F first saves `preparation_started_at` with a null `preparation_pgid` (**preparation starting**), then spawns, then saves the group ID. If recovery finds an attempt that is marked as starting but has no group ID, a child may exist that nobody can identify. Recovery then **fails closed**: it does not open admission, it leaves that attempt directory in place, and the CLI names the attempt and its directory. The user must stop the unknown process and then confirm with `--release-preparation <attemptId>`. A quiet directory proves nothing, because a blocked process may resume later, so codeboost does not rely on it. Before removing anything, it checks that no process has a file open or a working directory inside the attempt directory, using `lsof +D` on macOS or `/proc/*/fd` and `/proc/*/cwd` on Linux. If a process is found, or the check cannot run, it refuses and keeps admission closed. The path comes only from IDs that are already saved, so a crash cannot leave an unknown path. The job removes the directory when the attempt ends. Startup recovery handles the rest (step 4).
128
128
- An `AbortSignal` does not stop a child process by itself. Every preparation subprocess (for example `git clone`) is started in its own process group, owned by the job. On abort, F sends `SIGTERM` to the group, waits a fixed grace period (5 seconds), then sends `SIGKILL`, and awaits the `close` event. Preparation settles only after that. A child that ignores `SIGTERM` is therefore still bounded by the kill. Preparation never leaves work running after it settles.
129
-
- Preparation counts against two limits, so it cannot hold a slot forever. If the **task time budget** runs out first, the job records the first reason `time-limit`, then aborts the signal; the pending row ends `cancelled` and the task goes to `needs human`, as for a running attempt. If the **attempt deadline** passes first, the job aborts the signal with `timeout` and records no first reason; the row ends `failed` "Timed out while preparing".
129
+
- Preparation counts against two limits, so it cannot hold a slot forever. The task budget is already running during preparation: the admission transaction of the task's first attempt moves the task to `running` and sets `budget_deadline` (see "Proposed storage additions"), before preparation starts. If the **task time budget** runs out first, the job records the first reason `time-limit`, then aborts the signal; the pending row ends `cancelled` and the task goes to `needs human`, as for a running attempt. If the **attempt deadline** passes first, the job aborts the signal with `timeout` and records no first reason; the row ends `failed` "Timed out while preparing".
130
130
- Shutdown aborts every preparation signal in step 4.
131
131
132
132
| Case | What F does | When the slot is freed |
@@ -141,6 +141,8 @@ A `pending` attempt has no `settled` promise, so the slot rules need a separate
141
141
142
142
In every case, a failed terminal write leaves an unresolved marker (see "Slots and concurrency", rule 5).
143
143
144
+
**Launch setup must not block the server.** The D4 adapters do their Docker setup synchronously inside the start call: `createVendorNetwork`, profile creation and `startProfileInvocation` use `execFileSync` and `spawnSync` (`agents/network/network.ts`, `agents/container/`). While that runs, the event loop cannot set `closing`, cancel the launch, or begin the HTTP drain. **Prerequisite:** D's start call must return its handle at once and do the Docker setup asynchronously inside that handle's lifecycle. Then `cancel()` works during setup, and `settled` resolves only after setup cleanup. It is listed with the other D prerequisites under "Shutdown".
145
+
144
146
**Rule for D.** D's start call must keep this shape: it either throws with nothing left running, or it returns a handle that settles only after everything it started has stopped. An adapter that needs an asynchronous start must still follow this rule. Changing it is a change to D's contract.
145
147
146
148
### Rules for the running state
@@ -203,7 +205,7 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
203
205
2. Validate the output (schema, size, file-scope audit for writable attempts). Do not await anything between the last check and the transaction.
204
206
3. In one `Store` transaction: confirm that the attempt ID is the task's current attempt, the state is `running`, there is no first reason, the captured context is still current, the task is not closed, and `tasks.cancel_requested` is null (the same guard as the `running → completed` row). Do not compare the task's state version here; the attempt's own transitions have increased it. Then write `completed` and the result, and increase the state version.
205
207
4. If step 3 refuses, reread durable state and settle the row by the rules above. Keep the original diagnostic.
206
-
5. For a writable attempt, call `removeTaskFilesystems` after the terminal write, whatever the outcome. Free the slot only after it succeeds. If it fails, keep an unresolved marker, and startup recovery removes the storage.
208
+
5. For a completed writable attempt, F2's result extraction (committing the change and recording its ledger entry) is part of step 2 above. It runs from the still-present task storage before the step-3 transaction. If step 3 then refuses, F2's rules discard the extracted commit. Then, for every writable attempt, call `removeTaskFilesystems` after the terminal write, whatever the outcome. Free the slot only after it succeeds. If it fails, keep an unresolved marker, and startup recovery removes the storage.
207
209
208
210
**Irreversible actions.** Before each commit, push, PR open or merge, re-read the task's state version, the plan's `review_version` (which `saveReview` and `addReviewNote` advance) and the coordinator's `closing` flag **after the final await**. Stop if any of them changed. For a merge this means an approval or choice edit made during the final GitHub check blocks the merge. A check made before an await does not count. (This follows the AGENTS.md rules on guarded external actions.)
209
211
@@ -247,6 +249,8 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
247
249
248
250
This belongs to D (D5 or a D follow-up), not F. It changes `agents/contract.ts`, so it goes through D's contract tests.
249
251
252
+
**Also before F1 merges: asynchronous launch.** D's start call must return a handle immediately and run its Docker setup asynchronously and abortably (see "Launch setup must not block the server").
253
+
250
254
**Third prerequisite before F1 merges: abortable preparation helpers.** Both preparation helpers are synchronous today. `createTaskClone` runs git with `execFileSync`, and `prepareTaskFilesystems` runs `docker` with `execFileSync` and `spawnSync` (`agents/container/storage.ts`). Either would block the event loop, and cancel or shutdown could not abort it. D must add asynchronous variants of both that take an `AbortSignal`, spawn their subprocesses in their own process group, expose the group ID, and settle only after those processes have exited. Storage allocation must also label its volumes with the runner owner and attempt ID before it returns (see "Recovered storage handles"), so that storage allocated just before a crash is found by recovery.
251
255
252
256
**Prerequisite before F2's writable attempts: a bounded task-volume export.** D must add a bounded diff export over the task volumes (see "Partial output and interrupted rebases"). F1 does not need it, because F1 has no writable attempts yet.
@@ -299,7 +303,7 @@ F1 owns the production planning endpoints that G4 needs. They wrap E3's coordina
|`GET /api/plan/suggestions/:id`|`Store.getSuggestions`| reads only the request row |
302
-
|`POST /api/plan/suggestions/:id/cancel`|`pending`: `handle.cancel(reason)` on the `SuggestionHandle` that `SuggestionCoordinator.start()` returned. E3's coordinator has no cancel-by-ID method, so F keeps a map from request ID to handle, adds each handle when it starts the request, and removes it only after the handle's outcome settles. If the request is `pending` but F holds no handle (for example after a restart), F calls `Store.cancelSuggestions`: no provider runs in this process, so there is nothing to stop. `ready`: `Store.cancelSuggestions` (dismiss), because no invocation is running. | request is `pending` or `ready`, as `Store.cancelSuggestions` already allows |
306
+
| `POST /api/plan/suggestions/:id/cancel` | `pending`: `handle.cancel(reason)` on the `SuggestionHandle` that `SuggestionCoordinator.start()` returned. E3's coordinator has no cancel-by-ID method, so F keeps a map from request ID to handle, adds each handle when it starts the request, and removes it only after the handle's outcome settles. If the request is `pending` but F holds no handle (for example after a restart), F calls `Store.cancelSuggestions`. It is safe to do so because startup recovery has already stopped the crashed process's provider. A live planning provider runs only through D (G4 after D5), in a container labelled with this database's runner owner, and D's `recoverLeftovers` (startup step 2) stops it before admission opens. Until D5 there is no live planning provider at all ("Planning agent not available yet"). If a planning provider ever runs outside D, this endpoint must refuse such a request instead, until startup has reconciled it. `ready`: `Store.cancelSuggestions` (dismiss), because no invocation is running. | request is `pending` or `ready`, as `Store.cancelSuggestions` already allows |
303
307
|`POST /api/plan/suggestions/:id/apply`|`Store.applySuggestion`| request `ready` and bound to the current revision and snapshot |
304
308
305
309
Live planning invocation waits for D5. Until then, the server returns "Planning agent not available yet" instead of using a fake provider.
@@ -373,7 +377,7 @@ A merged v5 task also gets its `task-closed` event, keyed by the merge attempt I
373
377
374
378
| Table | Key columns |
375
379
|---|---|
376
-
|`tasks`|`plan_key` (primary key, references `plans(key)`), `status`, `requeue_pending` (boolean), `rebase_in_progress` (JSON or null; see "Partial output and interrupted rebases"), `budget_deadline` (absolute time the task budget ends; null until the budget starts, which is the task's first move to `running`, when it is set to that time plus the configured budget), `cancel_requested` (action ID or null), `state_version`, `context_generation`, `assignment_id`, `referenced_code_hash`, `current_attempt_id` (nullable; the composite foreign key `(plan_key, current_attempt_id)` references `attempts(plan_key, id)`, so a task can only point at its own attempt), `created_at`, `updated_at`|
380
+
|`tasks`|`plan_key` (primary key, references `plans(key)`), `status`, `requeue_pending` (boolean), `rebase_in_progress` (JSON or null; see "Partial output and interrupted rebases"), `budget_deadline` (absolute time the task budget ends; null until the budget starts: the admission transaction of the task's first attempt moves the task to `running` and sets it to that time plus the configured budget, so the budget covers preparation), `cancel_requested` (action ID or null), `state_version`, `context_generation`, `assignment_id`, `referenced_code_hash`, `current_attempt_id` (nullable; the composite foreign key `(plan_key, current_attempt_id)` references `attempts(plan_key, id)`, so a task can only point at its own attempt), `created_at`, `updated_at`|
377
381
|`attempts`|`id` (**primary key**; never reused), `deadline` (the attempt's absolute deadline, saved at admission), `plan_key` (references `tasks(plan_key)`; `(plan_key, id)` is also unique, for the composite key), `kind`, `phase`, `item`, `state`, `context` (JSON), `first_reason`, `stop_reason`, `exit_code`, `signal`, `result` (JSON, only for `completed`, 1 MiB or less), `diagnostic` (bounded), `diagnostic_ref` (partial-output file, or null), `preparation_pgid` and `preparation_started_at` (or null), `allocation_id` (task storage, or null), `created_at`, `started_at`, `settled_at`|
|`feedback_events`| the fields listed above, with `id` as primary key, with a unique index on `(plan_key, kind, action_id)`|
@@ -488,6 +492,10 @@ Each case needs a test that fails before the fix and passes after it. Each test
488
492
| Two starters after a crash (review round 23) | Runner killed (the OS releases its lock) → two processes start at once → exactly one gets the exclusive lock and the other exits; the lock file is never deleted |
489
493
| Retry versus recovery requeue (review round 24) | Restart lists the task and sets `requeue_pending` → user retry → refused → "Resume" and I3 requeue race → exactly one admits an attempt |
490
494
| Crash during a rebase (review round 25) | F3 sets `rebase_in_progress` → runner killed → restart → step 4 aborts that rebase and clears the field by compare-and-swap on its `attemptId` → only then is the task listed for I3 |
495
+
| Release an unknown preparation (review round 26) | A process still has a file open in the attempt directory → `--release-preparation` refuses and admission stays closed; after the process exits → released |
496
+
| Shutdown during launch setup (review round 26) | D start returns its handle → Docker setup still running → shutdown → `/api/runner` keeps answering, `cancel` reaches setup → setup cleanup → `settled`|
497
+
| Completed writable attempt keeps its work (review round 26) |`execute` exits 0 → F2 extracts the commit from task storage → `completed` → only then `removeTaskFilesystems`|
498
+
| Suggestion cancel after a restart (review round 26) | Provider container from the crashed process → startup step 2 removes it → pending request → cancel endpoint → `cancelled`|
491
499
| Old attempt settles after a retry (D/F contract) | Attempt A cancelled and settled → retry B admitted → a late publish from A is refused → B's row and the visible status are unchanged |
0 commit comments