You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 16b2d83
Browse filesBrowse the repository at this point in the historyBrowse files
- Durable requeue claim: recovery sets requeue_pending; retry is refused
until I3 or an explicit Resume claims it by compare-and-swap.
- The OS lock cannot name its holder; drop the process-ID diagnostic.
- The clean-restart regression keeps the lock file and releases the OS lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/implementation/runner-lifecycle.md
+7-6Lines changed: 7 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -185,7 +185,7 @@ A retry is a new attempt with a new attempt ID. The runner allows it only when a
185
185
186
186
| Check | Where it is checked |
187
187
|---|---|
188
-
| The last attempt for the task is `failed` or `cancelled` on disk, **and** the task's status is `running` or `queued`|`Store` transaction |
188
+
| The last attempt for the task is `failed` or `cancelled` on disk, **and** the task's status is `running` or `queued`, **and**`requeue_pending` is false|`Store` transaction |
189
189
| No in-memory job **and no unresolved marker** exists for the task | Coordinator, synchronously before the transaction |
190
190
| The caller's expected state version and attempt ID match the task's current values |`Store` transaction (prevents a double retry from two tabs) |
191
191
| The last attempt's captured context is still current: snapshot ID, plan ID, plan revision, assignment ID, referenced-code hash and context generation (every field in "Current") |`Store` transaction |
@@ -257,7 +257,7 @@ This belongs to D (D5 or a D follow-up), not F. It changes `agents/contract.ts`,
257
257
258
258
This runs before the coordinator opens.
259
259
260
-
1. Take the single-runner lock for this database (decision 1), **before opening the `Store`**, because opening runs migrations. If another live process holds it, exit with a message that names that process ID. Do not serve the review screen: it is not read-only, because `ReviewService.load()` records history when HEAD moves (`runner/review.ts`) and `act()` writes review actions. A true read-only mode would need `Store`-level write refusal and is out of scope for F1.
260
+
1. Take the single-runner lock for this database (decision 1), **before opening the `Store`**, because opening runs migrations. If another process holds it, exit with "Another codeboost runner is using this database." The OS lock cannot name the holder, so the message does not claim to. Do not serve the review screen: it is not read-only, because `ReviewService.load()` records history when HEAD moves (`runner/review.ts`) and `act()` writes review actions. A true read-only mode would need `Store`-level write refusal and is out of scope for F1.
261
261
2.**If D's recovery rejects, startup stops here**, before step 3 finalizes any row or step 6 hands anything to I3: close the `Store`, release the lock, and exit with D's diagnostic. The next start retries. Open the `Store` (migrations run here, under the lock). Read the runner owner token, or create it (see the table under "Shutdown"). Then call D's startup recovery with that token, and await it. **D does not provide this yet.**`agents/contract.ts` exposes only per-invocation handles, and the supervisor's cleanup ownership lives in memory, so it is lost when the process crashes. See the second prerequisite under "Shutdown".
262
262
3.**Export first.** For each task-storage recovery handle that matches an attempt row (see "Recovered storage handles"), run the bounded partial-output export and save the file. This makes no database writes. Before F2 there are no writable attempts, so there are no handles. Then finalize the unclean leftovers. These are `pending` or `running` rows left by a crash, or by a shutdown whose terminal write failed. **All of step 3 is one transaction.** If it fails, for example because of a storage error, startup stops there: it does not run step 4, does not open admission, closes the `Store`, releases the lock, and exits with the error. The next start runs recovery again from step 1. Finalize each one, storing its `diagnostic_ref` from the export in the same transaction, using the first-reason table in "Rules for the running state", rule 2:
263
263
- first reason `cancelled` → `cancelled`; `shutdown` → `cancelled` "Stopped by shutdown"; `stale` → `stale` with its cause;
@@ -268,7 +268,7 @@ This runs before the coordinator opens.
268
268
- first reason `time-limit` → `cancelled` and the task set to `needs human`, in the same transaction.
269
269
4. Clear every unresolved marker (these exist only in memory, so a restart has already cleared them; step 3 reconciles their rows). Abort every recorded interrupted rebase (F3). For each task-storage handle exported in step 3, call `removeTaskFilesystems`. Step 3's transaction already stored its `diagnostic_ref`. The coordinator does not open (step 8) until every removal has succeeded. Then, for every attempt row with a saved `preparation_pgid`: if a process group with that ID and start time is still alive, send `SIGTERM`, then `SIGKILL` after the grace period, and wait until it has exited. Only after that, for every directory under `<runner root>/<runner owner token>/attempts/` whose attempt is not marked as starting with a null group ID (step 7) (all attempts are terminal after step 3): check ownership (a real directory on the same device, not a symlink, named by a known attempt ID of this database), save a bounded partial-output diagnostic as for a hard stop, and remove it. Unknown entries are reported and left alone.
270
270
5. Insert any missing `task-closed` events. This applies **only** to tasks with a confirmed `merged` merge attempt (feedback-event rule 2). An attempt's terminal state never closes a task: a `cancelled` or `failed` attempt means only that the attempt ended. **Cancel attempt** (stop the current run; the task stays open) and **cancel task** (close the task and discard its workspace) are different user actions. Only cancel task closes a task. **If a runner attempt is `pending` or `running`, cancel task does not close the task at once.** It records the first reason `cancelled` on that attempt, and in the same transaction it sets `tasks.cancel_requested` to its `actionId`. The response says "Stopping, then cancelling". When that attempt settles, one transaction writes the attempt's terminal state, sets the task to `cancelled` and inserts `task-closed`. Only after that does F discard the workspace. With no active attempt, cancel task closes the task and writes `task-closed` in one local transaction. Either way it never needs reconciliation. Startup recovery finishes a pending `cancel_requested` in the same way after step 3. **Cancel task is refused while a merge attempt is `submitting` or `queued`**, including when its outcome is unclear. GitHub may still merge, and a closed task can never change to `merged`. The user can cancel the task after the merge coordinator records `merged`, `removed` or `failed`. If the result is `merged`, the task closes as merged instead. "Reject with feedback" never closes a task (see the status list).
271
-
6.**Requeue input.** Hand lane I3 every task whose status is not closed (`merged` or `cancelled`) and not human-gated, and whose latest attempt is either (a) `cancelled` with first reason `shutdown`, whether written by clean shutdown or by step 3, or (b) `failed` "Interrupted" by step 3. Attempts that end `cancelled` by the user or `stale` are not requeued; they wait for a user action. I3 rebuilds the workspace and requeues. F1 only makes the attempt rows terminal and produces this list.
271
+
6.**Requeue input.** Hand lane I3 every task whose status is not closed (`merged` or `cancelled`) and not human-gated, and whose latest attempt is either (a) `cancelled` with first reason `shutdown`, whether written by clean shutdown or by step 3, or (b) `failed` "Interrupted" by step 3. Attempts that end `cancelled` by the user or `stale` are not requeued; they wait for a user action. I3 rebuilds the workspace and requeues. F1 only makes the attempt rows terminal and produces this list.**Requeue claim:** the same transaction as step 3 sets `tasks.requeue_pending = true` for each listed task. While it is set, retry and every other admission for that task are refused, so recovery and a user retry cannot both start work on it. Exactly one path clears it, by an atomic compare-and-swap from `true` to `false` in the same transaction that admits the next attempt: I3's requeue, or, until lane I exists, the user's explicit "Resume" action (a `user_actions` row). The loser of that compare-and-swap is refused.
272
272
7.**Unowned preparation check.** If any attempt has `preparation_started_at` set and `preparation_pgid` null (see "Launch"), stop here: do not open the coordinator, name each attempt and directory, and wait for `--release-preparation`. Step 4 does not remove those directories.
273
273
8. Open the coordinator.
274
274
@@ -368,7 +368,7 @@ A merged v5 task also gets its `task-closed` event, keyed by the merge attempt I
368
368
369
369
| Table | Key columns |
370
370
|---|---|
371
-
|`tasks`|`plan_key` (primary key, references `plans(key)`), `status`, `budget_deadline` (absolute time the task budget ends), `cancel_requested` (action ID or null), `state_version`, `context_generation`, `assignment_id`, `referenced_code_hash`, `current_attempt_id` (nullable; the composite foreign key `(plan_key, current_attempt_id)` references `attempts(plan_key, id)`, so a task can only point at its own attempt), `created_at`, `updated_at`|
371
+
|`tasks`|`plan_key` (primary key, references `plans(key)`), `status`, `requeue_pending` (boolean), `budget_deadline` (absolute time the task budget ends), `cancel_requested` (action ID or null), `state_version`, `context_generation`, `assignment_id`, `referenced_code_hash`, `current_attempt_id` (nullable; the composite foreign key `(plan_key, current_attempt_id)` references `attempts(plan_key, id)`, so a task can only point at its own attempt), `created_at`, `updated_at`|
372
372
|`attempts`|`id` (**primary key**; never reused), `deadline` (the attempt's absolute deadline, saved at admission), `plan_key` (references `tasks(plan_key)`; `(plan_key, id)` is also unique, for the composite key), `kind`, `phase`, `item`, `state`, `context` (JSON), `first_reason`, `stop_reason`, `exit_code`, `signal`, `result` (JSON, only for `completed`, 1 MiB or less), `diagnostic` (bounded), `diagnostic_ref` (partial-output file, or null), `preparation_pgid` and `preparation_started_at` (or null), `allocation_id` (task storage, or null), `created_at`, `started_at`, `settled_at`|
|`feedback_events`| the fields listed above, with `id` as primary key, with a unique index on `(plan_key, kind, action_id)`|
@@ -414,7 +414,7 @@ Each case needs a test that fails before the fix and passes after it. Each test
414
414
| Choice changed twice at one revision (review round 5) | Assign segment to P1 → reassign to P2 at the same revision → two events saved, the second supersedes the first; replaying the second request adds none |
415
415
| Preparation hangs, then cancel (review round 6) | Clone preparation blocks → user cancels → the signal aborts and the clone subprocess is awaited → row `cancelled` → slot freed |
416
416
| Crash with a recorded first reason (review round 6) | User cancels (first reason saved) → process killed before settlement → restart → row `cancelled`, not `failed`, and not requeued |
417
-
| Clean stop, then restart (review round 6) | Shutdown completes → lock file removed → a new process takes the lock and starts |
417
+
| Clean stop, then restart (review round 6) | Shutdown completes → the lock connection is closed and the OS lock released; the lock file is still there → a new process takes the lock and starts |
418
418
| Preparation fails (review round 7) | Clone fails before the D start call → partial clone removed → row `failed` with the clone error → slot freed |
419
419
| Retry a closed or gated task (review round 7) | Task `cancelled` by cancel task (or `needs human` after the time limit) with a `cancelled` last attempt → retry refused; restart does not requeue it |
420
420
| Assignment changes alone (review round 7) | Work reassigned without a plan-revision or snapshot change → generation increases in the same transaction → the old attempt's result is refused and the row becomes `stale`|
@@ -481,6 +481,7 @@ Each case needs a test that fails before the fix and passes after it. Each test
481
481
| Cancel a pending suggestion (review round 23) | Suggestion running → cancel endpoint → the saved handle's `cancel` is called → provider settles → request `cancelled`, coordinator slot free |
482
482
| Invalid action ID (review round 23) | A 10 KB `actionId`, and one that is not a UUID → HTTP 400; no `user_actions` row |
483
483
| Two starters after a crash (review round 23) | Runner killed (the OS releases its lock) → two processes start at once → exactly one gets the exclusive lock and the other exits; the lock file is never deleted |
484
+
| Retry versus recovery requeue (review round 24) | Restart lists the task and sets `requeue_pending` → user retry → refused → "Resume" and I3 requeue race → exactly one admits an attempt |
484
485
| Old attempt settles after a retry (D/F contract) | Attempt A cancelled and settled → retry B admitted → a late publish from A is refused → B's row and the visible status are unchanged |
485
486
486
487
## Decisions (approved 2026-09-25)
@@ -492,7 +493,7 @@ The user approved the proposal for each of these four questions.
492
493
| 1 | How is one runner per database enforced across processes? | An exclusive lock file next to the database. **Updated in review round 23:** the lock is an operating-system lock held on that file, not a process-ID check. The approved proposal used a process ID and start time, but taking over a stale lock of that kind is not race-safe: two starters could both judge it stale, and one could delete the other's new lock. | Derive the lock path from the **canonical** database path: resolve the parent directory with `realpath`, and refuse a database path that is itself a symlink. Refuse a database file whose hard-link count is more than 1, both before and after opening, because two hard-linked names would give two lock paths for one file. Startup order, which also covers a database that does not exist yet:
493
494
494
495
0.**Before the lock file is opened:** resolve the parent directory with `realpath`, and check that it is owned by the current user and not writable by group or others. Otherwise exit. Only then is the lock path a safe boundary.
495
-
1. Open the lock file at the canonical path (`<database>.runner-lock`) as a small, dedicated SQLite database, with `O_NOFOLLOW` semantics checked as for the main database. Set `PRAGMA locking_mode = EXCLUSIVE`, and start a write transaction to take SQLite's exclusive POSIX lock on it. If that fails with `SQLITE_BUSY`, another runner holds it: exit. The operating system releases this lock when the process exits or crashes. **There is no stale lock and no takeover**: the lock file is never deleted, and a new starter only ever tries to take the OS lock. The lock database also stores the process ID and start time, for the error message only. No database file is needed for this step.
496
+
1. Open the lock file at the canonical path (`<database>.runner-lock`) as a small, dedicated SQLite database, with `O_NOFOLLOW` semantics checked as for the main database. Set `PRAGMA locking_mode = EXCLUSIVE`, and start a write transaction to take SQLite's exclusive POSIX lock on it. If that fails with `SQLITE_BUSY`, another runner holds it: exit. The operating system releases this lock when the process exits or crashes. **There is no stale lock and no takeover**: the lock file is never deleted, and a new starter only ever tries to take the OS lock. The lock database stores no owner details: while one process holds the exclusive lock, another cannot read it, so an owner record could never be shown. No database file is needed for this step.
496
497
1a. **Before SQLite opens anything:** if the database file exists, open it with `O_NOFOLLOW`, `fstat` the descriptor, and refuse a link count above 1. Two hard-linked names would each take a different lock, so this check has to run before migrations, not only after.
497
498
2. Open (and, for a new database, create) the `Store` under the lock. Migrations run here. **Path safety:** step 0 has already checked the parent directory. For a new database, F first creates the file itself with `O_CREAT | O_EXCL | O_NOFOLLOW` and keeps that descriptor. For an existing one, it opens it with `O_NOFOLLOW`. Only then does SQLite open the path. Step 4 compares the path's `lstat` with the descriptor's `fstat`. **What this defends against, and what it does not.** Because only this user can change the directory, another user cannot swap the path. `node:sqlite` can open a database only by path, not by descriptor, so a swap by the user's **own** processes between the descriptor check and SQLite's open is not prevented. The step-4 comparison detects it only afterwards, and migrations may already have run on the other file. Startup then exits before admission. This is accepted: the same user can already write the database file directly, so the lock guards against accidental double starts, not against the user's own processes.
498
499
3. Read the open file's device and inode numbers and link count. Refuse a link count above 1. Write the identity into the lock database through the connection opened in step 1, and keep that connection, and so the OS lock, open until shutdown step 8. The lock file is never replaced or deleted.
0 commit comments