You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c230853
Browse filesBrowse the repository at this point in the historyBrowse files
- Launch: a refused pending-to-running write caused by a first reason
cancels and settles normally; only other write failures leave a marker.
- F reserves the allocation ID on the attempt row before async allocation.
- Split recovery into an export phase (fixed-name files, no DB) and one
finalization transaction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/implementation/runner-lifecycle.md
+6-3Lines changed: 6 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -135,7 +135,8 @@ A `pending` attempt has no `settled` promise, so the slot rules need a separate
135
135
| Stop or context change before the D start call | Record the first reason. Await the preparation promise and remove anything it created. Then write the terminal state (`cancelled` or `stale`). | After that terminal write succeeds |
136
136
| The D start call throws | Write the terminal state from the first reason if one is recorded (a stop can be recorded while preparation is finishing); otherwise write `failed` with the launch error. D4's adapters throw only after their setup cleanup has succeeded. If cleanup is still unfinished, they return a handle instead, which settles when cleanup ends. So nothing is left running. | After that terminal write succeeds |
137
137
| The D start call returns a handle | Write `pending → running`. From here the running-state rules apply, even if the handle comes from failed setup that is still cleaning up. | After `settled` resolves and the terminal write succeeds |
138
-
| The handle arrives but the `pending → running` write fails | Keep the job with its handle, call `handle.cancel('capture-failure')`, and await `settled`. Then keep an unresolved marker for the task, because the row is still `pending`. | Only at startup recovery |
138
+
| The handle arrives, but `pending → running` is refused because a first reason is now recorded (a stop landed while preparation finished) | This is a normal stop, not a storage failure. Call `handle.cancel(...)` with the matching D reason, await `settled`, then write the terminal state from the first reason (`pending → cancelled` or `stale`). | After `settled` and that terminal write, as usual |
139
+
| The handle arrives but the `pending → running` write fails for any other reason (for example a storage error) | Keep the job with its handle, call `handle.cancel('capture-failure')`, and await `settled`. Then keep an unresolved marker for the task, because the row is still `pending`. | Only at startup recovery |
139
140
140
141
**Launch check.** Immediately before the D start call, in one synchronous turn with no await between them, F reads the task and plan rows (the `Store` is synchronous) and confirms that the attempt is still `pending`, has no first reason, has a current captured context, and has time left. For the time check it reads the clock in the same turn: if the task budget is spent, it records `time-limit` and ends the row `cancelled` (the task goes to `needs human`); if the attempt deadline has passed, it ends the row `failed` "Timed out while preparing". In both cases D is not called. Only when every check passes does it call D's start, which is also synchronous. If the context is no longer current, F takes `pending → stale` without launching. A writable invocation therefore never starts against an old plan, snapshot, assignment or referenced code. The later publish check cannot undo filesystem changes.
141
142
@@ -246,7 +247,7 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
246
247
|---|---|---|
247
248
| Runner owner token | F | A random ID of exactly 32 lowercase hexadecimal characters (`^[0-9a-f]{32}$`), stored in the database (`app_settings`). A stored value that does not match is refused before it is used as a label or in any path, and startup exits. It is stored together with the database file's device and inode numbers. It is read, or created, in startup recovery step 2, after the lock is taken and the `Store` is opened. If the stored device and inode don't match the open file (the database was copied), F creates a new token. So a copy never shares a token with its original. |
248
249
| Token in every request | D contract |`InvocationInput` gains a `runnerOwner` field, **and so does every allocator that runs before an invocation exists**: `prepareTaskFilesystems` (which today takes no owner, `agents/container/storage.ts`) and vendor network creation. Every resource D creates carries the label `io.codeboost.runner=<token>`: containers, networks, egress proxies, task-storage volumes and allocations. |
249
-
| Recovered storage handles | D | D tracks task-storage ownership in a process-local `WeakMap`, which is empty after a restart. So `recoverLeftovers` returns, for each kept volume set, an **authenticated recovery handle** that D registers in its ownership map after checking the volumes' runner and allocation labels. **Binding to an attempt:** task-storage volumes also carry `io.codeboost.attempt=<attemptId>`, set by `prepareTaskFilesystems` from a new `attemptId` parameter, and F saves the returned allocation ID as `attempts.allocation_id` in the same turn. The handle carries both IDs. F uses a handle only if its attempt ID and allocation ID both match one attempt row of this database. A handle that matches no row, or disagrees with the row, is reported and listed by `--list-unowned-agent-resources` for the user to remove by hand; it is never exported or removed automatically. `exportTaskDiff` and `removeTaskFilesystems` accept these handles. F never builds a `TaskFilesystems` value from report fields. |
250
+
| Recovered storage handles | D | D tracks task-storage ownership in a process-local `WeakMap`, which is empty after a restart. So `recoverLeftovers` returns, for each kept volume set, an **authenticated recovery handle** that D registers in its ownership map after checking the volumes' runner and allocation labels. **Binding to an attempt:** task-storage volumes also carry `io.codeboost.attempt=<attemptId>`, set by `prepareTaskFilesystems` from a new `attemptId` parameter. **F chooses the allocation ID, not D.** F generates it (a lowercase UUID v4) and saves it as `attempts.allocation_id` **before** it starts the asynchronous allocation, then passes it to `prepareTaskFilesystems`, which labels the volumes and keeper with it (`io.codeboost.allocation`). So a crash in the middle of allocation still leaves a row that matches whatever D created. The D prerequisite includes accepting a caller-provided allocation ID. The handle carries both IDs. F uses a handle only if its attempt ID and allocation ID both match one attempt row of this database. A handle that matches no row, or disagrees with the row, is reported and listed by `--list-unowned-agent-resources` for the user to remove by hand; it is never exported or removed automatically. `exportTaskDiff` and `removeTaskFilesystems` accept these handles. F never builds a `TaskFilesystems` value from report fields. |
250
251
| Scoped recovery | D |`recoverLeftovers(runnerOwner): Promise<RecoveryReport>` acts only on resources whose `io.codeboost.runner` label equals the token. It stops and removes agent containers, egress proxies and networks, and resolves only when they are gone. **It keeps task storage whole: the volumes and their keeper container**, which it recognises by its task-storage labels. The recovery handle depends on that keeper and lists them in the report by allocation, so F can export partial output first. F then calls `removeTaskFilesystems` for each one (startup recovery step 4). It rejects with a bounded diagnostic if anything cannot be stopped or removed. |
251
252
| Unowned resources | D and F | Resources with codeboost labels but no `io.codeboost.runner` label (from builds before this change) fail closed. The report lists them, and F refuses to open admission while any exist. **codeboost never removes them itself**, because it cannot prove that no older-build runner is alive: older builds take no lock this build can see, and an idle older runner may have no container running at the moment of a check. So `--list-unowned-agent-resources` prints each resource and the exact `docker rm` / `docker network rm` / `docker volume rm` commands. The user stops every older codeboost process, runs those commands, and starts again. There is no machine-wide lock. |
252
253
| When it runs | F | Only while holding this database's single-runner lock (startup recovery step 1 comes first). |
@@ -267,7 +268,7 @@ This runs before the coordinator opens.
267
268
268
269
1. Take the single-runner lock for this database (decision 1), **before opening the `Store`**, because opening runs migrations. If another process holds it, exit with "Another codeboost runner is using this database." The OS lock cannot name the holder, so the message does not claim to. Do not serve the review screen: it is not read-only, because `ReviewService.load()` records history when HEAD moves (`runner/review.ts`) and `act()` writes review actions. A true read-only mode would need `Store`-level write refusal and is out of scope for F1.
269
270
2.**If D's recovery rejects, startup stops here**, before step 3 finalizes any row or step 6 hands anything to I3: close the `Store`, release the lock, and exit with D's diagnostic. The next start retries. Open the `Store` (migrations run here, under the lock). Read the runner owner token, or create it (see the table under "Shutdown"). Then call D's startup recovery with that token, and await it. **D does not provide this yet.**`agents/contract.ts` exposes only per-invocation handles, and the supervisor's cleanup ownership lives in memory, so it is lost when the process crashes. See the second prerequisite under "Shutdown".
270
-
3.**Export first.** For each task-storage recovery handle that matches an attempt row (see "Recovered storage handles"), run the bounded partial-output export and save the file. This makes no database writes. Before F2 there are no writable attempts, so there are no handles. Then finalize the unclean leftovers. These are `pending` or `running` rows left by a crash, or by a shutdown whose terminal write failed. **All of step 3 is one transaction.** If it fails, for example because of a storage error, startup stops there: it does not run step 4, does not open admission, closes the `Store`, releases the lock, and exits with the error. The next start runs recovery again from step 1. Finalize each one, storing its `diagnostic_ref` from the export in the same transaction, using the first-reason table in "Rules for the running state", rule 2:
271
+
3. **Export phase, outside any transaction.** For each task-storage recovery handle that matches an attempt row (see "Recovered storage handles"), run the bounded partial-output export. Write it to the diagnostics directory under the fixed name `<attemptId>.diff`, so a retry overwrites instead of duplicating. An export failure is kept in memory as that attempt's diagnostic ("Partial output could not be exported: …"), with no file. This phase makes no database writes. Before F2 there are no writable attempts, so there are no handles. Then finalize the unclean leftovers. These are `pending` or `running` rows left by a crash, or by a shutdown whose terminal write failed. **Finalization phase: one transaction** that only finalizes rows and stores each `diagnostic_ref` (or the export-failure diagnostic). It does no D or file work. If it fails, for example because of a storage error, startup stops there: it does not run step 4, does not open admission, closes the `Store`, releases the lock, and exits with the error. The next start runs recovery again from step 1, and its export phase rewrites the same fixed-name files. A diagnostics file whose attempt row never got a `diagnostic_ref` is removed by the diagnostics directory's normal retention. Finalize each one, storing its `diagnostic_ref` from the export in the same transaction, using the first-reason table in "Rules for the running state", rule 2:
271
272
- first reason `cancelled` → `cancelled`; `stale` → `stale` with its cause;
272
273
- first reason `shutdown`: if the row has a saved D `stop_reason` of `timeout`, `output-limit` or `capture-failure`, **D's reason wins** → `failed`, as at settlement. Otherwise → `cancelled` "Stopped by shutdown". **Limit:** D's `InvocationHandle` reports its stop reason only in the settled result, and F saves it only with the terminal write. If the process dies after D stopped for its own reason but before settlement, D's reason is lost with the process. Recovery then cannot see it and maps the row to a shutdown cancellation, which is requeued. The next attempt has its own deadline, so a lost timeout costs one bounded re-run, not an unbounded one. Keeping D's reason across a crash would need D to report it before settlement; that is not a prerequisite;
273
274
- no first reason, and the captured context **is no longer current** → `stale` with its cause (the change happened, but the runner died before it recorded the reason). This is checked **first**, before both deadline fallbacks below, matching the settlement precedence;
@@ -505,6 +506,8 @@ Each case needs a test that fails before the fix and passes after it. Each test
505
506
| Ledger write in the publication transaction (review round 27) | Completed `execute` → the step-3 guard passes → `completed` and `recordHistory` in one transaction → generation +1. A stale variant → refused, no ledger entry, the commit is discarded with the storage |
506
507
| D timeout, then shutdown, then crash (review round 28) | (a) Row with a saved `stop_reason` of `timeout` and first reason `shutdown` → restart → `failed`, not requeued. (b) D timed out, then shutdown, then a crash before settlement (no `stop_reason` saved) → restart → `cancelled` "Stopped by shutdown" and requeued once; the requeued attempt has a fresh finite deadline |
507
508
| Recovery keeps the keeper (review round 29) | Crash with task storage → `recoverLeftovers` → the agent container and network are gone, but the volumes and their keeper container remain → export succeeds → `removeTaskFilesystems` removes both |
509
+
| Crash during async allocation (review round 30) |`allocation_id` saved → allocation starts → process killed → restart → the handle's attempt and allocation IDs match the row → exported and removed; never classed as unowned |
510
+
| Recovery finalization fails after export (review round 30) | Export writes `<attemptId>.diff` → the finalization transaction fails → exit → next start re-exports to the same name and finalizes once |
508
511
| Old attempt settles after a retry (D/F contract) | Attempt A cancelled and settled → retry B admitted → a late publish from A is refused → B's row and the visible status are unchanged |
0 commit comments