You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b2e515b
Browse filesBrowse the repository at this point in the historyBrowse files
- codeboost never removes unowned legacy resources itself: it lists them
with manual removal commands, and admission stays closed. The machine-wide
lock is dropped.
- Recovery applies D-reason precedence when stop_reason is saved; state
the limit when D's reason is lost with the process.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/implementation/runner-lifecycle.md
+6-4Lines changed: 6 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -246,9 +246,9 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
246
246
|---|---|---|
247
247
| Runner owner token | F | A random ID of exactly 32 lowercase hexadecimal characters (`^[0-9a-f]{32}$`), stored in the database (`app_settings`). A stored value that does not match is refused before it is used as a label or in any path, and startup exits. It is stored together with the database file's device and inode numbers. It is read, or created, in startup recovery step 2, after the lock is taken and the `Store` is opened. If the stored device and inode don't match the open file (the database was copied), F creates a new token. So a copy never shares a token with its original. |
248
248
| Token in every request | D contract |`InvocationInput` gains a `runnerOwner` field, **and so does every allocator that runs before an invocation exists**: `prepareTaskFilesystems` (which today takes no owner, `agents/container/storage.ts`) and vendor network creation. Every resource D creates carries the label `io.codeboost.runner=<token>`: containers, networks, egress proxies, task-storage volumes and allocations. |
249
-
| Recovered storage handles | D | D tracks task-storage ownership in a process-local `WeakMap`, which is empty after a restart. So `recoverLeftovers` returns, for each kept volume set, an **authenticated recovery handle** that D registers in its ownership map after checking the volumes' runner and allocation labels. **Binding to an attempt:** task-storage volumes also carry `io.codeboost.attempt=<attemptId>`, set by `prepareTaskFilesystems` from a new `attemptId` parameter, and F saves the returned allocation ID as `attempts.allocation_id` in the same turn. The handle carries both IDs. F uses a handle only if its attempt ID and allocation ID both match one attempt row of this database. A handle that matches no row, or disagrees with the row, is reported and left for`--remove-unowned-agent-resources`; it is never exported or removed automatically. `exportTaskDiff` and `removeTaskFilesystems` accept these handles. F never builds a `TaskFilesystems` value from report fields. |
249
+
| Recovered storage handles | D | D tracks task-storage ownership in a process-local `WeakMap`, which is empty after a restart. So `recoverLeftovers` returns, for each kept volume set, an **authenticated recovery handle** that D registers in its ownership map after checking the volumes' runner and allocation labels. **Binding to an attempt:** task-storage volumes also carry `io.codeboost.attempt=<attemptId>`, set by `prepareTaskFilesystems` from a new `attemptId` parameter, and F saves the returned allocation ID as `attempts.allocation_id` in the same turn. The handle carries both IDs. F uses a handle only if its attempt ID and allocation ID both match one attempt row of this database. A handle that matches no row, or disagrees with the row, is reported and listed by`--list-unowned-agent-resources` for the user to remove by hand; it is never exported or removed automatically. `exportTaskDiff` and `removeTaskFilesystems` accept these handles. F never builds a `TaskFilesystems` value from report fields. |
250
250
| Scoped recovery | D |`recoverLeftovers(runnerOwner): Promise<RecoveryReport>` acts only on resources whose `io.codeboost.runner` label equals the token. It stops and removes containers, egress proxies and networks, and resolves only when they are gone. **It keeps task-storage volumes** and lists them in the report by allocation, so F can export partial output first. F then calls `removeTaskFilesystems` for each one (startup recovery step 4). It rejects with a bounded diagnostic if anything cannot be stopped or removed. |
251
-
| Unowned resources | D and F | Resources with codeboost labels but no `io.codeboost.runner` label (from builds before this change) fail closed. The report lists them, F refuses to open admission, and the CLI prints the list with the instruction to stop every codeboost process and then run `--remove-unowned-agent-resources`. That flag needs proof that no other runner is live on the machine, and per-database locks cannot give that proof. So every runner of this build also holds a **shared** lock on one machine-wide lock database (`~/.codeboost/runners.lock`), through an open read transaction, for its whole lifetime. The flag takes an **exclusive** lock on the same file, which fails while any runner holds its shared lock. Runners from older builds do not take this lock, so the flag also refuses while any codeboost-labelled container is still running. It removes only stopped containers and the networks and volumes they no longer use. If either check cannot run, it refuses. It never adopts or reuses these resources. |
251
+
| Unowned resources | D and F | Resources with codeboost labels but no `io.codeboost.runner` label (from builds before this change) fail closed. The report lists them, and F refuses to open admission while any exist. **codeboost never removes them itself**, because it cannot prove that no older-build runner is alive: older builds take no lock this build can see, and an idle older runner may have no container running at the moment of a check. So `--list-unowned-agent-resources` prints each resource and the exact `docker rm` / `docker network rm` / `docker volume rm` commands. The user stops every older codeboost process, runs those commands, and starts again. There is no machine-wide lock. |
252
252
| When it runs | F | Only while holding this database's single-runner lock (startup recovery step 1 comes first). |
253
253
254
254
This belongs to D (D5 or a D follow-up), not F. It changes `agents/contract.ts`, so it goes through D's contract tests.
@@ -268,7 +268,8 @@ This runs before the coordinator opens.
268
268
1. Take the single-runner lock for this database (decision 1), **before opening the `Store`**, because opening runs migrations. If another process holds it, exit with "Another codeboost runner is using this database." The OS lock cannot name the holder, so the message does not claim to. Do not serve the review screen: it is not read-only, because `ReviewService.load()` records history when HEAD moves (`runner/review.ts`) and `act()` writes review actions. A true read-only mode would need `Store`-level write refusal and is out of scope for F1.
269
269
2.**If D's recovery rejects, startup stops here**, before step 3 finalizes any row or step 6 hands anything to I3: close the `Store`, release the lock, and exit with D's diagnostic. The next start retries. Open the `Store` (migrations run here, under the lock). Read the runner owner token, or create it (see the table under "Shutdown"). Then call D's startup recovery with that token, and await it. **D does not provide this yet.**`agents/contract.ts` exposes only per-invocation handles, and the supervisor's cleanup ownership lives in memory, so it is lost when the process crashes. See the second prerequisite under "Shutdown".
270
270
3.**Export first.** For each task-storage recovery handle that matches an attempt row (see "Recovered storage handles"), run the bounded partial-output export and save the file. This makes no database writes. Before F2 there are no writable attempts, so there are no handles. Then finalize the unclean leftovers. These are `pending` or `running` rows left by a crash, or by a shutdown whose terminal write failed. **All of step 3 is one transaction.** If it fails, for example because of a storage error, startup stops there: it does not run step 4, does not open admission, closes the `Store`, releases the lock, and exits with the error. The next start runs recovery again from step 1. Finalize each one, storing its `diagnostic_ref` from the export in the same transaction, using the first-reason table in "Rules for the running state", rule 2:
271
-
- first reason `cancelled` → `cancelled`; `shutdown` → `cancelled` "Stopped by shutdown"; `stale` → `stale` with its cause;
271
+
- first reason `cancelled` → `cancelled`; `stale` → `stale` with its cause;
272
+
- first reason `shutdown`: if the row has a saved D `stop_reason` of `timeout`, `output-limit` or `capture-failure`, **D's reason wins** → `failed`, as at settlement. Otherwise → `cancelled` "Stopped by shutdown". **Limit:** D's `InvocationHandle` reports its stop reason only in the settled result, and F saves it only with the terminal write. If the process dies after D stopped for its own reason but before settlement, D's reason is lost with the process. Recovery then cannot see it and maps the row to a shutdown cancellation, which is requeued. The next attempt has its own deadline, so a lost timeout costs one bounded re-run, not an unbounded one. Keeping D's reason across a crash would need D to report it before settlement; that is not a prerequisite;
272
273
- no first reason, and the captured context **is no longer current** → `stale` with its cause (the change happened, but the runner died before it recorded the reason). This is checked **first**, before both deadline fallbacks below, matching the settlement precedence;
273
274
- no first reason, context current, and the task's `budget_deadline` has passed → the `time-limit` mapping (task `needs human`);
274
275
- no first reason, context current, and the attempt's `deadline` has passed → `failed` "Timed out", not requeued;
@@ -500,8 +501,9 @@ Each case needs a test that fails before the fix and passes after it. Each test
500
501
| Shutdown during launch setup (review round 26) | D start returns its handle → Docker setup still running → shutdown → `/api/runner` keeps answering, `cancel` reaches setup → setup cleanup → `settled`|
501
502
| Completed writable attempt keeps its work (review round 26) |`execute` exits 0 → F2 extracts the commit from task storage → `completed` → only then `removeTaskFilesystems`|
502
503
| Suggestion cancel after a restart (review round 26) | Provider container from the crashed process → startup step 2 removes it → pending request → cancel endpoint → `cancelled`|
503
-
| Legacy cleanup with a live runner (review round 27) |Runner B holds its shared machine lock → `--remove-unowned-agent-resources` → exclusive lock fails → refused. A running unlabelled container → refused|
504
+
| Legacy resources are never removed automatically (review rounds 27–28) |Unlabelled resources exist → startup refuses admission and lists them with removal commands; no codeboost code path deletes them; after manual removal, the next start opens admission|
504
505
| Ledger write in the publication transaction (review round 27) | Completed `execute` → the step-3 guard passes → `completed` and `recordHistory` in one transaction → generation +1. A stale variant → refused, no ledger entry, the commit is discarded with the storage |
506
+
| D timeout, then shutdown, then crash (review round 28) | (a) Row with a saved `stop_reason` of `timeout` and first reason `shutdown` → restart → `failed`, not requeued. (b) D timed out, then shutdown, then a crash before settlement (no `stop_reason` saved) → restart → `cancelled` "Stopped by shutdown" and requeued once; the requeued attempt has a fresh finite deadline |
505
507
| Old attempt settles after a retry (D/F contract) | Attempt A cancelled and settled → retry B admitted → a late publish from A is refused → B's row and the visible status are unchanged |
0 commit comments