Skip to content

Commit c230853

Browse files
mchwangclaude
andcommitted
Address F1 contract review round 30
- Launch: a refused pending-to-running write caused by a first reason cancels and settles normally; only other write failures leave a marker. - F reserves the allocation ID on the attempt row before async allocation. - Split recovery into an export phase (fixed-name files, no DB) and one finalization transaction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 5f4d81a commit c230853

1 file changed

Lines changed: 6 additions & 3 deletions

File tree

‎docs/implementation/runner-lifecycle.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,8 @@ A `pending` attempt has no `settled` promise, so the slot rules need a separate
135135
| Stop or context change before the D start call | Record the first reason. Await the preparation promise and remove anything it created. Then write the terminal state (`cancelled` or `stale`). | After that terminal write succeeds |
136136
| The D start call throws | Write the terminal state from the first reason if one is recorded (a stop can be recorded while preparation is finishing); otherwise write `failed` with the launch error. D4's adapters throw only after their setup cleanup has succeeded. If cleanup is still unfinished, they return a handle instead, which settles when cleanup ends. So nothing is left running. | After that terminal write succeeds |
137137
| The D start call returns a handle | Write `pending → running`. From here the running-state rules apply, even if the handle comes from failed setup that is still cleaning up. | After `settled` resolves and the terminal write succeeds |
138-
| The handle arrives but the `pending → running` write fails | Keep the job with its handle, call `handle.cancel('capture-failure')`, and await `settled`. Then keep an unresolved marker for the task, because the row is still `pending`. | Only at startup recovery |
138+
| The handle arrives, but `pending → running` is refused because a first reason is now recorded (a stop landed while preparation finished) | This is a normal stop, not a storage failure. Call `handle.cancel(...)` with the matching D reason, await `settled`, then write the terminal state from the first reason (`pending → cancelled` or `stale`). | After `settled` and that terminal write, as usual |
139+
| The handle arrives but the `pending → running` write fails for any other reason (for example a storage error) | Keep the job with its handle, call `handle.cancel('capture-failure')`, and await `settled`. Then keep an unresolved marker for the task, because the row is still `pending`. | Only at startup recovery |
139140

140141
**Launch check.** Immediately before the D start call, in one synchronous turn with no await between them, F reads the task and plan rows (the `Store` is synchronous) and confirms that the attempt is still `pending`, has no first reason, has a current captured context, and has time left. For the time check it reads the clock in the same turn: if the task budget is spent, it records `time-limit` and ends the row `cancelled` (the task goes to `needs human`); if the attempt deadline has passed, it ends the row `failed` "Timed out while preparing". In both cases D is not called. Only when every check passes does it call D's start, which is also synchronous. If the context is no longer current, F takes `pending → stale` without launching. A writable invocation therefore never starts against an old plan, snapshot, assignment or referenced code. The later publish check cannot undo filesystem changes.
141142

@@ -246,7 +247,7 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
246247
|---|---|---|
247248
| Runner owner token | F | A random ID of exactly 32 lowercase hexadecimal characters (`^[0-9a-f]{32}$`), stored in the database (`app_settings`). A stored value that does not match is refused before it is used as a label or in any path, and startup exits. It is stored together with the database file's device and inode numbers. It is read, or created, in startup recovery step 2, after the lock is taken and the `Store` is opened. If the stored device and inode don't match the open file (the database was copied), F creates a new token. So a copy never shares a token with its original. |
248249
| Token in every request | D contract | `InvocationInput` gains a `runnerOwner` field, **and so does every allocator that runs before an invocation exists**: `prepareTaskFilesystems` (which today takes no owner, `agents/container/storage.ts`) and vendor network creation. Every resource D creates carries the label `io.codeboost.runner=<token>`: containers, networks, egress proxies, task-storage volumes and allocations. |
249-
| Recovered storage handles | D | D tracks task-storage ownership in a process-local `WeakMap`, which is empty after a restart. So `recoverLeftovers` returns, for each kept volume set, an **authenticated recovery handle** that D registers in its ownership map after checking the volumes' runner and allocation labels. **Binding to an attempt:** task-storage volumes also carry `io.codeboost.attempt=<attemptId>`, set by `prepareTaskFilesystems` from a new `attemptId` parameter, and F saves the returned allocation ID as `attempts.allocation_id` in the same turn. The handle carries both IDs. F uses a handle only if its attempt ID and allocation ID both match one attempt row of this database. A handle that matches no row, or disagrees with the row, is reported and listed by `--list-unowned-agent-resources` for the user to remove by hand; it is never exported or removed automatically. `exportTaskDiff` and `removeTaskFilesystems` accept these handles. F never builds a `TaskFilesystems` value from report fields. |
250+
| Recovered storage handles | D | D tracks task-storage ownership in a process-local `WeakMap`, which is empty after a restart. So `recoverLeftovers` returns, for each kept volume set, an **authenticated recovery handle** that D registers in its ownership map after checking the volumes' runner and allocation labels. **Binding to an attempt:** task-storage volumes also carry `io.codeboost.attempt=<attemptId>`, set by `prepareTaskFilesystems` from a new `attemptId` parameter. **F chooses the allocation ID, not D.** F generates it (a lowercase UUID v4) and saves it as `attempts.allocation_id` **before** it starts the asynchronous allocation, then passes it to `prepareTaskFilesystems`, which labels the volumes and keeper with it (`io.codeboost.allocation`). So a crash in the middle of allocation still leaves a row that matches whatever D created. The D prerequisite includes accepting a caller-provided allocation ID. The handle carries both IDs. F uses a handle only if its attempt ID and allocation ID both match one attempt row of this database. A handle that matches no row, or disagrees with the row, is reported and listed by `--list-unowned-agent-resources` for the user to remove by hand; it is never exported or removed automatically. `exportTaskDiff` and `removeTaskFilesystems` accept these handles. F never builds a `TaskFilesystems` value from report fields. |
250251
| Scoped recovery | D | `recoverLeftovers(runnerOwner): Promise<RecoveryReport>` acts only on resources whose `io.codeboost.runner` label equals the token. It stops and removes agent containers, egress proxies and networks, and resolves only when they are gone. **It keeps task storage whole: the volumes and their keeper container**, which it recognises by its task-storage labels. The recovery handle depends on that keeper and lists them in the report by allocation, so F can export partial output first. F then calls `removeTaskFilesystems` for each one (startup recovery step 4). It rejects with a bounded diagnostic if anything cannot be stopped or removed. |
251252
| Unowned resources | D and F | Resources with codeboost labels but no `io.codeboost.runner` label (from builds before this change) fail closed. The report lists them, and F refuses to open admission while any exist. **codeboost never removes them itself**, because it cannot prove that no older-build runner is alive: older builds take no lock this build can see, and an idle older runner may have no container running at the moment of a check. So `--list-unowned-agent-resources` prints each resource and the exact `docker rm` / `docker network rm` / `docker volume rm` commands. The user stops every older codeboost process, runs those commands, and starts again. There is no machine-wide lock. |
252253
| When it runs | F | Only while holding this database's single-runner lock (startup recovery step 1 comes first). |
@@ -267,7 +268,7 @@ This runs before the coordinator opens.
267268

268269
1. Take the single-runner lock for this database (decision 1), **before opening the `Store`**, because opening runs migrations. If another process holds it, exit with "Another codeboost runner is using this database." The OS lock cannot name the holder, so the message does not claim to. Do not serve the review screen: it is not read-only, because `ReviewService.load()` records history when HEAD moves (`runner/review.ts`) and `act()` writes review actions. A true read-only mode would need `Store`-level write refusal and is out of scope for F1.
269270
2. **If D's recovery rejects, startup stops here**, before step 3 finalizes any row or step 6 hands anything to I3: close the `Store`, release the lock, and exit with D's diagnostic. The next start retries. Open the `Store` (migrations run here, under the lock). Read the runner owner token, or create it (see the table under "Shutdown"). Then call D's startup recovery with that token, and await it. **D does not provide this yet.** `agents/contract.ts` exposes only per-invocation handles, and the supervisor's cleanup ownership lives in memory, so it is lost when the process crashes. See the second prerequisite under "Shutdown".
270-
3. **Export first.** For each task-storage recovery handle that matches an attempt row (see "Recovered storage handles"), run the bounded partial-output export and save the file. This makes no database writes. Before F2 there are no writable attempts, so there are no handles. Then finalize the unclean leftovers. These are `pending` or `running` rows left by a crash, or by a shutdown whose terminal write failed. **All of step 3 is one transaction.** If it fails, for example because of a storage error, startup stops there: it does not run step 4, does not open admission, closes the `Store`, releases the lock, and exits with the error. The next start runs recovery again from step 1. Finalize each one, storing its `diagnostic_ref` from the export in the same transaction, using the first-reason table in "Rules for the running state", rule 2:
271+
3. **Export phase, outside any transaction.** For each task-storage recovery handle that matches an attempt row (see "Recovered storage handles"), run the bounded partial-output export. Write it to the diagnostics directory under the fixed name `<attemptId>.diff`, so a retry overwrites instead of duplicating. An export failure is kept in memory as that attempt's diagnostic ("Partial output could not be exported: …"), with no file. This phase makes no database writes. Before F2 there are no writable attempts, so there are no handles. Then finalize the unclean leftovers. These are `pending` or `running` rows left by a crash, or by a shutdown whose terminal write failed. **Finalization phase: one transaction** that only finalizes rows and stores each `diagnostic_ref` (or the export-failure diagnostic). It does no D or file work. If it fails, for example because of a storage error, startup stops there: it does not run step 4, does not open admission, closes the `Store`, releases the lock, and exits with the error. The next start runs recovery again from step 1, and its export phase rewrites the same fixed-name files. A diagnostics file whose attempt row never got a `diagnostic_ref` is removed by the diagnostics directory's normal retention. Finalize each one, storing its `diagnostic_ref` from the export in the same transaction, using the first-reason table in "Rules for the running state", rule 2:
271272
- first reason `cancelled` → `cancelled`; `stale` → `stale` with its cause;
272273
- first reason `shutdown`: if the row has a saved D `stop_reason` of `timeout`, `output-limit` or `capture-failure`, **D's reason wins** → `failed`, as at settlement. Otherwise → `cancelled` "Stopped by shutdown". **Limit:** D's `InvocationHandle` reports its stop reason only in the settled result, and F saves it only with the terminal write. If the process dies after D stopped for its own reason but before settlement, D's reason is lost with the process. Recovery then cannot see it and maps the row to a shutdown cancellation, which is requeued. The next attempt has its own deadline, so a lost timeout costs one bounded re-run, not an unbounded one. Keeping D's reason across a crash would need D to report it before settlement; that is not a prerequisite;
273274
- no first reason, and the captured context **is no longer current** → `stale` with its cause (the change happened, but the runner died before it recorded the reason). This is checked **first**, before both deadline fallbacks below, matching the settlement precedence;
@@ -505,6 +506,8 @@ Each case needs a test that fails before the fix and passes after it. Each test
505506
| Ledger write in the publication transaction (review round 27) | Completed `execute` → the step-3 guard passes → `completed` and `recordHistory` in one transaction → generation +1. A stale variant → refused, no ledger entry, the commit is discarded with the storage |
506507
| D timeout, then shutdown, then crash (review round 28) | (a) Row with a saved `stop_reason` of `timeout` and first reason `shutdown` → restart → `failed`, not requeued. (b) D timed out, then shutdown, then a crash before settlement (no `stop_reason` saved) → restart → `cancelled` "Stopped by shutdown" and requeued once; the requeued attempt has a fresh finite deadline |
507508
| Recovery keeps the keeper (review round 29) | Crash with task storage → `recoverLeftovers` → the agent container and network are gone, but the volumes and their keeper container remain → export succeeds → `removeTaskFilesystems` removes both |
509+
| Crash during async allocation (review round 30) | `allocation_id` saved → allocation starts → process killed → restart → the handle's attempt and allocation IDs match the row → exported and removed; never classed as unowned |
510+
| Recovery finalization fails after export (review round 30) | Export writes `<attemptId>.diff` → the finalization transaction fails → exit → next start re-exports to the same name and finalizes once |
508511
| Old attempt settles after a retry (D/F contract) | Attempt A cancelled and settled → retry B admitted → a late publish from A is refused → B's row and the visible status are unchanged |
509512

510513
## Decisions (approved 2026-09-25)

0 commit comments

Comments
 (0)