Skip to content

Commit 20256a9

Browse files
committed
Merge branch 'feat/d-bounded-settlement' of https://github.com/codeabovelab/codeboost into feat/d-async-launch
# Conflicts: # docs/implementation/runner-lifecycle.md
2 parents 92a04c1 + da89c77 commit 20256a9

6 files changed

Lines changed: 20 additions & 11 deletions

File tree

‎agents/client-outcome.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
/** A full Docker object ID. Every ownership check compares against this form, never against a reusable name. */
2+
export const DOCKER_ID = /^[0-9a-f]{64}$/;
3+
14
/** Spawn errors: the Docker client process never started, so it cannot have sent any request to the daemon. */
25
// `ENOTSTARTED` is agents/docker.ts's code for a call it refused before spawning (already cancelled, bad deadline).
36
const NOT_STARTED = new Set(['ENOENT', 'EACCES', 'EPERM', 'EAGAIN', 'EMFILE', 'ENFILE', 'ENOMEM', 'E2BIG', 'ENOEXEC',

‎agents/container/run.ts‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { realpathSync } from 'node:fs';
2-
import { createOutcomeUnknown } from '../client-outcome.ts';
2+
import { createOutcomeUnknown, DOCKER_ID } from '../client-outcome.ts';
33
import type { UnreleasedResource } from '../contract.ts';
44
import { docker as runDockerCommand, pause, runDocker, type DockerOutcome } from '../docker.ts';
55
import { assertContainerProfile, assertContainerProfileAuthenticity, disposeContainerProfile,
@@ -55,7 +55,6 @@ const unsettledCreates = new WeakMap<ContainerProfile, number>();
5555
// network or staging cleanup is profile-only and never looks the name up again.
5656
const createdContainers = new WeakMap<ContainerProfile, string | undefined>();
5757

58-
const CONTAINER_ID = /^[0-9a-f]{64}$/;
5958
/**
6059
* The immutable ID of the agent container this profile created, when `docker create` returned one. Every operation
6160
* after the create should use it: the name can be taken over by a replacement container.
@@ -130,7 +129,7 @@ const removeContainerOrThrow = async (profile: ContainerProfile, waitForSettle =
130129
// Remove and confirm by the ID the daemon just reported for our container, never by the name: a same-named
131130
// replacement created after this inspect must not be deleted.
132131
const id = inspected.Id;
133-
if (!id || !CONTAINER_ID.test(id) || (capturedId && id !== capturedId))
132+
if (!id || !DOCKER_ID.test(id) || (capturedId && id !== capturedId))
134133
throw new Error('Failed to establish the agent container identity; staged credentials were retained.');
135134
const result = await runDocker(['rm', '--force', id], { timeoutMs: remaining() });
136135
if (result.status !== 0) {
@@ -146,7 +145,7 @@ const removeContainerOrThrow = async (profile: ContainerProfile, waitForSettle =
146145

147146
/**
148147
* Remove a validated invocation container, then its profile-owned staging and network resources. `timeoutMs` bounds
149-
* the container step and, separately, the network step.
148+
* the whole removal: the container step and the network step share one deadline.
150149
*/
151150
export async function disposeValidatedContainer(profile: ContainerProfile, timeoutMs = 30_000): Promise<void> {
152151
assertContainerProfileAuthenticity(profile);
@@ -358,7 +357,7 @@ export async function createValidatedContainer(profile: ContainerProfile, timeou
358357
try {
359358
const id = await docker(profile.args, { timeoutMs: createTimeout, secrets, signal });
360359
// An unexpected create output leaves the ID unknown, so cleanup falls back to a verified name lookup.
361-
createdContainers.set(profile, CONTAINER_ID.test(id) ? id : undefined);
360+
createdContainers.set(profile, DOCKER_ID.test(id) ? id : undefined);
362361
}
363362
catch (error) {
364363
// A nonzero exit means the daemon answered; a killed client leaves the request in flight.

‎agents/contract.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,8 @@ export interface InvocationResult {
5050
/**
5151
* Present only when cleanup was still failing when D's bounded retry window ended (`CLEANUP_RETRY_WINDOW_MS`,
5252
* 60 s after the first failure). Retries after the first are limited to what is left of the window, and every
53-
* cleanup subprocess is killed at its deadline, so settlement ends within about twice the window. These resources
53+
* cleanup subprocess is killed at its deadline, so settlement ends within about 90 s of cleanup starting: at most
54+
* 30 s for the first, failed attempt, then the 60 s window, which no retry outlasts. These resources
5455
* may still exist, and the agent container may still be running. `stopReason` is always set. The caller must
5556
* record them durably and keep them owned until their removal is confirmed.
5657
*/

‎agents/network/network.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { randomUUID } from 'node:crypto';
22
import { assertCapturedInvocation, type InvocationInput, type UnreleasedResource } from '../contract.ts';
3-
import { createOutcomeUnknown } from '../client-outcome.ts';
3+
import { createOutcomeUnknown, DOCKER_ID } from '../client-outcome.ts';
44
import { assertBuiltAgentImage } from '../container/image.ts';
55
import { docker as runDockerCommand, pause, runDocker, type DockerOutcome } from '../docker.ts';
66

@@ -92,7 +92,7 @@ const remove = async (object: 'container' | 'network', target: string, remaining
9292
const labels = inspected?.Labels ?? inspected?.Config?.Labels;
9393
if (labels?.['io.codeboost.egress'] !== allocationId) throw new Error(`Refused to remove unowned ${kind}.`);
9494
const id = inspected?.Id;
95-
if (!id || !/^[0-9a-f]{64}$/.test(id) || (/^[0-9a-f]{64}$/.test(target) && id !== target))
95+
if (!id || !DOCKER_ID.test(id) || (DOCKER_ID.test(target) && id !== target))
9696
throw new Error(`Failed to establish the identity of ${kind}.`);
9797
const result = await runDocker(object === 'container' ? ['rm', '--force', id] : ['network', 'rm', id],
9898
{ timeoutMs: remaining() });
@@ -199,7 +199,7 @@ export async function createVendorNetwork(invocation: InvocationInput, imageId:
199199
let networkId: string | undefined, proxyId: string | undefined, registered: VendorNetwork | undefined;
200200
const unsettled = new Set<string>(), created = new Set<string>();
201201
const createdId = (value: string, kind: string) => {
202-
if (!/^[0-9a-f]{64}$/.test(value)) throw new Error(`Docker did not return the created ${kind} ID.`);
202+
if (!DOCKER_ID.test(value)) throw new Error(`Docker did not return the created ${kind} ID.`);
203203
return value;
204204
};
205205
// Run one create step; a client killed by its deadline leaves the daemon outcome for `object` unknown.

‎docs/implementation/agent-isolation.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,13 @@ The caller must do the following:
8787
stopped and its cleanup has finished. If cleanup keeps failing, the supervisor retries it every second for
8888
60 seconds after the first failure, then settles anyway. Retries after the first get only what is left of
8989
the window, and each cleanup subprocess is killed at its deadline. That result has a `stopReason` and lists
90-
the resources it could not confirm removed in `unreleased`; the container may still be running.
90+
the resources it could not confirm removed in `unreleased`; the container may still be running. Worst case,
91+
settlement ends about 90 seconds after cleanup starts: up to 30 seconds for the first, failed attempt, then
92+
the 60-second window. Cleanup retries are asynchronous Docker calls (#51 item 2), so the event loop stays free
93+
and a cancel or shutdown is handled while a retry runs; the window bounds the total wait.
94+
- Every Docker setup failure settles the start call's handle rather than throwing (#51 item 2). If profile creation
95+
fails and its own cleanup fails too, that handle keeps retrying the cleanup and settles with `capture-failure`
96+
(plus `unreleased` if the window ends), so callers must not rely on a throw for this case.
9197
- When `unreleased` is present, record those resources durably and keep them owned until their removal is
9298
confirmed. Each Docker entry has its creation-time ID (when known) and its ownership label; remove one only
9399
if both still match. An object is listed only if this invocation created it or may have (its create succeeded,

‎docs/implementation/runner-lifecycle.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,7 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
227227
| 1 | Set `stopping` on the server and `closing` on every coordinator (runner, questions, suggestions, merge), in the same synchronous turn, before any active-work list is copied. New API requests get HTTP 503. Every coordinator's start method checks `closing` synchronously and throws, so a request admitted before shutdown cannot start new work after it. | server flag: yes. Coordinator barrier: **new**. Today `close()` sets only `stopping`, and `questions.close()` runs later, so a request that was still reading its body can call `questions.start()` after shutdown began. The server also rechecks `stopping` after reading a body only for `merge`, so `service.act`, `setQuestionProvider` and future planning writers can still change the `Store`. F1 adds a `stopping` check after the body is read and before **every** mutating dispatch, returning HTTP 503, and adds regressions for each writer. Two GET handlers also write: `/api/review` (`ReviewService.load()` calls `recordHistory` when HEAD moved) and `/api/merge` (queue polling records merge observations and outcomes). `/api/review` can also write after an await: `merges.displayStatus()` checks GitHub and can then record a direct merge (`finishMergeAttempt`, `runner/merge.ts`). Per-handler checks would miss paths like this, so F1 adds a **write gate on the `Store`**. Step 1 closes the gate in the same synchronous turn. After that, every `Store` write method throws "Shutting down" unless the caller passes the shutdown capability. The server hands that capability at construction to each coordinator's own settlement and `close()` code: the runner, questions (`finishAnswer` after abort), suggestions (`settleSuggestion` in E3's `close()`) and merge. Startup recovery holds it too. HTTP handlers and `ReviewService` never receive it. So after step 1, only work that is settling can write, and it can still record its terminal rows before step 7 closes the `Store`. The gate throws a distinct `ShuttingDownError`, which the server maps to HTTP 503 "The review server is shutting down.", never to the 409 used for review errors. So an admitted `/api/review` whose `load()` reaches `recordHistory` after step 1 returns 503 with no view, and the UI treats it like any other 503 during shutdown. There is no partial or read-only view. Merge reconciliation and queue polling hit the same gate. **Today both catch every error and turn it into a status** (`displayStatus()` and `#pollQueue()` in `runner/merge.ts`), which would swallow the gate error and answer HTTP 200. F1 changes each of these catch blocks to rethrow `ShuttingDownError` first, before any other handling, so these paths also end with 503. The next startup checks GitHub again, so a skipped merge observation is recovered, not lost. Regressions cover `/api/review` history, `/api/review` direct-merge reconciliation during the GitHub await, and `/api/merge` polling. |
228228
| 2 | Stop accepting connections, and wait for admitted requests up to the drain limit (at most 14.5 s, below the 15 s request timeout). | yes |
229229
| 3 | After the drain limit, abort the signals of the remaining requests, destroy requests that are still reading a body, then await request-owned work (the merge coordinator). | yes |
230-
| 4 | For every `pending` and `running` attempt that has an in-memory job, record the first reason `shutdown` **only if no first reason is set yet**. A job already marked `cancelled`, `stale` or `time-limit` keeps its reason. For `running`, call `cancel('shutdown')` and await `settled`. For `pending`, await its preparation promise (the D start call itself returns at once, so it is never in progress), remove only its host-side preparation files (task storage waits for the terminal write, as in step 5), and cancel any handle that start returned, then await its `settled` (the "Launch" table). F does not abandon a job after a timer (decision 4). **This ends:** D's supervisor escalates to a forced kill, and retries unfinished container, network or setup cleanup every second for at most 60 seconds after the first failure (`CLEANUP_RETRY_WINDOW_MS`, `agents/adapters/supervisor.ts`). Retries after the first get only what is left of the window, and every cleanup subprocess is killed at its deadline. Then `settled` resolves with a `stopReason` and an `unreleased` list, even if Docker is unreachable or hangs. F records `unreleased` durably before it frees the slot. See the prerequisite below. | new |
230+
| 4 | For every `pending` and `running` attempt that has an in-memory job, record the first reason `shutdown` **only if no first reason is set yet**. A job already marked `cancelled`, `stale` or `time-limit` keeps its reason. For `running`, call `cancel('shutdown')` and await `settled`. For `pending`, await its preparation promise (the D start call itself returns at once, so it is never in progress), remove only its host-side preparation files (task storage waits for the terminal write, as in step 5), and cancel any handle that start returned, then await its `settled` (the "Launch" table). F does not abandon a job after a timer (decision 4). **This ends:** D's supervisor escalates to a forced kill, and retries unfinished container, network or setup cleanup every second for at most 60 seconds after the first failure (`CLEANUP_RETRY_WINDOW_MS`, `agents/adapters/supervisor.ts`). Retries after the first get only what is left of the window, and every cleanup subprocess is killed at its deadline, so settlement ends about 90 seconds after cleanup starts at worst (up to 30 seconds for the first, failed attempt, then the 60-second window). Then `settled` resolves with a `stopReason` and an `unreleased` list, even if Docker is unreachable or hangs. F records `unreleased` durably before it frees the slot. See the prerequisite below. | new |
231231
| 5 | For each attempt from step 4, in the order set by the task-storage holder row: the partial-output export (writable attempts), then the terminal state from its first reason (a `shutdown` reason gives `cancelled` "Stopped by shutdown") with its `diagnostic_ref`, then `removeTaskFilesystems`, and only then free its slot. A failed write leaves the row non-terminal, and a failed removal keeps the slot held. Startup recovery handles both. | new |
232232
| 6 | Await server closure; await the question and suggestion coordinators' `close()`. | yes (questions); suggestions: new wiring |
233233
| 7 | Close the `Store`. | yes |

0 commit comments

Comments
 (0)