Skip to content

Commit da89c77

Browse files
mchwangclaude
andcommitted
Address review suggestions: shared Docker ID check, accurate bounds
Review of #69: - disposeValidatedContainer's comment said the container and network steps had separate budgets; they share one deadline. Comment corrected. - The 64-hex Docker ID check was written three times across run.ts and network.ts. One DOCKER_ID in agents/client-outcome.ts now backs all ownership checks. - agent-isolation.md now says cleanup retries are still synchronous (up to 30 s each), so the window bounds the total wait, not responsiveness, until #51 item 2. - The settlement bound is stated as a worked number (about 90 s after cleanup starts: 30 s first attempt, then the 60 s window) in contract.ts, agent-isolation.md and runner-lifecycle.md. - agent-isolation.md documents that a start call whose profile creation and cleanup both fail now returns a retaining handle instead of re-throwing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 1ca7e95 commit da89c77

6 files changed

Lines changed: 21 additions & 11 deletions

File tree

‎agents/client-outcome.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
/** A full Docker object ID. Every ownership check compares against this form, never against a reusable name. */
2+
export const DOCKER_ID = /^[0-9a-f]{64}$/;
3+
14
/** Spawn errors: the Docker client process never started, so it cannot have sent any request to the daemon. */
25
const NOT_STARTED = new Set(['ENOENT', 'EACCES', 'EPERM', 'EAGAIN', 'EMFILE', 'ENFILE', 'ENOMEM', 'E2BIG', 'ENOEXEC']);
36

‎agents/container/run.ts‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { execFileSync, spawnSync } from 'node:child_process';
22
import { realpathSync } from 'node:fs';
3-
import { createOutcomeUnknown } from '../client-outcome.ts';
3+
import { createOutcomeUnknown, DOCKER_ID } from '../client-outcome.ts';
44
import type { UnreleasedResource } from '../contract.ts';
55
import { assertContainerProfile, assertContainerProfileAuthenticity, disposeContainerProfile,
66
isContainerProfileAuthentic, profileTimeout,
@@ -59,7 +59,6 @@ const unsettledCreates = new WeakMap<ContainerProfile, number>();
5959
// network or staging cleanup is profile-only and never looks the name up again.
6060
const createdContainers = new WeakMap<ContainerProfile, string | undefined>();
6161

62-
const CONTAINER_ID = /^[0-9a-f]{64}$/;
6362
/**
6463
* The immutable ID of the agent container this profile created, when `docker create` returned one. Every operation
6564
* after the create should use it: the name can be taken over by a replacement container.
@@ -135,7 +134,7 @@ const removeContainerOrThrow = (profile: ContainerProfile, waitForSettle = false
135134
// Remove and confirm by the ID the daemon just reported for our container, never by the name: a same-named
136135
// replacement created after this inspect must not be deleted.
137136
const id = inspected.Id;
138-
if (!id || !CONTAINER_ID.test(id) || (capturedId && id !== capturedId))
137+
if (!id || !DOCKER_ID.test(id) || (capturedId && id !== capturedId))
139138
throw new Error('Failed to establish the agent container identity; staged credentials were retained.');
140139
const result = spawnSync('docker', ['rm', '--force', id], {
141140
encoding: 'utf8', timeout: remaining(), killSignal: 'SIGKILL', env: dockerEnvironment(),
@@ -157,7 +156,7 @@ const removeContainerOrThrow = (profile: ContainerProfile, waitForSettle = false
157156

158157
/**
159158
* Remove a validated invocation container, then its profile-owned staging and network resources. `timeoutMs` bounds
160-
* the container step and, separately, the network step.
159+
* the whole removal: the container step and the network step share one deadline.
161160
*/
162161
export function disposeValidatedContainer(profile: ContainerProfile, timeoutMs = 30_000): void {
163162
assertContainerProfileAuthenticity(profile);
@@ -360,7 +359,7 @@ export function createValidatedContainer(profile: ContainerProfile, timeoutMs =
360359
try {
361360
const id = docker(profile.args, { timeoutMs: createTimeout, secrets });
362361
// An unexpected create output leaves the ID unknown, so cleanup falls back to a verified name lookup.
363-
createdContainers.set(profile, CONTAINER_ID.test(id) ? id : undefined);
362+
createdContainers.set(profile, DOCKER_ID.test(id) ? id : undefined);
364363
}
365364
catch (error) {
366365
// A nonzero exit means the daemon answered; a killed client leaves the request in flight.

‎agents/contract.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,8 @@ export interface InvocationResult {
5050
/**
5151
* Present only when cleanup was still failing when D's bounded retry window ended (`CLEANUP_RETRY_WINDOW_MS`,
5252
* 60 s after the first failure). Retries after the first are limited to what is left of the window, and every
53-
* cleanup subprocess is killed at its deadline, so settlement ends within about twice the window. These resources
53+
* cleanup subprocess is killed at its deadline, so settlement ends within about 90 s of cleanup starting: at most
54+
* 30 s for the first, failed attempt, then the 60 s window, which no retry outlasts. These resources
5455
* may still exist, and the agent container may still be running. `stopReason` is always set. The caller must
5556
* record them durably and keep them owned until their removal is confirmed.
5657
*/

‎agents/network/network.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import { execFileSync, spawnSync } from 'node:child_process';
22
import { randomUUID } from 'node:crypto';
33
import { assertCapturedInvocation, type InvocationInput, type UnreleasedResource } from '../contract.ts';
4-
import { createOutcomeUnknown } from '../client-outcome.ts';
4+
import { createOutcomeUnknown, DOCKER_ID } from '../client-outcome.ts';
55
import { assertBuiltAgentImage } from '../container/image.ts';
66

77
export const VENDOR_HOSTS = Object.freeze({
@@ -95,7 +95,7 @@ const remove = (object: 'container' | 'network', target: string, remaining: () =
9595
const labels = inspected?.Labels ?? inspected?.Config?.Labels;
9696
if (labels?.['io.codeboost.egress'] !== allocationId) throw new Error(`Refused to remove unowned ${kind}.`);
9797
const id = inspected?.Id;
98-
if (!id || !/^[0-9a-f]{64}$/.test(id) || (/^[0-9a-f]{64}$/.test(target) && id !== target))
98+
if (!id || !DOCKER_ID.test(id) || (DOCKER_ID.test(target) && id !== target))
9999
throw new Error(`Failed to establish the identity of ${kind}.`);
100100
const result = spawnSync('docker', object === 'container' ? ['rm', '--force', id] : ['network', 'rm', id],
101101
{ encoding: 'utf8', timeout: remaining(), killSignal: 'SIGKILL', env: environment(), stdio: ['ignore', 'pipe', 'pipe'] });
@@ -197,7 +197,7 @@ export function createVendorNetwork(invocation: InvocationInput, imageId: string
197197
let networkId: string | undefined, proxyId: string | undefined;
198198
const unsettled = new Set<string>(), created = new Set<string>();
199199
const createdId = (value: string, kind: string) => {
200-
if (!/^[0-9a-f]{64}$/.test(value)) throw new Error(`Docker did not return the created ${kind} ID.`);
200+
if (!DOCKER_ID.test(value)) throw new Error(`Docker did not return the created ${kind} ID.`);
201201
return value;
202202
};
203203
// Run one create step; a client killed by its deadline leaves the daemon outcome for `object` unknown.

‎docs/implementation/agent-isolation.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,14 @@ The caller must do the following:
8484
stopped and its cleanup has finished. If cleanup keeps failing, the supervisor retries it every second for
8585
60 seconds after the first failure, then settles anyway. Retries after the first get only what is left of
8686
the window, and each cleanup subprocess is killed at its deadline. That result has a `stopReason` and lists
87-
the resources it could not confirm removed in `unreleased`; the container may still be running.
87+
the resources it could not confirm removed in `unreleased`; the container may still be running. Worst case,
88+
settlement ends about 90 seconds after cleanup starts: up to 30 seconds for the first, failed attempt, then
89+
the 60-second window. Cleanup retries are still synchronous Docker calls of up to 30 seconds each, so a cancel
90+
or shutdown waits behind a retry that is already running. The window bounds the total wait, not how quickly
91+
the process responds; asynchronous D helpers (#51 item 2) address that.
92+
- If profile creation fails and its own cleanup fails too, the Claude and Codex start calls return a handle that
93+
keeps retrying that cleanup and settles with `capture-failure` (plus `unreleased` if the window ends). They no
94+
longer retry once and re-throw the startup error, so callers must not rely on a throw for this case.
8895
- When `unreleased` is present, record those resources durably and keep them owned until their removal is
8996
confirmed. Each Docker entry has its creation-time ID (when known) and its ownership label; remove one only
9097
if both still match. An object is listed only if this invocation created it or may have (its create succeeded,

‎docs/implementation/runner-lifecycle.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,7 @@ The server computes `retryable` and sends it to the UI. The UI never works it ou
227227
| 1 | Set `stopping` on the server and `closing` on every coordinator (runner, questions, suggestions, merge), in the same synchronous turn, before any active-work list is copied. New API requests get HTTP 503. Every coordinator's start method checks `closing` synchronously and throws, so a request admitted before shutdown cannot start new work after it. | server flag: yes. Coordinator barrier: **new**. Today `close()` sets only `stopping`, and `questions.close()` runs later, so a request that was still reading its body can call `questions.start()` after shutdown began. The server also rechecks `stopping` after reading a body only for `merge`, so `service.act`, `setQuestionProvider` and future planning writers can still change the `Store`. F1 adds a `stopping` check after the body is read and before **every** mutating dispatch, returning HTTP 503, and adds regressions for each writer. Two GET handlers also write: `/api/review` (`ReviewService.load()` calls `recordHistory` when HEAD moved) and `/api/merge` (queue polling records merge observations and outcomes). `/api/review` can also write after an await: `merges.displayStatus()` checks GitHub and can then record a direct merge (`finishMergeAttempt`, `runner/merge.ts`). Per-handler checks would miss paths like this, so F1 adds a **write gate on the `Store`**. Step 1 closes the gate in the same synchronous turn. After that, every `Store` write method throws "Shutting down" unless the caller passes the shutdown capability. The server hands that capability at construction to each coordinator's own settlement and `close()` code: the runner, questions (`finishAnswer` after abort), suggestions (`settleSuggestion` in E3's `close()`) and merge. Startup recovery holds it too. HTTP handlers and `ReviewService` never receive it. So after step 1, only work that is settling can write, and it can still record its terminal rows before step 7 closes the `Store`. The gate throws a distinct `ShuttingDownError`, which the server maps to HTTP 503 "The review server is shutting down.", never to the 409 used for review errors. So an admitted `/api/review` whose `load()` reaches `recordHistory` after step 1 returns 503 with no view, and the UI treats it like any other 503 during shutdown. There is no partial or read-only view. Merge reconciliation and queue polling hit the same gate. **Today both catch every error and turn it into a status** (`displayStatus()` and `#pollQueue()` in `runner/merge.ts`), which would swallow the gate error and answer HTTP 200. F1 changes each of these catch blocks to rethrow `ShuttingDownError` first, before any other handling, so these paths also end with 503. The next startup checks GitHub again, so a skipped merge observation is recovered, not lost. Regressions cover `/api/review` history, `/api/review` direct-merge reconciliation during the GitHub await, and `/api/merge` polling. |
228228
| 2 | Stop accepting connections, and wait for admitted requests up to the drain limit (at most 14.5 s, below the 15 s request timeout). | yes |
229229
| 3 | After the drain limit, abort the signals of the remaining requests, destroy requests that are still reading a body, then await request-owned work (the merge coordinator). | yes |
230-
| 4 | For every `pending` and `running` attempt that has an in-memory job, record the first reason `shutdown` **only if no first reason is set yet**. A job already marked `cancelled`, `stale` or `time-limit` keeps its reason. For `running`, call `cancel('shutdown')` and await `settled`. For `pending`, await its preparation promise (and the D start call if it is in progress), remove only its host-side preparation files (task storage waits for the terminal write, as in step 5), and cancel any handle that start returned, then await its `settled` (the "Launch" table). F does not abandon a job after a timer (decision 4). **This ends:** D's supervisor escalates to a forced kill, and retries unfinished container, network or setup cleanup every second for at most 60 seconds after the first failure (`CLEANUP_RETRY_WINDOW_MS`, `agents/adapters/supervisor.ts`). Retries after the first get only what is left of the window, and every cleanup subprocess is killed at its deadline. Then `settled` resolves with a `stopReason` and an `unreleased` list, even if Docker is unreachable or hangs. F records `unreleased` durably before it frees the slot. See the prerequisite below. | new |
230+
| 4 | For every `pending` and `running` attempt that has an in-memory job, record the first reason `shutdown` **only if no first reason is set yet**. A job already marked `cancelled`, `stale` or `time-limit` keeps its reason. For `running`, call `cancel('shutdown')` and await `settled`. For `pending`, await its preparation promise (and the D start call if it is in progress), remove only its host-side preparation files (task storage waits for the terminal write, as in step 5), and cancel any handle that start returned, then await its `settled` (the "Launch" table). F does not abandon a job after a timer (decision 4). **This ends:** D's supervisor escalates to a forced kill, and retries unfinished container, network or setup cleanup every second for at most 60 seconds after the first failure (`CLEANUP_RETRY_WINDOW_MS`, `agents/adapters/supervisor.ts`). Retries after the first get only what is left of the window, and every cleanup subprocess is killed at its deadline, so settlement ends about 90 seconds after cleanup starts at worst (up to 30 seconds for the first, failed attempt, then the 60-second window). Then `settled` resolves with a `stopReason` and an `unreleased` list, even if Docker is unreachable or hangs. F records `unreleased` durably before it frees the slot. See the prerequisite below. | new |
231231
| 5 | For each attempt from step 4, in the order set by the task-storage holder row: the partial-output export (writable attempts), then the terminal state from its first reason (a `shutdown` reason gives `cancelled` "Stopped by shutdown") with its `diagnostic_ref`, then `removeTaskFilesystems`, and only then free its slot. A failed write leaves the row non-terminal, and a failed removal keeps the slot held. Startup recovery handles both. | new |
232232
| 6 | Await server closure; await the question and suggestion coordinators' `close()`. | yes (questions); suggestions: new wiring |
233233
| 7 | Close the `Store`. | yes |

0 commit comments

Comments
 (0)