Skip to content

Commit 9854fe3

Browse files
mchwangclaude
andauthored
D: abortable clone and storage preparation in process groups (#51 item 5) (#73)
* Add abortable clone and storage preparation that run in process groups (#51 item 5) createTaskCloneAsync and prepareTaskFilesystemsAsync take an AbortSignal and run every Git or Docker call as the leader of its own process group (runInProcessGroup in agents/process-group.ts). Each group is reported through onProcessGroup in the same turn as the spawn, so F can save it before anything else runs. On abort or at the deadline the group gets SIGTERM, then SIGKILL after 5 s, and the call settles only once the leader and every other member of its group have exited. An aborted clone removes its partial directory; an aborted allocation removes what it created with uncancelled cleanup. The clone and allocation are now step generators shared by both variants: the synchronous helpers keep their API and behaviour, and long file walks pause regularly so the asynchronous variants keep the event loop free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Kill a group the caller cannot record; never throw from group signals; report early aborts as cancelled runInProcessGroup ran onProcessGroup unguarded, so a throwing callback left the child running and unwatched. It now kills the group at once and settles only after it exits, with EUNRECORDED and the callback's error as the cause. signalGroup no longer throws (only ESRCH means gone), so a group member that cannot be signalled cannot crash the runner from a timer or hang the drain. An allocation ID check whose client was stopped or never started now reports why (for example cancelled) instead of a generic message. The real-Docker abort test checks groups with process.kill, which the kill utility cannot stand in for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Catch an abort raised during onProcessGroup; settle on the leader's exit, not its pipes An abort raised inside the onProcessGroup callback came before the listener existed and was never seen, so the child ran to its deadline. The helper now checks the signal right after attaching the listener. Settlement waited for 'close', which also needs every holder of the output pipes; a process that left the group with setsid but kept the pipe survived the group kill and hung the call forever. It now settles from the leader's exit: drain the group, wait up to 1 s for the pipes, then destroy them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Clear the SIGKILL timer once the group is empty; release the pipe wait; document settlement bounds The grace-period SIGKILL timer was cleared only after the up-to-1-second pipe wait, so a sub-second grace could fire it at an ID the group no longer held. It is now cleared as soon as the drain ends. The pipe-wait timer is cleared when the pipes close, so a finished call never keeps the process alive. The lifecycle doc now states the EGROUPALIVE and 1-second pipe bounds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pause while reading a large directory; start the deadline before the group is recorded The object audit and link check counted only paths taken off the queue, so reading one huge directory held the event loop (and an abort) for all of it. They now count every entry read too, and pause every 1,000 either way. The process deadline was armed only after onProcessGroup returned, so a slow durable write let the child run past timeoutMs; it is now armed at the spawn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add review rules from #73: reread whole functions, verify stated claims, independent review Copilot found three issues on #73 that several author self-reviews missed: a pause cadence claimed in comments but broken by an inner directory loop, and a deadline armed only after a synchronous callback. New review-readiness rules require rereading every changed function in each pass, verifying every behavioural claim with a failing-if-false test, and an independent review for concurrency, process and cleanup code. Two new required race cases cover synchronous lifecycle hooks and descendants that hold output pipes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Count a Git call as a yield in the clone walk-pause test The two object checks run before and after the clone call, which is awaited and so yields the event loop. The test counted reads across that boundary as one run and failed at 1,001 on Linux's object layout. It now resets the run at each Git spawn; it still fails at 2,506 on the pre-fix clone code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Report synchronous spawn failures as outcomes; stop nothing once the leader has exited spawn throws, rather than emitting 'error', for spawn failures such as ENOMEM or E2BIG. The throw rejected runInProcessGroup, so the step drivers skipped the generators' cleanup: an allocation left its volumes, stayed live and kept its claim. It is now reported as an outcome. stop() no longer runs after the leader exits, so output read during the pipe wait cannot signal a drained, possibly reused group ID or re-arm the SIGKILL timer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Report output held open past the group as an error, not a success When a process outside the group held the pipes past the 1 s wait, the call returned the leader's exit status with output that may be incomplete. It now settles with ESTDIOHELD and keeps what was read. Before deciding, it gives I/O one more event-loop turn, since after a long block the timer can fire before output that is already waiting is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Document ESTDIOHELD and synchronous spawn failures in the lifecycle bounds Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Settle truncated output as ENOBUFS; report every post-abort failure as a cancel Output crossing the limit after the leader exited was dropped while the call still reported success; any crossing now settles as ENOBUFS. An abort landing between two Git or Docker calls made the next call refuse to start, so the clone or allocation rejected with a plain error instead of an AbortError; both now rethrow any failure after the signal aborted as an AbortError with the original as its cause (an unsettled cleanup stays an AggregateError). Found by an independent review agent after the author's passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject deadlines longer than a Node timer; make the between-calls abort tests deterministic setTimeout treats delays above 2^31-1 ms as 1 ms, so a larger timeoutMs made the async allocation time out at once while the sync one worked. Both now refuse it up front, and runInProcessGroup reports it as not started. The between-calls abort tests aborted one turn after a child's exit, which in about a quarter of runs landed during the next call instead. They now wrap runInProcessGroup and abort after one call settles and before the next is requested, and assert that exactly one call was refused before it started. Found by a second independent review agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 90d8cec commit 9854fe3

11 files changed

Lines changed: 1029 additions & 96 deletions

‎AGENTS.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,17 @@ Before opening or updating a PR for asynchronous behavior, test every applicable
3939
- An abort error fires, then subprocess close arrives later.
4040
- Submit starts, then the user edits the composer or switches items, then the response returns.
4141
- Referenced code is reassigned or the snapshot changes, then retry or rendering occurs.
42+
- A synchronous caller hook inside a lifecycle (such as a callback that records a spawned process group) throws, blocks past the deadline, or aborts the signal, then the lifecycle continues. Timers and listeners that bound the lifecycle must already be armed when the hook runs, and a signal already aborted when its listener is attached must still take effect.
43+
- A child process exits, but a descendant outside its process group still holds its output pipes.
4244

4345
Every reproduced race requires a failing-before and passing-after regression. Assert both the visible result and the durable state when they can diverge.
4446

4547
## Review readiness
4648

4749
- Before requesting or re-requesting an automated Copilot review, self-review the full current diff, fix every issue found, and repeat the self-review and fix cycle until a complete pass finds no new issues. Re-run the relevant validation after fixes; only then request Copilot review.
50+
- Each self-review pass rereads every changed function in full against the base, not only the lines changed since the previous round. Code unchanged since the first commit of the PR still gets reviewed in every pass.
51+
- Treat every behavioural claim the change makes, in code comments, the PR body or docs (for example "pauses every 1,000 entries", "settles only after exit", "never throws", "bounded by N seconds"), as something to verify. Trace each claim through every path that can break it, including nested loops, callbacks, error paths and early returns, and give it a test that fails if the claim is false.
52+
- The author's self-review is not enough for concurrency, process, subprocess, timer or resource-cleanup code. Before requesting Copilot review, also run an independent review that does not share the author's context: `/codex review`, a separate review agent, or `/code-review` at `high` effort or above. Fix its findings like any other.
4853
- Run final validation against the exact pushed head after the last change.
4954
- Report current test counts separately from historical milestone counts.
5055
- Before requesting automated review, report the current head, CI state, mergeability, unresolved threads, and deferred follow-up issues.

‎agents/container/storage.ts‎

Lines changed: 158 additions & 62 deletions
Large diffs are not rendered by default.

‎agents/process-group.ts‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
import { spawn } from 'node:child_process';
2+
import { NOT_STARTED, type DockerOutcome } from './docker.ts';
3+
4+
/** A subprocess's process group, reported as soon as it exists so the caller can record it durably. */
5+
export interface ProcessGroup {
6+
/** The group ID, which is the leader's PID. */
7+
readonly pgid: number;
8+
/** `Date.now()` right after the spawn; with the ID, it tells this group from a later one that reuses the ID. */
9+
readonly startedAt: number;
10+
}
11+
export interface ProcessGroupOptions {
12+
readonly env: NodeJS.ProcessEnv;
13+
readonly timeoutMs: number;
14+
readonly cwd?: string;
15+
/** Aborting stops the group (SIGTERM, then SIGKILL after the grace period); the call still settles only after exit. */
16+
readonly signal?: AbortSignal;
17+
/** Called synchronously, in the same turn as the spawn, with the new group. */
18+
readonly onProcessGroup?: (group: ProcessGroup) => void;
19+
/** How long a stopped group gets between SIGTERM and SIGKILL. Default 5 s. */
20+
readonly graceMs?: number;
21+
/** Bytes kept of each of stdout and stderr; more stops the group. Default 16 MiB. */
22+
readonly maxBuffer?: number;
23+
}
24+
25+
const DEFAULT_GRACE_MS = 5_000;
26+
/** The longest delay a Node timer honours; `setTimeout` treats anything longer as 1 ms. */
27+
export const MAXIMUM_TIMER_MS = 2 ** 31 - 1;
28+
const DEFAULT_MAX_BUFFER = 16 * 1024 * 1024;
29+
// After the leader exits, how long to keep killing and polling other members of its group before giving up.
30+
const DRAIN_LIMIT_MS = 10_000;
31+
// Once the group is gone, how long to wait for stdout and stderr to reach end of file. Only a process outside the group
32+
// (one that called setsid but kept the inherited pipe) can hold them open after that, and it must not hold this call.
33+
const STDIO_CLOSE_MS = 1_000;
34+
const pause = (ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms));
35+
const notStarted = (message: string): DockerOutcome =>
36+
({ status: null, stdout: '', stderr: '', error: Object.assign(new Error(message), { code: NOT_STARTED }) });
37+
38+
// Signal every process in the group; false once the group no longer exists. Never throws: it runs in timers and
39+
// listeners, where a throw would crash the runner. Any failure other than ESRCH (such as EPERM, for a member that can
40+
// no longer be signalled) counts as still alive, so draining gives up at its limit and reports it.
41+
const signalGroup = (pgid: number, signal: NodeJS.Signals | 0) => {
42+
try { process.kill(-pgid, signal); return true; }
43+
catch (error) { return (error as NodeJS.ErrnoException).code !== 'ESRCH'; }
44+
};
45+
46+
// The leader can exit while other members of its group still run (for example a child it forked). A group ID stays
47+
// taken while any member exists, so until the group is empty the ID cannot name someone else's group.
48+
const drainGroup = async (pgid: number) => {
49+
const giveUpAt = performance.now() + DRAIN_LIMIT_MS;
50+
while (signalGroup(pgid, 0)) {
51+
signalGroup(pgid, 'SIGKILL');
52+
if (performance.now() >= giveUpAt) return false;
53+
await pause(20);
54+
}
55+
return true;
56+
};
57+
58+
/**
59+
* Run one command as the leader of a new process group. The group is reported through `onProcessGroup` before this
60+
* returns control to the event loop. On abort or at the deadline the whole group gets SIGTERM, then SIGKILL after the
61+
* grace period, so a child that ignores SIGTERM is still bounded. The promise settles only after the leader has
62+
* exited and every other member of its group is gone. Never rejects: failures are in the outcome, shaped like
63+
* `runDocker`'s, so `status` is a number only when the command ran to completion.
64+
*/
65+
export function runInProcessGroup(file: string, args: readonly string[],
66+
options: ProcessGroupOptions): Promise<DockerOutcome> {
67+
if (!Number.isSafeInteger(options.timeoutMs) || options.timeoutMs < 1 || options.timeoutMs > MAXIMUM_TIMER_MS)
68+
return Promise.resolve(notStarted(`Process deadline must be a positive integer of at most ${MAXIMUM_TIMER_MS} ms.`));
69+
if (options.signal?.aborted) return Promise.resolve(notStarted(`${file} was cancelled before it started.`));
70+
const graceMs = options.graceMs ?? DEFAULT_GRACE_MS, maxBuffer = options.maxBuffer ?? DEFAULT_MAX_BUFFER;
71+
return new Promise(resolve => {
72+
// `detached` makes the child the leader of a new process group (setsid), so signals to -pgid reach all it starts.
73+
// spawn throws, rather than emitting 'error', for invalid arguments and for spawn failures other than ENOENT,
74+
// EACCES, EAGAIN, EMFILE and ENFILE (for example ENOMEM or E2BIG). No child exists then; report it, never reject.
75+
let child: ReturnType<typeof spawn>;
76+
try {
77+
child = spawn(file, [...args], { cwd: options.cwd, env: options.env, detached: true,
78+
stdio: ['ignore', 'pipe', 'pipe'] });
79+
} catch (error) {
80+
resolve({ status: null, stdout: '', stderr: '', error: error as Error });
81+
return;
82+
}
83+
const pgid = child.pid;
84+
if (pgid === undefined) {
85+
// The spawn itself failed (for example ENOENT); nothing started, and 'error' carries the cause.
86+
child.once('error', error => resolve({ status: null, stdout: '', stderr: '', error }));
87+
return;
88+
}
89+
const out: Buffer[] = [], err: Buffer[] = [];
90+
let outBytes = 0, errBytes = 0, stopped: 'cancelled' | 'timeout' | 'output-limit' | 'unrecorded' | undefined;
91+
let graceTimer: ReturnType<typeof setTimeout> | undefined;
92+
// Set once the leader has exited: from then on the group is drained, not stopped, and its ID may be reused.
93+
let exited = false;
94+
const stop = (reason: NonNullable<typeof stopped>) => {
95+
if (stopped || exited) return;
96+
stopped = reason;
97+
signalGroup(pgid, 'SIGTERM');
98+
graceTimer = setTimeout(() => signalGroup(pgid, 'SIGKILL'), graceMs);
99+
};
100+
// Armed at the spawn, before the caller records the group: time spent recording counts against the deadline.
101+
const deadline = setTimeout(() => stop('timeout'), options.timeoutMs);
102+
// If the caller cannot record the group, the child must not outlive this call: it is killed at once below, and
103+
// the call still settles only after the group has exited, with the caller's error.
104+
let unrecorded: unknown;
105+
try { options.onProcessGroup?.(Object.freeze({ pgid, startedAt: Date.now() })); }
106+
catch (error) { unrecorded = error; }
107+
// Output past the limit stops a running group; once the leader has exited it only drops the excess. Either way the
108+
// output is incomplete, so the call settles as ENOBUFS, never as a success.
109+
let truncated = false;
110+
const collect = (chunks: Buffer[], add: (bytes: number) => number) => (chunk: Buffer) => {
111+
if (add(chunk.length) > maxBuffer) { truncated = true; stop('output-limit'); }
112+
else chunks.push(chunk);
113+
};
114+
child.stdout!.on('data', collect(out, bytes => (outBytes += bytes)));
115+
child.stderr!.on('data', collect(err, bytes => (errBytes += bytes)));
116+
const onAbort = () => stop('cancelled');
117+
options.signal?.addEventListener('abort', onAbort, { once: true });
118+
if (unrecorded !== undefined) { stopped = 'unrecorded'; signalGroup(pgid, 'SIGKILL'); }
119+
// An abort raised during onProcessGroup came before the listener existed, and an aborted signal never fires again.
120+
else if (options.signal?.aborted) stop('cancelled');
121+
let spawnError: Error | undefined;
122+
let pipesClosed = false;
123+
const closed = new Promise<void>(resolveClosed => child.once('close', () => { pipesClosed = true; resolveClosed(); }));
124+
// Settle from the leader's exit, not from 'close': 'close' also waits for every holder of the pipes, which can
125+
// include a process that left the group and so survives the group kill.
126+
let finished = false;
127+
const finish = (code: number | null, signal: NodeJS.Signals | null) => {
128+
if (finished) return;
129+
finished = true;
130+
exited = true;
131+
clearTimeout(deadline);
132+
options.signal?.removeEventListener('abort', onAbort);
133+
void drainGroup(pgid).then(async drained => {
134+
// The group is empty (or given up on): a later SIGKILL could reach a new group that reuses the ID.
135+
clearTimeout(graceTimer);
136+
let stdioTimer: ReturnType<typeof setTimeout> | undefined;
137+
await Promise.race([closed, new Promise<void>(done => { stdioTimer = setTimeout(done, STDIO_CLOSE_MS); })]);
138+
// Cleared so a finished call never keeps the process alive.
139+
clearTimeout(stdioTimer);
140+
// Timers run before I/O in each event-loop turn: after a long block the timer can win while the rest of the
141+
// output is already waiting. One more turn lets that I/O (and the end of the pipes) be read first.
142+
if (!pipesClosed) await new Promise(resolveTurn => setImmediate(resolveTurn));
143+
child.stdout!.destroy();
144+
child.stderr!.destroy();
145+
const stdout = Buffer.concat(out).toString('utf8'), stderr = Buffer.concat(err).toString('utf8');
146+
if (!drained) {
147+
resolve({ status: null, stdout, stderr, error: Object.assign(
148+
new Error(`${file} left processes in its group that did not exit after SIGKILL.`), { code: 'EGROUPALIVE' }) });
149+
return;
150+
}
151+
if (!pipesClosed) {
152+
// Only a process outside the group can still hold the pipes; the output may be incomplete, so this is not a
153+
// successful result even if the leader exited 0.
154+
resolve({ status: null, stdout, stderr, error: Object.assign(new Error(
155+
`${file} exited, but a process outside its group still holds its output; the output may be incomplete.`),
156+
{ code: 'ESTDIOHELD' }) });
157+
return;
158+
}
159+
if (stopped === 'unrecorded') {
160+
resolve({ status: null, stdout, stderr, error: Object.assign(new Error(
161+
`${file} was stopped because its process group could not be recorded.`, { cause: unrecorded }),
162+
{ code: 'EUNRECORDED' }) });
163+
return;
164+
}
165+
if (stopped === 'cancelled') {
166+
resolve({ status: null, stdout, stderr, error: Object.assign(new Error(`${file} ${args[0] ?? ''} was cancelled.`),
167+
{ name: 'AbortError', code: 'ABORT_ERR' }) });
168+
return;
169+
}
170+
if (truncated && !stopped) {
171+
resolve({ status: null, stdout, stderr, error: Object.assign(
172+
new Error(`${file} ${args[0] ?? ''} exceeded its output limit.`), { code: 'ENOBUFS' }) });
173+
return;
174+
}
175+
if (stopped) {
176+
resolve({ status: null, stdout, stderr, error: Object.assign(new Error(stopped === 'timeout'
177+
? `${file} ${args[0] ?? ''} ETIMEDOUT after ${options.timeoutMs} ms.`
178+
: `${file} ${args[0] ?? ''} exceeded its output limit.`), { code: stopped === 'timeout' ? 'ETIMEDOUT' : 'ENOBUFS' }) });
179+
return;
180+
}
181+
if (spawnError || code === null) {
182+
resolve({ status: null, stdout, stderr,
183+
error: spawnError ?? Object.assign(new Error(`${file} ${args[0] ?? ''} was killed by ${signal}.`), { signal }) });
184+
return;
185+
}
186+
resolve({ status: code, stdout, stderr });
187+
});
188+
};
189+
child.once('exit', finish);
190+
// A spawn that fails after the child had a PID emits 'error' and may never emit 'exit'.
191+
child.once('error', error => { spawnError = error; finish(null, null); });
192+
});
193+
}

‎docs/implementation/agent-isolation.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,8 +57,10 @@ metadata, read-only isolation and capacity probes had this defect.
5757

5858
Use only these entry points to run an agent:
5959

60-
1. `createTaskClone` creates a committed, standalone staging clone.
61-
2. `prepareTaskFilesystems` copies that clone into bounded task storage, labelled with the owner you pass: your
60+
1. `createTaskClone` creates a committed, standalone staging clone. The runner uses `createTaskCloneAsync`, which
61+
takes an `AbortSignal` and reports each Git process group through `onProcessGroup` (#51 item 5).
62+
2. `prepareTaskFilesystems` (or `prepareTaskFilesystemsAsync`, which takes `signal` and `onProcessGroup` in the same
63+
way) copies that clone into bounded task storage, labelled with the owner you pass: your
6264
runner token, the attempt ID, and an allocation ID (a lowercase UUID v4) you record first. Use each allocation
6365
ID once: before creating anything, D refuses an ID that another allocation in this process holds, or that any
6466
container, volume or network still carries (so a reuse after a restart is caught too). That check and the first

0 commit comments

Comments
 (0)