Skip to content

Commit 15ee8fa

Browse files
mchwangclaude
andauthored
D: label every resource with its runner, attempt and allocation (#51 item 3) (#71)
* Bound D cleanup retries so settlement always ends (#51 item 1) The supervisor retried unconfirmed container, network and setup cleanup every second with no limit, so an unreachable Docker daemon kept `settled` pending forever. Retries now stop 60 s after the first failure, and the handle settles with its stop reason and an `unreleased` list of resources whose removal was not confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fail closed on unreleased resources and report them precisely Fixes from two review rounds on the bounded-settlement change: - Ask counted a result with `unreleased` as an ordinary failure and could start another invocation. It now marks the resources as untracked leftovers, which turns Ask off until a restart finds none. - Container cleanup ran `spawnSync` without SIGKILL, so a Docker client that ignores SIGTERM could block past its deadline and keep `settled` pending. Cleanup subprocesses are now killed, and retries after the first get only what is left of the 60 s window. - The agent container was reported whenever container-level cleanup ran, including after a create the daemon refused because another invocation holds the name. It is now reported only if this profile's create ran. - Each unreleased Docker object now carries its creation-time ID and its ownership label, so recovery can check both instead of trusting a name. - A profile that settled with unconfirmed cleanup can no longer launch again. - Docs no longer claim startup recovery already removes these resources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Return D start handles at once and run Docker setup inside them (#51 item 2) The Claude and Codex start calls created the vendor network, the container profile and the container with execFileSync/spawnSync, so the server could not begin shutdown or cancel while they ran. Now: - agents/docker.ts runs Docker calls asynchronously, killing the client with SIGKILL at its deadline or when its AbortSignal fires. - Network, profile and container create/validate/dispose are async; setup calls take an AbortSignal, and cleanup is never cancelled. - startProfileInvocation returns its handle immediately and creates and validates the container inside it; a stop during setup kills the in-flight Docker call, cleans up, then settles. - launchInvocation gives the adapters the same shape for network and profile creation; setUpProfile replaces the duplicated adapter setup code. - A start call throws only when it allocated nothing; every other setup failure settles the handle with a stop reason and a diagnostic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep cleanup retries inside the window and report only unconfirmed parts Review fixes (Copilot, #69): - Retained setup cleanup ignored the budget the retry loop computed, so a retry could start near the end of the window and run a fresh 30 s. The budget now reaches every cleanup callback (network creation, profile creation, adapter setup), and no attempt starts once the window has ended. - A vendor network was reported as proxy plus network even after one part was confirmed removed, or when a create was refused. Removal is now tracked per object, and cleanup-error inventories are read when the window ends. - Ask turned off only for a non-empty `unreleased`; an empty list also means cleanup was not confirmed, so presence is now the signal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stop the handed-off handle when the launch budget ends Review fix (Copilot, #70): launchInvocation cleared its deadline timer once setup handed off to the supervisor, so a supervisor started without the adapter's invocationBudget could outlive the launch budget. The timer now lives until the handle settles and cancels the handed-off handle with `timeout`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remove and control the agent container by its ID, never its name Review fix (Copilot, #69): cleanup inspected the agent container by name and then ran `rm --force` by name, so a same-named replacement created between the two calls could be deleted. Cleanup now looks the container up by the ID that `docker create` returned (by name only when the create's outcome is unknown), checks the ownership label, and removes and confirms by the ID the daemon reported. The supervisor's validate, start, exec, stop, kill and rm calls and the Codex output read also target that ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Validate and start the agent container by its ID on every launch path Review fix (Copilot, #69): createValidatedContainer and the exported startValidatedContainer/runContainer path still validated and started the container by `profile.name` after the create, so a same-named replacement could run. Both now use the ID `docker create` returned and fail if it is unknown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Release a profile's own resources when its container create was refused Review fix (Copilot, #70 summary; the code is on this branch): when `docker create` was refused because another invocation holds the name, the profile owned no container, but cleanup still inspected that name, found the foreign container, and refused. The profile's network and staging (including the Codex credential copy) were then retried and reported as unreleased instead of being removed. A profile whose create never ran or was refused now skips the container step and removes only its own resources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Await profile cleanup when no container create ran Review fix (Copilot, #70): the merge of the refused-create fix brought its synchronous `disposeContainerProfile` call into this async branch without an `await`, so removeContainerOrThrow resolved before the profile's network and staging were removed, and a cleanup failure was detached. It is now awaited, so a refused create releases the profile before the handle settles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Do not clean up a vendor network or proxy whose create the daemon refused Review fix (Copilot, #70 summary; the code is on this branch): after a daemon-confirmed create failure, network setup cleanup still tried to remove the object by name. If the name belonged to someone else, cleanup refused, and the refused create came back as an unsettled cleanup to retry. Cleanup and the unreleased inventory now consider an object only if its create succeeded or its client was killed before the daemon answered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Report a setup failure caused by the deadline as a timeout Review fix (Copilot, #70 summary): during asynchronous setup, a Docker client killed at the invocation deadline can reject a moment before the supervisor's or launcher's own deadline timer fires, so setup was reported as `capture-failure` instead of `timeout`. A setup error caused by a deadline (ETIMEDOUT, or an expired budget, including through `cause` and the startup half of a cleanup aggregate) within 1 s of the invocation deadline is now a timeout. Earlier Docker timeouts, such as a 60 s step cap, stay failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin the container identity rule for profile-only cleanup retries Review follow-up (Copilot, #69): cleanup already falls back to the container name only for a create that ran with an unknown ID, and drops the ID record only after the container's removal is confirmed, so a retry of the profile's network or staging cleanup is profile-only. Document that invariant and add a real-Docker regression: our container is removed, the network removal fails, a same-named, same-labelled container appears, and the retry leaves it alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remove egress objects by inspected ID; keep a proxy whose start failed Review fixes (Copilot, #69): - Network and proxy cleanup inspected an object (by name when its create ID was unknown), checked the label, then removed and confirmed by that name, so a same-named replacement created in between could be deleted. Removal and confirmation now use the ID from that inspect, which must match the target when the target is an ID. - `docker run --detach` can create the proxy and then exit nonzero because the start failed. The refused-create rule treated that answered failure as "nothing created" and leaked the container. A failed proxy run now counts as possibly created, so cleanup looks it up by name behind the label check and reports it if removal is not confirmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Refuse every launch path for a profile that settled with leftovers Review fix (Copilot, #69): the guard against relaunching a profile whose invocation settled with `unreleased` resources lived only in startProfileInvocation. createValidatedContainer, startValidatedContainer and runContainer could still create or start the leftover container that recovery owns. The retired-profile state now lives in run.ts and every launch entry point checks it before doing any work; cleanup remains allowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Create and start the egress proxy separately; document Ask's host recovery Review fixes (Copilot, #69): - `docker run --detach` mixed two outcomes: a refused create (for example, a name held by another container) made nothing, while a failed start left our container. Treating both as "created" reported and inspected a foreign same-named container. The proxy is now created, then started: a refused create is absence, a failed start leaves a container owned by its ID. - Ask's handling of D's host staging directories was questioned. They are recovered through the Ask root (the worker's TMPDIR, recorded durably before setup and deleted at startup before Ask is enabled); say so at the point where Ask marks unreleased results. - Refresh the isolation doc's `unreleased` and retired-profile rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Refuse an untrusted image synchronously; fix stale launch docs Review fixes (Copilot, #70 summary): - The start-call rule is "throw only when nothing was allocated, for example invalid input", but an untrusted image ID was only rejected inside asynchronous setup, so the handle settled with a failure instead. Both adapters now check the image before launching. - runner-lifecycle.md still said D's adapters throw after their setup cleanup succeeds, and that shutdown may await a start call in progress. Both now describe the asynchronous start: it returns at once, and setup failures settle the handle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Treat a Docker client that never started as having created nothing Review fix (Copilot, #69 summary): a failed create with no exit status was read as "the request may have reached the daemon". A spawn failure (ENOENT, EACCES, EAGAIN and similar) sent nothing, but it too was treated as unknown, so cleanup waited out a settle window and could report a never-created agent container, network or proxy as unreleased. One shared classifier (`createOutcomeUnknown`) now separates a daemon answer, a client that never started, and a killed client, for both the agent container and the egress objects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Do not treat a create cancelled before it started as possibly created Review fix (Copilot, #70 summary: network cancellation race): when cancel() landed between two setup steps, the next Docker call saw an aborted signal and returned without starting a client. That refusal carried no exit status or spawn code, so the create classifier read it as "outcome unknown": the create that never ran (for example the proxy) counted as possibly created, the cancel waited out a settle window, and the object could be reported as unreleased. runDocker now marks calls it refuses before spawning with `ENOTSTARTED`, which the classifier treats as nothing created. A cancel that kills a client already in flight still leaves the outcome unknown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the daemon's exit status when a cancel lands after the client exited Review fix (Copilot, #70): runDocker checked the AbortSignal's current state before trusting a numeric exit code. If the Docker client had already exited with a daemon answer (for example a name conflict) and cancel() ran before the callback, the answer was discarded and the call was treated as killed with an unknown outcome, so a refused create counted as possibly created. Only Node's own abort error now marks a call as cancelled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Retry the profile cleanup once per attempt in the adapters Review fix (Copilot, #69): when profile creation and its cleanup both failed, each adapter retry ran the profile's cleanup (which already removes the vendor network) and then removed that network again with a fresh full budget, so a retained retry could run past the shrinking cleanup window. The adapters now retain exactly the profile's cleanup with the window's budget, as the network-creation path already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Make assertVendorNetwork async so a bad deadline rejects Fixes the real-docker CI failure on #70 (agent-network "rejects a copied network capability"): assertVendorNetwork returned a promise but was not declared async, so an invalid timeout threw synchronously from deadline() instead of rejecting. Every other async D entry point already rejects. 198 of 200 real-Docker tests passed in that run; this was the only failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review suggestions: shared Docker ID check, accurate bounds Review of #69: - disposeValidatedContainer's comment said the container and network steps had separate budgets; they share one deadline. Comment corrected. - The 64-hex Docker ID check was written three times across run.ts and network.ts. One DOCKER_ID in agents/client-outcome.ts now backs all ownership checks. - agent-isolation.md now says cleanup retries are still synchronous (up to 30 s each), so the window bounds the total wait, not responsiveness, until #51 item 2. - The settlement bound is stated as a worked number (about 90 s after cleanup starts: 30 s first attempt, then the 60 s window) in contract.ts, agent-isolation.md and runner-lifecycle.md. - agent-isolation.md documents that a start call whose profile creation and cleanup both fail now returns a retaining handle instead of re-throwing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Release the profile if the supervisor handoff throws; add signals to runContainer Review fixes (Copilot, #70): - setUpProfile returned `start(profile)` unguarded. If the handoff threw after the profile was created, the launcher treated it as a failure with nothing allocated and the profile's network and staging leaked. It now disposes the profile, or hands a retryable cleanup to the launcher. - startValidatedContainer and runContainer, the lower-level launch path, took no AbortSignal. Both now accept one and pass it to create, validate and start; cleanup still runs after a cancel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Label every D resource with its runner, attempt and allocation (#51 item 3) - InvocationInput gains a required runnerOwner (32 lowercase hex), checked by captureInvocation. - agents/labels.ts defines io.codeboost.runner, io.codeboost.attempt and io.codeboost.allocation, validates owners, and builds and checks the labels. - prepareTaskFilesystems takes an owner with a caller-chosen allocation ID (lowercase UUID v4) instead of generating one; its volumes, keeper and seeder carry all three labels. It also classifies failed allocations with createOutcomeUnknown. - createVendorNetwork takes a caller-chosen allocation ID (the adapters' request carries networkAllocationId); the network and proxy carry the invocation's runner and attempt and that allocation. - The agent container carries its own attempt and the task-storage allocation it mounts. Profile creation refuses storage from another runner. - Every validator checks the owner labels. - Ask passes a random per-session owner until F1d's per-database token exists, and fresh allocation IDs per question. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Refuse label-incompatible attempt IDs at capture; claim allocation IDs once Review fixes (#71 code review): - captureInvocation accepted any non-empty attempt ID, but the owner labels need a narrower form, so a bad ID was only rejected inside asynchronous setup and settled the handle instead of throwing. It is now refused at capture, after the existing identity check. - One allocation ID could name two allocations, which item 4's recovery would merge. claimAllocationId (labels.ts) now refuses a reused ID, shared by task storage and vendor networks, before anything is created. - Drop the stale doc comment on prepareTaskFilesystems and the duplicate allocationId field; the owner is the single source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check allocation reuse at the daemon; test every owner-label check Review fixes (#71 high-effort review): - Allocation-ID reuse was only caught within one process. Both allocators now ask the daemon (`allocationListCommands`) for any container, volume or network still labelled with the ID before creating anything, failing closed if Docker does not answer. The in-process claim now covers only allocations in progress (or whose cleanup did not settle), so it stays bounded. - UnreleasedResource reports every ownership label as `labels` (runner, attempt, allocation, and the invocation or egress label) instead of a single `owner` label, so recorded leftovers match what recovery will scan. The agent container's labels are captured at create time, so reporting cannot throw while a handle settles. - validateContainer computes the storage owner once and drops the allocation comparisons that hasOwnerLabels already makes. - New real-Docker tests strip the runner label from the real Docker call for the network, proxy, task volumes, keeper and agent container, and check that each validator rejects the result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test network allocation reuse; parallel, literal allocation lists Review suggestions (#71): - Add a real-Docker test that a second vendor network with a live network's allocation ID is refused before anything is created. - The network's three daemon list calls run in parallel. - allocationListCommands spells out its three commands instead of splitting strings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check every owner label before cleanup; re-check allocation IDs after the first create Cleanup of task storage and of a name-resolved vendor network or proxy now requires the runner, attempt and allocation labels together. Right after its first create, each allocator checks that its object is the only one carrying the allocation ID, so two processes racing past the first check back out. The adapters validate networkAllocationId before returning a handle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rewrap the network remove comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remove by captured ID regardless of labels; require every label by name A captured Docker ID proves the object is the one D created, so network, proxy and agent-container cleanup removes it even when validation refused its labels. An object found only by name must carry every label D wrote, including the agent container's owner labels. Storage cleanup targets only names whose create succeeded or whose outcome is unknown; tests that build mislabelled storage now expect cleanup to refuse it and remove it by hand. Document the limits of the allocation uniqueness checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let the Docker test shim's script run the command itself The killed-create tests edit the command before it runs; the shim ran the unedited command first, so a correctly labelled object was created and the edited create then failed on the name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Describe cleanup ownership as ID proof or all labels by name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remove task containers by the ID just inspected Storage cleanup inspected a keeper or seeder by name, then removed it by name, so a same-named replacement created in between could be deleted. It now removes and confirms by the inspected ID. Volumes have no ID beyond their name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Document the ID-or-labels ownership rule on UnreleasedResource Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Skip label checks on captured-ID container cleanup; release claims after a settled retry The agent container's cleanup still required the invocation label when its captured ID already proved ownership. A network setup cleanup that failed and later settled on retry never released its allocation claim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * State that unreleased labels are the labels D wrote, not observed ones Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Create and start the task keeper separately; clean up a run that failed after creating A keeper whose docker run created the container but failed to start it was tracked as never created, so cleanup skipped it and its volume references kept both volumes alive. The keeper is now created (tracked by name, with its ID returned) and started as a separate step. A seeder docker run that fails after the daemon answered may also have created its container, so cleanup checks it by name and owner labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 9292fe7 commit 15ee8fa

27 files changed

Lines changed: 798 additions & 151 deletions

‎agents/adapters/claude.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,7 @@ import type { InvocationHandle } from '../contract.ts';
22
import { createClaudeCommand, createPhasePolicy } from '../policy.ts';
33
import { launchInvocation } from './supervisor.ts';
44
import { setUpProfile } from './setup.ts';
5-
import { assertBuiltAgentImage } from '../container/image.ts';
6-
import { createAdapterInvocationBudget, type AgentAdapterOptions, type AgentAdapterRequest } from './types.ts';
5+
import { assertAdapterRequest, createAdapterInvocationBudget, type AgentAdapterOptions, type AgentAdapterRequest } from './types.ts';
76

87
export function parseClaudeOutput(raw: Buffer): { text: string; providerFailed: boolean } {
98
const envelope = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(raw)) as
@@ -19,7 +18,7 @@ export function startClaudeInvocation(request: AgentAdapterRequest,
1918
const policy = createPhasePolicy(request.invocation);
2019
const remaining = createAdapterInvocationBudget(request.invocation, options.timeoutMs);
2120
// Invalid input throws here, before anything is allocated; only Docker setup runs inside the handle.
22-
assertBuiltAgentImage(request.imageId);
21+
assertAdapterRequest(request);
2322
return launchInvocation(request.invocation, remaining, (signal, start) => setUpProfile(request, remaining, signal,
2423
network => ({ ...request, policy, network, command: createClaudeCommand(policy, request.prompt),
2524
claudeToken: oauthToken }),

‎agents/adapters/codex.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,7 @@ import { agentContainerId } from '../container/run.ts';
44
import { createCodexCommand, createPhasePolicy } from '../policy.ts';
55
import { launchInvocation, readBoundedContainerFile } from './supervisor.ts';
66
import { setUpProfile } from './setup.ts';
7-
import { assertBuiltAgentImage } from '../container/image.ts';
8-
import { createAdapterInvocationBudget, type AgentAdapterOptions, type AgentAdapterRequest } from './types.ts';
7+
import { assertAdapterRequest, createAdapterInvocationBudget, type AgentAdapterOptions, type AgentAdapterRequest } from './types.ts';
98

109
export const CODEX_OUTPUT_FILE = '/run/codeboost-output/final.txt';
1110
// Read the output from the container this invocation created, by ID; a same-named replacement must not answer.
@@ -28,7 +27,7 @@ export function startCodexInvocation(request: AgentAdapterRequest,
2827
const policy = createPhasePolicy(request.invocation);
2928
const remaining = createAdapterInvocationBudget(request.invocation, options.timeoutMs);
3029
// Invalid input throws here, before anything is allocated; only Docker setup runs inside the handle.
31-
assertBuiltAgentImage(request.imageId);
30+
assertAdapterRequest(request);
3231
return launchInvocation(request.invocation, remaining, (signal, start) => setUpProfile(request, remaining, signal,
3332
network => ({ ...request, policy, network, command: createCodexCommand(policy, request.prompt),
3433
codexAuthFile: authFile, deferredOutput: true }),

‎agents/adapters/setup.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@ const CLEANUP_TIMEOUT_MS = 30_000;
1919
export async function setUpProfile(request: AgentAdapterRequest, remaining: () => number, signal: AbortSignal,
2020
profileOptions: (network: VendorNetwork) => Omit<ProfileOptions, 'timeoutMs' | 'signal'>,
2121
start: (profile: ContainerProfile) => InvocationHandle): Promise<InvocationHandle> {
22-
const network = await createVendorNetwork(request.invocation, request.imageId, Math.min(60_000, remaining()), signal);
22+
const network = await createVendorNetwork(request.invocation, request.imageId, request.networkAllocationId,
23+
Math.min(60_000, remaining()), signal);
2324
let profile: ContainerProfile;
2425
try {
2526
profile = await createContainerProfile({ ...profileOptions(network), timeoutMs: Math.min(60_000, remaining()),

‎agents/adapters/types.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
import type { InvocationInput } from '../contract.ts';
22
import type { TaskFilesystems } from '../container/storage.ts';
3+
import { assertBuiltAgentImage } from '../container/image.ts';
4+
import { assertResourceOwner } from '../labels.ts';
35
import type { CaptureLimits } from './supervisor.ts';
46

57
export interface AgentAdapterRequest {
@@ -8,6 +10,17 @@ export interface AgentAdapterRequest {
810
readonly inputDirectory: string;
911
readonly imageId: string;
1012
readonly prompt: string;
13+
/**
14+
* The caller-chosen allocation ID (a lowercase UUID v4) for the vendor network and proxy, recorded before the start
15+
* call so recovery can match them to the attempt (#51 item 3).
16+
*/
17+
readonly networkAllocationId: string;
18+
}
19+
/** Check a start call's synchronous input, so invalid input throws before a handle exists. */
20+
export function assertAdapterRequest(request: AgentAdapterRequest): void {
21+
assertBuiltAgentImage(request.imageId);
22+
assertResourceOwner({ runnerOwner: request.invocation.runnerOwner, attemptId: request.invocation.attemptId,
23+
allocationId: request.networkAllocationId });
1124
}
1225
export interface AgentAdapterOptions {
1326
readonly timeoutMs?: number;

‎agents/container/profile.ts‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@ import { tmpdir } from 'node:os';
55
import { join } from 'node:path';
66
import { assertCapturedInvocation, type InvocationInput, type Phase, type UnreleasedResource } from '../contract.ts';
77
import { assertBuiltAgentImage } from './image.ts';
8-
import { assertTaskFilesystems, type TaskFilesystems } from './storage.ts';
8+
import { assertTaskFilesystems, taskFilesystemOwner, type TaskFilesystems } from './storage.ts';
9+
import { ownerLabelArgs, type ResourceOwner } from '../labels.ts';
910
import { assertVendorNetwork, removeVendorNetwork, vendorNetworkResources,
1011
type VendorNetwork } from '../network/network.ts';
1112
import { assertAgentCommand, assertPhasePolicy, type AgentCommand, type PhasePolicy } from '../policy.ts';
@@ -136,6 +137,12 @@ const captureInput = (directory: string): InputCapture => {
136137
return Object.freeze({ inputDirectory: canonical, schema, content: captured.content });
137138
};
138139

140+
/** The labels on an invocation's agent container: its runner and attempt, and the task-storage allocation it mounts. */
141+
export function agentContainerOwner(invocation: InvocationInput, filesystems: TaskFilesystems): ResourceOwner {
142+
return Object.freeze({ runnerOwner: invocation.runnerOwner, attemptId: invocation.attemptId,
143+
allocationId: taskFilesystemOwner(filesystems).allocationId });
144+
}
145+
139146
/** Prove that a profile object is the exact capability issued by this module. */
140147
export function assertContainerProfileAuthenticity(profile: ContainerProfile): void {
141148
if (!identities.has(profile)) throw new Error('Container profile was not created by the trusted profile builder.');
@@ -218,6 +225,11 @@ export async function createContainerProfile(options: ProfileOptions): Promise<C
218225
throw new Error('Container profile requires the immutable built image ID.');
219226
assertBuiltAgentImage(options.imageId);
220227
assertTaskFilesystems(filesystems, invocation.clone);
228+
// The storage must belong to this runner. The agent container is labelled with its own attempt and the allocation
229+
// it mounts, so recovery can find both from the container.
230+
const storageOwner = taskFilesystemOwner(filesystems);
231+
if (storageOwner.runnerOwner !== invocation.runnerOwner) throw new Error('Task filesystems belong to another runner.');
232+
const containerOwner = agentContainerOwner(invocation, filesystems);
221233
const invocationLeft = Math.floor(invocation.deadline - Date.now());
222234
if (invocationLeft < 1) throw new Error('Invocation deadline has passed.');
223235
await assertVendorNetwork(options.network, invocation, undefined, Math.min(options.timeoutMs ?? 30_000, invocationLeft),
@@ -265,7 +277,7 @@ export async function createContainerProfile(options: ProfileOptions): Promise<C
265277
'--cpus=1', '--shm-size=16m', '--ipc=private', '--cgroupns=private',
266278
`--network=${options.network.name}`, '--dns=127.0.0.1', '--env', 'HOME=/home/codeboost',
267279
'--env', `CODEBOOST_PHASE=${invocation.phase}`,
268-
'--label', `io.codeboost.invocation=${ownershipId}`,
280+
'--label', `io.codeboost.invocation=${ownershipId}`, ...ownerLabelArgs(containerOwner),
269281
'--env', `CODEBOOST_VENDOR=${invocation.vendor}`, '--env', 'npm_config_cache=/tmp/npm-cache',
270282
'--env', `HTTPS_PROXY=${options.network.proxyUrl}`, '--env', `HTTP_PROXY=${options.network.proxyUrl}`,
271283
'--env', 'NO_PROXY=localhost,127.0.0.1',

‎agents/container/run.ts‎

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@ import { realpathSync } from 'node:fs';
22
import { createOutcomeUnknown, DOCKER_ID } from '../client-outcome.ts';
33
import type { UnreleasedResource } from '../contract.ts';
44
import { docker as runDockerCommand, pause, runDocker, type DockerOutcome } from '../docker.ts';
5-
import { assertContainerProfile, assertContainerProfileAuthenticity, disposeContainerProfile,
5+
import { agentContainerOwner, assertContainerProfile, assertContainerProfileAuthenticity, disposeContainerProfile,
66
isContainerProfileAuthentic, profileTimeout,
77
type ContainerProfile } from './profile.ts';
88
import { BASE_IMAGE, CLAUDE_VERSION, CODEX_VERSION } from './image.ts';
9-
import { taskFilesystemAllocationId } from './storage.ts';
9+
import { taskFilesystemOwner } from './storage.ts';
10+
import { hasOwnerLabels, ownerLabels } from '../labels.ts';
11+
import { assertPhasePolicy } from '../policy.ts';
1012
export { prepareTaskFilesystems, removeTaskFilesystems } from './storage.ts';
1113
export type { TaskFilesystems, TaskStorageLimits } from './storage.ts';
1214

@@ -54,6 +56,8 @@ const unsettledCreates = new WeakMap<ContainerProfile, number>();
5456
// removed only once the container's removal (or absence) is confirmed, so any later retry of the profile's own
5557
// network or staging cleanup is profile-only and never looks the name up again.
5658
const createdContainers = new WeakMap<ContainerProfile, string | undefined>();
59+
// The ownership labels written on each profile's agent container, for reporting it if its removal is not confirmed.
60+
const containerLabels = new WeakMap<ContainerProfile, Readonly<Record<string, string>>>();
5761

5862
/**
5963
* The immutable ID of the agent container this profile created, when `docker create` returned one. Every operation
@@ -85,8 +89,9 @@ const assertLaunchable = (profile: ContainerProfile) => {
8589
export function agentContainerResources(profile: ContainerProfile): readonly UnreleasedResource[] {
8690
if (!createdContainers.has(profile)) return Object.freeze([]);
8791
const id = createdContainers.get(profile);
92+
// Labels captured at create time: this runs while a handle settles and must never throw.
8893
return Object.freeze([Object.freeze({ kind: 'container' as const, name: profile.name, ...(id ? { id } : {}),
89-
owner: Object.freeze({ label: 'io.codeboost.invocation', value: profile.ownershipId }) })]);
94+
labels: containerLabels.get(profile) ?? Object.freeze({ 'io.codeboost.invocation': profile.ownershipId }) })]);
9095
}
9196
// Cleanup is never cancelled: it runs to its own deadline so nothing is dropped.
9297
const removeContainerOrThrow = async (profile: ContainerProfile, waitForSettle = false, timeoutMs = 30_000) => {
@@ -124,11 +129,16 @@ const removeContainerOrThrow = async (profile: ContainerProfile, waitForSettle =
124129
}
125130
const inspected = JSON.parse(String(before.stdout || '[]'))[0] as
126131
{ Id?: string; Config?: { Labels?: Record<string, string> } } | undefined;
127-
if (inspected?.Config?.Labels?.['io.codeboost.invocation'] !== profile.ownershipId)
132+
// A captured ID proves the container is the one this profile created (Docker never reuses IDs), so it is removed
133+
// even if validation refused its labels. A container found by name (its create's ID never came back) must carry
134+
// every label this profile wrote.
135+
const expected = containerLabels.get(profile) ?? { 'io.codeboost.invocation': profile.ownershipId };
136+
if (!capturedId && (inspected?.Config?.Labels?.['io.codeboost.invocation'] !== profile.ownershipId
137+
|| Object.entries(expected).some(([label, value]) => inspected?.Config?.Labels?.[label] !== value)))
128138
throw new Error('Agent container name is held by another invocation; staged credentials were retained.');
129139
// Remove and confirm by the ID the daemon just reported for our container, never by the name: a same-named
130140
// replacement created after this inspect must not be deleted.
131-
const id = inspected.Id;
141+
const id = inspected?.Id;
132142
if (!id || !DOCKER_ID.test(id) || (capturedId && id !== capturedId))
133143
throw new Error('Failed to establish the agent container identity; staged credentials were retained.');
134144
const result = await runDocker(['rm', '--force', id], { timeoutMs: remaining() });
@@ -201,6 +211,7 @@ export async function validateContainer(container: string, profile: ContainerPro
201211
|| JSON.stringify(inspect.Config.Entrypoint) !== JSON.stringify(['/usr/local/bin/codeboost-container-probe'])
202212
|| JSON.stringify(inspect.Config.Cmd) !== JSON.stringify(profile.command)
203213
|| inspect.Config.Labels?.['io.codeboost.invocation'] !== profile.ownershipId
214+
|| !hasOwnerLabels(inspect.Config.Labels, agentContainerOwner(assertPhasePolicy(profile.policy), profile.filesystems))
204215
|| !host.ReadonlyRootfs || host.Privileged
205216
|| !host.CapDrop?.map(value => value.toUpperCase()).includes('ALL') || (host.CapAdd?.length ?? 0) !== 0
206217
|| !exactSecurityOptions(host.SecurityOpt)
@@ -262,7 +273,8 @@ export async function validateContainer(container: string, profile: ContainerPro
262273
const volumes = JSON.parse(await docker(['volume', 'inspect', work.Name!, metadata.Name!],
263274
{ timeoutMs: remaining(), signal })) as
264275
Array<{ Name: string; Driver: string; Labels: Record<string, string> | null; Options: Record<string, string> | null }>;
265-
const allocationId = taskFilesystemAllocationId(profile.filesystems);
276+
// One owner for the whole check: the storage's runner, attempt and allocation label every volume and the keeper.
277+
const storageOwner = taskFilesystemOwner(profile.filesystems);
266278
const expectedVolumes = new Map([
267279
[work.Name!, ['work', String(profile.filesystems.workBytes), String(profile.filesystems.workInodes)]],
268280
[metadata.Name!, ['metadata', String(profile.filesystems.metadataBytes), String(profile.filesystems.metadataInodes)]],
@@ -271,7 +283,7 @@ export async function validateContainer(container: string, profile: ContainerPro
271283
const expected = expectedVolumes.get(volume.Name), options = volume.Options ?? {}, optionString = options.o ?? '';
272284
if (!expected || volume.Driver !== 'local' || options.type !== 'tmpfs' || options.device !== 'tmpfs'
273285
|| volume.Labels?.['io.codeboost.task-storage'] !== expected[0]
274-
|| volume.Labels?.['io.codeboost.allocation'] !== allocationId
286+
|| !hasOwnerLabels(volume.Labels, storageOwner)
275287
|| !hasExactOptions(optionString, [`size=${expected[1]}`, `nr_inodes=${expected[2]}`,
276288
'uid=10001', 'gid=10001', 'mode=0755', 'nosuid', 'nodev']))
277289
throw new Error('Task volume does not match its bounded tmpfs allocation.');
@@ -286,7 +298,7 @@ export async function validateContainer(container: string, profile: ContainerPro
286298
const keeperVolumes = new Map((keeper?.Mounts ?? []).filter(item => item.Type === 'volume').map(item => [item.Destination, item]));
287299
if (!keeper?.State?.Running || keeper.Config?.Image !== profile.expectedImage || keeper.Config?.User !== '10001:10001'
288300
|| keeper.Config?.Labels?.['io.codeboost.task-storage'] !== 'keeper'
289-
|| keeper.Config?.Labels?.['io.codeboost.allocation'] !== allocationId || !keeper.HostConfig?.ReadonlyRootfs
301+
|| !hasOwnerLabels(keeper.Config?.Labels, storageOwner) || !keeper.HostConfig?.ReadonlyRootfs
290302
|| keeper.HostConfig.Privileged || keeper.HostConfig.NetworkMode !== 'none'
291303
|| !keeper.HostConfig.CapDrop?.map(value => value.toUpperCase()).includes('ALL')
292304
|| (keeper.HostConfig.CapAdd?.length ?? 0) !== 0
@@ -349,6 +361,8 @@ export async function createValidatedContainer(profile: ContainerProfile, timeou
349361
const remaining = createDeadline(profileTimeout(profile, timeoutMs));
350362
let createUnsettled = false;
351363
try {
364+
containerLabels.set(profile, Object.freeze({ 'io.codeboost.invocation': profile.ownershipId,
365+
...ownerLabels(agentContainerOwner(assertPhasePolicy(profile.policy), profile.filesystems)) }));
352366
validateSecrets(profile, secrets);
353367
await assertContainerProfile(profile, remaining(), signal);
354368
signal?.throwIfAborted();

0 commit comments

Comments
 (0)