Skip to content

Commit 1541797

Browse files
authored
Merge pull request #76 from codeabovelab/feat/d-task-diff-export
D: bounded task-storage diff export for stopped attempts (#51 item 6)
2 parents cd330f4 + 176a631 commit 1541797

8 files changed

Lines changed: 643 additions & 16 deletions

File tree

‎AGENTS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Every reproduced race requires a failing-before and passing-after regression. As
4949
- Before requesting or re-requesting an automated Copilot review, self-review the full current diff, fix every issue found, and repeat the self-review and fix cycle until a complete pass finds no new issues. Re-run the relevant validation after fixes; only then request Copilot review.
5050
- Each self-review pass rereads every changed function in full against the base, not only the lines changed since the previous round. Code unchanged since the first commit of the PR still gets reviewed in every pass.
5151
- Treat every behavioural claim the change makes, in code comments, the PR body or docs (for example "pauses every 1,000 entries", "settles only after exit", "never throws", "bounded by N seconds"), as something to verify. Trace each claim through every path that can break it, including nested loops, callbacks, error paths and early returns, and give it a test that fails if the claim is false.
52+
- A test's setup must leave the state the production path would: if production never runs a step (such as a commit that refreshes Git's index), the test must not run it before the behaviour under test either.
5253
- The author's self-review is not enough for concurrency, process, subprocess, timer or resource-cleanup code. Before requesting Copilot review, also run an independent review that does not share the author's context: `/codex review`, a separate review agent, or `/code-review` at `high` effort or above. Fix its findings like any other.
5354
- Run final validation against the exact pushed head after the last change.
5455
- Report current test counts separately from historical milestone counts.
@@ -96,6 +97,12 @@ Every reproduced race requires a failing-before and passing-after regression. As
9697
- Treat paths read from a durable record or discovered on disk as untrusted. Before deleting, opening or probing one, validate its exact location and name, not only its basename, and never follow a link to it. Keep files that other local users must not plant or swap, such as lock files, in a directory only the current user can write. Write durable records through a unique temporary file opened exclusively, and delete it if the write fails.
9798
- Exclude other processes with an OS-level lock held for the owner's lifetime, keyed by the resource's stable identity rather than a path spelling. A PID liveness check never authorizes taking over a lock. Run shared one-time startup work single-flight under that lock, and keep the lock until the work has finished.
9899

100+
## Agent-controlled content
101+
102+
- When a subprocess reports a problem only as a warning and carries on, decide pass or fail by what each message means for the result, not by whether anything was printed: fail on messages that mean it did less than it should (for example could not read a path), and let through messages about harmless input the agent controls. Test both a benign case and a failing case, and filter the output as it arrives so that no volume of benign messages can push a failure out of a bounded buffer.
103+
- Quote or escape agent-controlled text (file names, paths, branch names) wherever it lands in output that people or tools parse, such as diffs, notices, logs or reports, so it cannot forge that output's structure.
104+
- A hardened Git invocation must also keep Git out of nested repositories and populated submodules, whose own config and hooks are the agent's: pass `--ignore-submodules` on the command line (the config default does not bind plumbing or override `.gitmodules`), and never run Git with a nested repository as its working directory.
105+
99106
## Blinded experiments
100107

101108
- Keep experimental PRs as drafts with automated review disabled until the assigned human decision is recorded. An automated review invalidates reviewer blindness; replace the affected package rather than reusing it.

‎agents/container/storage.ts‎

Lines changed: 288 additions & 1 deletion
Large diffs are not rendered by default.

‎agents/recovery.ts‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ export interface UnownedResource extends RecoveredResource {
2424
readonly reason: 'no-runner-label' | 'unknown-kind' | 'inconsistent-storage';
2525
}
2626
export interface RecoveryReport {
27-
/** Agent containers, egress proxies, seeders and networks of this runner, now confirmed gone. */
27+
/** Agent containers, egress proxies, seeders, export containers and networks of this runner, now confirmed gone. */
2828
readonly removed: readonly RecoveredResource[];
2929
/** One handle per task-storage allocation of this runner, kept whole for export and `removeTaskFilesystems`. */
3030
readonly storage: readonly RecoveredTaskStorage[];
@@ -119,9 +119,10 @@ const removeById = async (resource: RecoveredResource, remaining: () => number)
119119

120120
/**
121121
* Crash recovery for one database (#51 item 4). Acts only on objects whose `io.codeboost.runner` label is
122-
* `runnerOwner`: it removes agent containers, egress proxies, seeders and networks, and resolves only once they are
123-
* gone. It keeps task storage whole (volumes and keeper) and returns a recovery handle per allocation. Objects from
124-
* older builds without a runner label, and anything it does not recognise, are reported and never touched.
122+
* `runnerOwner`: it removes agent containers, egress proxies, seeders, export containers and networks, and resolves
123+
* only once they are gone. It keeps task storage whole (volumes and keeper) and returns a recovery handle per
124+
* allocation. Objects from older builds without a runner label, and anything it does not recognise, are reported and
125+
* never touched.
125126
*
126127
* Call it only while holding the database's single-runner lock and before admitting work: it removes every agent
127128
* container of this runner. It refuses to run while this process holds task storage of the runner, which every agent
@@ -180,7 +181,9 @@ export async function recoverLeftovers(runnerOwner: string, timeoutMs = 120_000)
180181
for (const resource of owned.container) {
181182
const kind = kindOf(resource);
182183
if (kind === 'storage:keeper') keep(resource, 'keeper');
183-
else if (kind === 'storage:seeder' || kind === 'agent' || kind === 'egress') remove.push(resource);
184+
// Seeders and export containers are transient: a leftover one is removed, never kept with the storage.
185+
else if (kind === 'storage:seeder' || kind === 'storage:export' || kind === 'agent' || kind === 'egress')
186+
remove.push(resource);
184187
else unknown(resource);
185188
}
186189
for (const resource of owned.volume) {

‎docs/implementation/agent-isolation.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,9 +79,15 @@ Use only these entry points to run an agent:
7979
later failure settles the handle. `cancel()` during setup kills the in-flight Docker
8080
call. The request carries `networkAllocationId`, a UUID you choose for the vendor network. Pass the vendor
8181
credential only as the function argument.
82-
5. After a crash or restart, call `recoverLeftovers(runnerOwner)` (`agents/recovery.ts`) while holding the
82+
5. To keep a stopped writable attempt's partial output, call `exportTaskDiff(storage, { base, imageId })` before
83+
`removeTaskFilesystems`. `base` is the full ID of the last commit codeboost made in that storage. It returns at
84+
most 1 MiB of diff (`truncated` says whether it was cut), and takes `signal` and a deadline for the Docker work.
85+
A changed file over 8 MiB, an untracked nested repository, a path Git will not add, an entry named `.git`, a fifo
86+
or socket, and an ignored untracked path each appear as a `codeboost:` notice line, not a diff. When Git fails, the
87+
error carries Git's first error line.
88+
6. After a crash or restart, call `recoverLeftovers(runnerOwner)` (`agents/recovery.ts`) while holding the
8389
database's single-runner lock and before admitting work. It touches only objects labelled with that runner
84-
token. It removes agent containers, egress proxies, seeders and networks, and resolves once they are gone. It keeps
90+
token. It removes agent containers, egress proxies, seeders, export containers and networks, and resolves once they are gone. It keeps
8591
task storage whole (both volumes and the keeper) and returns one recovery handle per allocation, carrying its
8692
attempt and allocation IDs. `removeTaskFilesystems` accepts a handle as it accepts the value
8793
`prepareTaskFilesystems` returned. D issues a handle only after checking every part's owner labels. Objects without

0 commit comments

Comments
 (0)