Skip to content

Commit 30b2ec9

Browse files
runtime: run a repo's tests in a locked Docker box
runInSandbox() runs the test command with no network, 1 GB, 1 CPU, a read-only repo and a 60 s limit. Installs run first with repo scripts off. npm only for now.
1 parent 0c2add0 commit 30b2ec9

8 files changed

Lines changed: 562 additions & 12 deletions

File tree

‎README.md‎

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,8 @@ Week 1 (Oct 19 to 23, 2026), plus the first part of week 2.
8181
| Codex: `codex exec --json` to events | [runner/src/agents/codex.ts](runner/src/agents/codex.ts) | To do. `start()` with `agent: "codex"` fails cleanly for now |
8282
| `approval.requested` after `task.finished` | [runner/src/runner.ts](runner/src/runner.ts) | Done (contracts v0.2.0). Real and fake runners, never after a stop or a failure |
8383
| Repo memory: notes read before a run, a note added after | [runner/src/memory/](runner/src/memory/) | Done. `MemoryStore` with an in-memory stub and a file store. Stopped and failed runs add nothing, and a memory error never fails a run |
84-
| Repo profile: language, package manager, test command, Node version | [runner/src/memory/profile.ts](runner/src/memory/profile.ts) | Done. Read from files in the worktree, no model call. The sandbox reads it next |
84+
| Repo profile: language, package manager, test command, Node version | [runner/src/memory/profile.ts](runner/src/memory/profile.ts) | Done. Read from files in the worktree, no model call. The sandbox uses it to pick the image and decide whether to install |
85+
| Sandbox: run a repo's tests in a locked Docker box | [runner/src/runtime/](runner/src/runtime/) | Done. `runInSandbox()`: no network, 1 GB with no swap, 1 CPU, 256 processes, read-only repo and system, non-root, your timezone, 60 s. npm only. Nothing calls it yet: the engine's test step will |
8586

8687
Next, from the [design](https://github.com/devweekends/codehands-docs#who-owns-what): a model per phase, naming provider errors as one `provider.status` event (`rate_limited`, `auth_expired`, `usage_limit`, `subscription_inactive`, `down`), memory and user preferences, and two models on one task.
8788

@@ -124,6 +125,11 @@ const { branch, commit } = await runner.approve(sessionId, "Fix the date bug");
124125
| `formatNote`, `runNote`, `trimNotes`, `notesForPrompt` | How a note is written, built from a run, trimmed to about 200 lines, and capped at 6,000 characters for the prompt |
125126
| `detectRepoProfile(dir)`, `RepoProfile` | What a repo is, read from its files. Never throws: an unknown repo gives `language: "unknown"` |
126127
| `repoKey(repoPath)` | The repo's id, which names its worktree and memory folders |
128+
| `runInSandbox(options)`, `SandboxOptions`, `SandboxResult`, `SandboxError` | Runs the repo's test command in the locked Docker box and returns a result. Never throws: failing tests are `passed: false`, and a box that couldn't run them sets `error` (`sandbox.unavailable`, `sandbox.unsupported`, `sandbox.failed`, `install.failed`) |
129+
| `dockerArgs()`, `installArgs()`, `DEFAULT_LIMITS`, `BoxLimits` | The exact `docker run` flags for the test box and the install box, as plain data |
130+
| `dockerAvailable(docker?)`, `hostTimeZone()` | Whether the Docker engine answers within 5 s, and your IANA timezone (`Asia/Karachi`, not Windows' own name) |
131+
132+
When a repo needs its dependencies, a separate install box runs first, with the network on but the repo's install scripts off, then the test box runs with no network. `node_modules` must be git-ignored, or the sandbox refuses: everything not ignored lands in the commit you approve.
127133

128134
The daemon owns the real database. The runner keeps only what it needs to finish a session, in `~/.codehands/sessions/<sessionId>.json`, so `approve`, `reject`, `resume` and `diff` still work after a restart. The agent itself does not survive one.
129135

@@ -161,6 +167,9 @@ Some details that matter:
161167
| `~/.codehands/worktrees/<repoKey>/<first 8 of sessionId>` | The task's worktree. Outside your repo so worktrees never nest, and short because of Windows path limits |
162168
| `~/.codehands/memory/<repoKey>/notes.md` | Per-repo notes, newest last, trimmed to about 200 lines by dropping the oldest whole notes |
163169
| `~/.codehands/memory/<repoKey>/profile.json` | The repo profile from the latest run |
170+
| `~/.codehands/sessions/<sessionId>.json` | What the runner needs to finish a session after a restart |
171+
| `~/.codehands/runs/<sessionId>/test-<n>.json` | Each sandbox test run: passed, exit code, timed out, out of memory, the last 64 KB of output |
172+
| Docker containers `ch-<first 8 of sessionId>-<n>` | One per sandbox run, removed right after. `-install` on the end for the install box |
164173
| `codehands/<slug>-<first 8 of sessionId>` | The task's branch, in your repo |
165174

166175
`repoKey` is the first 12 characters of the SHA-1 of the repo's `origin` URL, or of its absolute path when there is no origin. Two clones of the same repo share a key.
@@ -169,7 +178,7 @@ All of it stays on this machine. Memory reaches Claude only as `--append-system-
169178

170179
## Safety rules
171180

172-
The agent runs on your machine with no sandbox until the Docker box lands in week 2. Until then:
181+
Tests run in the locked Docker box when the engine calls `runInSandbox()`. Claude Code itself still runs on your machine, not in a box, so:
173182

174183
- **Run it only on our own repos.**
175184
- Claude runs with `--permission-mode acceptEdits` and a short allow list: `Read, Edit, Write, Glob, Grep`, and only `npm test` in Bash or PowerShell. Anything else is blocked and shows up as a `message` with `denied: true`.
@@ -181,7 +190,7 @@ The agent runs on your machine with no sandbox until the Docker box lands in wee
181190

182191
## Set up
183192

184-
You need Node 22, pnpm 10, Git, and for real runs the [Claude Code](https://docs.claude.com/en/docs/claude-code) CLI, logged in.
193+
You need Node 22, pnpm 10, Git, and for real runs the [Claude Code](https://docs.claude.com/en/docs/claude-code) CLI, logged in. For the sandbox, Docker Desktop must be running. Without it, the sandbox tests are skipped, not failed.
185194

186195
The runner depends on the contracts by folder in week 1, so clone both side by side:
187196

@@ -206,7 +215,8 @@ Run these in `runner/`.
206215

207216
| Command | What | Costs money |
208217
| --- | --- | --- |
209-
| `pnpm test` | Every test. The parser replays a recorded run, the runner tests make real git repos in a temp folder and use a stand-in for `claude` | No |
218+
| `pnpm test` | Every test. The parser replays a recorded run, the runner tests make real git repos in a temp folder and use a stand-in for `claude`, and the sandbox tests start real containers when Docker is running | No |
219+
| `pnpm sandbox [repoPath] [--tz <zone>]` | Runs a repo's tests in the locked box once and prints the result. Defaults to the fixture. `--tz UTC` shows why the box needs your timezone | No |
210220
| `pnpm fake` | Prints the fake runner's events, one JSON object per line. This is exactly what the daemon receives | No |
211221
| `pnpm try [repoPath] [prompt] [--approve \| --reject]` | One real run through the runner, one line per event, then the diff stat, then approve, reject or keep | Yes, capped at $0.50 |
212222
| `pnpm build` | Compiles to `dist/` | No |
@@ -232,6 +242,8 @@ The scripts in `runner/scripts/`:
232242
| [memory.test.ts](runner/test/memory.test.ts) | The stub keeps notes in order and repos apart; trimming drops whole notes and keeps the newest; the prompt cap keeps the newest end; a note's summary stays on one line; the file store reads back from disk and treats missing files as empty |
233243
| [profile.test.ts](runner/test/profile.test.ts) | npm, pnpm, npm's placeholder test script, `.nvmrc`, `engines.node`, needs install, broken `package.json`, Python, empty folder |
234244
| [worktree.test.ts](runner/test/worktree.test.ts) | Branch names |
245+
| [docker-args.test.ts](runner/test/docker-args.test.ts) | Every security flag of the test box and the install box, with no Docker needed, so a lost flag fails in milliseconds |
246+
| [sandbox.test.ts](runner/test/sandbox.test.ts) | Real containers, skipped when Docker is off: pass and fail; a test that never ends is killed and nothing is left behind, even when the kill misses; out of memory; no network; read-only repo and system; the 64 KB output cap; timezone; result files numbered across restarts; no false pass from a leftover container; `sandbox.failed` when the box can't start; install, with no install scripts and no lockfile added; a failed install; `node_modules` not git-ignored; Docker unavailable |
235247

236248
To record a new fixture, run `scripts/spike.sh` (or a Codex equivalent), read every line of the output, then write tests that replay it. Never call a real agent from a test.
237249

@@ -245,10 +257,6 @@ To record a new fixture, run `scripts/spike.sh` (or a Codex equivalent), read ev
245257
| Cannot find module `@codehands/contracts` | Build the contracts repo first, and check it is cloned next to `codehands-agent` |
246258
| Child processes keep running after a stop on Windows | Killing the parent does not kill its children (`claude` → `npm test` → `node`). That is what `stopProcessTree` is for |
247259
| "nothing to approve: the agent made no changes" | The agent changed no files. Reject the session instead |
248-
249-
## Contributing
250-
251-
- One branch per Linear issue, for example `tahira/cod-12-runner-start`. Put the issue id in the PR title.
252-
- `main` is always green. CI runs build and test on every PR.
253-
- A change to `StartOptions`, `RunnerEvent` or `Runner` is a contract change. It goes to [codehands-contracts](https://github.com/devweekends/codehands-contracts#changing-a-contract) first, and `createFakeRunner()` is updated the same week.
254-
- The week by week plan is in [roadmap/tahira.md](https://github.com/devweekends/codehands-docs/blob/main/roadmap/tahira.md).
260+
| `sandbox.unavailable` | Docker Desktop isn't running. Start it and try again |
261+
| `sandbox.failed` | The box broke, not your code. `message` and `output` say why: often a wrong image tag from `.nvmrc`, or no internet for the first image pull |
262+
| "add node_modules/ to .gitignore first" | The sandbox won't install into a repo that doesn't ignore `node_modules`, because it would land in the commit |

‎runner/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
"typecheck": "tsc --noEmit",
1919
"test": "vitest run",
2020
"fake": "tsx scripts/fake.ts",
21-
"try": "tsx scripts/try.ts"
21+
"try": "tsx scripts/try.ts",
22+
"sandbox": "tsx scripts/sandbox.ts"
2223
},
2324
"dependencies": {
2425
"@codehands/contracts": "file:../../codehands-contracts",

‎runner/scripts/sandbox.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
// pnpm sandbox [repoPath] [--tz <zone>]: runs the repo's tests in the locked box once. No model calls, no cost.
2+
import { randomUUID } from "node:crypto";
3+
import { homedir } from "node:os";
4+
import { join } from "node:path";
5+
import { detectRepoProfile, runInSandbox } from "../src/index.js";
6+
7+
const args = process.argv.slice(2);
8+
const tzAt = args.indexOf("--tz");
9+
const timeZone = tzAt >= 0 ? args[tzAt + 1] : undefined;
10+
const repoPath = args.find((a, i) => !a.startsWith("--") && (tzAt < 0 || i !== tzAt + 1)) ?? join(homedir(), "codehands-fixtures", "date-bug");
11+
12+
const profile = await detectRepoProfile(repoPath);
13+
console.log("profile:", profile);
14+
const r = await runInSandbox({ sessionId: randomUUID(), worktree: repoPath, profile, timeZone });
15+
console.log(r.output.trim().split("\n").slice(-30).join("\n"));
16+
console.log({ passed: r.passed, exitCode: r.exitCode, timedOut: r.timedOut, oomKilled: r.oomKilled, durationMs: r.durationMs, error: r.error, message: r.message, resultFile: r.resultFile });
17+
process.exitCode = r.passed ? 0 : 1;

‎runner/src/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,5 @@ export { branchName, slugify } from "./git/worktree.js";
66
export { createMemoryStub, formatNote, notesForPrompt, runNote, trimNotes, type MemoryNote, type MemoryStore } from "./memory/store.js";
77
export { createFileMemory, repoKey } from "./memory/notes.js";
88
export { detectRepoProfile, type RepoProfile } from "./memory/profile.js";
9+
export { runInSandbox, type SandboxOptions, type SandboxResult, type SandboxError } from "./runtime/sandbox.js";
10+
export { dockerArgs, installArgs, dockerAvailable, hostTimeZone, DEFAULT_LIMITS, type BoxLimits } from "./runtime/docker.js";

‎runner/src/runtime/docker.ts‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
// Week 2: the locked box. Every flag here is checked in test/docker-args.test.ts.
2+
// Spike results (Nov 2): no --rm (OOMKilled must be read first), no pipes (they hide the exit code).
3+
import { execa } from "execa";
4+
5+
// Week 2: the locked box. Every flag here is checked in test/docker-args.test.ts.
6+
// Spike results (Nov 2): no --rm (OOMKilled must be read first), no pipes (they hide the exit code).
7+
8+
export interface BoxLimits {
9+
memoryMb: number; // 1024. Swap is set to the same, so there is no swap
10+
cpus: number; // 1
11+
pids: number; // 256
12+
}
13+
14+
export const DEFAULT_LIMITS: BoxLimits = { memoryMb: 1024, cpus: 1, pids: 256 };
15+
16+
// The test container: no network, read-only repo and system, non-root, no capabilities.
17+
export function dockerArgs(o: {
18+
name: string; worktree: string; image: string; command: string; timeZone: string; limits?: BoxLimits;
19+
}): string[] {
20+
const l = o.limits ?? DEFAULT_LIMITS;
21+
return [
22+
"run", "--name", o.name,
23+
"--network", "none",
24+
"--memory", `${l.memoryMb}m`, "--memory-swap", `${l.memoryMb}m`,
25+
"--cpus", String(l.cpus), "--pids-limit", String(l.pids),
26+
"--cap-drop", "ALL", "--security-opt", "no-new-privileges",
27+
"--user", "node",
28+
"--read-only", "--tmpfs", "/tmp:exec", "--tmpfs", "/home/node:uid=1000,gid=1000",
29+
"-e", `TZ=${o.timeZone}`, "-e", "CI=1",
30+
"-v", `${o.worktree}:/work:ro`, "-w", "/work",
31+
o.image, "sh", "-c", o.command,
32+
];
33+
}
34+
35+
// Installs dependencies before the test: network ON, repo writable, so the repo's own scripts must not run.
36+
export function installArgs(o: { name: string; worktree: string; image: string; command: string; limits?: BoxLimits }): string[] {
37+
const l = o.limits ?? DEFAULT_LIMITS;
38+
return [
39+
"run", "--name", o.name,
40+
"--memory", `${l.memoryMb}m`, "--memory-swap", `${l.memoryMb}m`,
41+
"--cpus", String(l.cpus), "--pids-limit", String(l.pids),
42+
"--cap-drop", "ALL", "--security-opt", "no-new-privileges",
43+
"--user", "node",
44+
"-e", "CI=1", "-e", "npm_config_cache=/tmp/.npm", "-e", "npm_config_ignore_scripts=true",
45+
"-v", `${o.worktree}:/work`, "-w", "/work",
46+
o.image, "sh", "-c", o.command,
47+
];
48+
}
49+
50+
// True when the Docker engine answers within 5 s. Docker Desktop not running -> false, never a hang.
51+
export async function dockerAvailable(docker = "docker"): Promise<boolean> {
52+
const r = await execa(docker, ["version", "--format", "{{.Server.Version}}"], { reject: false, timeout: 5_000, stdin: "ignore", windowsHide: true });
53+
return r.exitCode === 0 && r.stdout.trim() !== "";
54+
}
55+
56+
// The host's IANA zone, e.g. "Asia/Karachi". Windows' own name ("Pakistan Standard Time") means nothing to Linux.
57+
export function hostTimeZone(): string {
58+
return Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC";
59+
}

0 commit comments

Comments
 (0)