diff --git a/README.md b/README.md index 1f4fe8a..255c069 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,8 @@ Week 1 (Oct 19 to 23, 2026), plus the first part of week 2. | Piece | File | State | | --- | --- | --- | | Fake runner, the stand-in the daemon builds against | [runner/src/fake-runner.ts](runner/src/fake-runner.ts) | Done | -| Session records and legal state moves | [runner/src/session-store.ts](runner/src/session-store.ts) | Done | +| Session records and legal state moves | [runner/src/session-store.ts](runner/src/session-store.ts) | Done. The real runner saves them under `~/.codehands/sessions`, so they outlive a restart | +| `recover()`: clean up after a restart | [runner/src/runner.ts](runner/src/runner.ts) | Done. Not in the contract's `Runner`: the daemon calls it once at startup, on the type `createRunner` returns | | `seq` 1, 2, 3 … per session | [runner/src/events.ts](runner/src/events.ts) | Done | | Claude Code arguments and stream-json parser | [runner/src/agents/claude-code.ts](runner/src/agents/claude-code.ts) | Done, tested against a recorded run | | Worktrees: branch names, add, remove, prune | [runner/src/git/worktree.ts](runner/src/git/worktree.ts) | Done | @@ -80,7 +81,8 @@ Week 1 (Oct 19 to 23, 2026), plus the first part of week 2. | Codex: `codex exec --json` to events | [runner/src/agents/codex.ts](runner/src/agents/codex.ts) | To do. `start()` with `agent: "codex"` fails cleanly for now | | `approval.requested` after `task.finished` | [runner/src/runner.ts](runner/src/runner.ts) | Done (contracts v0.2.0). Real and fake runners, never after a stop or a failure | | Repo memory: notes read before a run, a note added after | [runner/src/memory/](runner/src/memory/) | Done. `MemoryStore` with an in-memory stub and a file store. Stopped and failed runs add nothing, and a memory error never fails a run | -| Repo profile: language, package manager, test command, Node version | [runner/src/memory/profile.ts](runner/src/memory/profile.ts) | Done. Read from files in the worktree, no model call. The sandbox reads it next | +| Repo profile: language, package manager, test command, Node version | [runner/src/memory/profile.ts](runner/src/memory/profile.ts) | Done. Read from files in the worktree, no model call. The sandbox uses it to pick the image and decide whether to install | +| Sandbox: run a repo's tests in a locked Docker box | [runner/src/runtime/](runner/src/runtime/) | Done. `runInSandbox()`: no network, 1 GB with no swap, 1 CPU, 256 processes, read-only repo and system, non-root, your timezone, 60 s. npm only. Nothing calls it yet: the engine's test step will | Next, from the [design](https://github.com/devweekends/codehands-docs#who-owns-what): a model per phase, naming provider errors as one `provider.status` event (`rate_limited`, `auth_expired`, `usage_limit`, `subscription_inactive`, `down`), memory and user preferences, and two models on one task. @@ -113,17 +115,25 @@ const { branch, commit } = await runner.approve(sessionId, "Fix the date bug"); | Export | What | | --- | --- | -| `createRunner(options?)` | The real runner. `worktreeRoot` overrides `~/.codehands/worktrees`. `memory` swaps where repo memory lives (default: files under `~/.codehands/memory`). `agentCommand` swaps the program it runs, which is how the tests replay a recording instead of calling Claude | +| `createRunner(options?)` | The real runner. `worktreeRoot` overrides `~/.codehands/worktrees`. `sessionRoot` overrides `~/.codehands/sessions`. `memory` swaps where repo memory lives (default: files under `~/.codehands/memory`). `agentCommand` swaps the program it runs, which is how the tests replay a recording instead of calling Claude | | `createFakeRunner({ delayMs? })` | Emits 12 scripted events, 300 ms apart, then `approval.requested`, and supports every method including stop and resume. Costs nothing, needs no repo. Build against it, and keep it as the demo backup | | `EventSequence` | Hands out events with the next `seq` | -| `SessionStore`, `SessionRecord` | In-memory session records with the legal-move check from the contracts | +| `SessionStore`, `SessionRecord` | Session records with the legal-move check from the contracts. Saved as files when given a root, otherwise in memory (the fake runner) | +| `RecoverableRunner`, `RecoveryReport` | What `createRunner` returns: the contract's `Runner` plus `recover()`, and what `recover()` reports | | `branchName(prompt, sessionId)`, `slugify(text)` | `codehands/fix-the-failing-test-3f2a9c1e` | | `MemoryStore`, `createMemoryStub()`, `createFileMemory(root?)` | Repo memory: `notes`, `addNote`, `profile`, `saveProfile`. The stub keeps it in a `Map` for tests | | `formatNote`, `runNote`, `trimNotes`, `notesForPrompt` | How a note is written, built from a run, trimmed to about 200 lines, and capped at 6,000 characters for the prompt | | `detectRepoProfile(dir)`, `RepoProfile` | What a repo is, read from its files. Never throws: an unknown repo gives `language: "unknown"` | | `repoKey(repoPath)` | The repo's id, which names its worktree and memory folders | +| `runInSandbox(options)`, `SandboxOptions`, `SandboxResult`, `SandboxError` | Runs the repo's test command in the locked Docker box and returns a result. Never throws: failing tests are `passed: false`, and a box that couldn't run them sets `error` (`sandbox.unavailable`, `sandbox.unsupported`, `sandbox.failed`, `install.failed`) | +| `dockerArgs()`, `installArgs()`, `DEFAULT_LIMITS`, `BoxLimits` | The exact `docker run` flags for the test box and the install box, as plain data | +| `dockerAvailable(docker?)`, `hostTimeZone()` | Whether the Docker engine answers within 5 s, and your IANA timezone (`Asia/Karachi`, not Windows' own name) | -The runner keeps session records in memory only. The daemon owns the real database. A runner that restarts forgets its sessions; stop, resume and crash recovery are where that changes. +When a repo needs its dependencies, a separate install box runs first, with the network on but the repo's install scripts off, then the test box runs with no network. `node_modules` must be git-ignored, or the sandbox refuses: everything not ignored lands in the commit you approve. + +The daemon owns the real database. The runner keeps only what it needs to finish a session, in `~/.codehands/sessions/.json`, so `approve`, `reject`, `resume` and `diff` still work after a restart. The agent itself does not survive one. + +When the daemon starts it should call `recover()` once, before anything else. Sessions still marked `queued` or `running` become `failed`, and their ids come back in `report.failed`. Failed sessions lose their worktree, along with anything uncommitted in it, because `failed` is final and nothing else would remove it. Their branch goes too, unless the agent committed to it. `awaiting_approval` and `stopped` sessions are left alone. A worktree that can't be removed comes back in `report.cleanupErrors` and is tried again next time. ### How Claude Code's output becomes events @@ -157,6 +167,9 @@ Some details that matter: | `~/.codehands/worktrees//` | The task's worktree. Outside your repo so worktrees never nest, and short because of Windows path limits | | `~/.codehands/memory//notes.md` | Per-repo notes, newest last, trimmed to about 200 lines by dropping the oldest whole notes | | `~/.codehands/memory//profile.json` | The repo profile from the latest run | +| `~/.codehands/sessions/.json` | What the runner needs to finish a session after a restart | +| `~/.codehands/runs//test-.json` | Each sandbox test run: passed, exit code, timed out, out of memory, the last 64 KB of output | +| Docker containers `ch--` | One per sandbox run, removed right after. `-install` on the end for the install box | | `codehands/-` | The task's branch, in your repo | `repoKey` is the first 12 characters of the SHA-1 of the repo's `origin` URL, or of its absolute path when there is no origin. Two clones of the same repo share a key. @@ -165,7 +178,7 @@ All of it stays on this machine. Memory reaches Claude only as `--append-system- ## Safety rules -The agent runs on your machine with no sandbox until the Docker box lands in week 2. Until then: +Tests run in the locked Docker box when the engine calls `runInSandbox()`. Claude Code itself still runs on your machine, not in a box, so: - **Run it only on our own repos.** - Claude runs with `--permission-mode acceptEdits` and a short allow list: `Read, Edit, Write, Glob, Grep`, and only `npm test` in Bash or PowerShell. Anything else is blocked and shows up as a `message` with `denied: true`. @@ -177,7 +190,7 @@ The agent runs on your machine with no sandbox until the Docker box lands in wee ## Set up -You need Node 22, pnpm 10, Git, and for real runs the [Claude Code](https://docs.claude.com/en/docs/claude-code) CLI, logged in. +You need Node 22, pnpm 10, Git, and for real runs the [Claude Code](https://docs.claude.com/en/docs/claude-code) CLI, logged in. For the sandbox, Docker Desktop must be running. Without it, the sandbox tests are skipped, not failed. The runner depends on the contracts by folder in week 1, so clone both side by side: @@ -202,7 +215,8 @@ Run these in `runner/`. | Command | What | Costs money | | --- | --- | --- | -| `pnpm test` | Every test. The parser replays a recorded run, the runner tests make real git repos in a temp folder and use a stand-in for `claude` | No | +| `pnpm test` | Every test. The parser replays a recorded run, the runner tests make real git repos in a temp folder and use a stand-in for `claude`, and the sandbox tests start real containers when Docker is running | No | +| `pnpm sandbox [repoPath] [--tz ]` | Runs a repo's tests in the locked box once and prints the result. Defaults to the fixture. `--tz UTC` shows why the box needs your timezone | No | | `pnpm fake` | Prints the fake runner's events, one JSON object per line. This is exactly what the daemon receives | No | | `pnpm try [repoPath] [prompt] [--approve \| --reject]` | One real run through the runner, one line per event, then the diff stat, then approve, reject or keep | Yes, capped at $0.50 | | `pnpm build` | Compiles to `dist/` | No | @@ -222,11 +236,14 @@ The scripts in `runner/scripts/`: | File | Covers | | --- | --- | | [claude-parser.test.ts](runner/test/claude-parser.test.ts) | The whole recorded run replays in order; each mapping above; blocked commands; failed tests hidden behind exit code 0; error results; broken lines; `raw` on every event | -| [runner.test.ts](runner/test/runner.test.ts) | A worktree on a new branch with your repo untouched; clean failures for a repo with no commits, an agent with no result line, a missing binary and an unsupported agent; diff with new files; approve commits on the branch and leaves `main` alone; reject deletes the branch; nothing to approve; stop kills the agent and its children, keeps the worktree and refuses a session that is not running; resume continues the numbering, passes the prompt, can be stopped again and refuses a session that is not stopped; `approval.requested` right after `task.finished`; memory: the second run gets the first run's note, the profile is saved, and failed or stopped runs add no note | +| [runner.test.ts](runner/test/runner.test.ts) | A worktree on a new branch with your repo untouched; clean failures for a repo with no commits, an agent with no result line, a missing binary and an unsupported agent; diff with new files; approve commits on the branch and leaves `main` alone; reject deletes the branch; nothing to approve; stop kills the agent and its children, keeps the worktree and refuses a session that is not running; resume continues the numbering, passes the prompt, can be stopped again and refuses a session that is not stopped; `approval.requested` right after `task.finished`; memory: the second run gets the first run's note, the profile is saved, and failed or stopped runs add no note; after a restart: approve, reject, diff and resume still work, and `recover()` fails interrupted sessions, removes their worktrees, keeps branches with commits, leaves live sessions alone, and reports what it can't clean or read | +| [session-store.test.ts](runner/test/session-store.test.ts) | A new store finds what the old one saved; the pid is never saved; moves are still checked after loading; ids that aren't plain names never reach the disk; no root means memory only | | [fake-runner.test.ts](runner/test/fake-runner.test.ts) | Event order and `seq`; approve only after finishing; stop then resume with `seq` continuing; no `approval.requested` after a stop | | [memory.test.ts](runner/test/memory.test.ts) | The stub keeps notes in order and repos apart; trimming drops whole notes and keeps the newest; the prompt cap keeps the newest end; a note's summary stays on one line; the file store reads back from disk and treats missing files as empty | | [profile.test.ts](runner/test/profile.test.ts) | npm, pnpm, npm's placeholder test script, `.nvmrc`, `engines.node`, needs install, broken `package.json`, Python, empty folder | | [worktree.test.ts](runner/test/worktree.test.ts) | Branch names | +| [docker-args.test.ts](runner/test/docker-args.test.ts) | Every security flag of the test box and the install box, with no Docker needed, so a lost flag fails in milliseconds | +| [sandbox.test.ts](runner/test/sandbox.test.ts) | Real containers, skipped when Docker is off: pass and fail; a test that never ends is killed and nothing is left behind, even when the kill misses; out of memory; no network; read-only repo and system; the 64 KB output cap; timezone; result files numbered across restarts; no false pass from a leftover container; `sandbox.failed` when the box can't start; install, with no install scripts and no lockfile added; a failed install; `node_modules` not git-ignored; Docker unavailable | To record a new fixture, run `scripts/spike.sh` (or a Codex equivalent), read every line of the output, then write tests that replay it. Never call a real agent from a test. @@ -240,10 +257,6 @@ To record a new fixture, run `scripts/spike.sh` (or a Codex equivalent), read ev | Cannot find module `@codehands/contracts` | Build the contracts repo first, and check it is cloned next to `codehands-agent` | | Child processes keep running after a stop on Windows | Killing the parent does not kill its children (`claude` → `npm test` → `node`). That is what `stopProcessTree` is for | | "nothing to approve: the agent made no changes" | The agent changed no files. Reject the session instead | - -## Contributing - -- One branch per Linear issue, for example `tahira/cod-12-runner-start`. Put the issue id in the PR title. -- `main` is always green. CI runs build and test on every PR. -- A change to `StartOptions`, `RunnerEvent` or `Runner` is a contract change. It goes to [codehands-contracts](https://github.com/devweekends/codehands-contracts#changing-a-contract) first, and `createFakeRunner()` is updated the same week. -- The week by week plan is in [roadmap/tahira.md](https://github.com/devweekends/codehands-docs/blob/main/roadmap/tahira.md). +| `sandbox.unavailable` | Docker Desktop isn't running. Start it and try again | +| `sandbox.failed` | The box broke, not your code. `message` and `output` say why: often a wrong image tag from `.nvmrc`, or no internet for the first image pull | +| "add node_modules/ to .gitignore first" | The sandbox won't install into a repo that doesn't ignore `node_modules`, because it would land in the commit | diff --git a/runner/package.json b/runner/package.json index 07ab696..07216d8 100644 --- a/runner/package.json +++ b/runner/package.json @@ -18,7 +18,8 @@ "typecheck": "tsc --noEmit", "test": "vitest run", "fake": "tsx scripts/fake.ts", - "try": "tsx scripts/try.ts" + "try": "tsx scripts/try.ts", + "sandbox": "tsx scripts/sandbox.ts" }, "dependencies": { "@codehands/contracts": "file:../../codehands-contracts", diff --git a/runner/scripts/sandbox.ts b/runner/scripts/sandbox.ts new file mode 100644 index 0000000..e737aaa --- /dev/null +++ b/runner/scripts/sandbox.ts @@ -0,0 +1,17 @@ +// pnpm sandbox [repoPath] [--tz ]: runs the repo's tests in the locked box once. No model calls, no cost. +import { randomUUID } from "node:crypto"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { detectRepoProfile, runInSandbox } from "../src/index.js"; + +const args = process.argv.slice(2); +const tzAt = args.indexOf("--tz"); +const timeZone = tzAt >= 0 ? args[tzAt + 1] : undefined; +const repoPath = args.find((a, i) => !a.startsWith("--") && (tzAt < 0 || i !== tzAt + 1)) ?? join(homedir(), "codehands-fixtures", "date-bug"); + +const profile = await detectRepoProfile(repoPath); +console.log("profile:", profile); +const r = await runInSandbox({ sessionId: randomUUID(), worktree: repoPath, profile, timeZone }); +console.log(r.output.trim().split("\n").slice(-30).join("\n")); +console.log({ passed: r.passed, exitCode: r.exitCode, timedOut: r.timedOut, oomKilled: r.oomKilled, durationMs: r.durationMs, error: r.error, message: r.message, resultFile: r.resultFile }); +process.exitCode = r.passed ? 0 : 1; diff --git a/runner/src/index.ts b/runner/src/index.ts index cbd66c3..ba52842 100644 --- a/runner/src/index.ts +++ b/runner/src/index.ts @@ -1,4 +1,4 @@ -export { createRunner, type RunnerOptions } from "./runner.js"; +export { createRunner, type RecoverableRunner, type RecoveryReport, type RunnerOptions } from "./runner.js"; export { createFakeRunner, type FakeRunnerOptions } from "./fake-runner.js"; export { EventSequence } from "./events.js"; export { SessionStore, type SessionRecord } from "./session-store.js"; @@ -6,3 +6,5 @@ export { branchName, slugify } from "./git/worktree.js"; export { createMemoryStub, formatNote, notesForPrompt, runNote, trimNotes, type MemoryNote, type MemoryStore } from "./memory/store.js"; export { createFileMemory, repoKey } from "./memory/notes.js"; export { detectRepoProfile, type RepoProfile } from "./memory/profile.js"; +export { runInSandbox, type SandboxOptions, type SandboxResult, type SandboxError } from "./runtime/sandbox.js"; +export { dockerArgs, installArgs, dockerAvailable, hostTimeZone, DEFAULT_LIMITS, type BoxLimits } from "./runtime/docker.js"; diff --git a/runner/src/runner.ts b/runner/src/runner.ts index 39677b3..f633f3d 100644 --- a/runner/src/runner.ts +++ b/runner/src/runner.ts @@ -1,11 +1,14 @@ import { randomUUID } from "node:crypto"; +import { existsSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; import { createInterface } from "node:readline"; import type { Runner, RunnerEvent, StartOptions } from "@codehands/contracts"; import { execa } from "execa"; import { claudeArgs, claudeResumeArgs, newClaudeParseState, parseClaudeLine } from "./agents/claude-code.js"; import { EventSequence } from "./events.js"; import { commitAll, diffSince } from "./git/diff.js"; -import { addWorktree, baseCommit, branchName, pruneWorktrees, removeWorktree, worktreePath } from "./git/worktree.js"; +import { addWorktree, baseCommit, branchName, git, pruneWorktrees, removeWorktree, worktreePath } from "./git/worktree.js"; import { createFileMemory, repoKey } from "./memory/notes.js"; import { detectRepoProfile } from "./memory/profile.js"; import { notesForPrompt, runNote, type MemoryStore } from "./memory/store.js"; @@ -16,6 +19,7 @@ import { SessionStore, type SessionRecord } from "./session-store.js"; export interface RunnerOptions { worktreeRoot?: string; // default ~/.codehands/worktrees + sessionRoot?: string; // default ~/.codehands/sessions. Session records, so they outlive a restart memory?: MemoryStore; // default: files under ~/.codehands/memory. Tests pass createMemoryStub() // Which program to run for a task. Tests swap in a script that replays a recording. // notes: the repo's memory for the prompt, undefined when there is none. @@ -23,6 +27,17 @@ export interface RunnerOptions { resumeCommand?: (o: StartOptions, prompt: string) => { file: string; args: string[] }; } +// What recover() did. failed: sessions a restart interrupted, for the daemon to mark failed too. +// cleanupErrors: sessions it couldn't clean up. They stay on disk and the next recover() tries again. +export interface RecoveryReport { + failed: string[]; + cleanupErrors: { sessionId: string; reason: string }[]; +} + +export interface RecoverableRunner extends Runner { + recover(): Promise; +} + const claudeCommand = (o: StartOptions, notes?: string) => ({ file: "claude", args: claudeArgs(o, notes) }); // start() passes our sessionId as Claude's --session-id, so the same id resumes it. const claudeResumeCommand = (o: StartOptions, prompt: string) => @@ -30,11 +45,12 @@ const claudeResumeCommand = (o: StartOptions, prompt: string) => export function createRunner({ worktreeRoot, + sessionRoot = join(homedir(), ".codehands", "sessions"), memory = createFileMemory(), agentCommand = claudeCommand, resumeCommand = claudeResumeCommand, -}: RunnerOptions = {}): Runner { - const store = new SessionStore(); +}: RunnerOptions = {}): RecoverableRunner { + const store = new SessionStore(sessionRoot); // The running agent of each session, so stop() can wait for it to exit. const exits = new Map>(); @@ -42,6 +58,7 @@ export function createRunner({ function emitters(record: SessionRecord, seq: EventSequence) { const emit = (event: RunnerEvent) => { record.lastSeq = seq.last; + store.save(record); return event; }; const fail = (reason: string, extra: Record = {}) => { @@ -67,6 +84,7 @@ export function createRunner({ record.branch = branchName(o.prompt, o.sessionId); record.repoKey = await repoKey(o.repoPath); record.worktreePath = worktreePath(record.repoKey, o.sessionId, worktreeRoot); + store.save(record); // before the worktree exists, so a crash here can't leave one nobody knows about await pruneWorktrees(o.repoPath); await addWorktree(o.repoPath, record.branch, record.worktreePath, record.baseCommit); } catch (err) { @@ -169,8 +187,40 @@ export function createRunner({ return notesForPrompt(await memory.notes(key)); } + // A failed session can't be approved or rejected, so nothing else would remove its worktree. + async function discardWorktree(record: SessionRecord): Promise { + const { repoPath } = record.options; + await removeWorktree(repoPath, record.worktreePath!); + if (existsSync(record.worktreePath!)) throw new Error(`could not remove ${record.worktreePath}`); + // The branch goes only if it never moved: commits the agent made are kept. + const tip = await git(repoPath, ["rev-parse", "--verify", "--quiet", `refs/heads/${record.branch}`]).catch(() => ""); + if (tip && tip === record.baseCommit) await git(repoPath, ["branch", "-D", record.branch!]); + record.worktreePath = undefined; + store.save(record); + } + return { start, + // Call once when the daemon starts, before it starts or commands anything. + // A session still marked queued or running lost its agent, so it becomes failed. + // awaiting_approval and stopped sessions are left alone: approve, reject and resume still work. + async recover() { + const report: RecoveryReport = { failed: [], cleanupErrors: [] }; + for (const sessionId of store.ids()) { + if (exits.has(sessionId)) continue; // its agent is alive in this process + try { + const record = store.get(sessionId); + if (record.status === "queued" || record.status === "running") { + store.transition(sessionId, "failed"); + report.failed.push(sessionId); + } + if (record.status === "failed" && record.worktreePath) await discardWorktree(record); + } catch (err) { + report.cleanupErrors.push({ sessionId, reason: (err as Error).message }); + } + } + return report; + }, // Kills the agent and everything it started. task.stopped comes out of the start() stream, // and the session is stopped by the time this resolves. The worktree stays. async stop(sessionId) { diff --git a/runner/src/runtime/docker.ts b/runner/src/runtime/docker.ts new file mode 100644 index 0000000..9fb277f --- /dev/null +++ b/runner/src/runtime/docker.ts @@ -0,0 +1,59 @@ +// Week 2: the locked box. Every flag here is checked in test/docker-args.test.ts. +// Spike results (Nov 2): no --rm (OOMKilled must be read first), no pipes (they hide the exit code). +import { execa } from "execa"; + +// Week 2: the locked box. Every flag here is checked in test/docker-args.test.ts. +// Spike results (Nov 2): no --rm (OOMKilled must be read first), no pipes (they hide the exit code). + +export interface BoxLimits { + memoryMb: number; // 1024. Swap is set to the same, so there is no swap + cpus: number; // 1 + pids: number; // 256 +} + +export const DEFAULT_LIMITS: BoxLimits = { memoryMb: 1024, cpus: 1, pids: 256 }; + +// The test container: no network, read-only repo and system, non-root, no capabilities. +export function dockerArgs(o: { + name: string; worktree: string; image: string; command: string; timeZone: string; limits?: BoxLimits; +}): string[] { + const l = o.limits ?? DEFAULT_LIMITS; + return [ + "run", "--name", o.name, + "--network", "none", + "--memory", `${l.memoryMb}m`, "--memory-swap", `${l.memoryMb}m`, + "--cpus", String(l.cpus), "--pids-limit", String(l.pids), + "--cap-drop", "ALL", "--security-opt", "no-new-privileges", + "--user", "node", + "--read-only", "--tmpfs", "/tmp:exec", "--tmpfs", "/home/node:uid=1000,gid=1000", + "-e", `TZ=${o.timeZone}`, "-e", "CI=1", + "-v", `${o.worktree}:/work:ro`, "-w", "/work", + o.image, "sh", "-c", o.command, + ]; +} + +// Installs dependencies before the test: network ON, repo writable, so the repo's own scripts must not run. +export function installArgs(o: { name: string; worktree: string; image: string; command: string; limits?: BoxLimits }): string[] { + const l = o.limits ?? DEFAULT_LIMITS; + return [ + "run", "--name", o.name, + "--memory", `${l.memoryMb}m`, "--memory-swap", `${l.memoryMb}m`, + "--cpus", String(l.cpus), "--pids-limit", String(l.pids), + "--cap-drop", "ALL", "--security-opt", "no-new-privileges", + "--user", "node", + "-e", "CI=1", "-e", "npm_config_cache=/tmp/.npm", "-e", "npm_config_ignore_scripts=true", + "-v", `${o.worktree}:/work`, "-w", "/work", + o.image, "sh", "-c", o.command, + ]; +} + +// True when the Docker engine answers within 5 s. Docker Desktop not running -> false, never a hang. +export async function dockerAvailable(docker = "docker"): Promise { + const r = await execa(docker, ["version", "--format", "{{.Server.Version}}"], { reject: false, timeout: 5_000, stdin: "ignore", windowsHide: true }); + return r.exitCode === 0 && r.stdout.trim() !== ""; +} + +// The host's IANA zone, e.g. "Asia/Karachi". Windows' own name ("Pakistan Standard Time") means nothing to Linux. +export function hostTimeZone(): string { + return Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC"; +} diff --git a/runner/src/runtime/sandbox.ts b/runner/src/runtime/sandbox.ts new file mode 100644 index 0000000..f959205 --- /dev/null +++ b/runner/src/runtime/sandbox.ts @@ -0,0 +1,188 @@ +import type { RepoProfile } from "../memory/profile.js"; +import { execa } from "execa"; +import { DEFAULT_LIMITS, dockerArgs, dockerAvailable, hostTimeZone, installArgs, type BoxLimits } from "./docker.js"; +import { existsSync, readdirSync } from "node:fs"; +import { homedir } from "node:os"; +import { join, resolve } from "node:path"; +import { mkdir, writeFile } from "node:fs/promises"; + +export interface SandboxOptions { + sessionId: string; + worktree: string; + profile: RepoProfile; + command?: string; // default: profile.testCommand + timeoutMs?: number; // default 60_000 + limits?: BoxLimits; // default DEFAULT_LIMITS + timeZone?: string; // default hostTimeZone() + runsRoot?: string; // default ~/.codehands/runs. Tests pass a temp folder + docker?: string; // default "docker". Tests pass a missing binary +} + +export type SandboxError = "sandbox.unavailable" | "sandbox.unsupported" | "sandbox.failed" | "install.failed"; + +export interface SandboxResult { + command: string; + image: string; + passed: boolean; // exit 0, not timed out, not killed for memory + exitCode: number | null; // null when the process never finished: it didn't start, or it had to be abandoned + timedOut: boolean; + oomKilled: boolean; + durationMs: number; + output: string; // stdout + stderr, the last 64 KB + outputTruncated: boolean; + error?: SandboxError; // set when the tests never ran properly + message?: string; // a human sentence for error + resultFile?: string; // where the JSON was saved +} + +const MAX_OUTPUT = 64 * 1024; +const SAFE_ID = /^[\w-]+$/; // the session id names a folder and a container + +// Every short docker command has a limit, so a stuck Docker can't hang us. +const cli = (docker: string, args: string[], timeout = 30_000) => + execa(docker, args, { reject: false, timeout, stdin: "ignore", windowsHide: true }); + +export async function runInSandbox(o: SandboxOptions): Promise { + const docker = o.docker ?? "docker"; + const command = o.command ?? o.profile.testCommand ?? ""; + const image = `node:${o.profile.nodeVersion}-slim`; + const base = { command, image, passed: false, exitCode: null, timedOut: false, oomKilled: false, durationMs: 0, output: "", outputTruncated: false }; + + if (!(await dockerAvailable(docker))) { + return { ...base, error: "sandbox.unavailable", message: "Docker is not running. Start Docker Desktop and try again." }; + } + if (o.profile.language !== "node" || o.profile.packageManager !== "npm" || !command) { + return { ...base, error: "sandbox.unsupported", message: `only npm repos with a test script for now (got ${o.profile.language}/${o.profile.packageManager ?? "none"})` }; + } + if (!SAFE_ID.test(o.sessionId)) { + return { ...base, error: "sandbox.failed", message: `bad session id: ${o.sessionId}` }; + } + // Docker would create a missing folder and mount it empty, which looks like broken tests. + const worktree = resolve(o.worktree); + if (!existsSync(worktree)) { + return { ...base, error: "sandbox.failed", message: `worktree not found: ${worktree}` }; + } + await ensureImage(docker, image); + + const dir = join(o.runsRoot ?? join(homedir(), ".codehands", "runs"), o.sessionId); + const n = nextRun(dir); + const name = `ch-${o.sessionId.slice(0, 8)}-${n}`; + const limits = o.limits ?? DEFAULT_LIMITS; + + if (o.profile.needsInstall) { + // git add -A takes everything that isn't ignored, so node_modules would land in the commit you approve. + const ignored = await execa("git", ["check-ignore", "-q", "node_modules/"], { cwd: worktree, reject: false, timeout: 10_000, stdin: "ignore", windowsHide: true }); + if (ignored.exitCode === 1) { + return { ...base, error: "sandbox.unsupported", message: "add node_modules/ to .gitignore first, or the install would land in the diff" }; + } + + const installCommand = existsSync(join(worktree, "package-lock.json")) ? "npm ci" : "npm install --no-package-lock"; + const { started: _started, ...install } = await runContainer(docker, `${name}-install`, installArgs({ + name: `${name}-install`, worktree, image, command: installCommand, limits, + }), 5 * 60_000); + if (install.exitCode !== 0 || install.timedOut || install.oomKilled) { + return save(dir, n, { ...base, ...install, command: installCommand, error: "install.failed", message: "npm could not install the dependencies. The output says why." }); + } + + } + const { started, ...ran } = await runContainer(docker, name, dockerArgs({ + name, worktree, image, command, timeZone: o.timeZone ?? hostTimeZone(), limits, + }), o.timeoutMs ?? 60_000); + + // A container that never started is not a failing test: the engine must not try to fix the code for it. + if (!started && !ran.timedOut) { + return save(dir, n, { ...base, ...ran, error: "sandbox.failed", message: "the container did not start. The output says why." }); + } + + return save(dir, n, { ...base, ...ran, passed: ran.exitCode === 0 && !ran.timedOut && !ran.oomKilled }); +} + +// Starts a named container, keeps the last 64 KB of output, kills it at the limit, +// reads what really happened from Docker, and always removes it. +async function runContainer(docker: string, name: string, args: string[], timeoutMs: number) { + // A crash can leave this name behind. docker run would refuse it, and inspect would read the old result. + await cli(docker, ["rm", "-f", name]); + + const started = Date.now(); + let output = ""; + let outputTruncated = false; + let timedOut = false; + + const child = execa(docker, args, { reject: false, all: true, buffer: false, stdin: "ignore", windowsHide: true }); + child.all!.setEncoding("utf8"); + child.all!.on("data", (chunk: string) => { + output += chunk; + if (output.length > MAX_OUTPUT) { + output = output.slice(-MAX_OUTPUT); + outputTruncated = true; + } + }); + + // docker kill does nothing if the container doesn't exist yet or Docker is stuck, so stop waiting after 10 s. + let giveUp: NodeJS.Timeout | undefined; + const timer = setTimeout(() => { + timedOut = true; + void cli(docker, ["kill", name]); + giveUp = setTimeout(() => child.kill("SIGKILL"), 10_000); + }, timeoutMs); + + try { + await child; + clearTimeout(timer); + clearTimeout(giveUp); + const state = await readState(docker, name); + const ran = state !== undefined && !state.Error && !state.StartedAt.startsWith("0001"); + return { + started: ran, + exitCode: ran && !state.Running ? state.ExitCode : null, + oomKilled: state?.OOMKilled ?? false, + timedOut, + durationMs: Date.now() - started, + output, + outputTruncated, + }; + } finally { + clearTimeout(timer); + clearTimeout(giveUp); + await cli(docker, ["rm", "-f", name]); + } +} + +// What Docker knows about a container. undefined when there is no such container. +async function readState(docker: string, name: string) { + const r = await cli(docker, ["inspect", name, "--format", "{{json .State}}"]); + if (r.exitCode !== 0) return undefined; + try { + return JSON.parse(r.stdout) as { ExitCode: number; OOMKilled: boolean; Running: boolean; Error: string; StartedAt: string }; + } catch { + return undefined; + } +} + +// Pulls the image before the timer starts, so a slow download never counts as a test timeout. +// A failed pull needs no check here: docker run fails next, and that becomes sandbox.failed. +async function ensureImage(docker: string, image: string): Promise { + const found = await cli(docker, ["image", "inspect", image]); + if (found.exitCode !== 0) await cli(docker, ["pull", "-q", image], 5 * 60_000); +} + +// Counted from the files, not memory, so a restart can't overwrite test-1.json. +function nextRun(dir: string): number { + try { + return readdirSync(dir).filter((file) => /^test-\d+\.json$/.test(file)).length + 1; + } catch { + return 1; + } +} + +// A full disk must not turn a result into a crash, so a failed save just leaves resultFile unset. +async function save(dir: string, n: number, result: SandboxResult): Promise { + const resultFile = join(dir, `test-${n}.json`); + try { + await mkdir(dir, { recursive: true }); + await writeFile(resultFile, JSON.stringify({ ...result, resultFile }, null, 2)); + return { ...result, resultFile }; + } catch { + return result; + } +} diff --git a/runner/src/session-store.ts b/runner/src/session-store.ts index bbabf96..05e86f0 100644 --- a/runner/src/session-store.ts +++ b/runner/src/session-store.ts @@ -1,3 +1,5 @@ +import { mkdirSync, readdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; import { canTransition, type SessionStatus, type StartOptions } from "@codehands/contracts"; // Only what the runner needs to stop, resume, diff, approve or reject. @@ -9,32 +11,61 @@ export interface SessionRecord { repoKey?: string; // set by start(). Where this repo's memory lives branch?: string; baseCommit?: string; - pid?: number; - stopRequested?: boolean; // set by stop(), so the agent's exit reads as task.stopped, not task.failed + pid?: number; // never saved: after a restart the number may belong to another program + stopRequested?: boolean; // set by stop(), so the agent's exit reads as task.stopped, not task.failed. Never saved lastSeq: number; diff?: { patch: string; stat: string }; } +// Session ids name files, so one like ../../x must never reach the disk. +const SAFE_ID = /^[\w-]+$/; + export class SessionStore { private sessions = new Map(); + private root?: string; + + // With a root, every record is also a file (/.json), so it outlives a restart. + // Without one it stays in memory, which is all the fake runner needs. + constructor(root?: string) { + this.root = root; + if (root) mkdirSync(root, { recursive: true }); + } create(options: StartOptions): SessionRecord { - if (this.sessions.has(options.sessionId)) throw new Error(`session ${options.sessionId} already exists`); + if (this.root && !SAFE_ID.test(options.sessionId)) throw new Error(`bad session id ${options.sessionId}`); + if (this.find(options.sessionId)) throw new Error(`session ${options.sessionId} already exists`); const record: SessionRecord = { options, status: "queued", lastSeq: 0 }; this.sessions.set(options.sessionId, record); + this.save(record); return record; } get(sessionId: string): SessionRecord { - const record = this.sessions.get(sessionId); + const record = this.find(sessionId); if (!record) throw new Error(`unknown session ${sessionId}`); return record; } + // Every saved session, including ones from before a restart. + ids(): string[] { + if (!this.root) return [...this.sessions.keys()]; + return readdirSync(this.root).filter((name) => name.endsWith(".json")).map((name) => name.slice(0, -".json".length)); + } + + // Writes to a temp file first, so a crash mid-write can't leave half a record. + save(record: SessionRecord): void { + if (!this.root) return; + const { pid: _pid, stopRequested: _stopRequested, ...saved } = record; + const file = this.file(record.options.sessionId); + writeFileSync(`${file}.tmp`, JSON.stringify(saved)); + renameSync(`${file}.tmp`, file); + } + // Refuses any move not in SESSION_TRANSITIONS, e.g. approving a running session. transition(sessionId: string, to: SessionStatus): SessionRecord { const record = this.check(sessionId, to); record.status = to; + this.save(record); return record; } @@ -46,4 +77,24 @@ export class SessionStore { } return record; } + + // Memory first, then the disk. undefined when it's in neither. + private find(sessionId: string): SessionRecord | undefined { + const known = this.sessions.get(sessionId); + if (known || !this.root || !SAFE_ID.test(sessionId)) return known; + let text: string; + try { + text = readFileSync(this.file(sessionId), "utf8"); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === "ENOENT") return undefined; + throw err; + } + const record = JSON.parse(text) as SessionRecord; + this.sessions.set(sessionId, record); + return record; + } + + private file(sessionId: string): string { + return join(this.root!, `${sessionId}.json`); + } } diff --git a/runner/test/docker-args.test.ts b/runner/test/docker-args.test.ts new file mode 100644 index 0000000..07bf404 --- /dev/null +++ b/runner/test/docker-args.test.ts @@ -0,0 +1,72 @@ +import { describe, expect, it } from "vitest"; +import { dockerArgs, hostTimeZone, installArgs } from "../src/runtime/docker.js"; + +const box = (extra = {}) => dockerArgs({ name: "ch-test-1", worktree: "C:\\Users\\My Name\\repo", image: "node:22-slim", command: "npm test", timeZone: "Asia/Karachi", ...extra }); +const pair = (args: string[], flag: string) => args[args.indexOf(flag) + 1]; + +describe("dockerArgs (the test box)", () => { + it("locks the box down", () => { + const a = box(); + expect(pair(a, "--network")).toBe("none"); + expect(pair(a, "--memory")).toBe("1024m"); + expect(pair(a, "--memory-swap")).toBe("1024m"); // same as memory: no swap + expect(pair(a, "--cpus")).toBe("1"); + expect(pair(a, "--pids-limit")).toBe("256"); + expect(pair(a, "--cap-drop")).toBe("ALL"); + expect(pair(a, "--security-opt")).toBe("no-new-privileges"); + expect(pair(a, "--user")).toBe("node"); + expect(pair(a, "-w")).toBe("/work"); + }); + + it("makes the system read-only, with memory-backed folders for scratch space", () => { + const a = box(); + expect(a).toContain("--read-only"); + expect(a).toContain("/tmp:exec"); + expect(a).toContain("/home/node:uid=1000,gid=1000"); + }); + + it("mounts the worktree read-only, as one argument even with spaces", () => { + expect(pair(box(), "-v")).toBe("C:\\Users\\My Name\\repo:/work:ro"); + }); + + it("keeps the container after it exits, so OOMKilled can be read", () => { + expect(box()).not.toContain("--rm"); + }); + + it("passes the timezone and CI, and runs the command last", () => { + const a = box(); + expect(a).toContain("TZ=Asia/Karachi"); + expect(a).toContain("CI=1"); + expect(a.slice(-4)).toEqual(["node:22-slim", "sh", "-c", "npm test"]); + }); + + it("uses the limits it is given", () => { + const a = box({ limits: { memoryMb: 64, cpus: 2, pids: 50 } }); + expect(pair(a, "--memory")).toBe("64m"); + expect(pair(a, "--memory-swap")).toBe("64m"); + expect(pair(a, "--cpus")).toBe("2"); + expect(pair(a, "--pids-limit")).toBe("50"); + }); +}); + +describe("installArgs (the install box)", () => { + const a = installArgs({ name: "ch-test-1-install", worktree: "/w", image: "node:22-slim", command: "npm ci" }); + + it("has the network and a writable repo, but stays non-root and limited", () => { + expect(a).not.toContain("--network"); + expect(pair(a, "-v")).toBe("/w:/work"); + expect(pair(a, "--user")).toBe("node"); + expect(pair(a, "--cap-drop")).toBe("ALL"); + expect(pair(a, "--memory")).toBe("1024m"); + }); + + it("never runs the repo's own install scripts", () => { + expect(a).toContain("npm_config_ignore_scripts=true"); + }); +}); + +describe("hostTimeZone", () => { + it("gives an IANA name, not a Windows one", () => { + expect(hostTimeZone()).toMatch(/^(UTC|[A-Za-z_]+\/[A-Za-z_\/+-]+)$/); + }); +}); diff --git a/runner/test/runner.test.ts b/runner/test/runner.test.ts index 1d8e69b..4f0af03 100644 --- a/runner/test/runner.test.ts +++ b/runner/test/runner.test.ts @@ -39,7 +39,7 @@ async function collect(o: StartOptions, runnerOptions: RunnerOptions = {}, runne } const newRunner = (runnerOptions: RunnerOptions = {}) => - createRunner({ worktreeRoot: join(tmp, "worktrees"), memory: createMemoryStub(), ...runnerOptions }); + createRunner({ worktreeRoot: join(tmp, "worktrees"), sessionRoot: join(tmp, "sessions"), memory: createMemoryStub(), ...runnerOptions }); const replayAgent = (file: string, exitCode = 0, mode = "", pidFile = ""): RunnerOptions["agentCommand"] => () => ({ file: process.execPath, args: [replay, file, String(exitCode), mode, pidFile] }); @@ -271,6 +271,150 @@ describe("runner.resume", () => { }); }); +describe("runner after a restart", () => { + // A new runner on the same sessionRoot is what a restarted daemon gets. + const sessionRootOf = () => ({ sessionRoot: mkdtempSync(join(tmp, "sessions-")) }); + const worktreeOf = (repo: string, sessionId: string) => git(repo, "worktree", "list", "--porcelain").split("\n") + .filter((l) => l.startsWith("worktree ")).map((l) => l.slice(9)).find((p) => p.includes(sessionId.slice(0, 8))); + const branchOf = (o: StartOptions) => `codehands/fix-the-failing-test-${o.sessionId.slice(0, 8)}`; + + async function finished(roots: RunnerOptions) { + const repo = makeRepo(); + const o = options(repo); + await collect(o, { ...roots, agentCommand: replayAgent(spike, 0, "edit") }); + return { repo, o }; + } + + // The daemon died mid-run: the file still says running. + function markRunning(sessionRoot: string, sessionId: string) { + const file = join(sessionRoot, `${sessionId}.json`); + writeFileSync(file, JSON.stringify({ ...JSON.parse(readFileSync(file, "utf8")), status: "running" })); + } + + it("approves a session from before the restart", async () => { + const roots = sessionRootOf(); + const { repo, o } = await finished(roots); + const worktree = worktreeOf(repo, o.sessionId)!; + + const { branch, commit } = await newRunner(roots).approve(o.sessionId, "Fix the failing test"); + expect(git(repo, "rev-parse", branch)).toBe(commit); + expect(existsSync(worktree)).toBe(false); + }); + + it("rejects a session from before the restart", async () => { + const roots = sessionRootOf(); + const { repo, o } = await finished(roots); + const worktree = worktreeOf(repo, o.sessionId)!; + + await newRunner(roots).reject(o.sessionId); + expect(existsSync(worktree)).toBe(false); + expect(git(repo, "branch", "--list", branchOf(o))).toBe(""); + }); + + it("still shows the diff, and remembers a session is finished", async () => { + const roots = sessionRootOf(); + const { o } = await finished(roots); + const restarted = newRunner(roots); + expect((await restarted.diff(o.sessionId)).patch).toContain("+world"); + + await restarted.approve(o.sessionId, "Fix the failing test"); + const again = newRunner(roots); + expect((await again.diff(o.sessionId)).patch).toContain("+world"); + await expect(again.approve(o.sessionId, "again")).rejects.toThrow(/illegal move/); + }); + + it("resumes a stopped session, and seq carries on", async () => { + const roots = sessionRootOf(); + const { o, events } = await stoppedRun(roots); + const restarted = newRunner({ ...roots, resumeCommand: () => ({ file: process.execPath, args: [replay, spike, "0", "edit"] }) }); + + const resumed = await drain(restarted.resume(o.sessionId)); + expect(resumed[0]!.seq).toBe(events.length + 1); + expect(resumed.at(-1)!.type).toBe("approval.requested"); + }); + + it("recover fails an interrupted session and removes its worktree and empty branch", async () => { + const roots = sessionRootOf(); + const { repo, o } = await stoppedRun(roots); + const worktree = worktreeOf(repo, o.sessionId)!; + markRunning(roots.sessionRoot, o.sessionId); + + const restarted = newRunner(roots); + expect(await restarted.recover()).toEqual({ failed: [o.sessionId], cleanupErrors: [] }); + expect(existsSync(worktree)).toBe(false); + expect(git(repo, "branch", "--list", branchOf(o))).toBe(""); + await expect(restarted.approve(o.sessionId, "msg")).rejects.toThrow(/no worktree/); + expect(await restarted.recover()).toEqual({ failed: [], cleanupErrors: [] }); + }); + + it("recover keeps a branch the agent committed to", async () => { + const roots = sessionRootOf(); + const { repo, o } = await stoppedRun(roots); + const worktree = worktreeOf(repo, o.sessionId)!; + writeFileSync(join(worktree, "partial.txt"), "work\n"); + git(worktree, "add", "-A"); + git(worktree, "commit", "-q", "-m", "partial"); + markRunning(roots.sessionRoot, o.sessionId); + + await newRunner(roots).recover(); + expect(existsSync(worktree)).toBe(false); + expect(git(repo, "log", "-1", "--format=%s", branchOf(o))).toBe("partial"); + }); + + it("recover removes the worktree of a run that had already failed", async () => { + const roots = sessionRootOf(); + const repo = makeRepo(); + const o = options(repo); + const empty = join(tmp, "empty-recover.jsonl"); + writeFileSync(empty, ""); + await collect(o, { ...roots, agentCommand: replayAgent(empty, 1) }); + const worktree = worktreeOf(repo, o.sessionId)!; + expect(existsSync(worktree)).toBe(true); + + expect(await newRunner(roots).recover()).toEqual({ failed: [], cleanupErrors: [] }); + expect(existsSync(worktree)).toBe(false); + }); + + it("recover leaves sessions alone that can still be approved or resumed", async () => { + const roots = sessionRootOf(); + const waiting = await finished(roots); + const stopped = await stoppedRun(roots); + + expect(await newRunner(roots).recover()).toEqual({ failed: [], cleanupErrors: [] }); + expect(existsSync(worktreeOf(waiting.repo, waiting.o.sessionId)!)).toBe(true); + expect(existsSync(worktreeOf(stopped.repo, stopped.o.sessionId)!)).toBe(true); + }); + + it("recover reports a worktree it can't remove, and tries again next time", async () => { + const roots = sessionRootOf(); + const { repo, o } = await stoppedRun(roots); + const worktree = worktreeOf(repo, o.sessionId)!; + git(repo, "worktree", "lock", worktree); // git refuses, like a folder a live process has open + markRunning(roots.sessionRoot, o.sessionId); + + const first = await newRunner(roots).recover(); + expect(first.failed).toEqual([o.sessionId]); + expect(first.cleanupErrors.map((e) => e.sessionId)).toEqual([o.sessionId]); + expect(existsSync(worktree)).toBe(true); + + git(repo, "worktree", "unlock", worktree); + expect(await newRunner(roots).recover()).toEqual({ failed: [], cleanupErrors: [] }); + expect(existsSync(worktree)).toBe(false); + }); + + it("recover carries on past a session it can't read", async () => { + const roots = sessionRootOf(); + const { repo, o } = await stoppedRun(roots); + markRunning(roots.sessionRoot, o.sessionId); + writeFileSync(join(roots.sessionRoot, "broken.json"), "{ not json"); + + const report = await newRunner(roots).recover(); + expect(report.failed).toEqual([o.sessionId]); + expect(report.cleanupErrors.map((e) => e.sessionId)).toEqual(["broken"]); + expect(worktreeOf(repo, o.sessionId)).toBeUndefined(); + }); +}); + describe("runner memory", () => { it("passes the first run's note to the second run", async () => { const memory = createMemoryStub(); diff --git a/runner/test/sandbox.test.ts b/runner/test/sandbox.test.ts new file mode 100644 index 0000000..12e0fe2 --- /dev/null +++ b/runner/test/sandbox.test.ts @@ -0,0 +1,203 @@ +import { execFileSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { execa } from "execa"; +import { afterAll, describe, expect, it } from "vitest"; +import type { RepoProfile } from "../src/memory/profile.js"; +import { dockerAvailable } from "../src/runtime/docker.js"; +import { runInSandbox } from "../src/runtime/sandbox.js"; + +// Real containers. Skipped when Docker isn't running, so `pnpm test` never fails just because Docker Desktop is closed. +const hasDocker = await dockerAvailable(); +const tmp = mkdtempSync(join(tmpdir(), "codehands-sandbox-")); +afterAll(() => rmSync(tmp, { recursive: true, force: true })); + +const npm: RepoProfile = { language: "node", packageManager: "npm", testCommand: "npm test", nodeVersion: "22", needsInstall: false }; +const needsInstall: RepoProfile = { ...npm, needsInstall: true }; + +// A tiny repo whose `npm test` runs the given script. +function repoWithTest(script: string, pkg: object = {}): string { + const dir = mkdtempSync(join(tmp, "repo-")); + writeFileSync(join(dir, "package.json"), JSON.stringify({ name: "t", private: true, scripts: { test: script }, ...pkg })); + return dir; +} + +// A dependency that needs no registry, so the install tests work offline. +function repoWithLocalDep(script: string, pkg: object = {}): string { + const dir = repoWithTest(script, { dependencies: { local: "file:./local" }, ...pkg }); + mkdirSync(join(dir, "local")); + writeFileSync(join(dir, "local", "package.json"), JSON.stringify({ name: "local", version: "1.0.0", main: "index.js" })); + writeFileSync(join(dir, "local", "index.js"), "module.exports = 1;\n"); + // A copy, not a link: Node on Windows can't delete a symlink a Linux container made (EPERM), and afterAll has to. + writeFileSync(join(dir, ".npmrc"), "install-links=true\n"); + return dir; +} + +const run = (script: string, extra = {}) => + runInSandbox({ sessionId: randomUUID(), worktree: repoWithTest(script), profile: npm, runsRoot: join(tmp, "runs"), ...extra }); + +const leftOver = async (sessionId: string) => + (await execa("docker", ["ps", "-a", "--filter", `name=ch-${sessionId.slice(0, 8)}`, "--format", "{{.Names}}"])).stdout.trim(); + +describe.skipIf(!hasDocker)("runInSandbox (needs Docker)", { timeout: 60_000 }, () => { + it("passes when the tests pass", async () => { + const r = await run(`node -e "process.exit(0)"`); + expect(r).toMatchObject({ passed: true, exitCode: 0, timedOut: false, oomKilled: false }); + expect(r.error).toBeUndefined(); + }); + + it("fails, as a result, when the tests fail", async () => { + const r = await run(`node -e "console.log('boom'); process.exit(3)"`); + expect(r).toMatchObject({ passed: false, exitCode: 3 }); + expect(r.error).toBeUndefined(); + expect(r.output).toContain("boom"); + }); + + it("kills a test that never ends and leaves no container behind", async () => { + const sessionId = randomUUID(); + const r = await runInSandbox({ sessionId, worktree: repoWithTest(`node -e "setInterval(() => {}, 1000)"`), profile: npm, runsRoot: join(tmp, "runs"), timeoutMs: 3_000 }); + expect(r).toMatchObject({ passed: false, timedOut: true, exitCode: 137 }); + expect(r.durationMs).toBeLessThan(15_000); + expect(await leftOver(sessionId)).toBe(""); + }); + + it("still returns when the limit passes before the container exists", async () => { + const sessionId = randomUUID(); + const r = await runInSandbox({ sessionId, worktree: repoWithTest(`node -e "setInterval(() => {}, 1000)"`), profile: npm, runsRoot: join(tmp, "runs"), timeoutMs: 1 }); + expect(r).toMatchObject({ passed: false, timedOut: true }); + expect(await leftOver(sessionId)).toBe(""); + }); + + it("reports a test that runs out of memory", async () => { + const r = await run(`node -e "const a = []; for (;;) a.push(Buffer.alloc(1e6, 1))"`, { limits: { memoryMb: 64, cpus: 1, pids: 256 } }); + expect(r).toMatchObject({ passed: false, oomKilled: true, timedOut: false }); + }); + + it("has no network", async () => { + const r = await run(`node -e "fetch('https://example.com').then(() => process.exit(0), () => process.exit(7))"`); + expect(r).toMatchObject({ passed: false, exitCode: 7 }); + }); + + it("can't change the repo", async () => { + const r = await run(`touch /work/written-by-test`); + expect(r.passed).toBe(false); + expect(r.output).toMatch(/Read-only file system/); + }); + + it("can only write to its scratch folders, and can run programs from /tmp", async () => { + const r = await run(`touch /tmp/a && touch ~/a && cp /bin/true /tmp/t && /tmp/t && touch /opt/a`); + expect(r.passed).toBe(false); + expect(r.output).toMatch(/\/opt\/a.*Read-only file system/); + }); + + it("keeps only the last 64 KB of a huge output", async () => { + const r = await run(`node -e "for (let i = 0; i < 3000; i++) console.log('x'.repeat(1000)); console.log('THE END')"`); + expect(r.outputTruncated).toBe(true); + expect(r.output.length).toBeLessThanOrEqual(64 * 1024); + expect(r.output.trim().endsWith("THE END")).toBe(true); + }); + + it("runs in the timezone it is given", async () => { + const check = `node -e "process.exit(Intl.DateTimeFormat().resolvedOptions().timeZone === 'Asia/Karachi' ? 0 : 1)"`; + expect((await run(check, { timeZone: "Asia/Karachi" })).passed).toBe(true); + expect((await run(check, { timeZone: "UTC" })).passed).toBe(false); + }); + + it("saves the result as test-1.json, then test-2.json", async () => { + const sessionId = randomUUID(); + const worktree = repoWithTest(`node -e "process.exit(0)"`); + const runsRoot = join(tmp, "runs-saved"); + await runInSandbox({ sessionId, worktree, profile: npm, runsRoot }); + const second = await runInSandbox({ sessionId, worktree, profile: npm, runsRoot }); + expect(readdirSync(join(runsRoot, sessionId)).sort()).toEqual(["test-1.json", "test-2.json"]); + expect(JSON.parse(readFileSync(second.resultFile!, "utf8"))).toMatchObject({ passed: true }); + }); + + it("carries on numbering after a restart, so it never overwrites a result", async () => { + const sessionId = randomUUID(); + const runsRoot = join(tmp, "runs-restart"); + mkdirSync(join(runsRoot, sessionId), { recursive: true }); + writeFileSync(join(runsRoot, sessionId, "test-1.json"), "{}"); + const r = await runInSandbox({ sessionId, worktree: repoWithTest(`node -e "process.exit(0)"`), profile: npm, runsRoot }); + expect(r.resultFile).toMatch(/test-2\.json$/); + expect(readFileSync(join(runsRoot, sessionId, "test-1.json"), "utf8")).toBe("{}"); + }); + + it("never reads the result of a container left behind by a crash", async () => { + const sessionId = randomUUID(); + const stale = `ch-${sessionId.slice(0, 8)}-1`; + await execa("docker", ["run", "--name", stale, "node:22-slim", "sh", "-c", "exit 0"]); + const r = await runInSandbox({ sessionId, worktree: repoWithTest(`node -e "process.exit(3)"`), profile: npm, runsRoot: join(tmp, "runs") }); + expect(r).toMatchObject({ passed: false, exitCode: 3 }); + expect(await leftOver(sessionId)).toBe(""); + }); + + it("says so when the container can't start, instead of blaming the tests", async () => { + const r = await runInSandbox({ sessionId: randomUUID(), worktree: repoWithTest(`node -e "0"`), profile: { ...npm, nodeVersion: "99.99.99" }, runsRoot: join(tmp, "runs") }); + expect(r).toMatchObject({ passed: false, exitCode: null, error: "sandbox.failed" }); + expect(r.output).toMatch(/99\.99\.99/); + }); + + it("refuses a worktree that doesn't exist, and doesn't create it", async () => { + const worktree = join(tmp, "not-there"); + const r = await runInSandbox({ sessionId: randomUUID(), worktree, profile: npm, runsRoot: join(tmp, "runs") }); + expect(r).toMatchObject({ passed: false, error: "sandbox.failed" }); + expect(existsSync(worktree)).toBe(false); + }); + + it("refuses a session id that isn't a plain name", async () => { + const r = await runInSandbox({ sessionId: "../../x", worktree: tmp, profile: npm, runsRoot: join(tmp, "runs") }); + expect(r).toMatchObject({ passed: false, error: "sandbox.failed" }); + }); + + it("refuses repos it can't handle yet", async () => { + const r = await runInSandbox({ sessionId: randomUUID(), worktree: tmp, profile: { ...npm, packageManager: "pnpm" }, runsRoot: join(tmp, "runs") }); + expect(r.error).toBe("sandbox.unsupported"); + }); + + it("installs first when the repo needs it, then tests", async () => { + const worktree = repoWithLocalDep(`node -e "process.exit(require('local') === 1 ? 0 : 1)"`); + const r = await runInSandbox({ sessionId: randomUUID(), worktree, profile: needsInstall, runsRoot: join(tmp, "runs") }); + expect(r).toMatchObject({ passed: true, exitCode: 0 }); + expect(existsSync(join(worktree, "node_modules"))).toBe(true); + expect(existsSync(join(worktree, "package-lock.json"))).toBe(false); // it would land in the diff + }); + + it("doesn't run the repo's install scripts", async () => { + const worktree = repoWithLocalDep(`node -e "0"`); + const pkg = JSON.parse(readFileSync(join(worktree, "package.json"), "utf8")); + pkg.scripts.postinstall = `node -e "require('fs').writeFileSync('planted.txt', 'x')"`; + writeFileSync(join(worktree, "package.json"), JSON.stringify(pkg)); + const r = await runInSandbox({ sessionId: randomUUID(), worktree, profile: needsInstall, runsRoot: join(tmp, "runs") }); + expect(r.passed).toBe(true); + expect(existsSync(join(worktree, "planted.txt"))).toBe(false); + }); + + it("reports a failed install, and doesn't run the tests", async () => { + const worktree = repoWithLocalDep(`node -e "console.log('TESTS RAN')"`); + writeFileSync(join(worktree, "package-lock.json"), "{ not json"); + const r = await runInSandbox({ sessionId: randomUUID(), worktree, profile: needsInstall, runsRoot: join(tmp, "runs") }); + expect(r).toMatchObject({ passed: false, error: "install.failed", command: "npm ci" }); + expect(r.output).not.toContain("TESTS RAN"); + expect(r.resultFile).toBeDefined(); + }); + + it("refuses to install when node_modules isn't git-ignored", async () => { + const worktree = repoWithLocalDep(`node -e "0"`); + execFileSync("git", ["init", "-q"], { cwd: worktree }); + const r = await runInSandbox({ sessionId: randomUUID(), worktree, profile: needsInstall, runsRoot: join(tmp, "runs") }); + expect(r).toMatchObject({ passed: false, error: "sandbox.unsupported" }); + expect(existsSync(join(worktree, "node_modules"))).toBe(false); + }); +}); + +describe("runInSandbox without Docker", () => { + it("says the box is unavailable, quickly", async () => { + const started = Date.now(); + const r = await runInSandbox({ sessionId: randomUUID(), worktree: tmp, profile: npm, runsRoot: join(tmp, "runs"), docker: "docker-does-not-exist" }); + expect(r).toMatchObject({ passed: false, error: "sandbox.unavailable" }); + expect(Date.now() - started).toBeLessThan(6_000); + }); +}); diff --git a/runner/test/session-store.test.ts b/runner/test/session-store.test.ts new file mode 100644 index 0000000..271e40e --- /dev/null +++ b/runner/test/session-store.test.ts @@ -0,0 +1,80 @@ +import { randomUUID } from "node:crypto"; +import { mkdtempSync, readdirSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { StartOptions } from "@codehands/contracts"; +import { afterAll, describe, expect, it } from "vitest"; +import { SessionStore } from "../src/index.js"; + +const tmp = mkdtempSync(join(tmpdir(), "codehands-store-")); +afterAll(() => rmSync(tmp, { recursive: true, force: true })); + +const newRoot = () => mkdtempSync(join(tmp, "root-")); +const options = (sessionId: string = randomUUID()): StartOptions => + ({ sessionId, repoPath: "/repo", prompt: "Fix it", agent: "claude-code" }); + +describe("SessionStore on disk", () => { + it("a new store finds what the old one saved", () => { + const root = newRoot(); + const o = options(); + const old = new SessionStore(root); + old.create(o); + old.transition(o.sessionId, "running"); + + const store = new SessionStore(root); + expect(store.get(o.sessionId)).toMatchObject({ options: o, status: "running", lastSeq: 0 }); + expect(store.ids()).toEqual([o.sessionId]); + }); + + it("never saves the pid or stopRequested", () => { + const root = newRoot(); + const o = options(); + const store = new SessionStore(root); + const record = store.create(o); + record.pid = 1234; + record.stopRequested = true; + store.save(record); + + const saved = JSON.parse(readFileSync(join(root, `${o.sessionId}.json`), "utf8")); + expect(saved).not.toHaveProperty("pid"); + expect(saved).not.toHaveProperty("stopRequested"); + expect(record.pid).toBe(1234); // the live record keeps them + }); + + it("leaves no temp file behind", () => { + const root = newRoot(); + const o = options(); + new SessionStore(root).create(o); + expect(readdirSync(root)).toEqual([`${o.sessionId}.json`]); + }); + + it("refuses a second session with an id already on disk", () => { + const root = newRoot(); + const o = options(); + new SessionStore(root).create(o); + expect(() => new SessionStore(root).create(o)).toThrow(/already exists/); + }); + + it("still checks moves after loading from disk", () => { + const root = newRoot(); + const o = options(); + new SessionStore(root).create(o); + expect(() => new SessionStore(root).transition(o.sessionId, "approved")).toThrow(/illegal move/); + }); + + it("treats an id that isn't a plain name as unknown, and won't write it", () => { + const store = new SessionStore(newRoot()); + expect(() => store.get("../../secrets")).toThrow(/unknown session/); + expect(() => store.create(options("../../secrets"))).toThrow(/bad session id/); + }); +}); + +describe("SessionStore in memory", () => { + it("writes nothing and forgets on restart", () => { + const o = options(); + const store = new SessionStore(); + store.create(o); + expect(store.ids()).toEqual([o.sessionId]); + expect(() => new SessionStore().get(o.sessionId)).toThrow(/unknown session/); + }); +});