Rethink orchestrate: investigate replacing the Workflow fleet with an implement-loop architecture (Sandcastle or a minimal runner)

[TBarregren]

Problem

orchestrate works but is unsatisfactory: runs take very long, spawn on the order of 100+ subagents, and consume tokens accordingly. It is also inherently Claude Code-only (Workflow tool, Task fleets, model/effort pinning). Before investing in porting it, we should question the architecture itself. This is an investigation ticket: the deliverable is a decision, not an implementation.

Hypothesis to test

An "implement-loop" architecture — a deterministic outer loop that feeds one issue at a time to a fresh-context agent session running a small implement protocol, with verification carried primarily by deterministic gates — achieves equal or better outcome quality at a small fraction of orchestrate's token and wall-clock cost, and is harness-independent as a bonus.

What we already know (context for the investigator)

• Matt Pocock's implement skill is 15 lines of markdown (source). Its entire content: implement the spec/ticket, use /tdd at pre-agreed seams, run typechecks and tests continuously and the full suite at the end, /code-review the result, commit. Its effectiveness comes from composition (tickets → TDD → gates → review) and from being run once per issue in a fresh context — not from orchestration machinery. The manual workflow it implies (/clear, /implement #N, repeat) is exactly what an outer loop automates: a headless invocation per issue (claude -p ..., codex exec ...) gives a fresh context for free. Note: implement sets disable-model-invocation: true (slash-command-only), but that does not hinder a runner — a slash command passed as the headless session's prompt is user-level invocation; the flag only stops the model from auto-invoking it mid-session.
• Most of orchestrate's agent count is verification fan-out. The lever is to let deterministic gates (tests, typecheck, audit.py, CI) carry verification, and spend agent review only on what gates cannot catch — plausibly one review pass per issue instead of an adversarial fleet. The gate-first principle should be a design constraint for v2, whatever engine wins.
• Sandcastle already exists and is essentially the "orchestration outside the AI" idea, built: a TypeScript library (sandcastle.run()) that runs coding agents in isolated sandboxes (Docker/Podman/Vercel providers) with agent providers for Claude Code, Codex, and Pi; branch strategies with merge-back; implement-then-review pipelines; session resume/fork for fan-out; structured output with schema validation; and shipped templates (simple-loop, parallel-planner) that loop over GitHub Issues. ~7k stars. Rule for this investigation: do not build a custom engine before Sandcastle has been evaluated and disqualified with reasons — maintaining a homegrown orchestrator engine is a known trap.
• The reusable assets in current orchestrate are deterministic and engine-independent: orchestrate.py's dependency-graph planning (queue ordering over ready-for-agent issues) and the quality-bar definitions. They survive any rewrite as the queue/planning step.
• The surrounding pipeline already exists: improve (or to-tickets) produces fully specified ready-for-agent issues; the runner loops an implement protocol over them; results integrate via PRs. Orchestrate v2 is only the middle piece.

Candidate architectures to evaluate

• A — Sandcastle as the engine: .sandcastle/ config in target repos or a shared harness repo; simple-loop/parallel-planner templates over GitHub Issues; Kntnt implement protocol (a small skill: TDD, gates, /code-review, commit) as the per-issue prompt; orchestrate.py for queue ordering.
• B — Minimal own runner: on the order of 100 lines of Python/bash: pick the next unblocked ready-for-agent issue (via orchestrate.py), create a worktree or container, invoke a headless agent CLI with the implement protocol, run the gates, open a PR, repeat; optionally N workers in parallel over disjoint issues. Evaluate honestly whether this stays ~100 lines once sandboxing, retries, and merge conflicts are handled — that is Sandcastle's whole reason to exist.
• C — Status quo: keep the Workflow engine, Claude-only, as the baseline to beat.

Non-negotiable design constraints for any winner

• Isolation: agents run in worktrees or containers, never directly in the working tree.
• Integration is PR-gated; the runner never pushes to the default branch.
• Gate-first verification; agent review is a bounded, per-issue pass, not a fleet.
• The per-issue review step is a protocol slot, not a specific product. Note a name collision: the /code-review that implement references is Matt Pocock's own portable skill (source — 89 lines of markdown spawning two general-purpose sub-agents for a Standards axis and a Spec axis; degradable to sequential execution), not Claude Code's built-in /code-review, which is harness-bound (bundled with Claude Code, typed findings surface, cloud ultra variant) and unavailable elsewhere. Workers on Claude Code may fill the slot with the built-in review; workers on other harnesses need a portable filler — Pocock's skill, or a Kntnt review procedure derived from coder/coding-standard. The recommended architecture must name its filler per supported harness.
• A hard token/cost budget per run, reported at the end.
• Billing mode is an explicit evaluation criterion. Per current docs (https://code.claude.com/docs/en/authentication.md, /costs.md, /agent-sdk/overview.md): headless claude -p with subscription OAuth is permitted and draws from the plan's separate monthly headless credit (Max 20x: $200/month) — but ANTHROPIC_API_KEY in the environment takes precedence and silently switches to per-token API billing, and the Claude Agent SDK does not allow subscription auth at all (API key required). Consequences to verify per candidate: whether Sandcastle's claudeCode() provider spawns the CLI (subscription possible if credentials are available inside the sandbox) or uses the SDK (API billing forced); whether a minimal own runner can keep workers on subscription auth within the headless credit; and what each architecture costs per benchmark issue under both billing modes. The ADR must state the recommended auth setup and its monthly ceiling.

Method

1. Pick a benchmark: 5–10 real ready-for-agent issues of mixed size in one repo.
2. Measure baseline (C): tokens, wall-clock, subagent count, and defects that escape to human review, on the benchmark.
3. Prototype A (and B only if A disqualifies): same benchmark, same metrics.
4. Compare and write an ADR in docs/adr/ with the recommendation, the numbers, and the migration path (including what happens to orchestrate.py, the Workflow engine, and the orchestrate skill's user surface).

Deliverables

• The ADR with a clear recommendation and the measured numbers.
• Follow-up implementation tickets for the chosen path (written to the same executable standard as this repo's other tickets).
• No implementation beyond the prototype in this ticket.

Not blocked by other tickets, but coordinate with #56 (which moves orchestrate's files and hides the skill from the standalone channel — both irrelevant to this investigation's outcome).

[TBarregren]

Kntnt/kntnt-skills#2 establishes the cross-collection dependency pattern (depend on upstream skills by name, promoted-only, availability preflight, recorded tested version). The recommended orchestrate v2 architecture should consume implement, tdd, and code-review from mattpocock/skills as named upstream dependencies per that pattern — not copy or reimplement them.