Skip to content

running.md

Latest commit

 

History

History
138 lines (115 loc) · 6.5 KB

running.md

File metadata and controls

138 lines (115 loc) · 6.5 KB
Raw

Operations guide

Running CodeyBox in something resembling production. This is intentionally short — the framework is small.

First-time host setup

  1. Linux host with KVM available (/dev/kvm).
  2. .NET 10 SDK if building from source; otherwise just the runtime.
  3. A sandbox provider — see sandboxes.md. Quick start:
    • Recommended for persistent/high-throughput headless operation: install Incus 6.3+ and configure an existing ZFS or Btrfs pool, then set CodeyBox.SandboxProvider=incus. Its baseline clones are copy-on-write, avoiding a full multi-gigabyte image copy for every pass. Confirm the service identity, storage pool, and host compatibility described in sandbox-providers.md before installing.
    • Simplest setup or graphical workloads: install Multipass with sudo snap install multipass, then run sudo scripts/setup-host-networks.sh to create the network profile bridges + nftables rules. Set CodeyBox.SandboxProvider=multipass. See host-firewall.md for the network setup.
    • Single-package fallback (shared-kernel, dev-friendly): sudo apt install bubblewrap, set CodeyBox.SandboxProvider=bubblewrap. No host firewall enforcement.
  4. git on the host (used by IGitHost for cloning seeds and pushing upstream).
  5. Service user with:
    • Write access to GitRootDirectory and StateDatabasePath parents.
    • Access to /dev/kvm (Multipass or Incus).
    • Membership in incus-admin for Incus, or permission to run multipass.
    • No SSH or other host privileges beyond the above.

The repository ships gitignored helper scripts under local/ for operator-side verification (local/setup-test-networks.sh, local/verify-host-firewall.sh, local/verify-internet-only.sh, local/teardown-test-networks.sh). Use these to confirm host-side egress enforcement is working before running real prompts.

Running

export CODEYBOX_CLAUDE_API_KEY=...
export CODEYBOX_GITHUB_TOKEN=...   # if upstream is github
dotnet run --project src/CodeyBox.Api

The service binds to the standard ASP.NET Core ports (http://localhost:5000 by default). Put a reverse proxy in front of it for TLS and auth.

Logs and observability

  • All state transitions log at Information. Pipe to your usual log sink.
  • SQLite at StateDatabasePath is the durable record. Backing this up is trivial (sqlite3 state.db ".backup state.db.bak").
  • The host bare repos under GitRootDirectory are large but not load- bearing — they can be regenerated by re-running work items if the SQLite state survives.

Restarting the orchestrator safely

Items in flight during a shutdown are not cancelled. They stay in their mid-flight state and the reaper resets each one to a safe restart point on the next startup, incrementing recoveryAttempts. After CodeyBox:DeadWorker:MaxRecoveryAttempts recoveries (default 10) without reaching a terminal state, an item lands in AbandonedAfterRecoveryAttempts and waits for POST /workitems/{id}/retry.

Per-state resume points, the reaper's fencing rules, and the caller-facing downtime window are in recovery.md.

The startup log tells you what happened:

INF Recovering abc123 from non-terminal state Working → Queued
    (recovery attempt 1, presumed lost on prior shutdown)

Every phase is safe to re-run. Work and rework stack new commits on the existing work branch, or start a fresh one if the branch is gone; auditors are stateless against the working tree; a merge re-run against an already-merged base passes post-merge verification unchanged; and the upstream push is idempotent.

Capacity

MaxConcurrentWorkers controls how many work items can be in the pipeline at once. A single item can still create additional audit, merge, smoke, and verification sandboxes, so host sizing should use CodeyBox:WorkerPool:MaxConcurrentSandboxes as the VM ceiling. With Multipass on KVM, plan for ~1 GB host RAM per admitted sandbox plus the agent's memory needs.

Failure modes you'll actually see

  • Agent produced no changes. Work phase fails with "Agent produced no changes to commit". Usually a too-narrow prompt.
  • Audit didn't converge. After MaxIterations of rework the audit phase still has Error-severity findings; the work item flips to AuditFailed. Inspect the auditor findings on the record and decide whether to relax the audit policy, fix the prompt, or merge by hand.
  • Tests cannot run in the sandbox. Audit scoring only counts tests that can run in the current audit environment; tests that cannot launch because required local infrastructure is absent are not part of the scoring or auditing criteria. csharp:test-pass filters near-zero-duration failures with no stack trace while still reporting real assertion and runtime failures.
  • Merge phase verification failed. The agent ran but the orchestrator's post-merge check (expected SHA / clean working tree) failed. The work item flips to Failed; nothing was pushed to the host bare repo's base branch.
  • Upstream push rejected. Often a branch protection rule (force-push blocked, required reviews). The local merge is still in place; either loosen the protection for the push user or move the upstream push behind a manual gate.
  • VM can't reach allowed host. Check nft list table inet codeybox for the resolved IPs and re-run setup-host-networks.sh if a CDN has rotated. See host-firewall.md troubleshooting.
  • NuGet restore fails with "unauthorized access" before any project builds. Failed to read NuGet.Config due to unauthorized access means the sandbox image has ~/.nuget owned by another uid, so the build user cannot create the settings directory NuGet writes before it reads any repository config. CodeyBox's build gates self-heal around it; a harness invoking dotnet directly does not. Symptoms, the self-heal paths, and the provisioning fix are in ../development/build-environment.md.

Backups

Two things to back up:

  1. The SQLite state DB.
  2. Any work item record you care about that hasn't completed (in-flight items). The bare repo can be regenerated by recloning from upstream; the work item record cannot.

Once a work item is Done and the upstream push has been mirrored elsewhere (your normal repo backups), CodeyBox has no further state of record. The bare repo can be deleted via IGitHost.DisposeRepositoryAsync.