feat(swarm): close conformance gaps and enforce spec review loop

Author: reoring

docs/spec/swarm.md

@@ -0,0 +1,189 @@
+# Swarm v1 Conformance Specification (Gap Closure)
+
+Status: implemented
+Last updated: 2026-03-02
+
+This document defines the conformance requirements needed for the current Swarm
+implementation to fully align with `docs/features/swarm.md`.
+
+It records review findings as normative requirements.
+Items already implemented in code are listed as baseline and are not redefined
+here unless clarification is needed.
+
+## 0. Scope
+
+This spec covers the remaining conformance work for:
+
+- `roboppi swarm` CLI behavior
+- coordinator lifecycle and cleanup guarantees
+- housekeeping/crash-recovery coverage
+- path and output safety rules
+- test strategy alignment
+
+Out of scope for this document:
+
+- multi-host swarm support
+- nested swarms
+- MCP transport delivery (`roboppi swarm mcp`) in v1
+
+## 1. Source Of Truth
+
+Primary design:
+
+- `docs/features/swarm.md`
+
+This document is a conformance addendum and MUST be interpreted together with
+that design document.
+
+## 2. Baseline (Already Implemented)
+
+The following are treated as implemented baseline:
+
+- File-backed mailbox/task store with atomic rename-based transitions.
+- Core CLI surface:
+  - `init`
+  - `members list`
+  - `message send|broadcast|recv|ack`
+  - `tasks add|list|claim|complete`
+  - `housekeep`
+- Metadata-only `_events.jsonl` emission for mailbox/tasks.
+- Capability wiring of `MAILBOX` / `TASKS` in workflow/daemon/type layers.
+- Workflow top-level `swarm:` integration and TUI swarm tab visibility.
+
+This spec focuses only on residual gaps.
+
+## 3. Normative Gap-Closure Requirements
+
+### 3.1 `message recv --wait-ms` MUST be implemented
+
+`roboppi swarm message recv` MUST support:
+
+- `--wait-ms <M>` where `M` is an integer `>= 0`.
+
+Behavior:
+
+- If messages are immediately available, return immediately.
+- Otherwise, poll until either:
+  - at least one message becomes available, or
+  - elapsed time reaches `wait-ms`.
+- On timeout, return success with an empty `messages` array.
+- `--wait-ms 0` MUST be equivalent to non-blocking current behavior.
+
+Polling interval MAY use a default constant; it MUST be deterministic and
+bounded.
+
+### 3.2 Coordinator shutdown MUST verify completion and apply cleanup policy
+
+Coordinator shutdown MUST satisfy all of:
+
+1. Send `shutdown_request` to all running teammates.
+2. Wait for teammate completion and/or explicit `shutdown_ack` evidence within a
+   bounded timeout.
+3. Force-abort only remaining non-settled teammates after timeout.
+4. Apply cleanup policy from `team.json.cleanup_policy`:
+   - if `retain_mailbox=false`, mailbox artifacts are removed at shutdown
+   - if `retain_tasks=false`, tasks artifacts are removed at shutdown
+5. Emit a final metadata-only cleanup event documenting completion and what was
+   retained/removed.
+
+Cleanup and event emission MUST be best-effort and MUST NOT leave partial
+deletions that violate store consistency guarantees.
+
+### 3.3 Task orphan recovery MUST include stale `in_progress` handling
+
+Housekeeping MUST handle stale tasks in `tasks/in_progress/`:
+
+- detect stale tasks via TTL
+- move stale tasks to either:
+  - `pending/` (requeue), or
+  - `orphaned/` (if introduced)
+- update task metadata atomically (`status`, timestamps, ownership fields)
+- emit metadata-only event for audit
+
+The operation MUST be idempotent and race-safe under concurrent housekeeping
+runs.
+
+### 3.4 CLI output contract MUST be JSON-safe for all failures
+
+For every `roboppi swarm` subcommand failure, including argument/usage errors:
+
+- stdout MUST contain exactly one JSON object:
+  - `{ "ok": false, "error": "<message>" }`
+- exit code MUST be non-zero.
+
+For help output (`-h` / `--help`):
+
+- help text MUST go to stderr
+- stdout MUST be empty
+- exit code MUST be `0`.
+
+### 3.5 Path safety MUST be explicit for tool-facing path inputs
+
+All tool-facing path inputs MUST be constrained to Swarm-safe boundaries.
+
+Requirements:
+
+- All filesystem mutations for swarm data MUST remain inside
+  `<context_dir>/_swarm`.
+- User-supplied path values (including artifact-like values) MUST reject:
+  - absolute paths (unless explicitly allowed by command contract)
+  - `..` traversal
+  - symlink escape outside allowed roots after resolution.
+
+Validation errors MUST follow the JSON error contract in section 3.4.
+
+### 3.6 Integration testing MUST match section 11 intent
+
+Test suite MUST include at least one supervised integration/AT scenario that:
+
+- runs with supervised mode enabled
+- uses `roboppi swarm ...` commands (not only direct store API calls)
+- includes two `CUSTOM` workers exchanging messages/tasks
+- verifies emitted `swarm_*` exec events and/or `_swarm/*/_events.jsonl`
+  artifacts
+
+## 4. Capability And Identity Clarifications
+
+### 4.1 `MAILBOX` and `TASKS` separation
+
+`MAILBOX` and `TASKS` MUST remain independently gateable from
+`RUN_COMMANDS`.
+
+It is acceptable in v1 that both map to the same restricted `roboppi swarm:*`
+surface, but adapters SHOULD evolve toward least-privilege separation by
+subcommand group.
+
+### 4.2 Identity anti-spoofing
+
+Current membership validation is mandatory baseline.
+
+Optional per-member token-file verification is a future hardening and is NOT
+required for v1 conformance.
+
+## 5. Conformance Checklist
+
+A release is considered conformant to this spec only when all items below are
+true:
+
+- [x] 3.1 implemented and covered by unit/AT tests
+  - `--wait-ms 0`, timeout, and arrival-during-wait AT coverage in `tests/at/swarm-cli.test.ts`
+- [x] 3.2 implemented and covered by lifecycle tests
+  - Shutdown bounded end-to-end: `teammateShutdownWaitMs` deadline + 5s settle cap
+  - All unsettled teammates force-aborted after deadline (regardless of ack evidence)
+  - Cleanup trash kept within `_swarm/.trash/` (not `os.tmpdir()`)
+  - `tests/unit/swarm/coordinator.test.ts` (bounded timeout, abort-only-unsettled, cleanup policy, cleanup event)
+- [x] 3.3 implemented and covered by housekeeping tests
+  - `tests/unit/swarm/housekeeping-tasks.test.ts` (stale detection, metadata rewrite, idempotency, concurrent safety)
+- [x] 3.4 implemented and covered by CLI error/help tests
+  - Unknown subcommand, missing flag, semantic failure AT coverage in `tests/at/swarm-cli.test.ts`
+- [x] 3.5 implemented and covered by path-safety tests
+  - `assertSwarmRootSafe()` + `validateMemberIdPath()` applied to all mutating commands including `init`, `recv`, `ack`, `housekeep`
+  - `init` validates `--lead` and all `--json-stdin` member IDs via `validateMemberIdPath()`
+  - Symlink escape + traversal rejection AT in `tests/at/swarm-cli.test.ts` (including `init` subcommand)
+- [x] 3.6 implemented and verified in CI test targets
+  - `tests/at/swarm-supervised-custom.test.ts` uses CLI subprocess calls (not direct store API)
+
+## 6. Compatibility
+
+This specification preserves compatibility with the `version: "1"` swarm
+on-disk objects and with the design direction in `docs/features/swarm.md`.

examples/swarm-impl-loop.yaml

@@ -45,6 +45,7 @@ steps:
       Read (by exact file path):
       - .roboppi-loop/request.md
       - docs/features/swarm.md
+      - docs/spec/swarm.md
       - docs/design.md
       - src/types/worker-task.ts
       - src/worker/adapters/claude-code-adapter.ts
@@ -60,6 +61,8 @@ steps:
       - Group by milestones (MVP first; then follow-ups)
       - Include file paths and symbols when possible
       - Include explicit test tasks (at least: make typecheck, make test-all)
+      - Add explicit conformance tasks for `docs/spec/swarm.md` section 3.1-3.6 and section 5 checklist.
+      - If any spec item is partial/missing, add at least one concrete task per gap (code + test).
       - Keep scope aligned to .roboppi-loop/request.md
     capabilities: [READ, EDIT]
     timeout: "25m"
@@ -94,13 +97,15 @@ steps:
       Read:
       - .roboppi-loop/request.md
       - docs/features/swarm.md
+      - docs/spec/swarm.md
       - .roboppi-loop/todo.md
       - .roboppi-loop/review.md (if present)
       - .roboppi-loop/fix.md (if present)
 
       Task:
       - If .roboppi-loop/fix.md exists and is non-empty, apply ONLY those fixes (minimal, targeted changes).
       - Otherwise, implement the TODO checklist items (focus on the MVP first).
+      - Give priority to unresolved MUST items in `docs/spec/swarm.md` section 3.
       - Update the TODO checkboxes as you complete items.
       - Keep changes focused; avoid drive-by refactors.
 
@@ -133,7 +138,7 @@ steps:
         Task:
         1) Run tests:
            - make test-all
-        2) Review the changes against the request + TODO + design doc.
+        2) Review the changes against the request + TODO + `docs/features/swarm.md` + `docs/spec/swarm.md`.
         3) Write/update:
            - .roboppi-loop/review.md
            - .roboppi-loop/review.verdict (single-line JSON; see Output rules)
@@ -154,6 +159,7 @@ steps:
         You must read at least:
         - .roboppi-loop/request.md
         - docs/features/swarm.md
+        - docs/spec/swarm.md
         - .roboppi-loop/todo.md
         - .roboppi-loop/review.base_ref
         - .roboppi-loop/review.diff
@@ -162,12 +168,19 @@ steps:
         - .roboppi-loop/review.untracked.diff
 
         Format for .roboppi-loop/review.md:
-        - Sections: Summary, Issues, Verification
+        - Sections: Summary, Conformance, Issues, Verification
+        - In Conformance, include `docs/spec/swarm.md` section 3.1-3.6 as:
+          - status: implemented | partial | missing
+          - evidence: file paths and symbols
+          - required fixes (for partial/missing)
 
         Output rules:
         - Write .roboppi-loop/review.verdict as a single-line JSON object:
           - complete => {"decision":"complete","check_id":"$ROBOPPI_COMPLETION_CHECK_ID"}
           - incomplete => {"decision":"incomplete","check_id":"$ROBOPPI_COMPLETION_CHECK_ID"}
+        - Verdict policy:
+          - If any MUST requirement in `docs/spec/swarm.md` section 3 is partial or missing, verdict MUST be `incomplete`.
+          - If all section 3 MUST requirements are implemented and tests pass, verdict MAY be `complete`.
         - When FAIL, also include:
           - "reasons": ["..."]
           - "fingerprints": ["..."]

scripts/self-dev/swarm/bootstrap.sh

@@ -25,9 +25,12 @@ rm -f \
 
 if [ ! -f "${LOOP_DIR}/request.md" ]; then
   cat > "${LOOP_DIR}/request.md" <<'EOF'
-# Request: Implement Swarm (Agent Teams) for Roboppi
+# Request: Swarm Gap Closure (Design + Conformance)
 
-Implement the Swarm feature described in `docs/features/swarm.md`.
+Implement/close remaining Swarm gaps so behavior aligns with both:
+
+- `docs/features/swarm.md`
+- `docs/spec/swarm.md`
 
 Swarm is a local, file-backed agent team system:
 
@@ -38,56 +41,56 @@ Swarm is a local, file-backed agent team system:
 This is inspired by Claude Code agent teams:
 - https://code.claude.com/docs/en/agent-teams
 
-## Scope (v1 / MVP)
+## Scope (v1 / MVP + Conformance)
 
 1) Swarm file store (maildir-like)
-- Create a file-backed mailbox + task store under `<context_dir>/_swarm/` as
-  specified in `docs/features/swarm.md`.
-- Use atomic operations (`tmp` + `rename`) to avoid corruption under concurrency.
+- Keep file-backed mailbox + task store under `<context_dir>/_swarm/`.
+- Keep atomic operations (`tmp` + `rename`) for concurrency safety.
 
 2) CLI tools (model-facing)
-- Add a `roboppi swarm` CLI group with JSON outputs:
+- Keep/extend `roboppi swarm` CLI group:
   - `swarm init`
   - `swarm members list`
   - `swarm message send|broadcast|recv|ack`
   - `swarm tasks add|list|claim|complete`
-  - `swarm housekeep` (requeue stale processing messages)
-- All commands MUST print machine-readable JSON to stdout on success.
-- Errors should be actionable and MUST NOT print non-JSON to stdout.
-
-3) Capability gating for Claude Code
-- Add a new `WorkerCapability` (name: `MAILBOX`) and map it so Claude Code gets
-  a restricted tool:
-  - allow `Bash(roboppi swarm:*)` (or equivalent narrow allowlist)
-  - do NOT grant full `Bash` just for mailbox access
-- Update documentation and tests accordingly.
-
-4) Tests
-- Add unit tests covering:
-  - message send -> inbox/new
-  - recv claim -> processing
-  - ack -> cur
-  - broadcast -> N recipients
-  - task add/claim/complete transitions
-  - housekeeping (stale processing requeue)
-- `make test-all` must pass.
+  - `swarm housekeep`
+- Implement `swarm message recv --wait-ms`.
+- Enforce JSON-safe output contract on all failures (including arg/usage errors).
+
+3) Lifecycle and housekeeping closure
+- Coordinator shutdown must include deterministic completion checks and cleanup
+  policy application.
+- Housekeeping must cover stale task recovery for `tasks/in_progress/` as
+  defined by spec.
+
+4) Capability/identity/safety closure
+- Keep `MAILBOX` / `TASKS` capability support and gating.
+- Keep metadata-only event policy.
+- Ensure tool-facing path handling remains traversal-safe.
+
+5) Tests and review quality gate
+- Add/extend tests for all spec MUST items.
+- `make test-all` should pass in normal local environments.
+- Completion review MUST include a conformance audit against
+  `docs/spec/swarm.md` section 3.1-3.6.
 
 ## Constraints
 
 - Keep changes focused; avoid repo-wide refactors.
 - Prefer ASCII in new files.
-- Validate all paths stay within `<context_dir>/_swarm/` (no traversal).
+- Validate paths for swarm mutations stay within `<context_dir>/_swarm/`.
 - Bound message/task sizes (e.g. 64KB message body; 256KB task file).
 
 ## Acceptance Criteria
 
-- [ ] `docs/features/swarm.md` remains accurate (update if implementation diverges)
-- [ ] `roboppi swarm init` creates the expected directory layout + config files
-- [ ] `roboppi swarm message send/recv/ack` works end-to-end (file semantics)
-- [ ] broadcast delivers to all members
-- [ ] `roboppi swarm tasks add/claim/complete` works and is race-safe
-- [ ] housekeeping requeues stale `processing/` messages
-- [ ] Claude Code capability `MAILBOX` enables only `roboppi swarm:*` bash usage
+- [ ] `docs/features/swarm.md` and `docs/spec/swarm.md` are both accurate
+- [ ] `docs/spec/swarm.md` section 3.1 (`recv --wait-ms`) is implemented + tested
+- [ ] `docs/spec/swarm.md` section 3.2 (shutdown/cleanup policy + final event) is implemented + tested
+- [ ] `docs/spec/swarm.md` section 3.3 (stale `in_progress` task recovery) is implemented + tested
+- [ ] `docs/spec/swarm.md` section 3.4 (JSON-safe CLI failure output) is implemented + tested
+- [ ] `docs/spec/swarm.md` section 3.5 (path safety for tool-facing inputs) is implemented + tested
+- [ ] `docs/spec/swarm.md` section 3.6 (supervised integration using `roboppi swarm`) is implemented + tested
+- [ ] `make typecheck` passes
 - [ ] `make test-all` passes
 EOF
 fi