feat(sensor): long-running soak harness + steady-state spool compaction (Wave A)

Closes the last open Wave A item (Implementation_Status.md sensor row:
"long-lived production-host soak").

scripts/sensor-soak.sh runs the real aegis-node-sensor against a real
gateway on a real Linux /proc for a sustained period under continuous
collector workload (tagged processes, real TCP connections, real fds,
secret-shaped env vars), periodic signed-kill enforcement round-trips
against real host processes, and a mid-soak gateway outage — then
asserts what only a long run proves: sensor survival with zero panics,
bounded RSS and fd counts (steady-state baseline vs final), bounded
spool disk, full spool drain, all kills verified/executed/ACKed, and
unattended post-outage recovery. Samples land in a CSV + summary JSON
artifact.

.github/workflows/sensor-soak.yml wires it three ways: nightly 15-min
soak, operator-dispatched arbitrary duration, and a short PR smoke soak
when sensor/harness paths change (so this PR proves the harness itself).

Building the harness surfaced a real steady-state defect, fixed here
TDD-first: spool lane files were only ever compacted on the normal
lane's over-budget drop path, so on a healthy long-lived host (enqueue →
ship → ack keeping pending_bytes near zero) both append-only lane files
grew without bound until disk exhaustion. SpoolQueue::compact_if_reclaimable
now reclaims the acked prefix once it exceeds a threshold, called from
the sensor's ship tick; compaction failure degrades to append-only and
retries, never crashes the sensor.

Docs: Implementation_Status.md sensor row + Wave A ledger,
current-vs-roadmap.md, and the PRP task list updated in the same change;
remaining honesty gap is an extended-duration soak on a production-grade
deployment (a CI runner soak is still not production load).

Author: lavkushry

.claude/PRPs/tasks/task.md

@@ -30,8 +30,13 @@ Blocks any claim of runtime control over non-cooperative agents.
       sandbox, emit lifecycle events, report status; package Dockerfile/Helm.
 - [ ] **Gateway cage lease APIs on main** — claim / heartbeat / ownership-scoped
       status (if not merged with cage PR).
-- [ ] **Sensor collectors + host enforcement** — real process/fs/net/secret
+- [x] **Sensor collectors + host enforcement** — real process/fs/net/secret
       signals; execute signed pause/kill/quarantine against the host/workload.
+      Done on `main` (`process_collector`/`net`/`fs`/`secret` + `ProcessEnforcer`
+      + `tests/real_host_integration.rs`); soak harness landed
+      (`scripts/sensor-soak.sh` + `sensor-soak.yml` nightly/dispatch/PR-smoke,
+      plus steady-state spool compaction fix). Remaining: extended-duration
+      soak on a production-grade deployment.
 - [ ] **E2E unknown-agent path** — cage + egress deny + control action +
       receipt/incident in compose or Playwright/integration harness.
 - [ ] **Egress forced path for caged runs** — not opt-in only (netns / default

.github/workflows/sensor-soak.yml

@@ -0,0 +1,69 @@
+name: Sensor Soak
+
+# Long-running node-sensor soak (Implementation_Status.md sensor row:
+# "long-lived production-host soak"). scripts/sensor-soak.sh runs the real
+# sensor against a real gateway on the runner's real Linux /proc under
+# continuous collector workload, signed-kill round-trips, and a mid-soak
+# gateway outage, then asserts bounded RSS/fd/spool-disk, zero panics,
+# full spool drain, and unattended outage recovery.
+#
+# Three entry points:
+#  - nightly schedule: the standing 15-minute soak;
+#  - workflow_dispatch: operator-chosen duration (e.g. multi-hour);
+#  - pull_request (sensor/harness paths only): a short smoke soak so
+#    changes to the sensor or the harness prove themselves in the PR.
+
+on:
+  schedule:
+    - cron: "17 3 * * *"
+  workflow_dispatch:
+    inputs:
+      duration_secs:
+        description: "Soak duration in seconds"
+        required: false
+        default: "900"
+  pull_request:
+    paths:
+      - "bins/aegis-node-sensor/**"
+      - "scripts/sensor-soak.sh"
+      - ".github/workflows/sensor-soak.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  sensor-soak:
+    name: Node sensor soak (real host)
+    runs-on: ubuntu-latest
+    timeout-minutes: 350
+    steps:
+      - uses: actions/checkout@v7
+      - name: Install protoc
+        run: sudo apt-get update && sudo apt-get install -y --no-install-recommends protobuf-compiler
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: .
+          key: sensor-soak
+      - name: Pick soak duration
+        id: duration
+        run: |
+          if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+            echo "secs=${{ github.event.inputs.duration_secs }}" >> "$GITHUB_OUTPUT"
+          elif [[ "${{ github.event_name }}" == "pull_request" ]]; then
+            echo "secs=180" >> "$GITHUB_OUTPUT"
+          else
+            echo "secs=900" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Run sensor soak
+        env:
+          SOAK_DURATION_SECS: ${{ steps.duration.outputs.secs }}
+          RUST_LOG: info
+        run: bash scripts/sensor-soak.sh
+      - name: Upload soak artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: sensor-soak-${{ github.run_id }}
+          path: target/sensor-soak/
+          if-no-files-found: ignore

bins/aegis-node-sensor/src/main.rs

@@ -38,6 +38,12 @@ const SHIP_TICK_INTERVAL: Duration = Duration::from_secs(2);
 /// How often the sensor polls the gateway for commands addressed to it.
 const COMMAND_POLL_INTERVAL: Duration = Duration::from_secs(5);
 
+/// Acked-prefix size at which a spool lane's log file is compacted after a
+/// ship tick. Acking only advances a watermark — without periodic
+/// compaction the append-only lane file grows without bound on a
+/// long-lived host even while `pending_bytes` stays near zero.
+const SPOOL_COMPACT_THRESHOLD_BYTES: u64 = 262_144;
+
 /// How often `/proc` (Linux) is scanned for `AEGIS_RUN_ID` host agents.
 const PROCESS_COLLECT_INTERVAL: Duration = Duration::from_secs(2);
 
@@ -259,6 +265,14 @@ async fn main() -> ExitCode {
                         Ok(n) => tracing::debug!(lane = ?lane, shipped = n, "shipped events"),
                         Err(e) => tracing::warn!(lane = ?lane, error = %e, "ship tick failed"),
                     }
+                    match spool.compact_if_reclaimable(lane, SPOOL_COMPACT_THRESHOLD_BYTES) {
+                        Ok(true) => tracing::debug!(lane = ?lane, "compacted spool lane"),
+                        Ok(false) => {}
+                        // Compaction failure never crashes the sensor: the
+                        // lane keeps working append-only and reclamation is
+                        // retried on the next tick.
+                        Err(e) => tracing::warn!(lane = ?lane, error = %e, "spool compaction failed"),
+                    }
                 }
             }
             _ = command_poll_tick.tick() => {

bins/aegis-node-sensor/src/spool.rs

@@ -327,6 +327,25 @@ impl SpoolQueue {
         self.lane_state(lane).compact(&self.dir, lane)
     }
 
+    /// Steady-state disk reclamation: compact `lane` only once at least
+    /// `min_reclaimable_bytes` of fully-acked prefix would be recovered.
+    /// Without a periodic call to this, the append-only lane file grows
+    /// without bound on a long-lived host even though `pending_bytes`
+    /// stays near zero — acking advances the watermark but never shrinks
+    /// the file. Returns whether a compaction ran.
+    pub fn compact_if_reclaimable(
+        &self,
+        lane: Lane,
+        min_reclaimable_bytes: u64,
+    ) -> Result<bool, SpoolError> {
+        let mut state = self.lane_state(lane);
+        if state.ack_offset == 0 || state.ack_offset < min_reclaimable_bytes {
+            return Ok(false);
+        }
+        state.compact(&self.dir, lane)?;
+        Ok(true)
+    }
+
     /// Bytes not yet acked in `lane` — exposed for heartbeat queue-depth
     /// reporting (Phase 3.2's `queue_depth_critical`/`queue_depth_normal`).
     pub fn pending_bytes(&self, lane: Lane) -> Result<u64, SpoolError> {
@@ -484,6 +503,69 @@ mod tests {
         assert_eq!(record.payload, vec![1u8; record_size]);
     }
 
+    #[test]
+    fn steady_state_ship_ack_cycles_keep_disk_bounded_with_compaction() {
+        // The long-running healthy path: events are enqueued, shipped, and
+        // acked continuously, so pending_bytes stays near zero — but the
+        // append-only log file itself must not grow without bound across
+        // days of uptime. compact_if_reclaimable is the steady-state
+        // reclamation hook the main loop calls after ship ticks.
+        let dir = tempfile::tempdir().unwrap();
+        let queue = SpoolQueue::open(dir.path(), 10_000_000).unwrap();
+        let record_size = 100usize;
+        let frame_size = HEADER_LEN + record_size as u64;
+        let threshold = frame_size * 10;
+
+        let log_path = dir.path().join("normal.log");
+        for _ in 0..500 {
+            queue
+                .enqueue(Lane::Normal, &vec![7u8; record_size])
+                .unwrap();
+            let record = queue.read_next(Lane::Normal).unwrap().unwrap();
+            queue.ack(Lane::Normal, &record).unwrap();
+            queue
+                .compact_if_reclaimable(Lane::Normal, threshold)
+                .unwrap();
+
+            let len = std::fs::metadata(&log_path).unwrap().len();
+            assert!(
+                len <= threshold + frame_size,
+                "log file grew past the compaction bound: {len} > {}",
+                threshold + frame_size
+            );
+        }
+    }
+
+    #[test]
+    fn compact_if_reclaimable_below_threshold_is_a_noop() {
+        let dir = tempfile::tempdir().unwrap();
+        let queue = SpoolQueue::open(dir.path(), 1_000_000).unwrap();
+        queue.enqueue(Lane::Normal, b"acked").unwrap();
+        queue.enqueue(Lane::Normal, b"pending").unwrap();
+        let first = queue.read_next(Lane::Normal).unwrap().unwrap();
+        queue.ack(Lane::Normal, &first).unwrap();
+
+        // Reclaimable bytes (one small acked frame) are below the threshold:
+        // nothing is rewritten and the unacked record is untouched.
+        let compacted = queue
+            .compact_if_reclaimable(Lane::Normal, 1_048_576)
+            .unwrap();
+        assert!(!compacted);
+        let second = queue.read_next(Lane::Normal).unwrap().unwrap();
+        assert_eq!(second.payload, b"pending");
+
+        // At-or-above threshold, the acked prefix is reclaimed and the
+        // unacked record survives the rewrite.
+        let compacted = queue.compact_if_reclaimable(Lane::Normal, 1).unwrap();
+        assert!(compacted);
+        let second = queue.read_next(Lane::Normal).unwrap().unwrap();
+        assert_eq!(second.payload, b"pending");
+        let len = std::fs::metadata(dir.path().join("normal.log"))
+            .unwrap()
+            .len();
+        assert_eq!(len, HEADER_LEN + b"pending".len() as u64);
+    }
+
     #[test]
     fn empty_queue_read_next_returns_none() {
         let dir = tempfile::tempdir().unwrap();

docs/Implementation_Status.md

@@ -66,7 +66,7 @@ production-authoritative.
 | Control commands (signed kill/pause/quarantine) | Partial | store + protocol + gateway issue routes; sensor poll/verify + **host ProcessEnforcer** (SIGTERM/STOP/CONT for registered PIDs); cage-runner Docker kill path | auto PID discovery/collectors; grace_period from command payload | Phased PR plan §5 | storage + sensor unit (real child kill) | beta | collectors |
 | Ban system (first-class store) | Implemented | `lib/storage/src/db/agent_bans.rs`, migration `0029`; enforced at every choke point: `POST /v1/authorize` (agent + tool bans → durable deterministic deny via `write_decision_and_audit`), `POST /v1/broker/execute` (tool ban → 403 before approval consumption), `POST /v1/agent-cage/runs` create + claim (agent ban → 403; claim re-checks so a ban created after registration still blocks start), `POST /v1/egress/check` (destination ban, pre-existing); sensor prop: `POST /v1/bans` with `target_type=agent` issues signed `kill_run` control commands for the agent's active runs (`list_active_agent_runs_for_agent`) | `fingerprint`/`image_digest`/`prompt_hash` target types stored but not yet consulted at a choke point | #1678 | storage + route (authorize/broker/runtime/control/egress) | beta | remaining target types |
 | Quarantine records | Partial | `lib/storage/src/db/quarantine.rs`, migration `0030`; agent-status quarantine; `quarantine_records` enforced at `POST /v1/authorize` (agent), `POST /v1/broker/execute` (tool), cage run create + claim (agent + run), egress (run/sandbox/agent, pre-existing) | workspace/sandbox evidence-freeze semantics (needs cage) | #1679 | storage + route | beta | cage integration |
-| Node sensor | Implemented | `bins/aegis-node-sensor` (main, spool, shipper, command_receiver); real `process`/`net`/`fs`/`secret` collectors (`AEGIS_RUN_ID`-tagged process discovery -> `ProcessEnforcer` registration + `network_connection`/filesystem/secret-signal runtime events), all polled from the main loop; Dockerfile, Helm, compose; `tests/real_host_integration.rs` proves `scan_host_aegis_processes`/`ProcessCollector`/`NetCollector`/`FsCollector`/`SecretCollector` against a real Linux host's actual `/proc`, a real signal-killed child, a real established TCP socket, a real open file descriptor, and a real secret-shaped env var (name reported, value never leaves the host) -- not the synthetic `/proc` tempdir fixtures the module unit tests use | long-lived production-host soak (this proves correctness on a real Linux VM, i.e. CI, not a production deployment under sustained load) | Phased PR plan §5 | unit (87 tests) + 5 real-host integration tests | beta | production soak |
+| Node sensor | Implemented | `bins/aegis-node-sensor` (main, spool, shipper, command_receiver); real `process`/`net`/`fs`/`secret` collectors (`AEGIS_RUN_ID`-tagged process discovery -> `ProcessEnforcer` registration + `network_connection`/filesystem/secret-signal runtime events), all polled from the main loop; Dockerfile, Helm, compose; `tests/real_host_integration.rs` proves `scan_host_aegis_processes`/`ProcessCollector`/`NetCollector`/`FsCollector`/`SecretCollector` against a real Linux host's actual `/proc`, a real signal-killed child, a real established TCP socket, a real open file descriptor, and a real secret-shaped env var (name reported, value never leaves the host) -- not the synthetic `/proc` tempdir fixtures the module unit tests use; `scripts/sensor-soak.sh` + CI `sensor-soak.yml` (nightly 15-min + dispatchable multi-hour + PR smoke) soak the real sensor on the runner's real `/proc` under continuous collector workload, signed-kill round-trips, and a mid-soak gateway outage, asserting bounded RSS/fds/spool disk, zero panics, full spool drain, and unattended recovery; steady-state spool compaction (`compact_if_reclaimable`, wired into the ship tick) keeps lane files from growing without bound on long-lived hosts | extended-duration soak on a production-grade deployment (the nightly CI soak runs on a CI runner, not a production host under production load) | Phased PR plan §5 | unit (89 tests) + 5 real-host integration tests + nightly CI soak | beta | production-deployment soak |
 | Agent cage runner | Partial | binary + DockerRuntime + Dockerfile + compose `cage` + Helm; claim lifecycle smoke; host-Docker review + hardened create; `scripts/cage-docker-e2e.sh` + CI job (quick finish + signed kill against real Docker) | sensor↔runner IPC; forced egress netns; product “untrusted→incident” narrative e2e | Phased PR plan §6 | unit (lib) + claim lifecycle + cage-docker-e2e + helm lint | beta (local/k8s) | forced egress + sensor enforce |
 | Egress proxy | Partial | `bins/aegis-egress-proxy` binary + Dockerfile + Helm + compose; `POST /v1/egress/check` | forced cage netns integration; always-on path | Phased PR plan §7 | unit + proxy tests | beta | cage net integration |
 | Tool broker | Partial | `routes/broker.rs` (owns tool CRUD, action-hash, approval consume, receipts), `lib/tool-broker-core` (shared types), `bins/aegis-tool-broker` (Phase 1: standalone connector-execution binary the gateway calls over HTTP with a service-to-service bearer token; gateway no longer links `lib/tool-broker-connectors` in production) | mandatory force path for privileged tools; per-run scoped agent tokens; reversed agent→broker→gateway topology | Phased PR plan §8 | unit (aegis-tool-broker) + route (mock-HTTP-broker) | beta | force-path + scoped tokens |
@@ -86,7 +86,7 @@ Living checklist (detail also in [`.claude/PRPs/tasks/task.md`](../.claude/PRPs/
 1. ~~Cage-runner binary + packaging + Helm~~ **Done**; ~~claim-path smoke~~ **Done**; ~~host Docker security review + sandbox create hardening~~ **Done** (`docs/AegisAgent_Cage_Docker_Security.md`); ~~full Docker e2e~~ **Done** — `scripts/cage-docker-e2e.sh`, CI job `cage-docker-e2e`  
 
 2. Gateway claim/heartbeat/lease APIs — present (`/v1/agent-cage/runs/:id/{claim,heartbeat,status}`)  
-3. ~~Sensor host enforce kill/pause/resume/quarantine~~ **Done** (`process_enforcer` + command_receiver); ~~process collectors that `register_run`~~ **Done** (`process_collector.rs` discovers `AEGIS_RUN_ID`-tagged host processes and registers them; `net`/`fs`/`secret` collectors alongside it, all polled from `main.rs`'s loop); ~~live-host proof~~ **Done** — `bins/aegis-node-sensor/tests/real_host_integration.rs` runs every collector against a real Linux host's actual `/proc`/sockets/files (verified against a real `rust:1.96-bookworm` container, not just the synthetic `/proc` tempdir fixtures the module unit tests use); **remaining:** long-running production-host soak testing  
+3. ~~Sensor host enforce kill/pause/resume/quarantine~~ **Done** (`process_enforcer` + command_receiver); ~~process collectors that `register_run`~~ **Done** (`process_collector.rs` discovers `AEGIS_RUN_ID`-tagged host processes and registers them; `net`/`fs`/`secret` collectors alongside it, all polled from `main.rs`'s loop); ~~live-host proof~~ **Done** — `bins/aegis-node-sensor/tests/real_host_integration.rs` runs every collector against a real Linux host's actual `/proc`/sockets/files (verified against a real `rust:1.96-bookworm` container, not just the synthetic `/proc` tempdir fixtures the module unit tests use); ~~soak harness~~ **Done** — `scripts/sensor-soak.sh` + `sensor-soak.yml` (nightly / dispatch / PR smoke): sustained collector workload, periodic signed-kill round-trips, mid-soak gateway outage, bounded RSS/fd/spool-disk + drain + zero-panic assertions; landing it surfaced and fixed a real steady-state defect (spool lane files were never compacted outside the over-budget drop path, so disk grew without bound on a healthy long-lived host — `compact_if_reclaimable` now runs after ship ticks); **remaining:** extended-duration soak on a production-grade deployment  
 
 4. ~~E2E: untrusted agent → cage → egress deny → control action → receipt/incident (Docker)~~ **Done** — `scripts/cage-wave-a-e2e.sh` (#1840), CI-wired as job `cage-wave-a-e2e`: `root_trust_level=untrusted_external` cage run, egress routed through `aegis-egress-proxy --gateway-url` (the real fail-closed `POST /v1/egress/check` path) so a durable deny event + `ActionReceiptRecord` is asserted via `GET /v1/egress/events`, then a signed kill control command. Landing this e2e surfaced a real pre-existing bug: a forced-egress sandbox's `--internal` Docker bridge has **no route to the host at all** (not just no internet), so `host.docker.internal` never actually worked for reaching a host-run proxy, in any environment — fixed by having `aegis-cage-runner` join the proxy as a **sidecar container** to each sandbox's dedicated bridge (`egress_proxy_container` config, `docker network connect`, proxy addressed by container name) instead. Verified with a real-Docker regression test (`docker_runtime::forced_egress_sandbox_reaches_the_sidecar_proxy_container`) and the CI job itself.