Skip to content

test(sensor): prove Wave A collectors against a real live Linux host #1857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lavkushry merged 1 commit into from
Jul 12, 2026
Merged

test(sensor): prove Wave A collectors against a real live Linux host #1857

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
226 changes: 226 additions & 0 deletions bins/aegis-node-sensor/tests/real_host_integration.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
//! Proves the Wave A collectors against a real live Linux host, not the
//! synthetic `/proc` trees the unit tests in each collector module use.
//! CI's `ubuntu-latest` runner is itself a real Linux VM, so these tests
//! exercise the actual production entrypoints (`scan_host_aegis_processes`,
//! `ProcessCollector::poll`, `NetCollector::poll`, `FsCollector::poll`) against
//! real child processes, a real SIGTERM/SIGKILL, a real established TCP
//! socket, and a real open file descriptor -- not the `*_in(root: &Path)`
//! test-only helpers that read a tempdir fixture instead of `/proc`.
//!
//! This closes the "not yet proven against a real live host" caveat named in
//! `docs/current-vs-roadmap.md` for the node sensor's collectors -- for the
//! definition of "real host" a CI runner satisfies (a real Linux kernel's
//! `/proc`, real signals, real sockets). It does not prove behavior on an
//! arbitrary long-lived production host under production load; that remains
//! a separate, larger follow-up.
//!
//! Linux-only: every collector here is a documented no-op on other
//! platforms, matching each module's own doc comment.

#![cfg(target_os = "linux")]

use std::process::{Child, Command, Stdio};
use std::sync::Arc;
use std::time::Duration;

use aegis_node_sensor::fs_collector::FsCollector;
use aegis_node_sensor::gateway_client::RuntimeEventPayload;
use aegis_node_sensor::net_collector::NetCollector;
use aegis_node_sensor::process_collector::{scan_host_aegis_processes, ProcessCollector};
use aegis_node_sensor::process_enforcer::{process_alive, ProcessEnforcer};
use aegis_node_sensor::secret_collector::SecretCollector;
use aegis_node_sensor::spool::{Lane, SpoolQueue};

fn unique_run_id(tag: &str) -> String {
format!(
"real-host-{tag}-{}-{}",
std::process::id(),
rand::random::<u32>()
)
}

fn spawn_sleep_with_run_id(run_id: &str) -> Child {
Command::new("sleep")
.arg("20")
.env("AEGIS_RUN_ID", run_id)
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
.expect("spawn real sleep child")
}
Comment on lines +42 to +50

@gemini-code-assist gemini-code-assist Bot Jul 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Spawning child processes in tests without a drop guard can lead to orphaned background processes if an assertion panics before child.kill() is reached. Introducing a simple KillOnDrop helper ensures that child processes are always cleaned up, even during test failures.

struct KillOnDrop(Child);

impl std::ops::Deref for KillOnDrop {
    type Target = Child;
    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

impl std::ops::DerefMut for KillOnDrop {
    fn deref_mut(&mut self) -> &mut Self::Target {
        &mut self.0
    }
}

impl Drop for KillOnDrop {
    fn drop(&mut self) {
        let _ = self.0.kill();
        let _ = self.0.wait();
    }
}

fn spawn_sleep_with_run_id(run_id: &str) -> KillOnDrop {
    KillOnDrop(
        Command::new("sleep")
            .arg("20")
            .env("AEGIS_RUN_ID", run_id)
            .stdout(Stdio::null())
            .stderr(Stdio::null())
            .spawn()
            .expect("spawn real sleep child"),
    )
}


#[test]
fn process_collector_discovers_a_real_child_via_the_actual_proc_filesystem() {
let run_id = unique_run_id("proc");
let mut child = spawn_sleep_with_run_id(&run_id);
let pid = child.id() as i32;

// Real /proc/<pid>/environ, populated by the kernel for a real child --
// not the tempdir fixture `scan_proc_fs`'s own unit tests use.
let found = scan_host_aegis_processes();
let hit = found.iter().find(|p| p.pid == pid);
assert!(
hit.is_some(),
"expected pid {pid} with AEGIS_RUN_ID={run_id} to be discovered via real /proc"
);
assert_eq!(hit.unwrap().run_id, run_id);

let _ = child.kill();
let _ = child.wait();
}

#[test]
fn process_collector_poll_registers_with_enforcer_and_can_kill_a_real_child() {
let run_id = unique_run_id("poll-kill");
let mut child = spawn_sleep_with_run_id(&run_id);
let pid = child.id() as i32;

let enforcer = Arc::new(ProcessEnforcer::new());
let collector = ProcessCollector::new(enforcer.clone());
let spool_dir = tempfile::tempdir().unwrap();
let spool = SpoolQueue::open(spool_dir.path(), 1_000_000).unwrap();

collector.poll(&spool);

assert_eq!(enforcer.pid_for(&run_id), Some(pid));

let rec = spool
.read_next(Lane::Normal)
.unwrap()
.expect("process_started event spooled for the real child");
let payload: RuntimeEventPayload = serde_json::from_slice(&rec.payload).unwrap();
assert_eq!(payload.event_type, "process_started");
assert_eq!(payload.run_id.as_deref(), Some(run_id.as_str()));

// Real SIGTERM (escalating to SIGKILL) against a real live host process.
enforcer.kill_run(&run_id).unwrap();
let _ = child.wait();
assert!(!process_alive(pid));
}

#[test]
fn net_collector_reports_a_real_established_tcp_connection() {
let run_id = unique_run_id("net");

let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = listener.local_addr().unwrap().port();
std::thread::spawn(move || {
if let Ok((stream, _)) = listener.accept() {
std::thread::sleep(Duration::from_secs(5));
drop(stream);
}
});

// bash's /dev/tcp pseudo-device opens a real TCP socket from a real
// child process without depending on netcat being on the runner image.
let Ok(mut child) = Command::new("bash")
.arg("-c")
.arg(format!("exec 3<>/dev/tcp/127.0.0.1/{port}; sleep 5"))
.env("AEGIS_RUN_ID", &run_id)
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
else {
eprintln!("bash not available -- skipping real TCP socket test");
return;
};

// Give the connection a moment to reach ESTABLISHED in /proc/net/tcp.
std::thread::sleep(Duration::from_millis(500));

let collector = NetCollector::new();
let spool_dir = tempfile::tempdir().unwrap();
let spool = SpoolQueue::open(spool_dir.path(), 1_000_000).unwrap();
collector.poll(&spool);

let rec = spool
.read_next(Lane::Normal)
.unwrap()
.expect("network_connection event spooled for the real socket");
let payload: RuntimeEventPayload = serde_json::from_slice(&rec.payload).unwrap();
Comment on lines +128 to +140

@gemini-code-assist gemini-code-assist Bot Jul 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a hardcoded sleep of 500ms to wait for the connection to reach ESTABLISHED can lead to flaky tests in busy CI environments. Since bash executes asynchronously, it is much more robust to poll the collector in a retry loop with a timeout.

    let collector = NetCollector::new();
    let spool_dir = tempfile::tempdir().unwrap();
    let spool = SpoolQueue::open(spool_dir.path(), 1_000_000).unwrap();

    let mut payload = None;
    for _ in 0..50 {
        collector.poll(&spool);
        if let Ok(Some(rec)) = spool.read_next(Lane::Normal) {
            let p: RuntimeEventPayload = serde_json::from_slice(&rec.payload).unwrap();
            if p.event_type == "network_connection" {
                payload = Some(p);
                break;
            }
        }
        std::thread::sleep(Duration::from_millis(100));
    }
    let payload = payload.expect("network_connection event spooled for the real socket");

assert_eq!(payload.event_type, "network_connection");
assert_eq!(payload.run_id.as_deref(), Some(run_id.as_str()));
assert!(payload
.reason
.as_deref()
.unwrap()
.contains(&format!("remote=127.0.0.1:{port}")));

let _ = child.kill();
let _ = child.wait();
}

#[test]
fn fs_collector_reports_a_real_open_file_descriptor() {
let run_id = unique_run_id("fs");
let path = std::env::temp_dir().join(format!("aegis-sensor-real-host-{run_id}.txt"));
let path_str = path.to_string_lossy().into_owned();

let Ok(mut child) = Command::new("bash")
.arg("-c")
.arg(format!("exec 3<>'{path_str}'; sleep 5"))
.env("AEGIS_RUN_ID", &run_id)
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
else {
eprintln!("bash not available -- skipping real file descriptor test");
return;
};

std::thread::sleep(Duration::from_millis(300));

let collector = FsCollector::new();
let spool_dir = tempfile::tempdir().unwrap();
let spool = SpoolQueue::open(spool_dir.path(), 1_000_000).unwrap();
collector.poll(&spool);

let rec = spool
.read_next(Lane::Normal)
.unwrap()
.expect("fs_open event spooled for the real file descriptor");
let payload: RuntimeEventPayload = serde_json::from_slice(&rec.payload).unwrap();
Comment on lines +171 to +182

@gemini-code-assist gemini-code-assist Bot Jul 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a hardcoded sleep of 300ms to wait for the file descriptor to be opened can lead to flaky tests in busy CI environments. Since bash executes asynchronously, it is much more robust to poll the collector in a retry loop with a timeout.

Suggested change
std::thread::sleep(Duration::from_millis(300));
let collector = FsCollector::new();
let spool_dir = tempfile::tempdir().unwrap();
let spool = SpoolQueue::open(spool_dir.path(), 1_000_000).unwrap();
collector.poll(&spool);
let rec = spool
.read_next(Lane::Normal)
.unwrap()
.expect("fs_open event spooled for the real file descriptor");
let payload: RuntimeEventPayload = serde_json::from_slice(&rec.payload).unwrap();
let collector = FsCollector::new();
let spool_dir = tempfile::tempdir().unwrap();
let spool = SpoolQueue::open(spool_dir.path(), 1_000_000).unwrap();
let mut payload = None;
for _ in 0..50 {
collector.poll(&spool);
if let Ok(Some(rec)) = spool.read_next(Lane::Normal) {
let p: RuntimeEventPayload = serde_json::from_slice(&rec.payload).unwrap();
if p.event_type == "fs_open" {
payload = Some(p);
break;
}
}
std::thread::sleep(Duration::from_millis(100));
}
let payload = payload.expect("fs_open event spooled for the real file descriptor");

assert_eq!(payload.event_type, "fs_open");
assert_eq!(payload.run_id.as_deref(), Some(run_id.as_str()));
assert!(payload.reason.as_deref().unwrap().contains(&path_str));

let _ = child.kill();
let _ = child.wait();
let _ = std::fs::remove_file(&path);
}

#[test]
fn secret_collector_reports_only_the_env_name_never_the_value_for_a_real_child() {
let run_id = unique_run_id("secret");
let secret_value = "definitely-not-a-real-secret-value-12345";
let mut child = Command::new("sleep")
.arg("20")
.env("AEGIS_RUN_ID", &run_id)
.env("GITHUB_TOKEN", secret_value)
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
.expect("spawn real sleep child with a secret-like env var");
let pid = child.id() as i32;

let collector = SecretCollector::new();
let spool_dir = tempfile::tempdir().unwrap();
let spool = SpoolQueue::open(spool_dir.path(), 1_000_000).unwrap();
collector.poll(&spool);

let rec = spool
.read_next(Lane::Normal)
.unwrap()
.expect("secret_signal event spooled for the real child's real /proc/<pid>/environ");
let payload: RuntimeEventPayload = serde_json::from_slice(&rec.payload).unwrap();
assert_eq!(payload.event_type, "secret_signal");
assert_eq!(payload.run_id.as_deref(), Some(run_id.as_str()));
let reason = payload.reason.as_deref().unwrap();
assert!(reason.contains(&format!("pid={pid}")));
assert!(reason.contains("GITHUB_TOKEN"));
// The whole point of the collector: the value never leaves the host.
assert!(!reason.contains(secret_value));

let _ = child.kill();
let _ = child.wait();
}
4 changes: 2 additions & 2 deletions docs/Implementation_Status.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
| Control commands (signed kill/pause/quarantine) | Partial | store + protocol + gateway issue routes; sensor poll/verify + **host ProcessEnforcer** (SIGTERM/STOP/CONT for registered PIDs); cage-runner Docker kill path | auto PID discovery/collectors; grace_period from command payload | Phased PR plan §5 | storage + sensor unit (real child kill) | beta | collectors |
| Ban system (first-class store) | Partial | `lib/storage/src/db/agent_bans.rs`, migration `0029` | enforcement at every choke point + sensor prop | #1678 | storage | beta | preflight wire-up |
| Quarantine records | Partial | `lib/storage/src/db/quarantine.rs`, migration `0030`; agent-status quarantine | workspace/sandbox quarantine (needs cage) | #1679 | storage | beta | cage integration |
| Node sensor | Implemented | `bins/aegis-node-sensor` (main, spool, shipper, command_receiver); real `process`/`net`/`fs`/`secret` collectors (`AEGIS_RUN_ID`-tagged process discovery -> `ProcessEnforcer` registration + `network_connection`/filesystem/secret-signal runtime events), all polled from the main loop; Dockerfile, Helm, compose | live-host e2e (current tests use synthetic `/proc` trees, not a real running process) | Phased PR plan §5 | unit (87 tests) | beta | live-host e2e |
| Node sensor | Implemented | `bins/aegis-node-sensor` (main, spool, shipper, command_receiver); real `process`/`net`/`fs`/`secret` collectors (`AEGIS_RUN_ID`-tagged process discovery -> `ProcessEnforcer` registration + `network_connection`/filesystem/secret-signal runtime events), all polled from the main loop; Dockerfile, Helm, compose; `tests/real_host_integration.rs` proves `scan_host_aegis_processes`/`ProcessCollector`/`NetCollector`/`FsCollector`/`SecretCollector` against a real Linux host's actual `/proc`, a real signal-killed child, a real established TCP socket, a real open file descriptor, and a real secret-shaped env var (name reported, value never leaves the host) -- not the synthetic `/proc` tempdir fixtures the module unit tests use | long-lived production-host soak (this proves correctness on a real Linux VM, i.e. CI, not a production deployment under sustained load) | Phased PR plan §5 | unit (87 tests) + 5 real-host integration tests | beta | production soak |
| Agent cage runner | Partial | binary + DockerRuntime + Dockerfile + compose `cage` + Helm; claim lifecycle smoke; host-Docker review + hardened create; `scripts/cage-docker-e2e.sh` + CI job (quick finish + signed kill against real Docker) | sensor↔runner IPC; forced egress netns; product “untrusted→incident” narrative e2e | Phased PR plan §6 | unit (lib) + claim lifecycle + cage-docker-e2e + helm lint | beta (local/k8s) | forced egress + sensor enforce |
| Egress proxy | Partial | `bins/aegis-egress-proxy` binary + Dockerfile + Helm + compose; `POST /v1/egress/check` | forced cage netns integration; always-on path | Phased PR plan §7 | unit + proxy tests | beta | cage net integration |
| Tool broker | Partial | `routes/broker.rs` (owns tool CRUD, action-hash, approval consume, receipts), `lib/tool-broker-core` (shared types), `bins/aegis-tool-broker` (Phase 1: standalone connector-execution binary the gateway calls over HTTP with a service-to-service bearer token; gateway no longer links `lib/tool-broker-connectors` in production) | mandatory force path for privileged tools; per-run scoped agent tokens; reversed agent→broker→gateway topology | Phased PR plan §8 | unit (aegis-tool-broker) + route (mock-HTTP-broker) | beta | force-path + scoped tokens |
Expand All @@ -67,7 +67,7 @@ Living checklist (detail also in [`.claude/PRPs/tasks/task.md`](../.claude/PRPs/
1. ~~Cage-runner binary + packaging + Helm~~ **Done**; ~~claim-path smoke~~ **Done**; ~~host Docker security review + sandbox create hardening~~ **Done** (`docs/AegisAgent_Cage_Docker_Security.md`); ~~full Docker e2e~~ **Done** — `scripts/cage-docker-e2e.sh`, CI job `cage-docker-e2e`

2. Gateway claim/heartbeat/lease APIs — present (`/v1/agent-cage/runs/:id/{claim,heartbeat,status}`)
3. ~~Sensor host enforce kill/pause/resume/quarantine~~ **Done** (`process_enforcer` + command_receiver); ~~process collectors that `register_run`~~ **Done** (`process_collector.rs` discovers `AEGIS_RUN_ID`-tagged host processes and registers them; `net`/`fs`/`secret` collectors alongside it, all polled from `main.rs`'s loop); **remaining:** a live-host e2e (today's tests use synthetic `/proc` trees)
3. ~~Sensor host enforce kill/pause/resume/quarantine~~ **Done** (`process_enforcer` + command_receiver); ~~process collectors that `register_run`~~ **Done** (`process_collector.rs` discovers `AEGIS_RUN_ID`-tagged host processes and registers them; `net`/`fs`/`secret` collectors alongside it, all polled from `main.rs`'s loop); ~~live-host proof~~ **Done** — `bins/aegis-node-sensor/tests/real_host_integration.rs` runs every collector against a real Linux host's actual `/proc`/sockets/files (verified against a real `rust:1.96-bookworm` container, not just the synthetic `/proc` tempdir fixtures the module unit tests use); **remaining:** long-running production-host soak testing

4. ~~E2E: untrusted agent → cage → egress deny → control action → receipt/incident (Docker)~~ **Done** — `scripts/cage-wave-a-e2e.sh` (#1840), CI-wired as job `cage-wave-a-e2e`: `root_trust_level=untrusted_external` cage run, egress routed through `aegis-egress-proxy --gateway-url` (the real fail-closed `POST /v1/egress/check` path) so a durable deny event + `ActionReceiptRecord` is asserted via `GET /v1/egress/events`, then a signed kill control command. Landing this e2e surfaced a real pre-existing bug: a forced-egress sandbox's `--internal` Docker bridge has **no route to the host at all** (not just no internet), so `host.docker.internal` never actually worked for reaching a host-run proxy, in any environment — fixed by having `aegis-cage-runner` join the proxy as a **sidecar container** to each sandbox's dedicated bridge (`egress_proxy_container` config, `docker network connect`, proxy addressed by container name) instead. Verified with a real-Docker regression test (`docker_runtime::forced_egress_sandbox_reaches_the_sidecar_proxy_container`) and the CI job itself.

Expand Down
7 changes: 3 additions & 4 deletions docs/current-vs-roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Strongest current capabilities:

### Partial (built, not end-to-end production)

- **Node sensor** — binary, register/heartbeat, command poll, shipper, host ProcessEnforcer (real kill/pause/resume for registered PIDs), real process/net/fs/secret collectors (`AEGIS_RUN_ID` auto-discovery -> enforcer registration); not yet proven against a real live host (tests use synthetic `/proc` trees)
- **Node sensor** — binary, register/heartbeat, command poll, shipper, host ProcessEnforcer (real kill/pause/resume for registered PIDs), real process/net/fs/secret collectors (`AEGIS_RUN_ID` auto-discovery -> enforcer registration); now proven against a real Linux host (`tests/real_host_integration.rs`: real `/proc`, real signal-killed child, real established TCP socket, real open file descriptor, real secret-shaped env var) rather than only synthetic `/proc` trees; long-running production-host soak still pending
- **Egress proxy** — binary + Helm; not forced for all caged traffic by default
- **Tool broker** — standalone `aegis-tool-broker` binary exists and is the only execution path when configured (gateway no longer links the connector-execution crate in production); no mandatory-force-path enforcement yet, no scoped per-run agent tokens
- **Agent cage runner** — binary + claim/execute loop; compose profile `cage` + Helm; host Docker security review + hardened create flags shipped (`docs/AegisAgent_Cage_Docker_Security.md`); full Docker e2e done (`cage-docker-e2e.sh` + `cage-wave-a-e2e.sh`, both CI-wired)
Expand All @@ -48,7 +48,6 @@ Strongest current capabilities:

### Roadmap / not done

- sensor collectors proven against a real live host, not just synthetic `/proc` trees (P0)
- transparent / netns forced egress (no raw-socket bypass) (P0 residual after proxy-env force)
- Postgres multi-replica GA (#1194) (P1) — remaining: real cross-instance failover/replication-lag validation, load validation, default Helm bundling
- SAML, multi-IdP, and per-SSO-user attribution/revocation for console login (P1)
Expand All @@ -75,11 +74,11 @@ Strongest current capabilities:
| TypeScript SDK | Available today | Canon + protect + client + receipt chain verifier (shared corpus). |
| Go SDK | Partial | Core path shipped; prompt/model emit parity TBD. |
| Full web console | Available today | Bun SPA panel suite + cage runs / ban / quarantine / egress / evidence-graph / policy / prompt-timeline / model-calls pages shipped (Phase 9.2/9.3 complete). |
| Node sensor | Partial | Binary + packaging; real process/net/fs/secret collectors + host enforce, unit-tested; not yet proven on a real live host. |
| Node sensor | Partial | Binary + packaging; real process/net/fs/secret collectors + host enforce, unit-tested and proven against a real Linux host (`tests/real_host_integration.rs`); production-host soak pending. |
| Agent cage runner | Partial | Binary + DockerRuntime + compose + Helm; Docker security review + create hardening done; full Docker e2e done (`cage-docker-e2e.sh` + `cage-wave-a-e2e.sh`, CI-wired). |
| Egress proxy | Partial | Binary + packaging; not default-forced for cages. |
| Tool broker | Partial | Standalone `aegis-tool-broker` binary + HTTP contract; gateway owns approval/receipts, broker owns credential resolution + connector execution; no mandatory force path, no scoped tokens. |
| Signed control commands | Partial | Issue + sensor poll + host PID enforce + auto PID discovery (`AEGIS_RUN_ID` process collector); not yet proven on a real live host. |
| Signed control commands | Partial | Issue + sensor poll + host PID enforce + auto PID discovery (`AEGIS_RUN_ID` process collector); real-host kill proven (`tests/real_host_integration.rs`), production soak pending. |
| Ban / quarantine centers | Partial | Stores/APIs; not every choke point + full UI. |
| Postgres production mode | Roadmap / partial | Code path now CI-validated against a live Postgres instance (was compile-check only); SQLite single-writer is still the default deploy. |
| Full Kubernetes multi-replica | Roadmap | Blocked on Postgres GA + broader Helm surface. |
Expand Down
Loading