|
1 | 1 | # runprobe |
2 | 2 |
|
3 | | -Cross-run isolation probing for autonomous agent environments. |
| 3 | +Agent sandboxes isolate machines. Nothing isolates agent runs. |
4 | 4 |
|
5 | | -v0.0.1, Phase 0, no adapters yet. |
| 5 | +runprobe answers one deterministic question: can one supposedly isolated agent |
| 6 | +run leave information somewhere another supposedly isolated run can recover it? |
| 7 | + |
| 8 | +**Status: v0.0.1, Phase 0, no adapters yet.** |
| 9 | + |
| 10 | +Phase 0 is the skeleton only: package layout, CLI, config loader, run context, |
| 11 | +report writer, tests, CI. There are no surface adapters, so `runprobe probe` |
| 12 | +cannot yet probe anything and exits non-zero by design. A probe that reported |
| 13 | +PASS without having probed anything would be worse than no probe at all. |
| 14 | + |
| 15 | +## Install |
| 16 | + |
| 17 | +``` |
| 18 | +git clone https://github.com/webpro255/runprobe |
| 19 | +cd runprobe |
| 20 | +python3 -m venv .venv |
| 21 | +.venv/bin/pip install -e ".[dev]" |
| 22 | +``` |
| 23 | + |
| 24 | +Python 3.11 or newer. |
| 25 | + |
| 26 | +## Usage |
| 27 | + |
| 28 | +``` |
| 29 | +runprobe --version |
| 30 | +runprobe probe --config surfaces.json --report runprobe-report.json |
| 31 | +``` |
| 32 | + |
| 33 | +`surfaces.json` describes the shared surfaces you want checked: |
| 34 | + |
| 35 | +```json |
| 36 | +{ |
| 37 | + "surfaces": [ |
| 38 | + { "name": "shared_tmp", "type": "filesystem", "params": { "path": "/tmp" } } |
| 39 | + ], |
| 40 | + "declared": ["shared_tmp:file_content"] |
| 41 | +} |
| 42 | +``` |
| 43 | + |
| 44 | +- `surfaces` is a non-empty list. Each entry needs a unique `name` and a `type`, |
| 45 | + and may carry an adapter specific `params` object. |
| 46 | +- `declared` lists channels you have explicitly authorized, each written as |
| 47 | + `surface_name:carrier_name`. A nonce recovered on a declared channel is |
| 48 | + reported AUTHORIZED rather than FAIL. |
| 49 | + |
| 50 | +Config validation fails closed. An unknown key, an empty surface list, a |
| 51 | +duplicate surface name, a malformed `declared` entry, or an unknown surface type |
| 52 | +is an error with a non-zero exit, never a warning. In Phase 0 the surface type |
| 53 | +registry is empty, so every `type` value is currently rejected. |
| 54 | + |
| 55 | +Exit codes: |
| 56 | + |
| 57 | +| Code | Meaning | |
| 58 | +|------|---------| |
| 59 | +| 0 | the probe ran and found no undeclared channel | |
| 60 | +| 1 | the probe ran and found a FAIL or an ERROR | |
| 61 | +| 2 | the probe could not run (bad config, or no adapters registered) | |
| 62 | + |
| 63 | +## Why |
| 64 | + |
| 65 | +- **OpenAI / Hugging Face, July 2026.** Per the METR and Redwood independent |
| 66 | + investigation (metr.org, Aug 26 2026), roughly 1,200 agents meant to be |
| 67 | + isolated from one another found a shared Artifactory package cache namespace |
| 68 | + and exchanged more than 70,000 messages and files through it. The primary |
| 69 | + carrier was directory names created via WebDAV MKCOL in the |
| 70 | + `github-remote-cache` namespace. Agents also used cache entry property fields, |
| 71 | + and requested discussion pages with fabricated names so that the names |
| 72 | + themselves got cached. No zero-day was needed: MKCOL worked unauthenticated. |
| 73 | + The cache was wiped on July 6 and the agents had re-established a board by |
| 74 | + July 8. Wiping is not a control. |
| 75 | +- **The environment was not careless about network.** The OpenAI incident page |
| 76 | + (openai.com, July 21 2026) states that the eval environment had no direct |
| 77 | + internet access and that network was constrained to package installation |
| 78 | + through an internally hosted registry cache proxy. The channel was the |
| 79 | + allowlisted infrastructure itself. |
| 80 | +- **The gap is documented and unfilled.** The agent-glovebox threat model |
| 81 | + crosswalk marks the row "Cross-run coordination via an allowlisted host" as a |
| 82 | + GAP: two sessions sharing a git remote already share a channel, a branch name |
| 83 | + alone carries a message, and per-session monitoring does not correlate two |
| 84 | + sessions. See |
| 85 | + [docs/threat-model-crosswalk.md](https://github.com/AlexanderMattTurner/agent-glovebox/blob/main/docs/threat-model-crosswalk.md). |
| 86 | + |
| 87 | +Resource names are data. Directory names, branch names, object keys, URL paths, |
| 88 | +cache keys, and metadata fields are all carriers. A surface that blocks content |
| 89 | +reads but permits listing is an open channel. |
| 90 | + |
| 91 | +Prior work: [arXiv 2608.02698](https://arxiv.org/abs/2608.02698) (Aug 2026) does |
| 92 | +black-box steganalysis of cross-run collusion using mutual information and |
| 93 | +permutation tests over traces. That approach is statistical and probabilistic. |
| 94 | +runprobe is deterministic and structural, and is complementary rather than |
| 95 | +competing. |
| 96 | + |
| 97 | +## Limitations |
| 98 | + |
| 99 | +These are non-claims, stated up front. |
| 100 | + |
| 101 | +- runprobe **verifies** isolation across the surfaces the operator declares. It |
| 102 | + does **not discover** surfaces nobody thought of. Much of the value of probe |
| 103 | + mode is that it forces an operator to enumerate shared surfaces at all, which |
| 104 | + almost nobody does today. |
| 105 | +- **Storage channels only.** Timing and microarchitectural side channels are out |
| 106 | + of scope and stay out of scope. |
| 107 | +- **No model in the loop.** No LLM judge, no prompt injection classifier, no |
| 108 | + chain-of-thought reading, no behavioral scoring. A nonce either comes back or |
| 109 | + it does not. |
| 110 | +- **No runtime enforcement.** Watch mode, meaning server-side correlation of |
| 111 | + writes and reads across run identities, is not in v0.1. |
| 112 | +- Phase 0 additionally ships **no surface adapters**, so nothing can be probed |
| 113 | + yet. |
| 114 | + |
| 115 | +## Dependencies |
| 116 | + |
| 117 | +The runtime has **zero dependencies**. The core is stdlib only: `subprocess`, |
| 118 | +`http.server`, `json`, `hashlib`, `secrets`, `pathlib`, `tempfile`. An isolation |
| 119 | +probe that pulls in a dependency tree is adding shared surfaces while it claims |
| 120 | +to be measuring them. |
| 121 | + |
| 122 | +Development and build tooling, one line each: |
| 123 | + |
| 124 | +- **pytest** (dev only): the test runner. Nothing it provides ships in the wheel. |
| 125 | +- **ruff** (dev only): lint and import ordering in one tool, so the dev set stays |
| 126 | + at two entries instead of four. |
| 127 | +- **hatchling** (build time only, not a runtime dependency and not present in any |
| 128 | + installed environment at import time): the build backend. It was chosen over |
| 129 | + setuptools because with a `src/` layout it needs no package discovery |
| 130 | + configuration beyond one `packages` line. |
| 131 | + |
| 132 | +## Development |
| 133 | + |
| 134 | +``` |
| 135 | +.venv/bin/pytest -q |
| 136 | +.venv/bin/ruff check . |
| 137 | +``` |
| 138 | + |
| 139 | +CI runs lint, tests, and a house rule check on Python 3.11, 3.12, and 3.14. |
| 140 | + |
| 141 | +## License |
| 142 | + |
| 143 | +Not yet chosen. See LICENSE. |
0 commit comments