Skip to content

Commit fa1cf8c

Browse files
committed
feat(isolation): enable the co-located RFC 0012 backend
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
1 parent fccbbc0 commit fa1cf8c

32 files changed

Lines changed: 3137 additions & 121 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,7 @@ Common findings:
173173
- Gateway process stopped: inspect exit status and logs.
174174
- Sandbox image missing or pull denied: verify image reference and registry credentials.
175175
- Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Root and missing identities are rejected.
176-
- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`.
176+
- Docker driver cannot initialize because it cannot find `openshell-sandbox` or its trusted helper runtime: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, and that the configured supervisor image contains both `/openshell-sandbox` and `/openshell-runtime`. A local supervisor binary without a valid sibling `openshell-runtime` still requires the image as the runtime source.
177177
- Sandbox never registers: check gateway logs and supervisor callback endpoint.
178178
- Supervisor image exits before printing `openshell-sandbox --version`: the image should be the scratch supervisor image from `deploy/docker/Dockerfile.supervisor` and must contain a static executable at `/openshell-sandbox`.
179179
- `mise run e2e:docker:gpu` fails with `docker info --format json did not report any discovered NVIDIA CDI GPU devices`: Docker may report `CDISpecDirs` while still having no generated NVIDIA CDI specs. Verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. NVIDIA recommends enabling the path and service units, and restarting `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs. If specs are generated but Docker still reports no discovered devices, restart Docker or reload the daemon and re-check `docker info`.
@@ -355,6 +355,20 @@ kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\
355355
kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
356356
```
357357

358+
For the RFC 0012 co-located backend, verify the trusted topology and confirm the
359+
driver placed the descriptor in the supervisor command rather than
360+
workload-controlled arguments:
361+
362+
```bash
363+
helm -n openshell get values openshell | grep topology
364+
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^topology\s*='
365+
kubectl -n <sandbox-namespace> get pod <sandbox-pod> -o jsonpath='{.spec.containers[?(@.name=="agent")].command}{"\n"}'
366+
```
367+
368+
Combined topology selects the in-pod backend; sidecar topology remains on its
369+
separate lifecycle. A combined pod command should include the driver-supplied
370+
topology descriptor arguments.
371+
358372
If `topology = "sidecar"` is rendered under `[openshell.drivers.kubernetes]`,
359373
sandbox pods should have an `openshell-network-init` init container running
360374
`--mode=network-init`, an `agent` container running

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,9 @@ mise run helm:skaffold:run
6565
mise run helm:skaffold:run:sidecar
6666
```
6767

68+
Combined topology selects RFC 0012's in-pod backend and supplies its descriptor;
69+
sidecar topology remains on its separate lifecycle.
70+
6871
**Supervisor sidecar topology with TLS/mTLS enabled** (build once and leave running):
6972
```bash
7073
mise run helm:skaffold:run:sidecar-mtls

‎.github/workflows/branch-checks.yml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,61 @@ jobs:
139139
fi
140140
exit 0
141141
142+
isolation-conformance:
143+
name: Isolation conformance (privileged Linux)
144+
needs: pr_metadata
145+
if: needs.pr_metadata.outputs.should_run == 'true'
146+
runs-on: linux-amd64-cpu8
147+
timeout-minutes: 20
148+
container:
149+
image: ghcr.io/nvidia/openshell/ci:latest
150+
options: --privileged
151+
credentials:
152+
username: ${{ github.actor }}
153+
password: ${{ secrets.GITHUB_TOKEN }}
154+
steps:
155+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
156+
157+
- name: Install tools and network helpers
158+
run: |
159+
apt-get update
160+
apt-get install -y --no-install-recommends iproute2 nftables iptables util-linux
161+
mise install --locked
162+
163+
- name: Materialize the Alpine trusted helper runtime fixture
164+
run: |
165+
alpine_root="${RUNNER_TEMP}/openshell-alpine-root"
166+
runtime="${RUNNER_TEMP}/openshell-runtime"
167+
archive="${RUNNER_TEMP}/alpine-minirootfs.tar.gz"
168+
curl -fsSL \
169+
https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/x86_64/alpine-minirootfs-3.22.5-x86_64.tar.gz \
170+
-o "$archive"
171+
echo "4b4daa9fe2fc696c4919c4412a4c3d3e770d8fb70292a004a2c72f5096175282 $archive" \
172+
| sha256sum -c -
173+
mkdir -p "$alpine_root" "$runtime"
174+
tar -xzf "$archive" -C "$alpine_root"
175+
cp /etc/resolv.conf "$alpine_root/etc/resolv.conf"
176+
chroot "$alpine_root" /sbin/apk add --no-cache \
177+
iproute2 nftables iptables iptables-legacy
178+
for path in /bin /sbin /lib /lib64 /usr/bin /usr/sbin /usr/lib /usr/lib64 /etc/iproute2 /usr/share/nftables; do
179+
if [ -e "$alpine_root$path" ]; then
180+
(cd "$alpine_root" && cp -aL --parents ".$path" "$runtime")
181+
fi
182+
done
183+
chmod -R go-w "$runtime"
184+
185+
- name: Exercise the live default-deny ceiling
186+
run: |
187+
cargo test -p openshell-isolation -p openshell-supervisor-process \
188+
-p openshell-supervisor-network -p openshell-sandbox
189+
OPENSHELL_TEST_TRUSTED_RUNTIME_ROOT="${RUNNER_TEMP}/openshell-runtime" \
190+
cargo test -p openshell-supervisor-process \
191+
installed_egress_ceiling_ -- \
192+
--ignored --nocapture --test-threads=1
193+
cargo test -p openshell-sandbox \
194+
pid_one_exit_kills_unregistered_setsid_descendant_within_bound -- \
195+
--ignored --nocapture --test-threads=1
196+
142197
rust-macos:
143198
name: Rust lint (macOS)
144199
needs: pr_metadata

‎.github/workflows/driver-vm-linux.yml‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,8 @@ jobs:
116116
username: ${{ github.actor }}
117117
password: ${{ secrets.GITHUB_TOKEN }}
118118
options: --privileged
119+
volumes:
120+
- /var/run/docker.sock:/var/run/docker.sock
119121
env:
120122
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
121123
OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }}
@@ -141,8 +143,12 @@ jobs:
141143
cache-directories: .cache/sccache
142144
cache-targets: "true"
143145

144-
- name: Install zstd
145-
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*
146+
- name: Install zstd and verify Docker
147+
run: |
148+
apt-get update
149+
apt-get install -y --no-install-recommends zstd
150+
rm -rf /var/lib/apt/lists/*
151+
docker info
146152
147153
- name: Download kernel runtime tarball
148154
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -168,14 +174,14 @@ jobs:
168174
- name: Verify embedded driver inputs
169175
run: |
170176
set -euo pipefail
171-
for file in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst openshell-sandbox.zst; do
177+
for file in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst umoci.zst openshell-sandbox.zst openshell-runtime.tar.zst; do
172178
test -s "target/vm-runtime-compressed/${file}"
173179
done
174180
175181
- name: Scope workspace to driver-vm crates
176182
run: |
177183
set -euo pipefail
178-
sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-driver-vm", "crates/openshell-core"]|' Cargo.toml
184+
sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-driver-vm", "crates/openshell-core", "crates/openshell-isolation"]|' Cargo.toml
179185
180186
- name: Patch workspace version
181187
if: ${{ inputs['cargo-version'] != '' }}

‎.github/workflows/driver-vm-macos.yml‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ jobs:
7575
credentials:
7676
username: ${{ github.actor }}
7777
password: ${{ secrets.GITHUB_TOKEN }}
78+
volumes:
79+
- /var/run/docker.sock:/var/run/docker.sock
7880
env:
7981
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
8082
OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }}
@@ -100,8 +102,12 @@ jobs:
100102
cache-directories: .cache/sccache
101103
cache-targets: "true"
102104

103-
- name: Install zstd
104-
run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/*
105+
- name: Install zstd and verify Docker
106+
run: |
107+
apt-get update
108+
apt-get install -y --no-install-recommends zstd
109+
rm -rf /var/lib/apt/lists/*
110+
docker info
105111
106112
- name: Build bundled supervisor
107113
run: |
@@ -116,7 +122,9 @@ jobs:
116122
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
117123
with:
118124
name: driver-vm-supervisor-arm64
119-
path: target/vm-runtime-compressed/openshell-sandbox.zst
125+
path: |
126+
target/vm-runtime-compressed/openshell-sandbox.zst
127+
target/vm-runtime-compressed/openshell-runtime.tar.zst
120128
retention-days: 1
121129

122130
build-driver-vm-macos:
@@ -180,12 +188,13 @@ jobs:
180188
run: |
181189
set -euo pipefail
182190
test -f target/vm-runtime-compressed-macos/openshell-sandbox.zst
183-
ls -lh target/vm-runtime-compressed-macos/openshell-sandbox.zst
191+
test -f target/vm-runtime-compressed-macos/openshell-runtime.tar.zst
192+
ls -lh target/vm-runtime-compressed-macos/openshell-{sandbox,runtime.tar}.zst
184193
185194
- name: Verify embedded driver inputs
186195
run: |
187196
set -euo pipefail
188-
for file in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst openshell-sandbox.zst; do
197+
for file in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst umoci.zst openshell-sandbox.zst openshell-runtime.tar.zst; do
189198
test -s "target/vm-runtime-compressed-macos/${file}"
190199
done
191200

‎Cargo.lock‎

Lines changed: 17 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/build.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -96,11 +96,13 @@ Runtime layout:
9696
as a release artifact. Linux GNU VM driver binaries must not reference
9797
`GLIBC_*` symbols newer than `GLIBC_2.28`; release workflows verify this
9898
before publishing artifacts.
99-
- **Supervisor**: Alpine base with `nftables`, static musl binary at
100-
`/openshell-sandbox`. Static linkage keeps the binary usable when the image
101-
is mounted/extracted into sandbox environments (Docker extraction, Podman
102-
image volumes, Kubernetes init-container copy-self), while `nftables` supports
103-
Kubernetes supervisor sidecar egress enforcement.
99+
- **Supervisor**: Alpine base with a static musl binary at
100+
`/openshell-sandbox` and a materialized, driver-controlled network-helper
101+
runtime at `/openshell-runtime`. Kubernetes mounts or copies both into
102+
workload containers so boundary setup never executes helpers supplied by a
103+
workload image. Podman mounts the same image runtime, Docker extracts it to
104+
a digest-keyed read-only bind mount, and the VM supervisor build embeds it
105+
beside the guest supervisor.
104106

105107
Gateway image builds bake the corresponding supervisor image tag into the
106108
gateway binary so Docker sandboxes do not depend on `:latest` by default.

‎architecture/sandbox.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,34 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
4949
The supervisor may enrich baseline filesystem allowances for runtime-required
5050
paths, such as proxy support files or GPU device paths when a GPU is present.
5151

52+
## Isolation Backend
53+
54+
[RFC 0012](../rfc/0012-isolation-backend/README.md) defines the Isolation
55+
Backend contract for topology-specific boundary construction and process
56+
operations. The contract uses consuming
57+
lifecycle states (`attach` → `Bound` → `confirm` → `Ready` → `start_agent` →
58+
`Running`) so untrusted workload execution cannot begin before standing
59+
enforcement is confirmed.
60+
61+
The logical supervisor remains the trusted bridge between the gateway and the
62+
workload. It drives the backend and applies approved network policy through
63+
supervisor-owned mediation; the backend routes workload egress to that
64+
mediation. The trusted Kubernetes driver selects the co-located backend for
65+
combined topology and supplies its topology descriptor to the supervisor.
66+
Sidecar topology remains on its pre-RFC lifecycle; a conforming backend for
67+
that placement requires separate design and implementation. Docker, Podman,
68+
and VM drivers provision the same co-located topology and supply its descriptor
69+
by default. The co-located backend requires the
70+
supervisor to own the execution environment's PID namespace so boundary
71+
teardown can terminate every remaining workload process.
72+
73+
For proxy-mode boundaries, the co-located backend verifies its default-deny
74+
kernel egress ceiling before exposing any workload execution surface and then
75+
rechecks it every 250 milliseconds. Each check has a two-second deadline.
76+
Verification failure or timeout ends the boundary and triggers process cleanup;
77+
the PID-1 supervisor exits so the kernel terminates the complete workload PID
78+
namespace. The topology's detection-and-termination bound is five seconds.
79+
5280
## Network and Inference
5381

5482
All ordinary agent egress is routed through the sandbox proxy. The proxy

‎crates/openshell-core/src/driver_mounts.rs‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ const RESERVED_MOUNT_TARGETS: &[&str] = &[
3030
"/etc/openshell",
3131
"/etc/openshell-tls",
3232
"/run/netns",
33+
"/proc",
3334
];
3435

3536
/// Validate a non-empty driver mount source.
@@ -118,9 +119,11 @@ pub fn validate_container_mount_target(target: &str) -> Result<(), String> {
118119
return Err("mount target '/sandbox' is reserved for the OpenShell workspace".to_string());
119120
}
120121
for reserved in RESERVED_MOUNT_TARGETS {
121-
if path_is_or_under(path, Path::new(reserved)) {
122+
let reserved = Path::new(reserved);
123+
if path_is_or_under(path, reserved) || path_is_or_under(reserved, path) {
122124
return Err(format!(
123-
"mount target '{target}' conflicts with reserved OpenShell path '{reserved}'"
125+
"mount target '{target}' conflicts with reserved OpenShell path '{}'",
126+
reserved.display()
124127
));
125128
}
126129
}
@@ -171,6 +174,24 @@ mod tests {
171174
assert!(err.contains("/etc/openshell"));
172175
}
173176

177+
#[test]
178+
fn container_target_rejects_parents_that_shadow_reserved_trees() {
179+
for target in ["/opt", "/etc", "/run"] {
180+
let err = validate_container_mount_target(target).unwrap_err();
181+
assert!(
182+
err.contains("reserved OpenShell path"),
183+
"expected {target} to be rejected: {err}"
184+
);
185+
}
186+
}
187+
188+
#[test]
189+
fn container_target_rejects_proc_shadowing() {
190+
for target in ["/proc", "/proc/self", "/"] {
191+
assert!(validate_container_mount_target(target).is_err());
192+
}
193+
}
194+
174195
#[test]
175196
fn container_target_does_not_prefix_match_unrelated_paths() {
176197
validate_container_mount_target("/etc/openshell-tools").unwrap();

‎crates/openshell-driver-docker/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ repository.workspace = true
1212

1313
[dependencies]
1414
openshell-core = { path = "../openshell-core", default-features = false }
15+
openshell-isolation = { path = "../openshell-isolation" }
1516

1617
tokio = { workspace = true }
1718
tonic = { workspace = true }

0 commit comments

Comments
 (0)