Skip to content

Commit f3fd9d9

Browse files
committed
feat(docker): adopt shared supervisor boundary mode
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
1 parent 9291888 commit f3fd9d9

5 files changed

Lines changed: 1031 additions & 184 deletions

File tree

‎crates/openshell-driver-docker/Cargo.toml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ path = "src/main.rs"
1616

1717
[dependencies]
1818
openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] }
19+
openshell-isolation-interface = { path = "../openshell-isolation-interface" }
1920
openshell-otel = { path = "../openshell-otel" }
2021

2122
opentelemetry = { workspace = true }
@@ -38,15 +39,17 @@ miette = { workspace = true }
3839
toml = { workspace = true }
3940
tower-http = { workspace = true }
4041
http = { workspace = true }
42+
rand = { workspace = true }
43+
sha2 = { workspace = true }
44+
tar = "0.4"
45+
tempfile = "3"
4146

4247
[dev-dependencies]
4348
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
4449
opentelemetry = { workspace = true }
4550
opentelemetry_sdk = { workspace = true, features = ["testing"] }
4651
prost-types = { workspace = true }
47-
tar = "0.4"
4852
temp-env = "0.3"
49-
tempfile = "3"
5053
tracing-subscriber = { workspace = true }
5154

5255
[lints]

‎crates/openshell-driver-docker/README.md‎

Lines changed: 53 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -29,18 +29,31 @@ no candidate responds.
2929

3030
## Runtime Model
3131

32-
The gateway runs as a host process. The Docker driver creates one container per
33-
sandbox and starts the `openshell-sandbox` supervisor inside that container. The
34-
supervisor then creates the nested sandbox namespace for the agent process.
32+
The gateway and Docker driver run on the host. For each sandbox, the driver
33+
starts `openshell-sandbox --mode=control` beside the gateway and
34+
`openshell-sandbox --mode=boundary` as PID 1 in the workload container. The two
35+
roles use the shared RFC 0012 protocol over a private bind-mounted Unix socket.
36+
37+
Control owns the gateway session, policy, SSH relay, and network proxy. The
38+
boundary owns the process tree, exec/signal/wait/PTY operations, loopback port
39+
forwarding, procfs binary identity, and the workload network namespace. The
40+
boundary receives no gateway JWT, client TLS private key, or policy authority.
41+
42+
The driver bind-mounts the boundary socket from a short, digest-keyed runtime
43+
directory under `/tmp/openshell-docker-boundaries`. This keeps the host endpoint
44+
below Linux's Unix-socket path limit even when the gateway state directory is
45+
deeply nested. The full protected topology and gateway material remain in the
46+
normal driver state directory, and sandbox cleanup removes both locations.
3547

3648
## Stop and Start
3749

38-
Stop stops the managed container without removing it. Docker retains the
39-
container writable layer, attached volumes, labels, token material, and restart
40-
policy. Start starts that same container, so files in the resolved OCI
41-
workspace remain available. A durably stopped sandbox is excluded from
42-
gateway startup recovery and stays stopped across gateway restarts. Delete
43-
continues to force-remove the container and clean up driver-owned material.
50+
Stop terminates host control and stops the managed container without removing
51+
it. Docker retains the container writable layer, attached volumes, labels, and
52+
the protected driver topology. Start starts that same container and recreates
53+
host control, so files in the resolved OCI workspace remain available. A
54+
durably stopped sandbox is excluded from gateway startup recovery and stays
55+
stopped across gateway restarts. Delete force-removes the container and the
56+
driver-owned boundary directory.
4457
Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
4558
phase requires running compute without changing that persisted intent. On
4659
startup, the gateway sends an idempotent `StartSandbox` request for the same
@@ -50,9 +63,9 @@ remain excluded.
5063
Before creating the container, the driver inspects the final sandbox image and
5164
captures its immutable image ID, raw OCI `Config.User`, and OCI
5265
`Config.WorkingDir`. Container creation uses that image ID, preventing a
53-
mutable tag from changing between inspection and launch. The supervisor runs as
54-
root, resolves omitted policy identity fields from the image declaration, and
55-
drops only agent children to the resulting identity. Named OCI components
66+
mutable tag from changing between inspection and launch. The boundary runs as
67+
root, resolves omitted policy identity fields from the image declaration inside
68+
the image filesystem, and drops only agent children to the resulting identity. Named OCI components
5669
remain names after validation; a missing group is filled with the user's
5770
numeric primary GID. Explicit `process.run_as_user` and
5871
`process.run_as_group` values take precedence independently.
@@ -83,14 +96,12 @@ are rejected, as are paths that overlap concrete OpenShell control resources.
8396
The workspace is the child cwd and `HOME`. The supervisor starts from `/`, then
8497
reports an invalid workdir as a readiness failure.
8598

86-
Docker containers join an OpenShell-managed bridge network. The driver injects
87-
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
88-
names for reaching the gateway host. On Docker Desktop, Colima, Rancher
89-
Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's
90-
`host-gateway` alias. The driver requests a separate IPv4 loopback callback
91-
listener when the primary listener does not already cover it. On native Linux
92-
Docker, the gateway also binds the bridge gateway IP so containers can call
93-
back to the host process.
99+
Boundary containers use Docker `network_mode=none`. The nested agent network
100+
namespace can reach only the boundary's mediation listener. Each accepted TCP
101+
stream and its procfs-derived executable identity cross the private Unix socket
102+
to host control, which applies policy before dialing the destination. Host
103+
control reaches the gateway directly; the existing managed bridge listener is
104+
retained for compatibility with legacy containers created before this split.
94105

95106
## Container Contract
96107

@@ -100,13 +111,15 @@ contract:
100111
| Setting | Purpose |
101112
|---|---|
102113
| `user = "0"` | The supervisor needs root inside the container to prepare namespaces, mounts, Landlock, and seccomp. |
103-
| `network_mode = openshell` | Places the supervisor on the managed Docker bridge network. |
114+
| `network_mode = none` | Removes direct container egress; the nested workload namespace can emit only through the authenticated boundary stream. |
104115
| `cap_add` | Grants supervisor-only capabilities required for namespace setup and process inspection. |
105116
| `apparmor=unconfined` | Avoids Docker's default profile blocking required mount operations. |
106117
| `restart_policy = no` | A canonical main-process exit remains terminal and is not silently restarted by Docker. |
107118
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. |
108119
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
109-
| `policy-dns-transparent-tcp` capability | Declares that the combined Docker supervisor can own namespace-local DNS/TCP capture and coupled workload restart. The shared supervisor still owns DNS eligibility, mappings, authorization, pinned dialing, relaying, and OCSF decisions. The marker is stripped from the workload environment. |
120+
| Read-only supervisor/runtime mounts | Supplies the boundary binary and trusted `ip`/`nft` helper runtime without trusting workload-image tools. |
121+
| Private boundary bind mount | Carries the protected boundary config and Unix control socket. The directory is host-owned mode `0700`; workload children cannot access it. |
122+
| `policy-dns-transparent-tcp` capability | Declares that the shared boundary can own namespace-local DNS/TCP capture while host control retains authorization, pinned dialing, relaying, and OCSF decisions. The marker is not exposed to the workload environment. |
110123

111124
The agent child process does not retain these supervisor privileges.
112125

@@ -127,8 +140,9 @@ mount types:
127140

128141
Host bind mounts are disabled by default because they expose gateway host
129142
paths to sandbox requests. Image mounts are not part of the Docker
130-
driver-config schema. The driver still uses internal bind mounts for
131-
OpenShell-owned supervisor, token, and TLS material.
143+
driver-config schema. The driver still uses internal bind mounts for the
144+
OpenShell-owned boundary binary, trusted helper runtime, and private socket
145+
directory. Gateway token and TLS material remain on the host.
132146

133147
Docker `bind` mounts accept `source`, `target`, optional `read_only`, and an
134148
optional `selinux_label` of `shared` (applies `:z`) or `private` (applies
@@ -154,8 +168,9 @@ openshell sandbox create \
154168

155169
## Supervisor Binary Resolution
156170

157-
The Docker driver bind-mounts a host-side Linux `openshell-sandbox` binary into
158-
each sandbox container. Resolution order is:
171+
The Docker driver bind-mounts a host-side Linux `openshell-sandbox` boundary
172+
binary and trusted helper runtime into each sandbox container. Resolution order
173+
for the Linux artifact is:
159174

160175
1. `supervisor_bin` in `[openshell.drivers.docker]`.
161176
2. `supervisor_image` in `[openshell.drivers.docker]`, extracting
@@ -164,22 +179,28 @@ each sandbox container. Resolution order is:
164179
4. A local Linux cargo target build for the Docker daemon architecture.
165180
5. The release-matched default supervisor image, extracting `/openshell-sandbox`.
166181

182+
When a selected binary has no valid sibling `openshell-runtime`, the driver
183+
extracts `/openshell-runtime` from the configured or release-matched supervisor
184+
image. Host control uses the native `openshell-sandbox` beside the gateway (or a
185+
local native cargo build); Linux hosts can reuse the boundary binary.
186+
167187
Release and Docker-image gateway builds bake the matching supervisor image tag
168188
into the binary at compile time. The default Docker supervisor image is not
169189
`:latest` unless a custom build explicitly sets that tag.
170190

171191
## Callback and TLS
172192

173-
`OPENSHELL_ENDPOINT` is injected from the gateway's configured gRPC endpoint.
193+
`OPENSHELL_ENDPOINT` is injected only into host control from the gateway's
194+
configured gRPC endpoint.
174195
When no endpoint is configured, the driver uses
175196
`host.openshell.internal:<gateway-port>` with the appropriate HTTP or HTTPS
176197
scheme. Set `host_gateway_ip` only when the host has an explicit, locally
177198
assigned address that containers should use for callbacks; package-managed
178199
macOS gateways should leave it unset.
179200

180201
For HTTPS endpoints, the server certificate must include the endpoint host as a
181-
subject alternative name. Docker sandboxes also need the client TLS bundle
182-
mounted into the container and exposed with:
202+
subject alternative name. The client TLS bundle is exposed only to host control
203+
with:
183204

184205
- `OPENSHELL_TLS_CA`
185206
- `OPENSHELL_TLS_CERT`
@@ -189,8 +210,9 @@ HTTP endpoints reject TLS material because the supervisor would not use it.
189210

190211
## Environment Ownership
191212

192-
The driver merges template environment and sandbox spec environment first, then
193-
overwrites security-critical keys:
213+
The driver writes template and sandbox environment into the protected boundary
214+
config. Boundary mode exposes that map only to workload children. Host control
215+
owns these security-critical keys:
194216

195217
- `OPENSHELL_ENDPOINT`
196218
- `OPENSHELL_SANDBOX_ID`
Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
//! Docker provisioning for the shared authenticated boundary protocol.
5+
//!
6+
//! Docker owns only the container/socket topology and immutable OCI resource
7+
//! claims. Lifecycle, process, network, identity, and wire behavior live in
8+
//! `openshell-isolation-interface` and `openshell-sandbox`.
9+
10+
use std::collections::{BTreeMap, HashMap};
11+
use std::net::IpAddr;
12+
use std::path::PathBuf;
13+
14+
use openshell_isolation_interface::boundary_protocol::{
15+
BOUNDARY_PROTOCOL_VERSION, BoundaryAgentIdentity, BoundaryConfig, BoundaryListener,
16+
BoundaryTopology, BoundaryTransport,
17+
};
18+
19+
/// Driver-owned inputs that bind one Docker container to one boundary.
20+
pub struct DockerBoundarySpec {
21+
pub boundary_id: String,
22+
pub bootstrap_token: String,
23+
pub container_id: String,
24+
pub image_identity: String,
25+
pub listener_socket: PathBuf,
26+
pub control_socket: PathBuf,
27+
pub host_gateway_ip: Option<IpAddr>,
28+
pub oci_user: String,
29+
pub trusted_runtime_root: PathBuf,
30+
pub child_env: HashMap<String, String>,
31+
}
32+
33+
/// Protected container config and matching host descriptor.
34+
pub struct DockerBoundaryProvisioning {
35+
pub boundary_config: BoundaryConfig,
36+
pub topology: BoundaryTopology,
37+
}
38+
39+
impl DockerBoundarySpec {
40+
/// Produce both sides of the common protocol from the same immutable
41+
/// Docker coordinates so attach cannot bind a different container.
42+
#[must_use]
43+
pub fn provision(self) -> DockerBoundaryProvisioning {
44+
let resource_claims = BTreeMap::from([
45+
("docker.container_id".to_string(), self.container_id),
46+
("docker.image_identity".to_string(), self.image_identity),
47+
]);
48+
DockerBoundaryProvisioning {
49+
boundary_config: BoundaryConfig {
50+
protocol_version: BOUNDARY_PROTOCOL_VERSION,
51+
boundary_id: self.boundary_id.clone(),
52+
bootstrap_token: self.bootstrap_token.clone(),
53+
listener: BoundaryListener::Unix {
54+
socket_path: self.listener_socket,
55+
},
56+
resource_claims: resource_claims.clone(),
57+
agent_identity: BoundaryAgentIdentity::OciUser {
58+
declaration: self.oci_user,
59+
},
60+
protect_config_file: true,
61+
trusted_runtime_root: self.trusted_runtime_root,
62+
child_env: self.child_env,
63+
},
64+
topology: BoundaryTopology {
65+
protocol_version: BOUNDARY_PROTOCOL_VERSION,
66+
boundary_id: self.boundary_id,
67+
transport: BoundaryTransport::Unix {
68+
socket_path: self.control_socket,
69+
},
70+
host_gateway_ip: self.host_gateway_ip,
71+
resource_claims,
72+
bootstrap_token: self.bootstrap_token,
73+
},
74+
}
75+
}
76+
}
77+
78+
#[cfg(test)]
79+
mod tests {
80+
use super::*;
81+
82+
#[test]
83+
fn provisioning_binds_container_and_image_claims() {
84+
let provisioned = DockerBoundarySpec {
85+
boundary_id: "sandbox-1".to_string(),
86+
bootstrap_token: "a".repeat(64),
87+
container_id: "sha256:container".to_string(),
88+
image_identity: "sha256:image".to_string(),
89+
listener_socket: PathBuf::from("/run/openshell/boundary/control.sock"),
90+
control_socket: PathBuf::from("/host/control.sock"),
91+
host_gateway_ip: Some(IpAddr::from([127, 0, 0, 1])),
92+
oci_user: "1000:1000".to_string(),
93+
trusted_runtime_root: PathBuf::from("/opt/openshell/bin/openshell-runtime"),
94+
child_env: HashMap::new(),
95+
}
96+
.provision();
97+
98+
assert_eq!(
99+
provisioned.boundary_config.resource_claims,
100+
provisioned.topology.resource_claims
101+
);
102+
assert_eq!(
103+
provisioned.topology.resource_claims["docker.container_id"],
104+
"sha256:container"
105+
);
106+
}
107+
}

0 commit comments

Comments
 (0)