Skip to content

Commit 9f08093

Browse files
committed
feat(docker): mediate transparent host-supervisor networking
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
1 parent 9a77cad commit 9f08093

23 files changed

Lines changed: 1660 additions & 3289 deletions

File tree

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/compute-runtimes.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
228228

229229
| Runtime | Best fit | Sandbox boundary | Notes |
230230
|---|---|---|---|
231-
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
231+
| Docker | Local Linux development with Docker available. | Network-disabled workload container plus native host supervisor. | Uses RFC 0012 supervisor-owned creation and OCI seccomp notification to carry policy DNS and transparent TCP without a proxy port or an OpenShell binary in the workload. |
232232
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
233233
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
234234
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |

‎architecture/sandbox.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,14 @@ generation-pinned authorization form the transparent TCP security boundary.
139139
Docker and Podman do not currently advertise usable IPv6 egress for this
140140
substrate, so AAAA queries return NOERROR/NODATA and IPv6 DNS remains fenced.
141141

142+
Isolation backends may provide the same substrate without namespace listeners.
143+
Their DNS source carries bounded wire exchanges directly to policy DNS, and
144+
their connection source supplies the captured synthetic destination and calling
145+
binary identity with each stream. The Docker host-supervisor backend uses OCI
146+
seccomp user notification to inject those streams while Docker networking stays
147+
disabled. The supervisor therefore binds no workload-visible proxy or DNS port;
148+
the synthetic mapping and pinned-destination checks remain unchanged.
149+
142150
Provider credential placeholders are resolved through the live provider state
143151
for each HTTP request, after destination and L7 policy admission. A static
144152
credential resolves only when the request host, port, and path match an endpoint

‎crates/openshell-driver-docker/Cargo.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ toml = { workspace = true }
4141
tower-http = { workspace = true }
4242
http = { workspace = true }
4343
async-trait = "0.1"
44+
base64 = { workspace = true }
45+
uuid = { workspace = true }
4446

4547
[target.'cfg(target_os = "linux")'.dependencies]
4648
libc = "0.2"
@@ -54,7 +56,6 @@ prost-types = { workspace = true }
5456
tar = "0.4"
5557
temp-env = "0.3"
5658
tempfile = "3"
57-
uuid = { workspace = true }
5859
tracing-subscriber = { workspace = true }
5960

6061
[lints]
Lines changed: 53 additions & 192 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# openshell-driver-docker
22

3-
Docker-backed compute driver for local OpenShell gateways.
3+
Docker-backed compute and isolation driver for local Linux OpenShell gateways.
44

55
When the gateway configures `[openshell.gateway.otlp]`, Docker compute-driver
66
spans export to the same OTLP/gRPC collector with the service name
@@ -17,216 +17,77 @@ The standalone `openshell-driver-docker` binary accepts
1717
collector, continues W3C trace context from gateway RPC metadata, and flushes
1818
spans during graceful shutdown.
1919

20-
The driver manages sandbox containers through the local Docker daemon with the
21-
`bollard` client. It is intended for developer environments where Docker is
22-
already available and running Kubernetes would be unnecessary.
23-
2420
The driver connects to `[openshell.drivers.docker].socket_path` when configured.
25-
Otherwise, it uses the first standard local Docker socket that responds to an
26-
API ping, which is the same selection mechanism used by gateway auto-detection.
27-
An explicitly selected Docker driver falls back to `/var/run/docker.sock` when
28-
no candidate responds.
29-
30-
## Runtime Model
21+
Otherwise, it selects a standard local Docker socket and falls back to
22+
`/var/run/docker.sock` when Docker is explicitly enabled.
3123

32-
The gateway runs as a host process. The Docker driver creates one container per
33-
sandbox and starts the `openshell-sandbox` supervisor inside that container. The
34-
supervisor then creates the nested sandbox namespace for the agent process.
24+
## Runtime model
3525

36-
### Experimental host-supervisor creation backend
26+
Docker uses RFC 0012 supervisor-owned boundary creation. The compute driver
27+
resolves the immutable workload image and launches a native
28+
`openshell-sandbox` process on the gateway host. That supervisor registers the
29+
Docker `IsolationBackend`, creates the container, confirms enforcement, starts
30+
the workload, and exposes exec through the normal supervisor relay.
3731

38-
The Linux-only `isolation` module is a proof of RFC 0012's optional
39-
supervisor-owned `create` path. It is not selected by the normal Docker compute
40-
driver. A host-side harness can register `DockerIsolationBackend`, pass a
41-
resolved image through `DockerBoundaryCreatePlan`, and drive the common
42-
`Bound -> Ready -> Running` lifecycle.
32+
The workload container contains no OpenShell binary, gateway credential, TLS
33+
private key, or supervisor capability set. The dedicated mode runs the
34+
workload as the gateway user's numeric UID and GID so the unprivileged host
35+
supervisor can capture syscall arguments and hash the calling executable. It
36+
uses `/` as the initial working directory and does not provide driver mounts.
4337

44-
The proof binds a private host Unix socket before Docker creates the container,
38+
The backend binds a private host Unix socket before container creation and
4539
passes OCI seccomp `listenerPath`, `listenerMetadata`, and `SCMP_ACT_NOTIFY`
46-
through the custom Docker seccomp profile, creates the container stopped, and
47-
starts it only from `ReadyBoundary::start_agent`. runc sends the listener FD
48-
directly to the host over `SCM_RIGHTS`; the prototype denies notified syscalls
49-
with `EPERM`. The workload image contains no OpenShell supervisor binary.
50-
51-
The proof disables container networking and drops every capability. It does not
52-
yet implement mediated networking, binary identity, exec, port forwarding,
53-
running-boundary recovery, or the durable ownership record needed for complete
54-
cleanup. It therefore does not claim RFC 0012 conformance and does not change
55-
the production runtime model above.
56-
57-
Run the ignored local-daemon smoke after pre-pulling an Alpine-compatible image:
58-
59-
```shell
60-
OPENSHELL_DOCKER_POC_IMAGE=alpine:latest \
61-
cargo test -p openshell-driver-docker \
62-
local_docker_denies_notified_uname_without_in_container_supervisor \
63-
-- --ignored --nocapture
64-
```
40+
through the Docker seccomp profile. runc sends the listener FD directly to the
41+
host supervisor with `SCM_RIGHTS` before the workload starts. The container
42+
drops every Linux capability and has direct networking disabled.
6543

66-
## Stop and Start
67-
68-
Stop stops the managed container without removing it. Docker retains the
69-
container writable layer, attached volumes, labels, token material, and restart
70-
policy. Start starts that same container, so files in the resolved OCI
71-
workspace remain available. A durably stopped sandbox is excluded from
72-
gateway startup recovery and stays stopped across gateway restarts. Delete
73-
continues to force-remove the container and clean up driver-owned material.
74-
Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
75-
phase requires running compute without changing that persisted intent. On
76-
startup, the gateway sends an idempotent `StartSandbox` request for the same
77-
sandboxes, restarting their retained containers. Explicitly stopped sandboxes
78-
remain excluded.
79-
80-
Before creating the container, the driver inspects the final sandbox image and
81-
captures its immutable image ID, raw OCI `Config.User`, and OCI
82-
`Config.WorkingDir`. Container creation uses that image ID, preventing a
83-
mutable tag from changing between inspection and launch. The supervisor runs as
84-
root, resolves omitted policy identity fields from the image declaration, and
85-
drops only agent children to the resulting identity. Named OCI components
86-
remain names after validation; a missing group is filled with the user's
87-
numeric primary GID. Explicit `process.run_as_user` and
88-
`process.run_as_group` values take precedence independently.
89-
90-
An absolute OCI working directory becomes the agent workspace. An empty,
91-
root (`/`), or explicit `/sandbox` declaration uses `/sandbox`, which OpenShell
92-
creates when necessary and owns as a compatibility workspace. Any other image
93-
workdir must already exist without symlink components. The completed identity,
94-
including supplementary groups, must already be able to traverse every parent
95-
and write and enter the workdir. OpenShell does not change its ownership or
96-
mode.
97-
98-
OpenShell deliberately asks the Linux kernel to make this access decision
99-
under the completed sandbox identity instead of reproducing permission rules
100-
from ownership and mode bits. Mode-bit inspection alone can reject authority
101-
granted by a POSIX ACL or overlook a denial imposed by a Linux Security Module
102-
such as SELinux or AppArmor. OpenShell does not configure or otherwise manage
103-
ACLs or LSM policy here; the one-shot validator only observes the kernel's
104-
effective decision. This keeps the no-authority-expansion invariant aligned
105-
with the access the eventual workload will receive without adding a separate,
106-
incomplete permission model to OpenShell.
107-
108-
Image `VOLUME` declarations must not cover the workdir or one of its parents
109-
because Docker would mount the volume before the supervisor could validate the
110-
immutable image path.
111-
Workdirs under the standard OCI runtime namespaces `/proc`, `/sys`, and `/dev`
112-
are rejected, as are paths that overlap concrete OpenShell control resources.
113-
The workspace is the child cwd and `HOME`. The supervisor starts from `/`, then
114-
reports an invalid workdir as a readiness failure.
115-
116-
Docker containers join an OpenShell-managed bridge network. The driver injects
117-
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
118-
names for reaching the gateway host. On Docker Desktop, Colima, Rancher
119-
Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's
120-
`host-gateway` alias. The driver requests a separate IPv4 loopback callback
121-
listener when the primary listener does not already cover it. On native Linux
122-
Docker, the gateway also binds the bridge gateway IP so containers can call
123-
back to the host process.
124-
125-
## Container Contract
126-
127-
The driver-controlled container settings are part of the sandbox security
128-
contract:
129-
130-
| Setting | Purpose |
131-
|---|---|
132-
| `user = "0"` | The supervisor needs root inside the container to prepare namespaces, mounts, Landlock, and seccomp. |
133-
| `network_mode = openshell` | Places the supervisor on the managed Docker bridge network. |
134-
| `cap_add` | Grants supervisor-only capabilities required for namespace setup and process inspection. |
135-
| `apparmor=unconfined` | Avoids Docker's default profile blocking required mount operations. |
136-
| `restart_policy = no` | A canonical main-process exit remains terminal and is not silently restarted by Docker. |
137-
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. |
138-
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
139-
| `policy-dns-transparent-tcp` capability | Declares that the combined Docker supervisor can own namespace-local DNS/TCP capture and coupled workload restart. The shared supervisor still owns DNS eligibility, mappings, authorization, pinned dialing, relaying, and OCSF decisions. The marker is stripped from the workload environment. |
140-
141-
The agent child process does not retain these supervisor privileges.
142-
143-
## Driver Config Mounts
144-
145-
The gateway forwards the `docker` block from `--driver-config-json` to this
146-
driver. The driver accepts user-supplied `mounts` entries with these Docker
147-
mount types:
148-
149-
- `bind`: mounts an absolute host path when `[openshell.drivers.docker]`
150-
has `enable_bind_mounts = true`.
151-
- `volume`: mounts an existing Docker named volume. The driver validates that
152-
the volume exists before provisioning and never creates or removes it.
153-
Docker local-driver volumes created with bind options are treated as host
154-
bind mounts and require `enable_bind_mounts = true`.
155-
- `tmpfs`: mounts an in-memory filesystem with optional `options`,
156-
`size_bytes`, and `mode`.
157-
158-
Host bind mounts are disabled by default because they expose gateway host
159-
paths to sandbox requests. Image mounts are not part of the Docker
160-
driver-config schema. The driver still uses internal bind mounts for
161-
OpenShell-owned supervisor, token, and TLS material.
162-
163-
Docker `bind` mounts accept `source`, `target`, optional `read_only`, and an
164-
optional `selinux_label` of `shared` (applies `:z`) or `private` (applies
165-
`:Z`) for SELinux-enforcing hosts. Docker `volume` mounts may include
166-
`subpath`. User-supplied bind and volume mounts are read-only by default; set
167-
`read_only: false` to make them writable. Mount `source`, `target`, and
168-
`subpath` values must not contain surrounding whitespace. Mount targets must be
169-
absolute container paths and must not replace or contain the resolved workspace
170-
root. Nested workspace mounts remain valid. Mounts also must not overlap the
171-
configured SSH socket or the reserved `/opt/openshell`, `/etc/openshell`,
172-
`/etc/openshell-tls`, `/run/openshell`, `/run/openshell-sidecar`, and network
173-
namespace roots.
174-
175-
Example named-volume usage:
44+
The listener injects connected sockets for workload TCP and DNS operations.
45+
DNS queries travel directly to the supervisor policy-DNS service and return
46+
short-lived synthetic addresses. A later connection to a synthetic address is
47+
bound to the queried hostname, calling-binary identity, allowed port, policy
48+
generation, and pinned real addresses. The supervisor consumes these streams
49+
without binding a container-visible proxy port or setting proxy environment
50+
variables. Direct real-IP and non-mediated UDP traffic fail closed.
17651

177-
```shell
178-
docker volume create openshell-work
52+
For TLS termination, the backend read-only mounts the generated public CA and
53+
combined public trust bundle. CA private key material remains in host
54+
supervisor memory.
17955

180-
openshell sandbox create \
181-
--driver-config-json '{"docker":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \
182-
-- claude
183-
```
56+
The current implementation supports create, confirm, start, wait,
57+
signal-main-process, delete, Docker exec, policy DNS, transparent TCP, and TLS
58+
termination. Port forwarding, exec signaling, GPU devices, resource limits,
59+
driver mounts, images that require root, and durable running-boundary recovery
60+
remain unsupported and fail closed.
18461

185-
## Supervisor Binary Resolution
62+
## Supervisor binary resolution
18663

187-
The Docker driver bind-mounts a host-side Linux `openshell-sandbox` binary into
188-
each sandbox container. Resolution order is:
64+
The native host supervisor is resolved in this order:
18965

19066
1. `supervisor_bin` in `[openshell.drivers.docker]`.
19167
2. `supervisor_image` in `[openshell.drivers.docker]`, extracting
192-
`/openshell-sandbox` from that image.
193-
3. A sibling `openshell-sandbox` next to the running `openshell-gateway` binary.
68+
`/openshell-sandbox` to a host cache.
69+
3. A sibling `openshell-sandbox` next to `openshell-gateway`.
19470
4. A local Linux cargo target build for the Docker daemon architecture.
195-
5. The release-matched default supervisor image, extracting `/openshell-sandbox`.
196-
197-
Release and Docker-image gateway builds bake the matching supervisor image tag
198-
into the binary at compile time. The default Docker supervisor image is not
199-
`:latest` unless a custom build explicitly sets that tag.
200-
201-
## Callback and TLS
71+
5. The release-matched default supervisor image.
20272

203-
`OPENSHELL_ENDPOINT` is injected from the gateway's configured gRPC endpoint.
204-
When no endpoint is configured, the driver uses
205-
`host.openshell.internal:<gateway-port>` with the appropriate HTTP or HTTPS
206-
scheme. Set `host_gateway_ip` only when the host has an explicit, locally
207-
assigned address that containers should use for callbacks; package-managed
208-
macOS gateways should leave it unset.
73+
The resolved binary executes on the host; it is never mounted into the
74+
workload container.
20975

210-
For HTTPS endpoints, the server certificate must include the endpoint host as a
211-
subject alternative name. Docker sandboxes also need the client TLS bundle
212-
mounted into the container and exposed with:
76+
## Gateway authentication
21377

214-
- `OPENSHELL_TLS_CA`
215-
- `OPENSHELL_TLS_CERT`
216-
- `OPENSHELL_TLS_KEY`
78+
The compute driver writes the sandbox JWT to a host-only state directory and
79+
passes that path only to the native supervisor. HTTPS CA, certificate, and key
80+
paths likewise remain on the host. The supervisor connects to the gateway over
81+
the configured `grpc_endpoint`; host aliases such as
82+
`host.openshell.internal` are normalized to loopback for the native process.
21783

218-
HTTP endpoints reject TLS material because the supervisor would not use it.
84+
## Testing
21985

220-
## Environment Ownership
86+
The standard Docker runner exercises this implementation:
22187

222-
The driver merges template environment and sandbox spec environment first, then
223-
overwrites security-critical keys:
224-
225-
- `OPENSHELL_ENDPOINT`
226-
- `OPENSHELL_SANDBOX_ID`
227-
- `OPENSHELL_SANDBOX`
228-
- `OPENSHELL_SSH_SOCKET_PATH`
229-
- `OPENSHELL_MAIN_PROCESS_SPEC`
230-
- TLS path variables when HTTPS is enabled
88+
```shell
89+
mise run e2e:docker
90+
```
23191

232-
Do not allow sandbox images or templates to override these values.
92+
Set `OPENSHELL_E2E_SANDBOX_IMAGE` to test another workload image supported by
93+
the smoke scenario.

0 commit comments

Comments
 (0)