Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,10 @@ rationale, configured and effective modes, active generation, and the explicit
```bash
docker info
docker ps --filter name=openshell
docker logs <container> --tail=200
docker inspect <container> --format '{{json .Config.Labels}} {{json .State}}'
find "${XDG_STATE_HOME:-$HOME/.local/state}/openshell/docker-sandbox-tokens" \
-name 'supervisor*.log' -type f -print
tail -n 200 "${XDG_STATE_HOME:-$HOME/.local/state}/openshell/docker-sandbox-tokens"/*/*/supervisor.err.log
docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version
openshell status
```
Expand Down Expand Up @@ -184,7 +187,8 @@ Common findings:
- Docker also rejects an image `VOLUME` that covers the workdir or one of its parents because the runtime would mask the immutable path before validation. Move the `VOLUME` below the workspace or remove the declaration.
- A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error.
- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`.
- Sandbox never registers: check gateway logs and supervisor callback endpoint.
- Docker attach fails before readiness: inspect the native supervisor error log for topology-descriptor validation, stopped-container conflicts, or OCI seccomp listener setup. The descriptor carries compute-driver-prepared inputs; `attach` establishes or locates the stopped container, while the compute driver retains deletion and reconciliation ownership.
- Sandbox never registers: check gateway logs, the native host-supervisor logs under the Docker sandbox-token state directory, and the supervisor callback endpoint. The workload container does not contain the supervisor.
- On macOS, repeated `Policy fetch failed after 5 attempts` messages with a
Homebrew gateway bound to `[::1]:17670` indicate that the Docker
`host-gateway` IPv4 route has no matching callback listener. Current releases
Expand Down
18 changes: 18 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.

| Runtime | Best fit | Sandbox boundary | Notes |
|---|---|---|---|
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
| Docker | Local Linux development with Docker available. | Network-disabled workload container plus native host supervisor. | Uses RFC 0012 attach-time boundary establishment and OCI seccomp notification to carry policy DNS and transparent TCP without a proxy port or an OpenShell binary in the workload. |
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
Expand Down
15 changes: 13 additions & 2 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,11 @@ enforcement is confirmed.
The logical supervisor remains the trusted bridge between the gateway and the
workload. It drives the backend and applies approved network policy through
supervisor-owned mediation; the backend routes workload egress to that
mediation. Existing container placements remain on their legacy lifecycle
while the VM driver prototypes the backend contract.
mediation. A trusted topology descriptor can identify an existing resource or
carry prepared inputs that the backend uses to establish the boundary during
`attach`. The compute driver retains durable deletion and reconciliation
ownership. Podman and Kubernetes remain on their legacy lifecycles while the
Docker and VM drivers prototype the backend contract.

VM uses delegated placement. The logical supervisor stays on
the host and sends admitted policy, workload state, and proxy CA material over
Expand Down Expand Up @@ -139,6 +142,14 @@ generation-pinned authorization form the transparent TCP security boundary.
Docker and Podman do not currently advertise usable IPv6 egress for this
substrate, so AAAA queries return NOERROR/NODATA and IPv6 DNS remains fenced.

Isolation backends may provide the same substrate without namespace listeners.
Their DNS source carries bounded wire exchanges directly to policy DNS, and
their connection source supplies the captured synthetic destination and calling
binary identity with each stream. The Docker host-supervisor backend uses OCI
seccomp user notification to inject those streams while Docker networking stays
disabled. The supervisor therefore binds no workload-visible proxy or DNS port;
the synthetic mapping and pinned-destination checks remain unchanged.

Provider credential placeholders are resolved through the live provider state
for each HTTP request, after destination and L7 policy admission. A static
credential resolves only when the request host, port, and path match an endpoint
Expand Down
9 changes: 9 additions & 0 deletions crates/openshell-driver-docker/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] }
openshell-isolation = { path = "../openshell-isolation" }
openshell-otel = { path = "../openshell-otel" }

opentelemetry = { workspace = true }
Expand All @@ -30,6 +31,7 @@ tracing-subscriber = { workspace = true }
bytes = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
sha2 = { workspace = true }
prost-types = { workspace = true }
bollard = { version = "0.20" }
url = { workspace = true }
Expand All @@ -38,6 +40,13 @@ miette = { workspace = true }
toml = { workspace = true }
tower-http = { workspace = true }
http = { workspace = true }
async-trait = "0.1"
base64 = { workspace = true }
uuid = { workspace = true }

[target.'cfg(target_os = "linux")'.dependencies]
libc = "0.2"
nix = { workspace = true, features = ["socket", "uio"] }

[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
Expand Down
Loading
Loading