You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9291888
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: crates/openshell-driver-vm/README.md
+39-28Lines changed: 39 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,41 +1,48 @@
1
1
# openshell-driver-vm
2
2
3
-
> Status: Experimental. The VM compute driver is under active development and the interface still has VM-specific plumbing that will be generalized.
3
+
> Status: Experimental. The VM compute driver is under active development.
4
4
5
-
Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess, talks to it over a Unix domain socket with the `openshell.compute.v1.ComputeDriver` gRPC surface, and lets it manage per-sandbox microVMs. The runtime (libkrun + libkrunfw + gvproxy), guest OCI unpacker, and sandbox supervisor are embedded directly in the binary; each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir.
5
+
Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess and talks to it over the `openshell.compute.v1.ComputeDriver` Unix-socket surface. `openshell-sandbox --mode=control` runs as a native host process. The same binary runs as `--mode=boundary` inside each microVM and applies guest-local isolation through the shared RFC 0012 boundary protocol over virtio-vsock.
6
+
7
+
The driver still embeds libkrun, libkrunfw, gvproxy, the guest OCI unpacker, the portable guest leaf, and the existing custom kernel runtime. Each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir.
Sandbox guests execute `/opt/openshell/bin/openshell-sandbox` as PID 1 inside the VM. gvproxy exposes a single inbound SSH port (`host:<allocated>` → `guest:2222`) and provides virtio-net egress.
35
+
The control role owns gateway credentials, admitted policy, provider resolution, middleware, the network proxy, and relay registration. The boundary role receives no gateway JWT or mTLS key. It receives a driver-authored boundary token and only the policy/environment state needed to launch the workload.
36
+
37
+
VM-specific RFC 0012 code under `src/isolation/` only chooses the vsock transport and binds immutable VM generation and image claims into the protected guest config and host descriptor. Lifecycle, authentication, process control, binary identity, forwarding, and streaming come from `openshell-isolation-interface` and `openshell-sandbox`.
31
38
32
39
## Quick start (recommended)
33
40
34
41
```shell
35
42
mise run gateway:vm
36
43
```
37
44
38
-
First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/gvproxy/umoci and `mise run vm:supervisor` builds the bundled guest supervisor. Subsequent runs are cached.
45
+
First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/gvproxy/umoci and `mise run vm:supervisor` builds the portable Linux guest leaf plus its trusted helper runtime. The development task also builds the native host supervisor. Subsequent runs are cached.
@@ -142,7 +149,7 @@ Select the VM driver with `--drivers vm`, `OPENSHELL_DRIVERS=vm`, or `compute_dr
142
149
143
150
| Configuration key | Default | Purpose |
144
151
|---|---|---|
145
-
|`grpc_endpoint`| empty | Required. URL the sandbox guest dials to reach the gateway. Use `http://host.containers.internal:<port>` (or `host.docker.internal` / `host.openshell.internal`) so traffic flows through gvproxy's host-loopback NAT (HostIP `192.168.127.254` → host `127.0.0.1`). Loopback URLs like `http://127.0.0.1:<port>` are rewritten automatically by the driver. The bare gateway IP (`192.168.127.1`) only carries gvproxy's own services and will not reach host-bound ports. |
152
+
|`grpc_endpoint`| empty | Required. URL the native host supervisor uses to reach the gateway. Host loopback such as `http://127.0.0.1:<port>`is valid. This endpoint is never sent into the VM. |
|`guest_tls_ca`|unset|Historical key name for the host supervisor's gateway CA certificate. Required when `grpc_endpoint` uses `https://`; never copied into the guest. |
162
+
| `guest_tls_cert` | unset | Historical key name for the host supervisor's client certificate; never copied into the guest. |
163
+
|`guest_tls_key`|unset|Historical key name for the host supervisor's client private key; never copied into the guest. |
157
164
158
165
See [`openshell-gateway --help`](../openshell-server/src/cli.rs) for the gateway process flag surface.
The VM guest's serial console is appended to `<state-dir>/<sandbox-id>/console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions and removes same-owner stale sockets. On clean shutdown, the gateway sends the managed driver `SIGTERM`, waits up to five seconds for it to flush telemetry and exit, then force-kills it if necessary and removes the socket. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.
235
+
The VM serial console is appended to `<state-dir>/sandboxes/<sandbox-id>/rootfs-console.log`. Host-supervisor stdout and stderr are written beside it as `supervisor.log` and `supervisor.err.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions, removes same-owner stale sockets, and the gateway removes the socket on clean shutdown via `ManagedDriverProcess::drop`. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.
228
236
229
237
## Host-side nftables rules
230
238
231
-
The VM driver creates a per-VM nftables table on the host (`openshell_vm_vmtap_<id>`) with three chains. These rules serve two purposes: NAT infrastructure (required for VM connectivity) and defense-in-depth host isolation. Primary security enforcement — proxy-only egress and bypass detection — is handled by the sandbox supervisor's own nftables rules inside the VM guest.
239
+
This section applies to the QEMU/VFIO path, which uses a host TAP device. The normal libkrun path uses gvproxy and needs KVM access but does not require `CAP_NET_ADMIN`. The host supervisor performs primary policy enforcement and receives guest-originated connections through the authenticated network-mediation stream.
240
+
241
+
The QEMU path creates a per-VM nftables table on the host (`openshell_vm_vmtap_<id>`) with three chains for NAT infrastructure and defense-in-depth host isolation.
232
242
233
243
**`postrouting` (NAT):** Masquerades outbound VM traffic so it can be routed from the VM's private subnet to the external network. This chain handles forwarded traffic (VM → internet), not traffic destined for the host.
234
244
235
-
**`forward` (defense-in-depth):** Accepts all outbound traffic from the VM (security enforcement happens guest-side) and accepts established/related response traffic back to the VM. Drops unsolicited inbound connections to the VM from the broader network. This chain handles forwarded traffic only — packets transiting the host between the TAP interface and other interfaces.
245
+
**`forward` (defense-in-depth):** Accepts outbound traffic from the VM and established/related response traffic back to the VM. Drops unsolicited inbound connections to the VM from the broader network. This chain handles forwarded traffic only — packets transiting the host between the TAP interface and other interfaces.
236
246
237
247
**`input` (defense-in-depth):** Accepts traffic from the VM to the gateway port on the host. Drops all other traffic from the VM destined for the host itself. This limits what a compromised guest can reach on the host to the gateway service only.
238
248
@@ -250,9 +260,9 @@ Each table is created atomically via `nft -f` on VM start and torn down atomical
250
260
- macOS on Apple Silicon, or Linux on aarch64/x86_64 with KVM
251
261
- Rust toolchain
252
262
- e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk image creation and QEMU environment injection
253
-
- Guest-supervisor cross-compile toolchain (needed on macOS, and on Linux when host arch ≠ guest arch):
263
+
- Guest-leaf cross-compile toolchain (needed on macOS, and on Linux when host arch differs from the guest):
254
264
- Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or `x86_64-unknown-linux-gnu` for an amd64 guest)
255
-
- `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` uses `cargo zigbuild` to cross-compile the in-VM`openshell-sandbox` supervisor binary.
265
+
- `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` uses `cargo zigbuild` to cross-compile the Linux `openshell-sandbox --mode=boundary` binary.
256
266
- [mise](https://mise.jdx.dev/) task runner
257
267
- Docker or Podman socket on the local CLI/gateway host when using
258
268
`openshell sandbox create --from ./Dockerfile` or `--from ./dir`; the CLI
@@ -283,11 +293,12 @@ The RPM gateway package is configured for the Podman driver.
283
293
284
294
On Apple Silicon macOS, `install.sh` stages the generated `openshell.rb`
285
295
formula from the selected release in the `nvidia/openshell` Homebrew tap.
286
-
Homebrew installs `openshell`, `openshell-gateway`, and
287
-
`openshell-driver-vm`, ad-hoc signs the driver with the Hypervisor entitlement
288
-
in`post_install`, and owns the `brew services` gateway lifecycle. The service
289
-
also leaves `OPENSHELL_DRIVERS`unset so driver choice remains automatic unless
290
-
the user explicitly overrides it.
296
+
Homebrew installs `openshell`, `openshell-gateway`, `openshell-driver-vm`, and
297
+
the native `openshell-sandbox` host supervisor beside the driver. It ad-hoc
298
+
signs the driver with the Hypervisor entitlement in `post_install` and owns the
299
+
`brew services` gateway lifecycle. The service also leaves `OPENSHELL_DRIVERS`
300
+
unset so driver choice remains automatic unless the user explicitly overrides
0 commit comments