diff --git a/.agents/skills/test-release-canary/SKILL.md b/.agents/skills/test-release-canary/SKILL.md index 7f279778c4..2a996bf0b5 100644 --- a/.agents/skills/test-release-canary/SKILL.md +++ b/.agents/skills/test-release-canary/SKILL.md @@ -14,9 +14,10 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti | Job | Runner | Verifies | |---|---|---| | `macos` | `macos-latest-xlarge` | Installs the dev Homebrew artifacts, reaches the VM gateway, and creates, executes in, and deletes a sandbox. | -| `ubuntu` | `ubuntu-latest` | Installs the dev Debian package, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. | +| `ubuntu-deb` | `ubuntu-latest` | Installs the dev Debian package, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. | | `fedora` | `fedora:latest` container | Installs the dev RPM packages, reaches the Podman gateway, and creates, executes in, and deletes a sandbox. | -| `ubuntu-snap` | `ubuntu-latest` | Installs the Release Dev Snap, connects its interfaces, reaches the Docker gateway, and creates, executes in, and deletes a sandbox. | +| `ubuntu-snap-system-docker` | `ubuntu-latest` | Uses `install.sh` to install the snap from `latest/edge`, reuses system Docker, reaches the Docker gateway, and creates, executes in, and deletes a sandbox, and verifies that the Docker snap is not installed. | +| `ubuntu-snap-docker-preflight` | `ubuntu-latest` | Verifies that `install.sh` rejects the OpenShell Snap path when Docker is absent or supplied by the Docker snap, without installing OpenShell. | | `kubernetes` | `ubuntu-latest` + kind | Installs the dev Helm chart, reaches the in-cluster gateway, and creates, executes in, and deletes a sandbox using the published runtime images. | All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject @@ -24,9 +25,12 @@ All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject Kubernetes job installs with `server.telemetryEnabled=false`, so smoke traffic does not contribute to product usage metrics. -The workflow sets `OPENSHELL_VERSION=dev`, so every `install.sh` job consumes the -rolling dev release produced by the triggering workflow. Kubernetes pins the -matching `0.0.0-dev` chart and `:dev` images. +The workflow sets `OPENSHELL_VERSION=dev` for every `install.sh` job. Positive +jobs consume the rolling dev release produced by the triggering workflow. The +system-Docker Snap lane tracks `latest/edge`; the missing-Docker lane exits +before installing a snap. The Debian and Kubernetes CLI lanes remove snapd so +they continue to exercise the dev Debian package. Kubernetes pins the matching +`0.0.0-dev` chart and `:dev` images. The host-package jobs exercise fresh installs, not upgrades from a persisted schema-v1 gateway config. Validate Homebrew and RPM exact-default migration with @@ -50,7 +54,7 @@ on: ``` - **Automatic.** Every successful `Release Dev` run (on `main` or a manual dispatch of Release Dev) fires the canary. Each job gates on `github.event.workflow_run.conclusion == 'success'` so a failed Release Dev does not run the canary. -- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. To include `ubuntu-snap`, supply `release-dev-run-id` for a successful Release Dev run whose Snap artifact should be tested; without it, that job is skipped because no artifact is available. +- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. When dispatched manually, `github.event.workflow_run.head_sha` is empty and the workflow falls back to `github.sha` (the branch tip) for the `install.sh` URL. @@ -62,13 +66,6 @@ Run the canary as-is on the current branch: gh workflow run release-canary.yml --ref "$(git branch --show-current)" ``` -To exercise the Ubuntu Snap job, pass the successful Release Dev run ID: - -```shell -gh workflow run release-canary.yml --ref "$(git branch --show-current)" \ - -f release-dev-run-id= -``` - Watch the run that starts: ```shell @@ -87,7 +84,7 @@ gh run view --log-failed When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published dev artifacts* (`0.0.0-dev` chart, `:dev` images, and the `dev` GitHub release). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts. -Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag. +Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though packages come from the latest public dev release and the snap comes from `latest/edge`. ## Testing artifacts from a specific SHA @@ -137,10 +134,11 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i | Symptom | Likely cause | Where to look | |---|---|---| -| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Dev release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | +| `macos`/`ubuntu-deb`/`fedora` job fails on `install.sh` | Dev release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | | Sandbox create or exec fails | Published sandbox and supervisor artifacts are missing, incompatible, or cannot establish the protected runtime channel. | Gateway logs plus Docker, Podman, VM, Snap, or Kubernetes runtime diagnostics for the job. | -| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. | -| `ubuntu-snap` fails after interface connection | The gateway did not recover after Docker became available, or did not become reachable within the 30-second bound. | Failure diagnostics dump Snap service/connection/change state, gateway and snapd journals, Snap logs, and port 17670 listeners. | +| `macos`/`ubuntu-deb`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. | +| `ubuntu-snap-system-docker` fails during `install.sh` | System Docker was unavailable, the edge revision or automatic interfaces were unavailable, or the gateway did not become reachable. | Failure diagnostics dump system Docker, snap service/connection/change state, gateway and snapd journals, snap logs, and port 17670 listeners. | +| `ubuntu-snap-docker-preflight` unexpectedly succeeds | The installer no longer fails before installing the OpenShell snap when Docker is absent or supplied by the Docker snap. | Inspect `install.log`, `docker-snap.log`, `snap list`, and snapd changes. | | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. | | `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. | | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. | diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 2935042776..cd3821c05a 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -276,6 +276,20 @@ jobs: runner: linux-arm64-cpu8 conformance-artifact-prefix: openshell-conformance + podman-e2e: + needs: [pr_metadata, build-binaries, build-images] + if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' + permissions: + actions: read + contents: read + packages: read + uses: ./.github/workflows/e2e-podman-test.yml + with: + image-tag: ${{ github.sha }} + conformance-artifact-prefix: openshell-conformance + suite-matrix: >- + [{"suite":"ci","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:ci"}] + vm-e2e: needs: [pr_metadata, build-binaries, build-vm-driver] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' @@ -453,6 +467,7 @@ jobs: - feature-specific-integration - driver-specific-integration - docker-e2e + - podman-e2e - vm-e2e - docker-external-driver-e2e - podman-external-driver-e2e diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index a8a8df7b51..723bb4b005 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -2,11 +2,6 @@ name: Release Canary on: workflow_dispatch: - inputs: - release-dev-run-id: - description: "Successful Release Dev run ID whose Snap artifact to test" - required: false - type: string workflow_run: workflows: ["Release Dev"] types: [completed] @@ -58,12 +53,22 @@ jobs: fi done - ubuntu: - name: Ubuntu Docker + ubuntu-deb: + name: Ubuntu DEB if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} runs-on: ubuntu-latest timeout-minutes: 20 steps: + - name: Remove snapd + run: | + set -euo pipefail + snaps=$(snap list 2>/dev/null | awk 'NR > 1 { print $1 }') + if [ -n "$snaps" ]; then + sudo snap remove $snaps + fi + sudo apt-get -y --purge remove snapd + ! command -v snap >/dev/null 2>&1 + - name: Ensure Docker run: | if ! command -v docker >/dev/null 2>&1; then @@ -186,9 +191,9 @@ jobs: run: | docker rm -f "${FEDORA_CANARY_CONTAINER}" >/dev/null 2>&1 || true - ubuntu-snap: - name: Ubuntu Snap - if: ${{ github.event.workflow_run.conclusion == 'success' || inputs.release-dev-run-id != '' }} + ubuntu-snap-system-docker: + name: Ubuntu Snap with system Docker + if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} runs-on: ubuntu-latest timeout-minutes: 20 steps: @@ -201,54 +206,38 @@ jobs: sudo systemctl start snapd sudo snap wait system seed.loaded - - name: Install Docker snap + - name: Ensure Docker run: | set -euo pipefail - sudo snap install docker - - - name: Download snap from release-dev artifacts - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - github-token: ${{ github.token }} - run-id: ${{ inputs.release-dev-run-id || github.event.workflow_run.id }} - pattern: snap-linux-amd64 - path: release/ - merge-multiple: true + if ! command -v docker >/dev/null 2>&1; then + sudo apt-get update + sudo apt-get install -y docker.io + fi + sudo systemctl start docker || sudo service docker start + docker info - - name: Install snap (dangerous — from release, not store) + - name: Install and check status run: | set -euo pipefail sudo systemctl set-environment \ "OPENSHELL_TELEMETRY_ENABLED=${OPENSHELL_TELEMETRY_ENABLED}" - sudo snap install ./release/*.snap --dangerous - - - name: Connect interfaces - run: | - set -euo pipefail - sudo snap connect openshell:docker docker:docker-daemon - sudo snap connect openshell:log-observe - sudo snap connect openshell:system-observe - - - name: Register snap gateway and check status - run: | - set -euo pipefail + curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh + sudo snap list openshell + if sudo snap list docker >/dev/null 2>&1; then + echo "install.sh unexpectedly installed the Docker snap" >&2 + exit 1 + fi + snap info openshell | grep -E '^tracking: +latest/edge$' + docker info + sudo snap connections openshell | grep -E '^docker +openshell:docker +:docker +' openshell --version sudo snap services openshell - openshell gateway add http://127.0.0.1:17670 --local --name snap-docker - openshell gateway select snap-docker - for _ in $(seq 1 30); do - if openshell status; then - exit 0 - fi - sleep 1 - done - echo "Gateway did not become ready within 30 seconds" >&2 - exit 1 + openshell status - name: Create and exercise a sandbox run: | set -euo pipefail - sandbox="rc-${GITHUB_RUN_ID}" + sandbox="rc-system-docker-${GITHUB_RUN_ID}" openshell sandbox create --name "$sandbox" --detach openshell sandbox exec --name "$sandbox" --no-tty -- true openshell sandbox delete "$sandbox" @@ -257,6 +246,9 @@ jobs: if: failure() run: | set +e + docker info + sudo systemctl status docker.service --no-pager + sudo journalctl -b -u docker.service --no-pager -n 300 sudo snap services openshell sudo snap connections openshell sudo snap changes @@ -266,6 +258,66 @@ jobs: sudo snap logs openshell.gateway -n=300 sudo ss -ltnp '( sport = :17670 )' + ubuntu-snap-docker-preflight: + name: Ubuntu Snap Docker preflight + if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - name: Install snapd + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y snapd + sudo systemctl enable --now snapd.socket + sudo systemctl start snapd + sudo snap wait system seed.loaded + + - name: Remove system Docker + run: | + set -euo pipefail + sudo systemctl stop docker.service docker.socket 2>/dev/null || true + sudo apt-get -y --purge remove docker-ce docker-ce-cli containerd.io \ + docker-buildx-plugin docker-compose-plugin docker.io 2>/dev/null || true + ! command -v docker >/dev/null 2>&1 + + - name: Verify installer rejects missing Docker + run: | + set -euo pipefail + curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh -o install.sh + if OPENSHELL_VERSION=dev sh install.sh >install.log 2>&1; then + echo "install.sh unexpectedly succeeded without Docker" >&2 + cat install.log >&2 + exit 1 + fi + cat install.log + grep -F "Docker is required before installing the OpenShell snap" install.log + ! sudo snap list docker >/dev/null 2>&1 + ! sudo snap list openshell >/dev/null 2>&1 + + - name: Verify installer rejects Docker snap + run: | + set -euo pipefail + sudo snap install docker + if OPENSHELL_VERSION=dev sh install.sh >docker-snap.log 2>&1; then + echo "install.sh unexpectedly succeeded with the Docker snap" >&2 + cat docker-snap.log >&2 + exit 1 + fi + cat docker-snap.log + grep -F "the Docker snap is not currently compatible with OpenShell" docker-snap.log + sudo snap list docker >/dev/null + ! sudo snap list openshell >/dev/null 2>&1 + + - name: Collect Snap diagnostics + if: failure() + run: | + set +e + cat install.log + cat docker-snap.log + sudo snap changes + sudo journalctl -b -u snapd.service --no-pager -n 300 + kubernetes: name: Kubernetes Helm (kind) if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} @@ -336,6 +388,16 @@ jobs: cat port-forward.log >&2 exit 1 + - name: Remove snapd + run: | + set -euo pipefail + snaps=$(snap list 2>/dev/null | awk 'NR > 1 { print $1 }') + if [ -n "$snaps" ]; then + sudo snap remove $snaps + fi + sudo apt-get -y --purge remove snapd + ! command -v snap >/dev/null 2>&1 + - name: Install OpenShell CLI run: | set -euo pipefail diff --git a/CI.md b/CI.md index 1324ca9d7f..7555935a9c 100644 --- a/CI.md +++ b/CI.md @@ -399,7 +399,7 @@ These workflows run after merge to publish dev/tagged artifacts and verify them. |---|---| | `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | | `.github/workflows/release-tag.yml` | Publishes tagged stable releases and manually dispatched pre-releases. Its automatic tag trigger excludes `-pre.*`. Security and integration failures do not block pre-release artifact publication. Stable publication requires the currently implemented qualification profile to pass; the summary identifies the remaining RFC 0014 coverage. | -| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Each job reaches its gateway and creates, exercises, and deletes a sandbox. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref `). See the `test-release-canary` skill for the playbook and local kind reproduction. | +| `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts in the `macos`, `ubuntu-deb`, `ubuntu-snap-system-docker`, `fedora`, and `kubernetes` (kind + Helm) jobs. Each job reaches its gateway and creates, exercises, and deletes a sandbox. The Snap lanes verify a compatible system Docker lifecycle and `ubuntu-snap-docker-preflight` tests fail-fast behavior when Docker is absent or supplied by the Docker snap. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref `). See the `test-release-canary` skill for the playbook and local kind reproduction. | ## Required status contexts diff --git a/architecture/build.md b/architecture/build.md index 081f11af7b..a447b03d60 100644 --- a/architecture/build.md +++ b/architecture/build.md @@ -318,6 +318,19 @@ configure the gateway to use them. The Ubuntu `deb` installer consumes for direct executable installation on every environment. Release Dev and Release Tag run Ubuntu conformance through the Debian package, while Fedora continues using direct executable installation until RPM coverage is available. +The release canary separately exercises the public installer on Ubuntu. The +OpenShell Snap requires a compatible, preinstalled non-Snap Docker daemon. Its +positive canary uses system Docker; negative preflight coverage verifies that +the installer rejects both missing Docker and the Docker Snap before installing +OpenShell. Its Debian lane removes snapd before running the installer so Snap +precedence cannot change the package under test. +Snapd runs the gateway as a root-owned system service. Its generated client +certificates reside in root-owned snap state and are unavailable to ordinary CLI +users, so the Snap uses plaintext loopback transport and enables unauthenticated +local users by default. Debian and RPM packages instead run systemd user services +and use user-owned mTLS material. Bootstrap creates the default configuration +only when it is missing. Sandbox-to-gateway sessions remain authenticated with +gateway-minted JWTs. The Debian qualification profile keeps candidate-image overrides outside the operator-owned gateway configuration: it writes a harness-owned file under `/var/lib/openshell-qualification` and selects it through the packaged systemd diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index a341ef2ba2..f4745b3ac7 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -356,7 +356,16 @@ VM runtime state paths are derived only from driver-validated sandbox IDs matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a private `run/` directory plus Unix peer UID/PID checks. Standalone unauthenticated TCP mode is disabled unless explicitly enabled for local -development. +development. The VM image cache is owner-only. When the host assembles a +bootstrap rootfs from OCI layers, cross-layer symlinks may resolve only within +that rootfs; absolute or escaping targets reject the image before a later layer +can write through them. Rootfs traversal and mutation use opened directory +handles with no-follow file creation, so later copies, permission changes, and +whiteouts cannot be redirected by replacing a validated pathname component. +The bootstrap rootfs comes only from the operator-configured `bootstrap_image` +or `default_image`; a sandbox-requested image never becomes the VM bootstrap +image. The gateway rejects configurations without either trusted source, and +the standalone driver independently fails startup for the same condition. Runtime-specific implementation notes belong in the driver crate README: diff --git a/architecture/gateway.md b/architecture/gateway.md index 689bd4b493..2f626286e6 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -358,8 +358,12 @@ can recover that same successor for 30 seconds when the request matches, but it cannot authorize ordinary RPCs or choose another successor. Advancing the successor removes that retry path across every gateway replica. Short `gateway_jwt.ttl_secs` lifetimes still bound the exposure of a current bearer -that has not yet been refreshed. Omitting `gateway_jwt.ttl_secs` uses a -900-second lifetime. Explicit zero is rejected. +that has not yet been refreshed. Omitting `gateway_jwt.ttl_secs` selects +non-expiring launch-scoped gateway and Sandbox Protocol tokens for local +single-player Docker, Podman, and VM gateways; both token profiles carry +`exp = 0`. Typed extension JWTs retain a 900-second default when the field is +omitted. Kubernetes and other shared deployments should set a positive TTL. +Explicit zero is rejected. Gateway JWT signing-key rotation is currently an offline operator action. The runtime loads one active signing key and one matching public verification key @@ -741,11 +745,28 @@ Gateway and Sandbox Protocol token responses follow the same convention: a present expiration timestamp carries the absolute deadline, while absence means the issued token does not expire. +On-disk SQLite databases run in WAL journal mode with `synchronous=FULL`. +The adapter switches the file to WAL on a single connection before the pool +opens, then applies both settings to every pooled connection. WAL lets readers +proceed while a writer commits and reduces each commit to one WAL `fsync`, +which matters because gateway hot paths such as SSH session issuance and +revocation are many small autocommit writes. `synchronous` stays at `FULL` +because some of those writes tighten authorization: under `NORMAL`, a power +loss could roll back an acknowledged SSH session revocation and make the token +valid again. Writes that are safe to lose, currently only SSH session issuance +through `Store::create_relaxed`, use a second single-connection pool with +`synchronous=NORMAL`. Losing a minted token only invalidates it, and because +both pools share one WAL, the next `FULL` commit also makes earlier relaxed +commits durable. Deployments that need multiple replicas use Postgres, where +`create_relaxed` is an ordinary durable insert. WAL requires a local filesystem with working shared +memory, so the SQLite file must not live on a network mount, and backups must +use `sqlite3 .backup` or `VACUUM INTO` rather than copying the main file alone. + The SQLite adapter tightens the on-disk database file to mode `0o600` on every connect so that provider API keys, SSH session tokens, and sandbox metadata are not readable by other local users on shared hosts. The same restriction is -reapplied to the `-wal` and `-shm` sidecars (created by SQLite's -default WAL journal mode), which mirror the same sensitive contents. +reapplied to the `-wal` and `-shm` sidecars that WAL mode creates, +which mirror the same sensitive contents. Persisted state includes sandboxes, providers, provider profiles, provider credential refresh state, SSH sessions, policy revisions, settings, deployment diff --git a/architecture/sandbox.md b/architecture/sandbox.md index c2de9a1d6d..26a446e4f0 100644 --- a/architecture/sandbox.md +++ b/architecture/sandbox.md @@ -55,6 +55,12 @@ TCP mediation accepts use the same authenticated transport recovery as process w A renewed Sandbox Protocol bearer is authenticated even when its credential epoch is unchanged. The supervisor confirms that bearer on the active physical connection and records its fingerprint only after confirmation succeeds, preserving pending streams and the mediation session. Changing the credential epoch still requires an authenticated replacement connection. +Local single-player Docker, Podman, and VM gateways propagate an omitted +`gateway_jwt.ttl_secs` value to both launch-scoped credential profiles. Those +credentials use `exp = 0`, so host suspension cannot strand the supervisor +after a refresh deadline passes. Shared deployments retain expiring credentials +and a durable compute-platform bootstrap identity. + Unauthenticated TLS handshakes have a separate bounded asynchronous pool and five-second deadline, never consuming authenticated control slots or threads. The socket broker reserves the TCP control-listener port against workload @@ -256,6 +262,12 @@ qualification output (`seccomp_listener_mode`). DNS uses an exact sandbox-local resolver at `127.0.0.53:53`. The driver sets the nameserver and permits an unprivileged bind to port 53. UDP and TCP DNS requests are forwarded through the supervisor, which applies hostname-based DNS policy. +The Podman driver supplies that resolver configuration as a driver-owned, +read-only secret mounted at `/etc/resolv.conf`; the workload remains on +`network=none` and receives no host aliases directly. For +`host.openshell.internal`, the supervisor returns the trusted concrete host +destination carried in its runtime descriptor rather than relying on Podman's +workload-side host-gateway injection. DNS sender identity is explicitly unavailable: native writes can come from an inheriting process or after exec, and neither the connecting binary nor a later descriptor-owner snapshot proves who sent an already queued query. Consumers @@ -669,6 +681,12 @@ sandbox workload directly. The relay supports: buffer, and a single stdin lease across client disconnects. Ctrl-C interrupts the foreground process. For read-only attachments, Ctrl-C only exits the current viewer. +- Supervised CLI attachment. After an established SSH transport fails, the CLI + remains alive, requests a fresh SSH session from the gateway, and reattaches + to the same canonical main process within a bounded recovery window. It does + not stop or restart the sandbox to recover the client connection. The same + deadline bounds replacement-session RPCs. Process-targeted termination is + forwarded to the SSH child, which the CLI reaps before exiting. - Independent interactive shell sessions. - Command execution. Commands run through a login shell (`bash -lc`) by default, so the first of the user's `.bash_profile`, `.bash_login`, or `.profile` is diff --git a/crates/openshell-cli/src/ssh.rs b/crates/openshell-cli/src/ssh.rs index 947c84cedc..204c8f6366 100644 --- a/crates/openshell-cli/src/ssh.rs +++ b/crates/openshell-cli/src/ssh.rs @@ -14,8 +14,8 @@ use openshell_core::forward::{ validate_ssh_session_response, write_forward_pid, }; use openshell_core::proto::{ - CreateSshSessionRequest, GetSandboxRequest, SshRelayTarget, TcpForwardFrame, TcpForwardInit, - tcp_forward_init, + CreateSshSessionRequest, GetSandboxRequest, SandboxPhase, SshRelayTarget, TcpForwardFrame, + TcpForwardInit, tcp_forward_init, }; use std::fs; use std::future::Future; @@ -23,8 +23,8 @@ use std::io::{IsTerminal, Write}; #[cfg(unix)] use std::os::unix::process::CommandExt; use std::path::{Path, PathBuf}; -use std::process::{Command, Stdio}; -use std::time::Duration; +use std::process::{Command, ExitStatus, Stdio}; +use std::time::{Duration, Instant}; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::TcpStream; use tokio::process::{Child, Command as TokioCommand}; @@ -44,6 +44,20 @@ const FORWARD_LISTENER_CONNECT_TIMEOUT: Duration = Duration::from_millis(200); /// command has already reported its terminal result. const TERMINAL_RELAY_REGISTRATION_TIMEOUT: Duration = Duration::from_secs(5); const TERMINAL_RELAY_REGISTRATION_INTERVAL: Duration = Duration::from_millis(50); +/// An SSH client that remained alive for this long was attached successfully, +/// rather than failing during initial authentication or setup. +const CONNECT_ESTABLISHED_DURATION: Duration = Duration::from_secs(2); +/// Briefly wait for the gateway lifecycle projection to catch up with an SSH +/// exit status before treating status 255 as a transport failure. +const CONNECT_TERMINAL_STATUS_TIMEOUT: Duration = Duration::from_millis(500); +const CONNECT_TERMINAL_STATUS_INTERVAL: Duration = Duration::from_millis(50); +/// Time allowed to restore an established canonical-main attachment after its +/// transport is lost. +const CONNECT_RECOVERY_TIMEOUT: Duration = Duration::from_mins(1); +const CONNECT_RETRY_INITIAL_DELAY: Duration = Duration::from_millis(250); +const CONNECT_RETRY_MAX_DELAY: Duration = Duration::from_secs(5); +const CONNECT_CHILD_TERMINATION_TIMEOUT: Duration = Duration::from_secs(5); +const SSH_TRANSPORT_FAILURE_EXIT_CODE: i32 = 255; const SYNC_RETRY_ATTEMPTS: usize = 4; const SYNC_RETRY_DELAY: Duration = Duration::from_secs(2); @@ -281,23 +295,7 @@ fn exec_or_wait(mut command: Command, replace_process: bool) -> Result { Ok(status.code().unwrap_or(1)) } -async fn sandbox_connect_with_mode( - server: &str, - name: &str, - tls: &TlsOptions, - replace_process: bool, - workspace: &str, - terminal_relay_registration_timeout: Option, -) -> Result { - let session = ssh_session_config( - server, - name, - tls, - workspace, - terminal_relay_registration_timeout, - ) - .await?; - +fn main_attach_command(session: &SshSessionConfig) -> Command { let mut command = ssh_base_command(&session.proxy_command); if session.main_terminal { command.arg("-tt").arg("-o").arg("RequestTTY=force"); @@ -313,12 +311,379 @@ async fn sandbox_connect_with_mode( .stdin(Stdio::inherit()) .stdout(Stdio::inherit()) .stderr(Stdio::inherit()); + command +} - let exit_code = tokio::task::spawn_blocking(move || exec_or_wait(command, replace_process)) +async fn run_main_attach(session: &SshSessionConfig, replace_process: bool) -> Result { + let command = main_attach_command(session); + tokio::task::spawn_blocking(move || exec_or_wait(command, replace_process)) .await - .into_diagnostic()??; + .into_diagnostic()? +} + +fn process_exit_code(status: ExitStatus) -> i32 { + status.code().unwrap_or(1) +} + +#[cfg(unix)] +struct TerminationSignals { + interrupt: tokio::signal::unix::Signal, + quit: tokio::signal::unix::Signal, + terminate: tokio::signal::unix::Signal, +} + +#[cfg(unix)] +impl TerminationSignals { + fn new() -> Result { + use tokio::signal::unix::{SignalKind, signal}; + + Ok(Self { + interrupt: signal(SignalKind::interrupt()).into_diagnostic()?, + quit: signal(SignalKind::quit()).into_diagnostic()?, + terminate: signal(SignalKind::terminate()).into_diagnostic()?, + }) + } + + async fn recv(&mut self) -> Signal { + tokio::select! { + _ = self.interrupt.recv() => Signal::SIGINT, + _ = self.quit.recv() => Signal::SIGQUIT, + _ = self.terminate.recv() => Signal::SIGTERM, + } + } +} + +struct ConnectCancellation { + #[cfg(unix)] + signals: TerminationSignals, +} + +impl ConnectCancellation { + fn new() -> Result { + Ok(Self { + #[cfg(unix)] + signals: TerminationSignals::new()?, + }) + } + + async fn wait(&mut self, future: F) -> std::result::Result + where + F: Future, + { + #[cfg(unix)] + { + tokio::select! { + biased; + result = future => Ok(result), + signal = self.signals.recv() => Err(128 + signal as i32), + } + } + + #[cfg(not(unix))] + { + Ok(future.await) + } + } +} - Ok(exit_code) +#[cfg(unix)] +fn forward_signal_to_child(child: &Child, signal: Signal) -> Result<()> { + let Some(pid) = child.id() else { + return Ok(()); + }; + let pid = i32::try_from(pid).into_diagnostic()?; + match nix::sys::signal::kill(nix::unistd::Pid::from_raw(pid), signal) { + Ok(()) | Err(nix::errno::Errno::ESRCH) => Ok(()), + Err(error) => Err(error).into_diagnostic(), + } +} + +#[cfg(unix)] +async fn terminate_and_reap_child(child: &mut Child, signal: Signal) -> Result { + forward_signal_to_child(child, signal)?; + if let Ok(status) = tokio::time::timeout(CONNECT_CHILD_TERMINATION_TIMEOUT, child.wait()).await + { + status.into_diagnostic()?; + } else { + child.kill().await.into_diagnostic()?; + child.wait().await.into_diagnostic()?; + } + Ok(128 + signal as i32) +} + +async fn run_main_attach_supervised( + session: &SshSessionConfig, + cancellation: &mut ConnectCancellation, +) -> Result { + #[cfg(not(unix))] + let _ = cancellation; + + let mut command = TokioCommand::from(main_attach_command(session)); + command.kill_on_drop(true); + + let mut child = command.spawn().into_diagnostic()?; + + #[cfg(unix)] + { + tokio::select! { + biased; + status = child.wait() => Ok(process_exit_code(status.into_diagnostic()?)), + signal = cancellation.signals.recv() => terminate_and_reap_child(&mut child, signal).await, + } + } + + #[cfg(not(unix))] + { + Ok(process_exit_code(child.wait().await.into_diagnostic()?)) + } +} + +async fn await_before_recovery_deadline( + deadline: Option, + future: F, +) -> std::result::Result +where + F: Future, +{ + let Some(deadline) = deadline else { + return Ok(future.await); + }; + tokio::time::timeout_at(tokio::time::Instant::from_std(deadline), future) + .await + .map_err(|_| ()) +} + +fn print_connect_recovery_timeout() { + eprintln!( + "Unable to restore the sandbox connection within {} seconds.", + CONNECT_RECOVERY_TIMEOUT.as_secs() + ); +} + +fn should_recover_connect( + exit_code: i32, + attached_for: Duration, + recovery_active: bool, + main_has_terminal_result: bool, +) -> bool { + !main_has_terminal_result + && exit_code == SSH_TRANSPORT_FAILURE_EXIT_CODE + && (recovery_active || attached_for >= CONNECT_ESTABLISHED_DURATION) +} + +async fn canonical_main_has_terminal_result( + server: &str, + name: &str, + tls: &TlsOptions, + workspace: &str, +) -> bool { + let probe = async { + let Ok(mut client) = grpc_client(server, tls).await else { + return false; + }; + loop { + let Ok(response) = client + .get_sandbox(GetSandboxRequest { + name: name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + workspace.to_string(), + )), + }) + .await + else { + return false; + }; + let Some(sandbox) = response.into_inner().sandbox else { + return false; + }; + let phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); + if sandbox + .status + .as_ref() + .is_some_and(|status| status.exit_code.is_some()) + || matches!( + phase, + SandboxPhase::Completed + | SandboxPhase::Error + | SandboxPhase::Stopping + | SandboxPhase::Stopped + | SandboxPhase::Deleting + ) + { + return true; + } + tokio::time::sleep(CONNECT_TERMINAL_STATUS_INTERVAL).await; + } + }; + + tokio::time::timeout(CONNECT_TERMINAL_STATUS_TIMEOUT, probe) + .await + .unwrap_or(false) +} + +fn should_start_connect_recovery_window(attached_for: Duration, recovery_active: bool) -> bool { + !recovery_active || attached_for >= CONNECT_RECOVERY_TIMEOUT +} + +fn next_connect_retry_delay(delay: Duration) -> Duration { + std::cmp::min(delay.saturating_mul(2), CONNECT_RETRY_MAX_DELAY) +} + +fn connect_error_is_retryable(error: &Report) -> bool { + let message = format!("{error:?}").to_ascii_lowercase(); + [ + "broken pipe", + "connection aborted", + "connection closed", + "connection refused", + "connection reset", + "deadline exceeded", + "h2 protocol", + "http2", + "reset before headers", + "service is currently unavailable", + "timed out", + "timeout", + "transport error", + "unexpected eof", + "unavailable", + "upstream connect error", + ] + .iter() + .any(|needle| message.contains(needle)) +} + +async fn sandbox_connect_supervised( + server: &str, + name: &str, + tls: &TlsOptions, + workspace: &str, +) -> Result { + let mut recovery_deadline = None; + let mut retry_delay = CONNECT_RETRY_INITIAL_DELAY; + let mut cancellation = ConnectCancellation::new()?; + + loop { + if recovery_deadline.is_some_and(|deadline| Instant::now() >= deadline) { + print_connect_recovery_timeout(); + return Ok(SSH_TRANSPORT_FAILURE_EXIT_CODE); + } + + let session_attempt = match Box::pin(cancellation.wait(await_before_recovery_deadline( + recovery_deadline, + ssh_session_config(server, name, tls, workspace, None), + ))) + .await + { + Ok(attempt) => attempt, + Err(exit_code) => return Ok(exit_code), + }; + let session = match session_attempt { + Err(()) => { + print_connect_recovery_timeout(); + return Ok(SSH_TRANSPORT_FAILURE_EXIT_CODE); + } + Ok(Ok(session)) => session, + Ok(Err(error)) + if recovery_deadline.is_some_and(|deadline| Instant::now() < deadline) + && connect_error_is_retryable(&error) => + { + tracing::warn!( + sandbox = name, + error = %error, + "failed to create replacement SSH session; retrying" + ); + let now = Instant::now(); + let delay = recovery_deadline + .map_or(retry_delay, |deadline| retry_delay.min(deadline - now)); + if let Err(exit_code) = cancellation.wait(tokio::time::sleep(delay)).await { + return Ok(exit_code); + } + retry_delay = next_connect_retry_delay(retry_delay); + continue; + } + Ok(Err(error)) => return Err(error), + }; + + let attach_started = Instant::now(); + let exit_code = run_main_attach_supervised(&session, &mut cancellation).await?; + let attached_for = attach_started.elapsed(); + let recovery_active = recovery_deadline.is_some(); + let main_has_terminal_result = if exit_code == SSH_TRANSPORT_FAILURE_EXIT_CODE { + match cancellation + .wait(canonical_main_has_terminal_result( + server, name, tls, workspace, + )) + .await + { + Ok(has_result) => has_result, + Err(exit_code) => return Ok(exit_code), + } + } else { + false + }; + + if !should_recover_connect( + exit_code, + attached_for, + recovery_active, + main_has_terminal_result, + ) { + return Ok(exit_code); + } + + // Once a replacement attachment survives the full recovery interval, + // treat a later failure as a new interruption. Keeping the existing + // deadline for shorter attempts prevents SSH connect timeouts from + // extending recovery forever. + if should_start_connect_recovery_window(attached_for, recovery_active) { + recovery_deadline = Some(Instant::now() + CONNECT_RECOVERY_TIMEOUT); + retry_delay = CONNECT_RETRY_INITIAL_DELAY; + eprintln!( + "Connection to sandbox lost; reconnecting. To disconnect, press Ctrl-P then Ctrl-Q after reattachment; press Ctrl-C while retrying to cancel." + ); + } + + let Some(deadline) = recovery_deadline else { + return Ok(exit_code); + }; + let now = Instant::now(); + if now >= deadline { + print_connect_recovery_timeout(); + return Ok(exit_code); + } + + if let Err(exit_code) = cancellation + .wait(tokio::time::sleep(std::cmp::min( + retry_delay, + deadline - now, + ))) + .await + { + return Ok(exit_code); + } + retry_delay = next_connect_retry_delay(retry_delay); + } +} + +async fn sandbox_connect_with_mode( + server: &str, + name: &str, + tls: &TlsOptions, + replace_process: bool, + workspace: &str, + terminal_relay_registration_timeout: Option, +) -> Result { + let session = ssh_session_config( + server, + name, + tls, + workspace, + terminal_relay_registration_timeout, + ) + .await?; + + run_main_attach(&session, replace_process).await } /// Connect to a sandbox via SSH. @@ -328,7 +693,7 @@ pub async fn sandbox_connect( tls: &TlsOptions, workspace: &str, ) -> Result { - sandbox_connect_with_mode(server, name, tls, true, workspace, None).await + sandbox_connect_supervised(server, name, tls, workspace).await } pub(crate) async fn sandbox_connect_without_exec( @@ -1868,6 +2233,91 @@ mod tests { assert!(!sync_error_is_retryable(&error)); } + #[test] + fn connect_recovery_starts_only_for_established_transport_failures() { + assert!(should_recover_connect( + SSH_TRANSPORT_FAILURE_EXIT_CODE, + CONNECT_ESTABLISHED_DURATION, + false, + false + )); + assert!(!should_recover_connect( + SSH_TRANSPORT_FAILURE_EXIT_CODE, + CONNECT_ESTABLISHED_DURATION + .checked_sub(Duration::from_millis(1)) + .unwrap(), + false, + false + )); + assert!(!should_recover_connect( + 0, + CONNECT_ESTABLISHED_DURATION, + false, + false + )); + assert!(!should_recover_connect( + SSH_TRANSPORT_FAILURE_EXIT_CODE, + CONNECT_ESTABLISHED_DURATION, + false, + true + )); + } + + #[test] + fn connect_recovery_continues_after_fast_replacement_failure() { + assert!(should_recover_connect( + SSH_TRANSPORT_FAILURE_EXIT_CODE, + Duration::ZERO, + true, + false + )); + assert!(!should_start_connect_recovery_window( + CONNECT_ESTABLISHED_DURATION, + true + )); + assert!(should_start_connect_recovery_window( + CONNECT_RECOVERY_TIMEOUT, + true + )); + } + + #[test] + fn connect_retry_delay_is_capped() { + assert_eq!( + next_connect_retry_delay(CONNECT_RETRY_INITIAL_DELAY), + Duration::from_millis(500) + ); + assert_eq!( + next_connect_retry_delay(CONNECT_RETRY_MAX_DELAY), + CONNECT_RETRY_MAX_DELAY + ); + } + + #[test] + fn connect_retry_filter_rejects_authentication_failures() { + let transient = miette::miette!("transport error: connection reset by peer"); + assert!(connect_error_is_retryable(&transient)); + + let authentication = miette::miette!("status: Unauthenticated"); + assert!(!connect_error_is_retryable(&authentication)); + + let lifecycle = miette::miette!("sandbox is not ready"); + assert!(!connect_error_is_retryable(&lifecycle)); + } + + #[tokio::test] + async fn connect_recovery_deadline_bounds_stalled_session_creation() { + let deadline = Instant::now() + Duration::from_millis(20); + let result = await_before_recovery_deadline( + Some(deadline), + std::future::pending::>(), + ) + .await; + + assert!(result.is_err()); + assert!(Instant::now() >= deadline); + } + #[test] #[allow(unsafe_code)] // Test-only: env vars require unsafe in Rust 2024. fn install_ssh_config_adds_include_once_and_updates_managed_file() { diff --git a/crates/openshell-core/src/config.rs b/crates/openshell-core/src/config.rs index 1c844864a4..0d4a9f4325 100644 --- a/crates/openshell-core/src/config.rs +++ b/crates/openshell-core/src/config.rs @@ -800,19 +800,25 @@ pub struct GatewayJwtConfig { /// `openshell`. #[serde(default = "default_gateway_id")] pub gateway_id: String, - /// Token lifetime in seconds. Defaults to 15 minutes when omitted. + /// Token lifetime in seconds. Omission selects non-expiring sandbox + /// session credentials and the default lifetime for extension tokens. /// Explicit zero is invalid. #[serde(default, skip_serializing_if = "Option::is_none")] pub ttl_secs: Option, } impl GatewayJwtConfig { - /// Effective token lifetime. + /// Effective typed extension-token lifetime. pub fn token_ttl(&self) -> Duration { self.ttl_secs.map_or(Duration::from_mins(15), |ttl| { Duration::from_secs(ttl.get()) }) } + + /// Effective sandbox session-token lifetime. `None` is non-expiring. + pub fn sandbox_token_ttl(&self) -> Option { + self.ttl_secs.map(|ttl| Duration::from_secs(ttl.get())) + } } fn default_gateway_id() -> String { @@ -1215,7 +1221,7 @@ mod tests { } #[test] - fn gateway_jwt_ttl_defaults_to_fifteen_minutes() { + fn gateway_jwt_omitted_ttl_defaults_extension_and_nonexpiring_session_tokens() { let cfg: GatewayJwtConfig = serde_json::from_value(serde_json::json!({ "signing_key_path": "/tmp/signing.pem", "public_key_path": "/tmp/public.pem", @@ -1225,6 +1231,7 @@ mod tests { assert_eq!(cfg.ttl_secs, None); assert_eq!(cfg.token_ttl(), Duration::from_mins(15)); + assert_eq!(cfg.sandbox_token_ttl(), None); let serialized = serde_json::to_value(&cfg).expect("gateway JWT config serializes"); assert!(serialized.get("ttl_secs").is_none()); @@ -1241,6 +1248,7 @@ mod tests { .expect("gateway JWT config should deserialize with positive ttl"); assert_eq!(cfg.token_ttl(), Duration::from_hours(1)); + assert_eq!(cfg.sandbox_token_ttl(), Some(Duration::from_hours(1))); let serialized = serde_json::to_value(&cfg).expect("gateway JWT config serializes"); assert_eq!(serialized["ttl_secs"], 3600); } diff --git a/crates/openshell-core/src/grpc_client.rs b/crates/openshell-core/src/grpc_client.rs index cbd6de5670..46b2b0dd56 100644 --- a/crates/openshell-core/src/grpc_client.rs +++ b/crates/openshell-core/src/grpc_client.rs @@ -133,13 +133,19 @@ fn validate_sandbox_refresh( ) -> std::result::Result { let token = crate::jwt::SecretJwt::parse(response.sandbox_token.clone())?; let credential_epoch = crate::jwt::CredentialEpoch::new(response.credential_epoch)?; - let expiration_time = response + let expires_at = response .sandbox_expiration_time .as_ref() - .ok_or(crate::jwt::SessionJwtError::InvalidLifetime)?; - crate::time::validate_timestamp(expiration_time) - .map_err(|_| crate::jwt::SessionJwtError::InvalidLifetime)?; - let expires_at = expiration_time.seconds; + .map(|expiration_time| { + crate::time::validate_timestamp(expiration_time) + .map_err(|_| crate::jwt::SessionJwtError::InvalidLifetime)?; + if expiration_time.seconds == 0 { + return Err(crate::jwt::SessionJwtError::InvalidLifetime); + } + Ok(expiration_time.seconds) + }) + .transpose()? + .unwrap_or(0); crate::jwt::SessionBearerTokenSlot::new(token.clone(), expires_at, credential_epoch)?; Ok(ValidatedSandboxRefresh { token, @@ -710,17 +716,15 @@ mod auth_tests { #[cfg(feature = "jwt")] #[test] - fn sandbox_refresh_validation_rejects_missing_expiration() { + fn sandbox_refresh_validation_accepts_missing_expiration_as_non_expiring() { let response = crate::proto::RefreshSandboxTokenResponse { sandbox_token: "sandbox-token".to_string(), credential_epoch: 2, ..Default::default() }; - assert_eq!( - validate_sandbox_refresh(&response).err(), - Some(crate::jwt::SessionJwtError::InvalidLifetime) - ); + let refresh = validate_sandbox_refresh(&response).expect("non-expiring refresh"); + assert_eq!(refresh.expires_at, 0); } #[cfg(feature = "jwt")] diff --git a/crates/openshell-core/src/jwt.rs b/crates/openshell-core/src/jwt.rs index 5cd8c60438..d1b8d22226 100644 --- a/crates/openshell-core/src/jwt.rs +++ b/crates/openshell-core/src/jwt.rs @@ -299,7 +299,7 @@ mod session { impl SupervisorAuthBundle { pub fn validate(&self) -> Result<(), SessionJwtError> { - if self.gateway_expires_at <= 0 || self.sandbox_expires_at <= 0 { + if self.gateway_expires_at < 0 || self.sandbox_expires_at < 0 { return Err(SessionJwtError::InvalidLifetime); } SandboxGenerationId::parse(self.runtime_generation.to_string()) @@ -385,7 +385,7 @@ mod session { expires_at: i64, credential_epoch: CredentialEpoch, ) -> Result<(), SessionJwtError> { - if expires_at <= 0 { + if expires_at < 0 { return Err(SessionJwtError::InvalidLifetime); } let mut stored = self @@ -439,7 +439,7 @@ mod session { .read() .unwrap_or_else(std::sync::PoisonError::into_inner); let stored = stored.as_ref().ok_or(SessionJwtError::TokenUnavailable)?; - if stored.expires_at <= SystemJwtClock.now_unix_seconds() { + if stored.expires_at > 0 && stored.expires_at <= SystemJwtClock.now_unix_seconds() { return Err(SessionJwtError::Expired); } format!("Bearer {}", stored.token.expose_secret()) @@ -487,7 +487,7 @@ mod session { encoding_key: EncodingKey, key_id: String, issuer: String, - ttl: Duration, + ttl: Option, clock: Arc, } @@ -507,11 +507,13 @@ mod session { signing_key_pem: &[u8], key_id: impl Into, gateway_id: &str, - ttl: Duration, + ttl: Option, clock: Arc, ) -> Result { install_crypto_provider(); - validate_ttl(ttl)?; + if let Some(ttl) = ttl { + validate_ttl(ttl)?; + } let key_id = validate_key_id(key_id.into())?; let gateway_id = validate_gateway_id(gateway_id)?; let encoding_key = EncodingKey::from_ed_pem(signing_key_pem) @@ -583,9 +585,9 @@ mod session { token_id: Uuid, issued_at: i64, ) -> Result { - let expires_at = issued_at.saturating_add( - i64::try_from(self.ttl.as_secs()).map_err(|_| SessionJwtError::InvalidLifetime)?, - ); + let expires_at = self.ttl.map_or(0, |ttl| { + issued_at.saturating_add(i64::try_from(ttl.as_secs()).unwrap_or(i64::MAX)) + }); let claims = SessionClaims { iss: self.issuer.clone(), sub: format!("{SANDBOX_SUBJECT_PREFIX}{}", identity.sandbox_id), @@ -718,20 +720,22 @@ mod session { SandboxGenerationId::parse(claims.runtime_generation.to_string()) .map_err(|_| SessionJwtError::InvalidRuntimeIdentity)?; let token_id = Uuid::parse_str(&claims.jti).map_err(|_| SessionJwtError::InvalidJti)?; - if claims.exp <= claims.iat { - return Err(SessionJwtError::InvalidLifetime); - } - let lifetime = claims.exp.saturating_sub(claims.iat); - if lifetime > i64::try_from(MAX_SESSION_TOKEN_TTL.as_secs()).unwrap_or(i64::MAX) { - return Err(SessionJwtError::InvalidLifetime); - } let now = self.clock.now_unix_seconds(); let leeway = i64::try_from(MAX_SESSION_CLOCK_LEEWAY.as_secs()).unwrap_or(30); if claims.iat > now.saturating_add(leeway) { return Err(SessionJwtError::IssuedInFuture); } - if claims.exp < now.saturating_sub(leeway) { - return Err(SessionJwtError::Expired); + if claims.exp != 0 { + if claims.exp <= claims.iat { + return Err(SessionJwtError::InvalidLifetime); + } + let lifetime = claims.exp.saturating_sub(claims.iat); + if lifetime > i64::try_from(MAX_SESSION_TOKEN_TTL.as_secs()).unwrap_or(i64::MAX) { + return Err(SessionJwtError::InvalidLifetime); + } + if claims.exp < now.saturating_sub(leeway) { + return Err(SessionJwtError::Expired); + } } Ok(AuthenticatedSandboxSession { sandbox_id: claims.sandbox_id, @@ -772,7 +776,7 @@ mod session { InvalidSigningKey, #[error("Ed25519 verification key is invalid")] InvalidVerificationKey, - #[error("session token lifetime must be between 60 and 3600 seconds")] + #[error("session token lifetime must be non-expiring or between 60 and 3600 seconds")] InvalidLifetime, #[error("session token profile does not match its claims")] ProfileMismatch, @@ -924,7 +928,7 @@ mod tests { key.serialize_pem().as_bytes(), "current", "test", - DEFAULT_SESSION_TOKEN_TTL, + Some(DEFAULT_SESSION_TOKEN_TTL), clock.clone(), ) .expect("issuer"); @@ -976,6 +980,64 @@ mod tests { ); } + #[test] + fn non_expiring_session_tokens_remain_usable() { + let key = KeyPair::generate_for(&PKCS_ED25519).expect("generate Ed25519 key"); + let public_key_pem = key.public_key_pem().into_bytes(); + let issuer = SessionJwtIssuer::from_ed25519_pem( + key.serialize_pem().as_bytes(), + "current", + "test", + None, + Arc::new(FixedClock(1_900_000_000)), + ) + .expect("issuer"); + let verifier = SessionJwtVerifier::new( + "test", + SessionTokenProfile::Sandbox, + [SessionVerificationKey { + key_id: "current".to_string(), + public_key_pem, + }], + Arc::new(FixedClock(2_000_000_000)), + ) + .expect("verifier"); + let identity = SandboxRuntimeIdentity { + sandbox_id: SandboxId::parse("sandbox-a").expect("sandbox ID"), + runtime_generation: SandboxGenerationId::parse("generation-1") + .expect("runtime generation"), + auth_epoch: CredentialEpoch::new(1).expect("auth epoch"), + }; + let pair = issuer.mint_pair(&identity).expect("token pair"); + + assert_eq!(pair.gateway.expires_at, 0); + assert_eq!(pair.sandbox.expires_at, 0); + verifier + .verify(pair.sandbox.token.expose_secret()) + .expect("non-expiring token remains valid"); + + let slot = SessionBearerTokenSlot::new( + pair.sandbox.token.clone(), + pair.sandbox.expires_at, + pair.auth_epoch, + ) + .expect("non-expiring bearer slot"); + slot.authorization_metadata() + .expect("non-expiring bearer remains available"); + + let bundle = SupervisorAuthBundle { + session_id: SandboxSessionId::new(), + session_rotation: SessionRotation::new(1).expect("session rotation"), + runtime_generation: identity.runtime_generation, + auth_epoch: pair.auth_epoch, + gateway_token: pair.gateway.token, + gateway_expires_at: pair.gateway.expires_at, + sandbox_token: pair.sandbox.token, + sandbox_expires_at: pair.sandbox.expires_at, + }; + bundle.validate().expect("non-expiring auth bundle"); + } + #[test] fn token_debug_is_redacted() { let (issuer, _gateway, _sandbox, identity) = fixture(); diff --git a/crates/openshell-driver-podman/NETWORKING.md b/crates/openshell-driver-podman/NETWORKING.md index be86ac335e..37361395aa 100644 --- a/crates/openshell-driver-podman/NETWORKING.md +++ b/crates/openshell-driver-podman/NETWORKING.md @@ -24,6 +24,14 @@ TCP opens, TCP byte streams, DNS requests/replies, and lifecycle operations share the authenticated gRPC channel. DNS is resolved and authorized by the supervisor. General UDP is unsupported. +The driver mounts a read-only, driver-owned `/etc/resolv.conf` into the +workload with `nameserver 127.0.0.53`. This prevents the disconnected Podman +network namespace from replacing the policy-DNS relay with a runtime resolver. +The supervisor answers `host.openshell.internal` with a trusted concrete +destination: `127.0.0.1` on native Linux, the configured Podman Machine host +address on macOS. That destination is part of the supervisor bootstrap and is +reapplied when a sandbox restarts. + ## Supervisor network The supervisor companion uses Podman's host network. On Linux it connects to diff --git a/crates/openshell-driver-podman/src/config.rs b/crates/openshell-driver-podman/src/config.rs index 75dc2aa77e..e5a7802d2e 100644 --- a/crates/openshell-driver-podman/src/config.rs +++ b/crates/openshell-driver-podman/src/config.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -use std::net::IpAddr; +use std::net::{IpAddr, Ipv4Addr}; use std::num::{NonZeroI64, NonZeroU64}; use std::path::PathBuf; @@ -54,12 +54,12 @@ pub struct PodmanComputeConfig { pub ssh_socket_path: String, /// Name of the Podman bridge network used for driver-managed resources. pub network_name: String, - /// Host gateway IP used for sandbox host aliases. + /// Trusted supervisor-side destination for sandbox host-gateway aliases. /// - /// Empty uses Podman's `host-gateway` resolver. macOS defaults to - /// gvproxy's host-loopback IP because stale Podman machines may fail to - /// resolve `host-gateway` while still serving `host.containers.internal` - /// through gvproxy. + /// Empty uses supervisor loopback on Linux and gvproxy's host-loopback IP + /// with Podman Machine. The resolved address is pinned into the protected + /// runtime descriptor; sandbox policy DNS never trusts workload resolver + /// state for `host.openshell.internal`. pub host_gateway_ip: String, /// Container stop timeout in seconds (SIGTERM → SIGKILL). pub stop_timeout_secs: u32, @@ -439,16 +439,46 @@ impl PodmanComputeConfig { } pub fn validate_host_gateway_ip(&self) -> Result<(), crate::client::PodmanApiError> { + self.resolved_host_gateway_ip().map(|_| ()) + } + + /// Resolve the trusted supervisor-side destination for host-gateway aliases. + pub fn resolved_host_gateway_ip(&self) -> Result { let trimmed = self.host_gateway_ip.trim(); - if trimmed.is_empty() { - return Ok(()); + let address = if trimmed.is_empty() { + #[cfg(target_os = "macos")] + { + MACOS_PODMAN_MACHINE_HOST_GATEWAY_IP + .parse::() + .expect("Podman Machine host gateway constant must be an IP address") + } + #[cfg(not(target_os = "macos"))] + { + IpAddr::V4(Ipv4Addr::LOCALHOST) + } + } else { + trimmed.parse::().map_err(|err| { + crate::client::PodmanApiError::InvalidInput(format!( + "invalid host_gateway_ip value '{trimmed}': {err}" + )) + })? + }; + + let normalized = match address { + IpAddr::V6(ip) => ip.to_ipv4_mapped().map_or(address, IpAddr::V4), + IpAddr::V4(_) => address, + }; + if address.is_unspecified() + || address.is_multicast() + || normalized == IpAddr::V4(Ipv4Addr::BROADCAST) + || normalized == IpAddr::V4(Ipv4Addr::new(169, 254, 169, 254)) + { + return Err(crate::client::PodmanApiError::InvalidInput(format!( + "invalid host_gateway_ip value '{address}': address is not a safe concrete host destination" + ))); } - trimmed.parse::().map(|_| ()).map_err(|err| { - crate::client::PodmanApiError::InvalidInput(format!( - "invalid host_gateway_ip value '{trimmed}': {err}" - )) - }) + Ok(address) } /// Resolve the default host gateway override for the current platform. @@ -648,6 +678,12 @@ mod tests { let cfg = PodmanComputeConfig::default(); assert_eq!(cfg.host_gateway_ip, MACOS_PODMAN_MACHINE_HOST_GATEWAY_IP); assert!(cfg.validate_host_gateway_ip().is_ok()); + assert_eq!( + cfg.resolved_host_gateway_ip().unwrap(), + MACOS_PODMAN_MACHINE_HOST_GATEWAY_IP + .parse::() + .unwrap() + ); } #[test] @@ -656,6 +692,10 @@ mod tests { let cfg = PodmanComputeConfig::default(); assert!(cfg.host_gateway_ip.is_empty()); assert!(cfg.validate_host_gateway_ip().is_ok()); + assert_eq!( + cfg.resolved_host_gateway_ip().unwrap(), + IpAddr::V4(Ipv4Addr::LOCALHOST) + ); } #[test] @@ -668,6 +708,33 @@ mod tests { assert!(err.to_string().contains("host_gateway_ip")); } + #[test] + fn host_gateway_ip_resolution_preserves_explicit_address() { + let cfg = PodmanComputeConfig { + host_gateway_ip: "192.168.127.254".to_string(), + ..PodmanComputeConfig::default() + }; + assert_eq!( + cfg.resolved_host_gateway_ip().unwrap(), + "192.168.127.254".parse::().unwrap() + ); + } + + #[test] + fn host_gateway_ip_validation_rejects_unsafe_destinations() { + for address in ["0.0.0.0", "224.0.0.1", "255.255.255.255", "169.254.169.254"] { + let cfg = PodmanComputeConfig { + host_gateway_ip: address.to_string(), + ..PodmanComputeConfig::default() + }; + let error = cfg.validate_host_gateway_ip().unwrap_err(); + assert!( + error.to_string().contains("safe concrete host destination"), + "unexpected validation error for {address}: {error}" + ); + } + } + #[test] fn runtime_limit_rejects_invalid_pids_limits() { let zero = serde_json::from_value::(serde_json::json!({ diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index 1c7d0f62db..bb4ee88c9f 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -53,6 +53,7 @@ const VOLUME_PREFIX: &str = "openshell-sandbox-"; /// Secret name prefix for per-sandbox gateway JWTs. const TOKEN_SECRET_PREFIX: &str = "openshell-token-"; const PROXY_AUTH_SECRET_PREFIX: &str = "openshell-proxy-auth-"; +const RESOLVER_SECRET_PREFIX: &str = "openshell-resolver-"; const TLS_CA_SECRET_PREFIX: &str = "openshell-tls-ca-"; const TLS_CERT_SECRET_PREFIX: &str = "openshell-tls-cert-"; const TLS_KEY_SECRET_PREFIX: &str = "openshell-tls-key-"; @@ -64,6 +65,7 @@ const TLS_KEY_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_KEY_MOUNT_PAT const SANDBOX_TOKEN_MOUNT_PATH: &str = openshell_core::driver_utils::SANDBOX_TOKEN_MOUNT_PATH; const UPSTREAM_PROXY_AUTH_MOUNT_PATH: &str = openshell_core::driver_utils::UPSTREAM_PROXY_AUTH_MOUNT_PATH; +const RESOLV_CONF_PATH: &str = "/etc/resolv.conf"; const PROXY_CA_MOUNT_PATH: &str = openshell_core::driver_utils::PROXY_CA_MOUNT_PATH; const PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR: &str = openshell_core::driver_utils::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR; @@ -193,6 +195,12 @@ pub fn proxy_auth_secret_name(sandbox_id: &str) -> String { format!("{PROXY_AUTH_SECRET_PREFIX}{sandbox_id}") } +/// Build the per-sandbox Podman secret name for the mediated DNS resolver. +#[must_use] +pub fn resolver_secret_name(sandbox_id: &str) -> String { + format!("{RESOLVER_SECRET_PREFIX}{sandbox_id}") +} + /// Build per-sandbox Podman secret names for TLS CA, cert, and key. #[must_use] pub fn tls_secret_names(sandbox_id: &str) -> [String; 3] { @@ -253,14 +261,17 @@ pub struct ContainerSpec { /// File-mounted Podman secrets. secrets: Vec, stop_timeout: u32, - /// Extra /etc/hosts entries. Used to inject `host.containers.internal` - /// via Podman's `host-gateway` magic so sandbox containers can reach - /// the gateway server running on the host in rootless mode. + /// Extra /etc/hosts entries for the networked supervisor container. + /// The isolated workload resolves host aliases through policy DNS. hostadd: Vec, /// Search domains written to `/etc/resolv.conf` by Podman. dns_search: Vec, /// Resolver options written to `/etc/resolv.conf` by Podman. dns_option: Vec, + /// Preserve the image resolver path so a driver-owned read-only resolver + /// file can be mounted there without Podman replacing it. + #[serde(skip_serializing_if = "Option::is_none")] + use_image_resolve_conf: Option, netns: NetNS, // Matches libpod's network spec format, which is `{name: {opts}}` where // empty opts is a unit struct rather than `()`. Keep as a map so JSON @@ -1269,15 +1280,16 @@ fn build_base_spec( secrets }, stop_timeout: config.stop_timeout_secs, - // Inject stable host aliases into /etc/hosts so sandbox containers can - // reach services on the host. `host.openshell.internal` is the driver- - // neutral alias used by policies and e2e tests. + // Inject stable host aliases into the networked supervisor container. + // The workload clears these entries and resolves the driver-neutral + // alias through the policy-DNS relay instead. hostadd: hostadd_entries(config), // Preserve Podman's resolver defaults for both policy-DNS and ordinary // sandboxes. Namespace-local capture supports UDP and TCP, so it must // not depend on a libc-specific option or alter short-name searches. dns_search: Vec::new(), dns_option: Vec::new(), + use_image_resolve_conf: None, netns: NetNS { nsmode: "bridge".to_string(), }, @@ -1424,6 +1436,7 @@ pub struct IsolationSpecInput<'a> { pub sandbox: &'a DriverSandbox, pub config: &'a PodmanComputeConfig, pub token_secret: Option<&'a str>, + pub resolver_secret: &'a str, pub gpu_devices: Option<&'a [String]>, pub requested_image: &'a str, pub image_id: &'a str, @@ -1539,6 +1552,14 @@ pub fn build_isolation_specs( workload.hostadd.clear(); workload.secret_env.clear(); workload.secrets.clear(); + workload.secrets.push(SecretMount { + source: input.resolver_secret.to_string(), + target: RESOLV_CONF_PATH.into(), + uid: 0, + gid: 0, + mode: 0o444, + }); + workload.use_image_resolve_conf = Some(true); workload.healthconfig.test = vec!["NONE".into()]; workload .mounts @@ -1803,6 +1824,7 @@ mod tests { sandbox: &sandbox, config: &config, token_secret: Some("jwt"), + resolver_secret: "resolver", gpu_devices: None, requested_image: "image:latest", image_id: "sha256:image", @@ -1866,6 +1888,7 @@ mod tests { sandbox: &sandbox, config: &config, token_secret: Some("jwt"), + resolver_secret: "resolver", gpu_devices: None, requested_image: "image:latest", image_id: "sha256:image", @@ -1907,7 +1930,14 @@ mod tests { assert!(specs.supervisor.portmappings.is_empty()); assert!(specs.workload.env.is_empty()); assert_eq!(specs.workload.unsetenv, vec!["LD_PRELOAD", "HTTP_PROXY"]); - assert!(specs.workload.secrets.is_empty()); + assert_eq!(specs.workload.secrets.len(), 1); + assert_eq!(specs.workload.secrets[0].source, "resolver"); + assert_eq!(specs.workload.secrets[0].target, RESOLV_CONF_PATH); + assert_eq!(specs.workload.secrets[0].uid, 0); + assert_eq!(specs.workload.secrets[0].gid, 0); + assert_eq!(specs.workload.secrets[0].mode, 0o444); + assert_eq!(specs.workload.use_image_resolve_conf, Some(true)); + assert_eq!(specs.supervisor.use_image_resolve_conf, None); assert!( specs .workload diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index 4493192b2e..3788c756f4 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -32,6 +32,7 @@ use tracing::{Instrument as _, debug, info, warn}; const STOP_COMPLETION_POLL_INTERVAL: Duration = Duration::from_millis(50); const STOP_COMPLETION_TIMEOUT_HEADROOM: Duration = Duration::from_secs(5); +const POLICY_DNS_RESOLV_CONF: &[u8] = b"nameserver 127.0.0.53\n"; #[derive(Clone, Copy, Debug, Eq, PartialEq)] enum PodmanEndpointEnvironment { @@ -185,6 +186,28 @@ async fn cleanup_sandbox_token_secret(client: &PodmanClient, secret_name: &str) } } +async fn create_sandbox_resolver_secret( + client: &PodmanClient, + sandbox_id: &str, +) -> Result { + let secret_name = container::resolver_secret_name(sandbox_id); + client + .create_secret(&secret_name, POLICY_DNS_RESOLV_CONF) + .await + .map_err(ComputeDriverError::from)?; + Ok(secret_name) +} + +async fn cleanup_sandbox_resolver_secret(client: &PodmanClient, secret_name: &str) { + if let Err(err) = client.remove_secret(secret_name).await { + warn!( + secret = %secret_name, + error = %err, + "Failed to remove Podman sandbox resolver secret" + ); + } +} + /// Read the operator's proxy credentials file and stage it as a per-sandbox /// Podman secret, so the credentials reach the supervisor through a root-only /// mount rather than the container environment. @@ -977,6 +1000,10 @@ impl PodmanComputeDriver { // failure. The supervisor independently validates the certificate // content at startup. validate_sandbox_proxy_ca_bundle(&self.config).await?; + let host_gateway_ip = self + .config + .resolved_host_gateway_ip() + .map_err(ComputeDriverError::from)?; let identity = self .resolve_workload_identity(sandbox, &immutable_image_id, &image_user) @@ -1004,22 +1031,32 @@ impl PodmanComputeDriver { )); } - // Create workspace volume and per-sandbox token secret. - let (token_secret_name, proxy_auth_secret_name) = async { + // Create the workspace volume and per-sandbox runtime files. + let (resolver_secret_name, token_secret_name, proxy_auth_secret_name) = async { let phase_status = openshell_otel::ErrorStatusGuard::current(); let result = async { self.client .create_owned_volume(&vol_name, &sandbox.id, &sandbox.workspace) .await .map_err(ComputeDriverError::from)?; - let token_secret_name = - match create_sandbox_token_secret(&self.client, sandbox).await { + let resolver_secret_name = + match create_sandbox_resolver_secret(&self.client, &sandbox.id).await { Ok(name) => name, Err(e) => { let _ = self.client.remove_volume(&vol_name).await; return Err(e); } }; + let token_secret_name = match create_sandbox_token_secret(&self.client, sandbox) + .await + { + Ok(name) => name, + Err(e) => { + let _ = self.client.remove_volume(&vol_name).await; + cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name).await; + return Err(e); + } + }; let proxy_auth_secret_name = match create_sandbox_proxy_auth_secret(&self.client, &self.config, sandbox) .await @@ -1030,10 +1067,16 @@ impl PodmanComputeDriver { if let Some(secret) = token_secret_name.as_deref() { cleanup_sandbox_token_secret(&self.client, secret).await; } + cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name) + .await; return Err(e); } }; - Ok((token_secret_name, proxy_auth_secret_name)) + Ok(( + resolver_secret_name, + token_secret_name, + proxy_auth_secret_name, + )) } .await; phase_status.finish(result) @@ -1046,14 +1089,15 @@ impl PodmanComputeDriver { )) .await?; - // Clean up the volume and both per-sandbox secrets on any failure past - // this point. + // Clean up the volume and per-sandbox secrets on any failure past this + // point. let channel_owned = std::sync::atomic::AtomicBool::new(false); let cleanup_created = || async { if channel_owned.load(std::sync::atomic::Ordering::Relaxed) { let _ = self.client.remove_volume(&channel_volume).await; } let _ = self.client.remove_volume(&vol_name).await; + cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name).await; if let Some(secret) = token_secret_name.as_deref() { cleanup_sandbox_token_secret(&self.client, secret).await; } @@ -1111,6 +1155,7 @@ impl PodmanComputeDriver { sandbox, config: &runtime_config, token_secret: token_secret_name.as_deref(), + resolver_secret: &resolver_secret_name, gpu_devices: gpu_devices.as_deref(), requested_image: &image, image_id: &immutable_image_id, @@ -1154,16 +1199,21 @@ impl PodmanComputeDriver { .and_then(|spec| { decode_launch_authentication(&spec.launch_authentication) })?; + let generation = uuid::Uuid::new_v4().to_string(); let archives = crate::isolation::bootstrap_archives( - &sandbox.id, - &workload_id, - &uuid::Uuid::new_v4().to_string(), - &identity, - crate::isolation::userns_preserves_host_groups( - self.config.userns.as_deref(), - ), - child_env, - &launch_authentication, + crate::isolation::BootstrapArchivesInput { + sandbox_id: &sandbox.id, + container_id: &workload_id, + generation: &generation, + host_gateway_ip, + identity: &identity, + allow_extra_supplementary_groups: + crate::isolation::userns_preserves_host_groups( + self.config.userns.as_deref(), + ), + child_env, + launch_authentication: &launch_authentication, + }, )?; self.client .copy_to_container( @@ -1499,15 +1549,23 @@ impl PodmanComputeDriver { let bundle = extract_first_tar_entry(&archive).map_err(ComputeDriverError::Precondition)?; let restart_metadata = crate::isolation::restart_metadata_from_slice(&bundle)?; - let archives = crate::isolation::bootstrap_archives( - sandbox_id, - &container_id, - generation.as_str(), - &restart_metadata.workload_identity, - crate::isolation::userns_preserves_host_groups(self.config.userns.as_deref()), - restart_metadata.child_env, - &launch_authentication, - )?; + let archives = + crate::isolation::bootstrap_archives(crate::isolation::BootstrapArchivesInput { + sandbox_id, + container_id: &container_id, + generation: generation.as_str(), + host_gateway_ip: self + .config + .resolved_host_gateway_ip() + .map_err(ComputeDriverError::from)?, + identity: &restart_metadata.workload_identity, + allow_extra_supplementary_groups: + crate::isolation::userns_preserves_host_groups( + self.config.userns.as_deref(), + ), + child_env: restart_metadata.child_env, + launch_authentication: &launch_authentication, + })?; self.client .copy_to_container( &container_id, @@ -1575,6 +1633,11 @@ impl PodmanComputeDriver { } cleanup_sandbox_token_secret(&self.client, &container::token_secret_name(sandbox_id)) .await; + cleanup_sandbox_resolver_secret( + &self.client, + &container::resolver_secret_name(sandbox_id), + ) + .await; cleanup_sandbox_proxy_auth_secret( &self.client, &container::proxy_auth_secret_name(sandbox_id), @@ -1617,6 +1680,8 @@ impl PodmanComputeDriver { ); } cleanup_sandbox_token_secret(&self.client, &container::token_secret_name(sandbox_id)).await; + cleanup_sandbox_resolver_secret(&self.client, &container::resolver_secret_name(sandbox_id)) + .await; cleanup_sandbox_proxy_auth_secret( &self.client, &container::proxy_auth_secret_name(sandbox_id), @@ -3335,7 +3400,7 @@ mod tests { assert!(!environment.keys().any(|key| key.starts_with("OPENSHELL_"))); } - fn secret_delete_request(sandbox_id: &str) -> String { + fn proxy_auth_secret_delete_request(sandbox_id: &str) -> String { format!( "DELETE {}", api_path(&format!( @@ -3345,6 +3410,16 @@ mod tests { ) } + fn resolver_secret_delete_request(sandbox_id: &str) -> String { + format!( + "DELETE {}", + api_path(&format!( + "/libpod/secrets/{}", + container::resolver_secret_name(sandbox_id) + )) + ) + } + fn restart_responses() -> Vec { let mut archive = tar::Builder::new(Vec::new()); let identity = openshell_isolation_interface::contract::ResolvedWorkloadIdentity::new( @@ -3482,6 +3557,7 @@ mod tests { StubResponse::new(StatusCode::NOT_FOUND, ""), // no existing private workspace StubResponse::new(StatusCode::CREATED, "{}"), // workspace volume owned_volume_response(&container::volume_name(sandbox_id), sandbox_id), + StubResponse::new(StatusCode::CREATED, "{}"), // resolver secret ]; if proxy_secret { responses.push(StubResponse::new(StatusCode::CREATED, "{}")); @@ -3580,6 +3656,7 @@ mod tests { StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, "create failed"), StubResponse::new(StatusCode::NO_CONTENT, ""), // channel StubResponse::new(StatusCode::NO_CONTENT, ""), // workspace + StubResponse::new(StatusCode::NO_CONTENT, ""), // resolver secret StubResponse::new(StatusCode::NO_CONTENT, ""), // proxy secret ]) .collect(), @@ -3602,9 +3679,13 @@ mod tests { .expect("request log lock should not be poisoned") .clone(); assert!( - requests.contains(&secret_delete_request(sandbox_id)), + requests.contains(&proxy_auth_secret_delete_request(sandbox_id)), "proxy-auth secret must be removed on container-create failure: {requests:?}" ); + assert!( + requests.contains(&resolver_secret_delete_request(sandbox_id)), + "resolver secret must be removed on container-create failure: {requests:?}" + ); let _ = fs::remove_file(&auth_file); let _ = fs::remove_file(socket_path); } @@ -3626,6 +3707,7 @@ mod tests { StubResponse::new(StatusCode::NO_CONTENT, ""), // workload StubResponse::new(StatusCode::NO_CONTENT, ""), // channel StubResponse::new(StatusCode::NO_CONTENT, ""), // workspace + StubResponse::new(StatusCode::NO_CONTENT, ""), // resolver secret StubResponse::new(StatusCode::NO_CONTENT, ""), // proxy secret ]) .collect(), @@ -3648,9 +3730,13 @@ mod tests { .expect("request log lock should not be poisoned") .clone(); assert!( - requests.contains(&secret_delete_request(sandbox_id)), + requests.contains(&proxy_auth_secret_delete_request(sandbox_id)), "proxy-auth secret must be removed on start failure: {requests:?}" ); + assert!( + requests.contains(&resolver_secret_delete_request(sandbox_id)), + "resolver secret must be removed on start failure: {requests:?}" + ); let _ = fs::remove_file(&auth_file); let _ = fs::remove_file(socket_path); } @@ -3668,6 +3754,7 @@ mod tests { StubResponse::new(StatusCode::OK, "[]"), // list_containers (not found) StubResponse::new(StatusCode::NO_CONTENT, ""), // remove volume StubResponse::new(StatusCode::NO_CONTENT, ""), // remove token secret + StubResponse::new(StatusCode::NO_CONTENT, ""), // remove resolver secret StubResponse::new(StatusCode::NO_CONTENT, ""), // remove proxy-auth secret ], ); @@ -3684,9 +3771,13 @@ mod tests { .expect("request log lock should not be poisoned") .clone(); assert!( - requests.contains(&secret_delete_request(sandbox_id)), + requests.contains(&proxy_auth_secret_delete_request(sandbox_id)), "proxy-auth secret must be removed on delete: {requests:?}" ); + assert!( + requests.contains(&resolver_secret_delete_request(sandbox_id)), + "resolver secret must be removed on delete: {requests:?}" + ); let _ = fs::remove_file(socket_path); } diff --git a/crates/openshell-driver-podman/src/isolation.rs b/crates/openshell-driver-podman/src/isolation.rs index 2524186cbe..ae33796ea6 100644 --- a/crates/openshell-driver-podman/src/isolation.rs +++ b/crates/openshell-driver-podman/src/isolation.rs @@ -195,17 +195,32 @@ pub struct RestartMetadata { pub(crate) child_env: HashMap, } +pub struct BootstrapArchivesInput<'a> { + pub sandbox_id: &'a str, + pub container_id: &'a str, + pub generation: &'a str, + pub host_gateway_ip: std::net::IpAddr, + pub identity: &'a ResolvedWorkloadIdentity, + pub allow_extra_supplementary_groups: bool, + pub child_env: HashMap, + pub launch_authentication: &'a openshell_core::jwt::SandboxLaunchAuthentication, +} + /// The shared volume contains only sandbox credentials. Supervisor credentials, /// gateway authorization, and the restart copy never enter that volume. pub fn bootstrap_archives( - sandbox_id: &str, - container_id: &str, - generation: &str, - identity: &ResolvedWorkloadIdentity, - allow_extra_supplementary_groups: bool, - child_env: HashMap, - launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication, + input: BootstrapArchivesInput<'_>, ) -> Result { + let BootstrapArchivesInput { + sandbox_id, + container_id, + generation, + host_gateway_ip, + identity, + allow_extra_supplementary_groups, + child_env, + launch_authentication, + } = input; launch_authentication.validate().map_err(invalid)?; let session_id = launch_authentication.supervisor.session_id; let tls = generate_sandbox_tls_material(session_id).map_err(invalid)?; @@ -277,7 +292,7 @@ pub fn bootstrap_archives( server_name: tls.server_name, trust_anchor_pem: tls.trust_anchor_pem, }, - host_gateway_ip: None, + host_gateway_ip: Some(host_gateway_ip), resource_claims, workload_identity: identity.clone(), outer_fence, @@ -505,15 +520,16 @@ mod tests { .unwrap(); let authentication = authentication(); let child_env = HashMap::from([("PATH".to_string(), "/agent/bin".to_string())]); - let archives = bootstrap_archives( - "sandbox", - "container", - "generation-1", - &identity, - false, - child_env.clone(), - &authentication, - ) + let archives = bootstrap_archives(BootstrapArchivesInput { + sandbox_id: "sandbox", + container_id: "container", + generation: "generation-1", + host_gateway_ip: "127.0.0.1".parse().unwrap(), + identity: &identity, + allow_extra_supplementary_groups: false, + child_env: child_env.clone(), + launch_authentication: &authentication, + }) .unwrap(); let workload = files(&archives.channel); let supervisor = files(&archives.supervisor); @@ -552,6 +568,10 @@ mod tests { assert_eq!(config.session_id, runtime_descriptor.session_id); assert_eq!(config.outer_fence, runtime_descriptor.outer_fence); assert_eq!(config.workload_identity, identity); + assert_eq!( + runtime_descriptor.host_gateway_ip, + Some("127.0.0.1".parse().unwrap()) + ); runtime_descriptor .outer_fence .validate(&runtime_descriptor.generation) diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs index d6ad260d0e..d04b20cea7 100644 --- a/crates/openshell-driver-podman/src/main.rs +++ b/crates/openshell-driver-podman/src/main.rs @@ -73,9 +73,10 @@ struct Args { )] gateway_port: u16, - /// Host gateway IP used for sandbox host aliases. + /// Trusted supervisor-side destination used for sandbox host aliases. /// - /// Empty uses Podman's `host-gateway` resolver. + /// Empty uses loopback on native Linux and the gvproxy host address on + /// macOS Podman Machine. #[arg(long, env = "OPENSHELL_PODMAN_HOST_GATEWAY_IP")] host_gateway_ip: Option, diff --git a/crates/openshell-driver-vm/Cargo.toml b/crates/openshell-driver-vm/Cargo.toml index 9c2ef8c81d..ad676f8302 100644 --- a/crates/openshell-driver-vm/Cargo.toml +++ b/crates/openshell-driver-vm/Cargo.toml @@ -56,7 +56,7 @@ serde = { workspace = true } serde_json = { workspace = true } oci-client = { version = "0.16", optional = true } libc = "0.2" -rustix = { workspace = true, optional = true } +rustix = { workspace = true, features = ["fs"], optional = true } libloading = { version = "0.8", optional = true } tar = { version = "0.4", optional = true } flate2 = { version = "1", optional = true } diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index 3f483577a8..0ec5758fac 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -161,7 +161,7 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`, | `state_dir` | `target/openshell-vm-driver` | Per-sandbox overlay disks, console logs, image cache, and private `run/compute-driver.sock` UDS. Relative paths are resolved to absolute paths at driver startup. | | `driver_dir` | unset | Override the directory searched for `openshell-driver-vm`. | | `default_image` | OpenShell base image | Sandbox image used when a create request omits one. | -| `bootstrap_image` | unset | VM runtime image used as the immutable bootstrap root disk. Defaults to the sandbox image when unset. | +| `bootstrap_image` | unset | VM runtime image used as the immutable bootstrap root disk. Falls back only to the operator-configured `default_image`; the driver refuses to start when both are empty. | | `vcpus` | `2` | vCPUs per sandbox. | | `mem_mib` | `2048` | Memory per sandbox, in MiB. | | `overlay_disk_mib` | `4096` | Sparse writable overlay disk size per sandbox, in MiB. | @@ -212,7 +212,14 @@ and reports the image-prep console tail. Set `OPENSHELL_VM_IMAGE_PULL_CONCURRENC tune registry layer download parallelism (default `4`, maximum `16`). Both caches are scoped by source image identity and OpenShell version, so an OpenShell upgrade builds a fresh guest rootfs instead of reusing one with an old -embedded supervisor. +embedded supervisor. Host-side bootstrap layer assembly preserves relative +symlinks that remain inside the rootfs, but rejects absolute or escaping +cross-layer symlink traversal before applying later files or whiteouts. The +layer applier performs rootfs mutations relative to opened directory handles +and creates destination files without following symlinks. +The requested sandbox image is never selected as the bootstrap image. Operators +must configure either `bootstrap_image` or `default_image`; when both are empty, +the driver fails during startup. Each sandbox gets its own sparse writable `/sandboxes//overlay.ext4`. Guest init mounts overlayfs as `/` diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 78e0b7b63c..8cdabb22c5 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -6,6 +6,7 @@ use crate::gpu::{GpuInventory, allocate_vsock_cid}; use crate::isolation::VmBoundarySpec; +use crate::layer_applier::apply_layer_dir_to_rootfs; use crate::lifecycle::{ BackendFeature, GuestInitDropin, LaunchAbortReason, LaunchPlan, LifecycleExtensionRegistry, RestoreContext, extension_state_dir, @@ -199,7 +200,7 @@ const GUEST_IMAGE_CONFIG_DIR: &str = "openshell-image"; const GUEST_IMAGE_OCI_LAYOUT_DIR: &str = "oci"; const GUEST_IMAGE_OCI_REF: &str = "openshell"; const IMAGE_EXPORT_ROOTFS_ARCHIVE: &str = "source-rootfs.tar"; -const BOOTSTRAP_IMAGE_CACHE_LAYOUT_VERSION: &str = "sandbox-bootstrap-rootfs-ext4-v4"; +const BOOTSTRAP_IMAGE_CACHE_LAYOUT_VERSION: &str = "sandbox-bootstrap-rootfs-ext4-v5"; const PREPARED_IMAGE_CACHE_LAYOUT_VERSION: &str = "sandbox-prepared-rootfs-ext4-umoci-v3"; const IMAGE_IDENTITY_FILE: &str = "image-identity"; const IMAGE_REFERENCE_FILE: &str = "image-reference"; @@ -433,6 +434,16 @@ impl VmDriverConfig { Ok(()) } + pub fn validate_bootstrap_image_config(&self) -> Result<(), String> { + if self.bootstrap_image.trim().is_empty() && self.default_image.trim().is_empty() { + return Err( + "vm driver requires bootstrap_image or default_image; the sandbox image cannot be used as the VM bootstrap image" + .to_string(), + ); + } + Ok(()) + } + pub fn validate_rootfs_tar_config(&self) -> Result<(), String> { if self .rootfs_tar_staging_dir @@ -690,6 +701,7 @@ impl VmDriver { .map_err(|err| err.message().to_string())?; config.validate_sandbox_identity()?; config.validate_runtime_security_config()?; + config.validate_bootstrap_image_config()?; config.validate_rootfs_tar_config()?; if config.grpc_endpoint.trim().is_empty() { return Err("openshell endpoint is required".to_string()); @@ -711,7 +723,7 @@ impl VmDriver { ) })?; let image_cache_root = image_cache_root_dir(&config.state_dir); - tokio::fs::create_dir_all(&image_cache_root) + create_private_dir_all(&image_cache_root) .await .map_err(|err| { format!( @@ -1316,7 +1328,7 @@ impl VmDriver { "cannot restore rootfs-tar sandbox: persisted image identity not found: {err}" )) })?; - let bootstrap_image_ref = self.bootstrap_image_ref(&image_ref); + let bootstrap_image_ref = self.bootstrap_image_ref()?; let bootstrap_image_identity = self .ensure_cached_bootstrap_rootfs_image(&sandbox.id, &bootstrap_image_ref) .await?; @@ -2703,7 +2715,7 @@ impl VmDriver { rootfs_tar_path: Option<&Path>, ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); - let bootstrap_image_ref = self.bootstrap_image_ref(image_ref); + let bootstrap_image_ref = self.bootstrap_image_ref()?; let bootstrap_image_identity = self .ensure_cached_bootstrap_rootfs_image(sandbox_id, &bootstrap_image_ref) .await?; @@ -2741,9 +2753,13 @@ impl VmDriver { })) } - fn bootstrap_image_ref(&self, sandbox_image_ref: &str) -> String { + fn bootstrap_image_ref(&self) -> Result { self.bootstrap_image_ref_default() - .unwrap_or_else(|| sandbox_image_ref.to_string()) + .ok_or_else(|| { + Status::failed_precondition( + "vm driver requires bootstrap_image or default_image; the sandbox image cannot be used as the VM bootstrap image", + ) + }) } fn bootstrap_image_ref_default(&self) -> Option { @@ -5492,142 +5508,6 @@ fn layer_compression_from_media_type(media_type: &str) -> Result Result<(), String> { - merge_layer_directory(layer_root, rootfs) -} - -fn merge_layer_directory(source_dir: &Path, target_dir: &Path) -> Result<(), String> { - fs::create_dir_all(target_dir) - .map_err(|err| format!("create {}: {err}", target_dir.display()))?; - - let mut entries = fs::read_dir(source_dir) - .map_err(|err| format!("read {}: {err}", source_dir.display()))? - .collect::, _>>() - .map_err(|err| format!("read {}: {err}", source_dir.display()))?; - entries.sort_by_key(fs::DirEntry::file_name); - - if entries - .iter() - .any(|entry| entry.file_name().to_string_lossy() == ".wh..wh..opq") - { - clear_directory_contents(target_dir)?; - } - - for entry in entries { - let file_name = entry.file_name(); - let name = file_name.to_string_lossy(); - if name == ".wh..wh..opq" { - continue; - } - if let Some(hidden_name) = name.strip_prefix(".wh.") { - remove_path_if_exists(&target_dir.join(hidden_name))?; - continue; - } - - let source_path = entry.path(); - let dest_path = target_dir.join(&file_name); - let metadata = fs::symlink_metadata(&source_path) - .map_err(|err| format!("stat {}: {err}", source_path.display()))?; - let file_type = metadata.file_type(); - - if file_type.is_dir() { - if let Ok(dest_metadata) = fs::symlink_metadata(&dest_path) - && !dest_metadata.file_type().is_dir() - && !path_is_dir_or_symlink_to_dir(&dest_path)? - { - remove_path_if_exists(&dest_path)?; - } - fs::create_dir_all(&dest_path) - .map_err(|err| format!("create {}: {err}", dest_path.display()))?; - merge_layer_directory(&source_path, &dest_path)?; - if fs::symlink_metadata(&dest_path) - .map_err(|err| format!("stat {}: {err}", dest_path.display()))? - .file_type() - .is_dir() - { - fs::set_permissions(&dest_path, metadata.permissions()) - .map_err(|err| format!("chmod {}: {err}", dest_path.display()))?; - } - } else if file_type.is_file() { - remove_path_if_exists(&dest_path)?; - if let Some(parent) = dest_path.parent() { - fs::create_dir_all(parent) - .map_err(|err| format!("create {}: {err}", parent.display()))?; - } - fs::copy(&source_path, &dest_path).map_err(|err| { - format!( - "copy {} to {}: {err}", - source_path.display(), - dest_path.display() - ) - })?; - fs::set_permissions(&dest_path, metadata.permissions()) - .map_err(|err| format!("chmod {}: {err}", dest_path.display()))?; - } else if file_type.is_symlink() { - copy_symlink(&source_path, &dest_path)?; - } else { - return Err(format!( - "unsupported layer entry type at {}", - source_path.display() - )); - } - } - - Ok(()) -} - -fn path_is_dir_or_symlink_to_dir(path: &Path) -> Result { - match fs::metadata(path) { - Ok(metadata) => Ok(metadata.file_type().is_dir()), - Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(false), - Err(err) => Err(format!("stat {}: {err}", path.display())), - } -} - -fn clear_directory_contents(dir: &Path) -> Result<(), String> { - if !dir.exists() { - return Ok(()); - } - for entry in fs::read_dir(dir).map_err(|err| format!("read {}: {err}", dir.display()))? { - let entry = entry.map_err(|err| format!("read {}: {err}", dir.display()))?; - remove_path_if_exists(&entry.path())?; - } - Ok(()) -} - -fn remove_path_if_exists(path: &Path) -> Result<(), String> { - let Ok(metadata) = fs::symlink_metadata(path) else { - return Ok(()); - }; - if metadata.file_type().is_dir() { - fs::remove_dir_all(path).map_err(|err| format!("remove {}: {err}", path.display())) - } else { - fs::remove_file(path).map_err(|err| format!("remove {}: {err}", path.display())) - } -} - -#[cfg(unix)] -fn copy_symlink(source_path: &Path, dest_path: &Path) -> Result<(), String> { - let target = fs::read_link(source_path) - .map_err(|err| format!("readlink {}: {err}", source_path.display()))?; - remove_path_if_exists(dest_path)?; - if let Some(parent) = dest_path.parent() { - fs::create_dir_all(parent).map_err(|err| format!("create {}: {err}", parent.display()))?; - } - std::os::unix::fs::symlink(&target, dest_path).map_err(|err| { - format!( - "symlink {} to {}: {err}", - target.display(), - dest_path.display() - ) - }) -} - -#[cfg(not(unix))] -fn copy_symlink(_source_path: &Path, _dest_path: &Path) -> Result<(), String> { - Err("symlink layers are only supported on Unix hosts".to_string()) -} - #[derive(Clone, Copy, Debug, Eq, PartialEq)] enum LayerCompression { None, @@ -7497,6 +7377,7 @@ mod tests { let temp = tempfile::tempdir().unwrap(); let mut driver = test_driver_with_extensions(LifecycleExtensionRegistry::new()); driver.config.state_dir = temp.path().to_path_buf(); + driver.config.bootstrap_image = "invalid bootstrap image reference".to_string(); let sandbox = Sandbox { id: "sb-spawned-trace".to_string(), name: "spawned-trace".to_string(), @@ -9191,7 +9072,7 @@ mod tests { }; assert_eq!( - driver.bootstrap_image_ref("ghcr.io/example/app:latest"), + driver.bootstrap_image_ref().unwrap(), "openshell/sandbox-bootstrap:latest" ); } @@ -9215,13 +9096,13 @@ mod tests { }; assert_eq!( - driver.bootstrap_image_ref("ghcr.io/example/app:latest"), + driver.bootstrap_image_ref().unwrap(), "openshell/sandbox:default" ); } #[test] - fn bootstrap_image_ref_falls_back_to_requested_image() { + fn bootstrap_image_ref_rejects_missing_trusted_image() { let (socket_root, socket_root_fd) = test_socket_root(); let driver = VmDriver { config: VmDriverConfig::default(), @@ -9235,10 +9116,41 @@ mod tests { lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), }; - assert_eq!( - driver.bootstrap_image_ref("ghcr.io/example/app:latest"), - "ghcr.io/example/app:latest" - ); + let error = driver + .bootstrap_image_ref() + .expect_err("sandbox images must not become VM bootstrap images"); + assert_eq!(error.code(), Code::FailedPrecondition); + assert!(error.message().contains("sandbox image cannot be used")); + } + + #[tokio::test] + async fn vm_driver_startup_rejects_missing_trusted_bootstrap_image() { + let Err(error) = VmDriver::new(VmDriverConfig::default()).await else { + panic!("driver startup must reject an empty bootstrap configuration"); + }; + assert!(error.contains("sandbox image cannot be used")); + } + + #[test] + fn bootstrap_image_config_requires_a_trusted_image() { + let error = VmDriverConfig::default() + .validate_bootstrap_image_config() + .expect_err("an empty bootstrap configuration must fail closed"); + assert!(error.contains("sandbox image cannot be used")); + + VmDriverConfig { + default_image: "openshell/sandbox:default".to_string(), + ..Default::default() + } + .validate_bootstrap_image_config() + .expect("the operator-controlled default image is a valid fallback"); + + VmDriverConfig { + bootstrap_image: "openshell/sandbox-bootstrap:latest".to_string(), + ..Default::default() + } + .validate_bootstrap_image_config() + .expect("an explicit bootstrap image is valid"); } #[test] @@ -9573,6 +9485,214 @@ mod tests { let _ = fs::remove_dir_all(base); } + #[cfg(unix)] + #[test] + fn apply_layer_dir_to_rootfs_does_not_write_through_escaping_symlink() { + let base = unique_temp_dir(); + let rootfs = base.join("rootfs"); + let layer = base.join("layer"); + let outside = base.join("outside"); + + fs::create_dir_all(&rootfs).unwrap(); + fs::create_dir_all(layer.join("escape")).unwrap(); + fs::create_dir_all(&outside).unwrap(); + fs::write(outside.join("sentinel"), "unchanged").unwrap(); + std::os::unix::fs::symlink(&outside, rootfs.join("escape")).unwrap(); + fs::write(layer.join("escape/payload"), "escaped").unwrap(); + + let error = apply_layer_dir_to_rootfs(&layer, &rootfs).err(); + + let sentinel = fs::read_to_string(outside.join("sentinel")).unwrap(); + let payload_escaped = outside.join("payload").exists(); + let _ = fs::remove_dir_all(base); + + let error = error.expect("escaping symlink should reject the layer"); + assert!( + error.contains("absolute symlink"), + "unexpected error: {error}" + ); + assert_eq!(sentinel, "unchanged"); + assert!( + !payload_escaped, + "upper-layer payload escaped the rootfs through a lower-layer symlink" + ); + } + + #[cfg(unix)] + #[test] + fn apply_layer_dir_to_rootfs_does_not_whiteout_through_escaping_symlink() { + let base = unique_temp_dir(); + let outside_entry = base.join("outside-entry"); + let outside_opaque = base.join("outside-opaque"); + fs::create_dir_all(&outside_entry).unwrap(); + fs::create_dir_all(&outside_opaque).unwrap(); + fs::write(outside_entry.join("victim"), "entry").unwrap(); + fs::write(outside_opaque.join("victim"), "opaque").unwrap(); + + let entry_rootfs = base.join("entry-rootfs"); + let entry_layer = base.join("entry-layer"); + fs::create_dir_all(&entry_rootfs).unwrap(); + fs::create_dir_all(entry_layer.join("escape")).unwrap(); + std::os::unix::fs::symlink(&outside_entry, entry_rootfs.join("escape")).unwrap(); + fs::write(entry_layer.join("escape/.wh.victim"), "").unwrap(); + let entry_error = apply_layer_dir_to_rootfs(&entry_layer, &entry_rootfs).err(); + + let opaque_rootfs = base.join("opaque-rootfs"); + let opaque_layer = base.join("opaque-layer"); + fs::create_dir_all(&opaque_rootfs).unwrap(); + fs::create_dir_all(opaque_layer.join("escape")).unwrap(); + std::os::unix::fs::symlink(&outside_opaque, opaque_rootfs.join("escape")).unwrap(); + fs::write(opaque_layer.join("escape/.wh..wh..opq"), "").unwrap(); + let opaque_error = apply_layer_dir_to_rootfs(&opaque_layer, &opaque_rootfs).err(); + + let entry_remained = outside_entry.join("victim").exists(); + let opaque_remained = outside_opaque.join("victim").exists(); + let _ = fs::remove_dir_all(base); + + let entry_error = + entry_error.expect("escaping symlink should reject an individual whiteout layer"); + let opaque_error = + opaque_error.expect("escaping symlink should reject an opaque whiteout layer"); + assert!( + entry_error.contains("absolute symlink"), + "unexpected error: {entry_error}" + ); + assert!( + opaque_error.contains("absolute symlink"), + "unexpected error: {opaque_error}" + ); + assert!(entry_remained, "individual whiteout escaped the rootfs"); + assert!(opaque_remained, "opaque whiteout escaped the rootfs"); + } + + #[cfg(unix)] + #[test] + fn apply_layer_dir_to_rootfs_rejects_relative_symlink_escape() { + let base = unique_temp_dir(); + let rootfs = base.join("rootfs"); + let layer = base.join("layer"); + let outside = base.join("outside"); + + fs::create_dir_all(&rootfs).unwrap(); + fs::create_dir_all(layer.join("escape")).unwrap(); + fs::create_dir_all(&outside).unwrap(); + std::os::unix::fs::symlink("../outside", rootfs.join("escape")).unwrap(); + fs::write(layer.join("escape/payload"), "escaped").unwrap(); + + let error = apply_layer_dir_to_rootfs(&layer, &rootfs).err(); + let payload_escaped = outside.join("payload").exists(); + let _ = fs::remove_dir_all(base); + + let error = error.expect("escaping symlink should reject the layer"); + assert!( + error.contains("escapes rootfs"), + "unexpected error: {error}" + ); + assert!(!payload_escaped, "relative symlink escaped the rootfs"); + } + + #[cfg(unix)] + #[test] + fn apply_layer_dir_to_rootfs_rejects_directory_symlink_cycles() { + let base = unique_temp_dir(); + let rootfs = base.join("rootfs"); + let layer = base.join("layer"); + fs::create_dir_all(&rootfs).unwrap(); + fs::create_dir_all(layer.join("a")).unwrap(); + fs::write(layer.join("a/payload"), "payload").unwrap(); + std::os::unix::fs::symlink("b", rootfs.join("a")).unwrap(); + std::os::unix::fs::symlink("a", rootfs.join("b")).unwrap(); + + let error = apply_layer_dir_to_rootfs(&layer, &rootfs) + .expect_err("directory symlink cycle must reject the layer"); + + assert!( + error.contains("too many symlinks"), + "unexpected error: {error}" + ); + let _ = fs::remove_dir_all(base); + } + + #[cfg(unix)] + #[test] + fn extracted_layers_do_not_write_through_escaping_symlink() { + let base = unique_temp_dir(); + let rootfs = base.join("rootfs"); + let lower = base.join("lower"); + let upper = base.join("upper"); + // GNU tar's legacy symlink field is limited to 100 bytes, while the + // macOS temporary directory path is already close to that limit. + let outside = Path::new("/tmp").join(format!( + "openshell-vm-layer-test-{}-{:x}", + std::process::id(), + rand::random::() + )); + fs::create_dir_all(&outside).unwrap(); + fs::write(outside.join("sentinel"), "unchanged").unwrap(); + + let mut lower_tar = tar::Builder::new(Vec::new()); + append_test_tar_symlink(&mut lower_tar, "escape", &outside); + let lower_tar = lower_tar.into_inner().expect("finish lower tar"); + extract_tar_reader_to_dir(std::io::Cursor::new(lower_tar), &lower).unwrap(); + + let upper_tar = tar_bytes_with_file("escape/payload", b"escaped"); + extract_tar_reader_to_dir(std::io::Cursor::new(upper_tar), &upper).unwrap(); + + apply_layer_dir_to_rootfs(&lower, &rootfs).unwrap(); + let error = apply_layer_dir_to_rootfs(&upper, &rootfs) + .expect_err("upper layer must not traverse the lower absolute symlink"); + + assert!( + error.contains("absolute symlink"), + "unexpected error: {error}" + ); + assert_eq!( + fs::read_to_string(outside.join("sentinel")).unwrap(), + "unchanged" + ); + assert!(!outside.join("payload").exists()); + + let _ = fs::remove_dir_all(base); + let _ = fs::remove_dir_all(outside); + } + + #[cfg(unix)] + #[test] + fn extracted_layer_implicit_parent_preserves_lower_directory_symlink() { + let base = unique_temp_dir(); + let rootfs = base.join("rootfs"); + let lower = base.join("lower"); + let upper = base.join("upper"); + + let mut lower_tar = tar::Builder::new(Vec::new()); + append_test_tar_file(&mut lower_tar, "usr/bin/bash", b"bash"); + append_test_tar_symlink(&mut lower_tar, "bin", Path::new("usr/bin")); + let lower_tar = lower_tar.into_inner().expect("finish lower tar"); + extract_tar_reader_to_dir(std::io::Cursor::new(lower_tar), &lower).unwrap(); + + // The tar contains no explicit `bin/` entry. Extraction necessarily + // materializes it as an implicit parent for `bin/tool`. + let upper_tar = tar_bytes_with_file("bin/tool", b"tool"); + extract_tar_reader_to_dir(std::io::Cursor::new(upper_tar), &upper).unwrap(); + + apply_layer_dir_to_rootfs(&lower, &rootfs).unwrap(); + apply_layer_dir_to_rootfs(&upper, &rootfs).unwrap(); + + assert!( + fs::symlink_metadata(rootfs.join("bin")) + .unwrap() + .file_type() + .is_symlink(), + "implicit upper parent must not replace the lower /bin symlink" + ); + assert_eq!( + fs::read_to_string(rootfs.join("usr/bin/tool")).unwrap(), + "tool" + ); + + let _ = fs::remove_dir_all(base); + } + #[test] fn layer_compression_from_media_type_supports_common_formats() { assert_eq!( @@ -9969,7 +10089,7 @@ mod tests { fn bootstrap_image_cache_identity_includes_rootfs_layout_version_and_guest_runtime() { let identity = bootstrap_image_cache_identity("sha256:bootstrap-image"); assert!(identity.starts_with(&format!( - "sandbox-bootstrap-rootfs-ext4-v4:openshell-{}:guest-", + "sandbox-bootstrap-rootfs-ext4-v5:openshell-{}:guest-", openshell_core::VERSION ))); assert!(identity.ends_with(":sha256:bootstrap-image")); @@ -10363,6 +10483,11 @@ mod tests { /// Build an uncompressed tar holding a single file. fn tar_bytes_with_file(name: &str, contents: &[u8]) -> Vec { let mut builder = tar::Builder::new(Vec::new()); + append_test_tar_file(&mut builder, name, contents); + builder.into_inner().expect("finish tar") + } + + fn append_test_tar_file(builder: &mut tar::Builder>, name: &str, contents: &[u8]) { let mut header = tar::Header::new_gnu(); header.set_size(u64::try_from(contents.len()).expect("tar entry size fits u64")); header.set_mode(0o644); @@ -10370,7 +10495,18 @@ mod tests { builder .append_data(&mut header, name, contents) .expect("append tar entry"); - builder.into_inner().expect("finish tar") + } + + fn append_test_tar_symlink(builder: &mut tar::Builder>, name: &str, target: &Path) { + let mut header = tar::Header::new_gnu(); + header.set_entry_type(tar::EntryType::Symlink); + header.set_size(0); + header.set_mode(0o777); + header.set_link_name(target).expect("set symlink target"); + header.set_cksum(); + builder + .append_data(&mut header, name, std::io::empty()) + .expect("append symlink entry"); } fn gzip_bytes(bytes: &[u8]) -> Vec { diff --git a/crates/openshell-driver-vm/src/layer_applier.rs b/crates/openshell-driver-vm/src/layer_applier.rs new file mode 100644 index 0000000000..2742c78169 --- /dev/null +++ b/crates/openshell-driver-vm/src/layer_applier.rs @@ -0,0 +1,780 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Applies extracted OCI layers to a VM root filesystem. + +use std::collections::VecDeque; +use std::ffi::OsString; +use std::fs; +use std::io::Write as _; +#[cfg(unix)] +use std::os::unix::ffi::OsStrExt as _; +#[cfg(unix)] +use std::os::unix::fs::PermissionsExt; +use std::path::{Component, Path, PathBuf}; + +pub fn apply_layer_dir_to_rootfs(layer_root: &Path, rootfs: &Path) -> Result<(), String> { + LayerApplier::new(rootfs)?.apply(layer_root) +} + +/// Owns all filesystem mutations performed while merging an extracted OCI +/// layer into a rootfs. +/// +/// Keeping the mutation boundary here makes the distinction between symlinks +/// supplied by the current layer and directory symlinks retained from lower +/// layers explicit. The latter are followed only through a +/// [`DirectoryMergePlan::FollowLowerLayerSymlink`] decision. +struct LayerApplier { + rootfs: PathBuf, + #[cfg(unix)] + root_dir: rustix::fd::OwnedFd, +} + +#[cfg(not(unix))] +#[derive(Debug)] +enum DirectoryMergePlan { + CreateDirectory { path: PathBuf }, + MergeDirectory { path: PathBuf }, + FollowLowerLayerSymlink { target: PathBuf }, + ReplaceLowerLayerEntry { path: PathBuf }, +} + +#[cfg(not(unix))] +impl DirectoryMergePlan { + fn path(&self) -> &Path { + match self { + Self::CreateDirectory { path } + | Self::MergeDirectory { path } + | Self::ReplaceLowerLayerEntry { path } => path, + Self::FollowLowerLayerSymlink { target, .. } => target, + } + } + + fn applies_directory_metadata(&self) -> bool { + !matches!(self, Self::FollowLowerLayerSymlink { .. }) + } +} + +#[cfg(unix)] +struct ResolvedDirectory { + fd: rustix::fd::OwnedFd, + components: Vec, +} + +#[cfg(unix)] +enum DirectoryMergePlan { + CreateDirectory, + MergeDirectory(ResolvedDirectory), + FollowLowerLayerSymlink(ResolvedDirectory), + ReplaceLowerLayerEntry, +} + +impl LayerApplier { + fn new(rootfs: &Path) -> Result { + fs::create_dir_all(rootfs).map_err(|err| format!("create {}: {err}", rootfs.display()))?; + let rootfs = fs::canonicalize(rootfs) + .map_err(|err| format!("canonicalize rootfs {}: {err}", rootfs.display()))?; + #[cfg(unix)] + let root_dir = rustix::fs::open( + &rootfs, + rustix::fs::OFlags::RDONLY + | rustix::fs::OFlags::DIRECTORY + | rustix::fs::OFlags::NOFOLLOW + | rustix::fs::OFlags::CLOEXEC, + rustix::fs::Mode::empty(), + ) + .map_err(|err| format!("open rootfs {}: {err}", rootfs.display()))?; + Ok(Self { + rootfs, + #[cfg(unix)] + root_dir, + }) + } + + fn apply(&self, layer_root: &Path) -> Result<(), String> { + #[cfg(unix)] + { + let root = ResolvedDirectory { + fd: self.open_root_dir()?, + components: Vec::new(), + }; + self.merge_directory(layer_root, &root) + } + #[cfg(not(unix))] + { + self.merge_directory(layer_root, &self.rootfs) + } + } + + #[cfg(not(unix))] + fn merge_directory(&self, source_dir: &Path, target_dir: &Path) -> Result<(), String> { + fs::create_dir_all(target_dir) + .map_err(|err| format!("create {}: {err}", target_dir.display()))?; + + let mut entries = fs::read_dir(source_dir) + .map_err(|err| format!("read {}: {err}", source_dir.display()))? + .collect::, _>>() + .map_err(|err| format!("read {}: {err}", source_dir.display()))?; + entries.sort_by_key(fs::DirEntry::file_name); + + if entries + .iter() + .any(|entry| entry.file_name().to_string_lossy() == ".wh..wh..opq") + { + self.clear_directory_contents(target_dir)?; + } + + for entry in entries { + let file_name = entry.file_name(); + let name = file_name.to_string_lossy(); + if name == ".wh..wh..opq" { + continue; + } + if let Some(hidden_name) = name.strip_prefix(".wh.") { + self.remove_path_if_exists(&target_dir.join(hidden_name))?; + continue; + } + + let source_path = entry.path(); + let dest_path = target_dir.join(&file_name); + let metadata = fs::symlink_metadata(&source_path) + .map_err(|err| format!("stat {}: {err}", source_path.display()))?; + let file_type = metadata.file_type(); + + if file_type.is_dir() { + let plan = self.plan_directory_merge(&dest_path)?; + if matches!(plan, DirectoryMergePlan::ReplaceLowerLayerEntry { .. }) { + self.remove_path_if_exists(plan.path())?; + } + fs::create_dir_all(plan.path()) + .map_err(|err| format!("create {}: {err}", plan.path().display()))?; + self.merge_directory(&source_path, plan.path())?; + if plan.applies_directory_metadata() { + fs::set_permissions(plan.path(), metadata.permissions()) + .map_err(|err| format!("chmod {}: {err}", plan.path().display()))?; + } + } else if file_type.is_file() { + self.remove_path_if_exists(&dest_path)?; + if let Some(parent) = dest_path.parent() { + fs::create_dir_all(parent) + .map_err(|err| format!("create {}: {err}", parent.display()))?; + } + fs::copy(&source_path, &dest_path).map_err(|err| { + format!( + "copy {} to {}: {err}", + source_path.display(), + dest_path.display() + ) + })?; + fs::set_permissions(&dest_path, metadata.permissions()) + .map_err(|err| format!("chmod {}: {err}", dest_path.display()))?; + } else if file_type.is_symlink() { + self.copy_symlink(&source_path, &dest_path)?; + } else { + return Err(format!( + "unsupported layer entry type at {}", + source_path.display() + )); + } + } + + Ok(()) + } + + #[cfg(unix)] + fn merge_directory( + &self, + source_dir: &Path, + target_dir: &ResolvedDirectory, + ) -> Result<(), String> { + let mut entries = fs::read_dir(source_dir) + .map_err(|err| format!("read {}: {err}", source_dir.display()))? + .collect::, _>>() + .map_err(|err| format!("read {}: {err}", source_dir.display()))?; + entries.sort_by_key(fs::DirEntry::file_name); + + if entries + .iter() + .any(|entry| entry.file_name().to_string_lossy() == ".wh..wh..opq") + { + self.clear_directory_contents(target_dir)?; + } + + for entry in entries { + let file_name = entry.file_name(); + let name = file_name.to_string_lossy(); + if name == ".wh..wh..opq" { + continue; + } + if let Some(hidden_name) = name.strip_prefix(".wh.") { + self.remove_entry(target_dir, std::ffi::OsStr::new(hidden_name))?; + continue; + } + + let source_path = entry.path(); + let metadata = fs::symlink_metadata(&source_path) + .map_err(|err| format!("stat {}: {err}", source_path.display()))?; + let file_type = metadata.file_type(); + + if file_type.is_dir() { + let plan = self.plan_directory_merge(target_dir, &file_name)?; + let apply_metadata = + !matches!(plan, DirectoryMergePlan::FollowLowerLayerSymlink(_)); + let child = match plan { + DirectoryMergePlan::CreateDirectory => { + self.create_child_directory(target_dir, &file_name)? + } + DirectoryMergePlan::MergeDirectory(directory) + | DirectoryMergePlan::FollowLowerLayerSymlink(directory) => directory, + DirectoryMergePlan::ReplaceLowerLayerEntry => { + self.remove_entry(target_dir, &file_name)?; + self.create_child_directory(target_dir, &file_name)? + } + }; + self.merge_directory(&source_path, &child)?; + if apply_metadata { + rustix::fs::fchmod(&child.fd, Self::mode_from_metadata(&metadata)).map_err( + |err| format!("chmod {}: {err}", self.directory_path(&child).display()), + )?; + } + } else if file_type.is_file() { + self.copy_file(&source_path, target_dir, &file_name, &metadata)?; + } else if file_type.is_symlink() { + self.copy_symlink(&source_path, target_dir, &file_name)?; + } else { + return Err(format!( + "unsupported layer entry type at {}", + source_path.display() + )); + } + } + + Ok(()) + } + + #[cfg(unix)] + fn plan_directory_merge( + &self, + parent: &ResolvedDirectory, + name: &std::ffi::OsStr, + ) -> Result { + let entry_path = self.entry_path(parent, name); + let stat = match rustix::fs::statat(&parent.fd, name, rustix::fs::AtFlags::SYMLINK_NOFOLLOW) + { + Ok(stat) => stat, + Err(rustix::io::Errno::NOENT) => return Ok(DirectoryMergePlan::CreateDirectory), + Err(err) => return Err(format!("stat {}: {err}", entry_path.display())), + }; + let file_type = rustix::fs::FileType::from_raw_mode(stat.st_mode); + if file_type.is_dir() { + return self + .open_child_directory(parent, name) + .map(DirectoryMergePlan::MergeDirectory); + } + if file_type.is_symlink() { + let mut components = parent.components.clone(); + components.push(name.to_os_string()); + if let Some(directory) = self.resolve_directory(&components)? { + return Ok(DirectoryMergePlan::FollowLowerLayerSymlink(directory)); + } + } + Ok(DirectoryMergePlan::ReplaceLowerLayerEntry) + } + + #[cfg(unix)] + fn open_root_dir(&self) -> Result { + rustix::fs::openat( + &self.root_dir, + ".", + Self::directory_open_flags(), + rustix::fs::Mode::empty(), + ) + .map_err(|err| format!("open rootfs {}: {err}", self.rootfs.display())) + } + + #[cfg(unix)] + fn open_child_directory( + &self, + parent: &ResolvedDirectory, + name: &std::ffi::OsStr, + ) -> Result { + let path = self.entry_path(parent, name); + let fd = rustix::fs::openat( + &parent.fd, + name, + Self::directory_open_flags(), + rustix::fs::Mode::empty(), + ) + .map_err(|err| format!("open directory {}: {err}", path.display()))?; + let mut components = parent.components.clone(); + components.push(name.to_os_string()); + Ok(ResolvedDirectory { fd, components }) + } + + #[cfg(unix)] + fn create_child_directory( + &self, + parent: &ResolvedDirectory, + name: &std::ffi::OsStr, + ) -> Result { + let path = self.entry_path(parent, name); + rustix::fs::mkdirat(&parent.fd, name, rustix::fs::Mode::from_raw_mode(0o700)) + .map_err(|err| format!("create {}: {err}", path.display()))?; + self.open_child_directory(parent, name) + } + + #[cfg(unix)] + fn resolve_directory( + &self, + components: &[OsString], + ) -> Result, String> { + const MAX_SYMLINKS: usize = 40; + + let requested = components + .iter() + .fold(self.rootfs.clone(), |path, component| path.join(component)); + let mut pending = components + .iter() + .cloned() + .map(RootfsPathComponent::Normal) + .collect::>(); + let mut resolved = Vec::::new(); + let mut stack = vec![self.open_root_dir()?]; + let mut symlinks = 0_usize; + + while let Some(component) = pending.pop_front() { + match component { + RootfsPathComponent::Parent => { + if resolved.pop().is_none() { + return Err(format!( + "layer destination {} escapes rootfs {}", + requested.display(), + self.rootfs.display() + )); + } + stack.pop(); + } + RootfsPathComponent::Normal(name) => { + let parent = stack.last().expect("root directory handle remains"); + let candidate = resolved + .iter() + .fold(self.rootfs.clone(), |path, component| path.join(component)) + .join(&name); + let stat = match rustix::fs::statat( + parent, + &name, + rustix::fs::AtFlags::SYMLINK_NOFOLLOW, + ) { + Ok(stat) => stat, + Err(rustix::io::Errno::NOENT | rustix::io::Errno::NOTDIR) => { + return Ok(None); + } + Err(err) => return Err(format!("stat {}: {err}", candidate.display())), + }; + let file_type = rustix::fs::FileType::from_raw_mode(stat.st_mode); + if file_type.is_symlink() { + symlinks += 1; + if symlinks > MAX_SYMLINKS { + return Err(format!( + "too many symlinks while resolving layer destination {}", + requested.display() + )); + } + let target = rustix::fs::readlinkat(parent, &name, Vec::new()) + .map_err(|err| format!("readlink {}: {err}", candidate.display()))?; + let target = Path::new(std::ffi::OsStr::from_bytes(target.to_bytes())); + if target.is_absolute() { + return Err(format!( + "absolute symlink {} -> {} cannot be traversed while applying a layer", + candidate.display(), + target.display() + )); + } + let mut target_components = rootfs_path_components(target)?; + target_components.append(&mut pending); + pending = target_components; + } else if file_type.is_dir() { + let fd = rustix::fs::openat( + parent, + &name, + Self::directory_open_flags(), + rustix::fs::Mode::empty(), + ) + .map_err(|err| format!("open directory {}: {err}", candidate.display()))?; + resolved.push(name); + stack.push(fd); + } else { + return Ok(None); + } + } + } + } + + let fd = stack.pop().expect("resolved directory handle remains"); + Ok(Some(ResolvedDirectory { + fd, + components: resolved, + })) + } + + #[cfg(unix)] + fn copy_file( + &self, + source_path: &Path, + parent: &ResolvedDirectory, + name: &std::ffi::OsStr, + metadata: &fs::Metadata, + ) -> Result<(), String> { + self.remove_entry(parent, name)?; + let path = self.entry_path(parent, name); + let mut source = fs::File::open(source_path) + .map_err(|err| format!("open {}: {err}", source_path.display()))?; + let fd = rustix::fs::openat( + &parent.fd, + name, + rustix::fs::OFlags::WRONLY + | rustix::fs::OFlags::CREATE + | rustix::fs::OFlags::EXCL + | rustix::fs::OFlags::NOFOLLOW + | rustix::fs::OFlags::CLOEXEC, + rustix::fs::Mode::from_raw_mode(0o600), + ) + .map_err(|err| format!("create {}: {err}", path.display()))?; + let mut destination = fs::File::from(fd); + std::io::copy(&mut source, &mut destination).map_err(|err| { + format!( + "copy {} to {}: {err}", + source_path.display(), + path.display() + ) + })?; + destination + .flush() + .map_err(|err| format!("flush {}: {err}", path.display()))?; + rustix::fs::fchmod(&destination, Self::mode_from_metadata(metadata)) + .map_err(|err| format!("chmod {}: {err}", path.display())) + } + + #[cfg(unix)] + fn clear_directory_contents(&self, directory: &ResolvedDirectory) -> Result<(), String> { + let mut stream = rustix::fs::Dir::read_from(&directory.fd) + .map_err(|err| format!("read {}: {err}", self.directory_path(directory).display()))?; + let mut names = Vec::new(); + for entry in &mut stream { + let entry = entry.map_err(|err| { + format!("read {}: {err}", self.directory_path(directory).display()) + })?; + let name = entry.file_name().to_bytes(); + if name != b"." && name != b".." { + names.push(std::ffi::OsStr::from_bytes(name).to_os_string()); + } + } + for name in names { + self.remove_entry(directory, &name)?; + } + Ok(()) + } + + #[cfg(unix)] + fn remove_entry( + &self, + parent: &ResolvedDirectory, + name: &std::ffi::OsStr, + ) -> Result<(), String> { + let path = self.entry_path(parent, name); + let stat = match rustix::fs::statat(&parent.fd, name, rustix::fs::AtFlags::SYMLINK_NOFOLLOW) + { + Ok(stat) => stat, + Err(rustix::io::Errno::NOENT) => return Ok(()), + Err(err) => return Err(format!("stat {}: {err}", path.display())), + }; + if rustix::fs::FileType::from_raw_mode(stat.st_mode).is_dir() { + let directory = self.open_child_directory(parent, name)?; + self.clear_directory_contents(&directory)?; + rustix::fs::unlinkat(&parent.fd, name, rustix::fs::AtFlags::REMOVEDIR) + .map_err(|err| format!("remove {}: {err}", path.display())) + } else { + rustix::fs::unlinkat(&parent.fd, name, rustix::fs::AtFlags::empty()) + .map_err(|err| format!("remove {}: {err}", path.display())) + } + } + + #[cfg(unix)] + fn copy_symlink( + &self, + source_path: &Path, + parent: &ResolvedDirectory, + name: &std::ffi::OsStr, + ) -> Result<(), String> { + let target = fs::read_link(source_path) + .map_err(|err| format!("readlink {}: {err}", source_path.display()))?; + self.remove_entry(parent, name)?; + let path = self.entry_path(parent, name); + rustix::fs::symlinkat(&target, &parent.fd, name) + .map_err(|err| format!("symlink {} to {}: {err}", target.display(), path.display())) + } + + #[cfg(unix)] + fn directory_open_flags() -> rustix::fs::OFlags { + rustix::fs::OFlags::RDONLY + | rustix::fs::OFlags::DIRECTORY + | rustix::fs::OFlags::NOFOLLOW + | rustix::fs::OFlags::CLOEXEC + } + + #[cfg(unix)] + fn mode_from_metadata(metadata: &fs::Metadata) -> rustix::fs::Mode { + let raw = metadata.permissions().mode(); + let mut mode = rustix::fs::Mode::empty(); + for (bit, flag) in [ + (0o4000, rustix::fs::Mode::SUID), + (0o2000, rustix::fs::Mode::SGID), + (0o1000, rustix::fs::Mode::SVTX), + (0o0400, rustix::fs::Mode::RUSR), + (0o0200, rustix::fs::Mode::WUSR), + (0o0100, rustix::fs::Mode::XUSR), + (0o0040, rustix::fs::Mode::RGRP), + (0o0020, rustix::fs::Mode::WGRP), + (0o0010, rustix::fs::Mode::XGRP), + (0o0004, rustix::fs::Mode::ROTH), + (0o0002, rustix::fs::Mode::WOTH), + (0o0001, rustix::fs::Mode::XOTH), + ] { + if raw & bit != 0 { + mode |= flag; + } + } + mode + } + + #[cfg(unix)] + fn directory_path(&self, directory: &ResolvedDirectory) -> PathBuf { + directory + .components + .iter() + .fold(self.rootfs.clone(), |path, component| path.join(component)) + } + + #[cfg(unix)] + fn entry_path(&self, parent: &ResolvedDirectory, name: &std::ffi::OsStr) -> PathBuf { + self.directory_path(parent).join(name) + } + + #[cfg(not(unix))] + fn plan_directory_merge(&self, path: &Path) -> Result { + self.ensure_rootfs_destination(path)?; + let metadata = match fs::symlink_metadata(path) { + Ok(metadata) => metadata, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => { + return Ok(DirectoryMergePlan::CreateDirectory { + path: path.to_path_buf(), + }); + } + Err(err) => return Err(format!("stat {}: {err}", path.display())), + }; + if metadata.file_type().is_dir() { + return Ok(DirectoryMergePlan::MergeDirectory { + path: path.to_path_buf(), + }); + } + if metadata.file_type().is_symlink() { + let target = self.resolve_path_beneath_rootfs(path)?; + match fs::metadata(&target) { + Ok(metadata) if metadata.file_type().is_dir() => { + return Ok(DirectoryMergePlan::FollowLowerLayerSymlink { target }); + } + Ok(_) => {} + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => return Err(format!("stat {}: {err}", target.display())), + } + } + Ok(DirectoryMergePlan::ReplaceLowerLayerEntry { + path: path.to_path_buf(), + }) + } + + #[cfg(not(unix))] + fn clear_directory_contents(&self, dir: &Path) -> Result<(), String> { + if !dir.exists() { + return Ok(()); + } + for entry in fs::read_dir(dir).map_err(|err| format!("read {}: {err}", dir.display()))? { + let entry = entry.map_err(|err| format!("read {}: {err}", dir.display()))?; + self.remove_path_if_exists(&entry.path())?; + } + Ok(()) + } + + #[cfg(not(unix))] + fn remove_path_if_exists(&self, path: &Path) -> Result<(), String> { + self.ensure_rootfs_destination(path)?; + let Ok(metadata) = fs::symlink_metadata(path) else { + return Ok(()); + }; + if metadata.file_type().is_dir() { + fs::remove_dir_all(path).map_err(|err| format!("remove {}: {err}", path.display())) + } else { + fs::remove_file(path).map_err(|err| format!("remove {}: {err}", path.display())) + } + } + + #[cfg(not(unix))] + fn ensure_rootfs_destination(&self, path: &Path) -> Result<(), String> { + path.strip_prefix(&self.rootfs).map(|_| ()).map_err(|_| { + format!( + "layer destination {} is outside rootfs {}", + path.display(), + self.rootfs.display() + ) + }) + } + + #[cfg(not(unix))] + fn resolve_path_beneath_rootfs(&self, path: &Path) -> Result { + const MAX_SYMLINKS: usize = 40; + + // A symlink from an earlier OCI layer has container path semantics, + // not ambient access to the host filesystem. Resolve it component by + // component without following it, and reject targets that leave the + // private image staging root before a later layer performs any + // filesystem operation. + let relative = path.strip_prefix(&self.rootfs).map_err(|_| { + format!( + "layer destination {} is outside rootfs {}", + path.display(), + self.rootfs.display() + ) + })?; + let mut pending = rootfs_path_components(relative)?; + let mut resolved = Vec::::new(); + let mut symlinks = 0_usize; + + while let Some(component) = pending.pop_front() { + match component { + RootfsPathComponent::Parent => { + if resolved.pop().is_none() { + return Err(format!( + "layer destination {} escapes rootfs {}", + path.display(), + self.rootfs.display() + )); + } + } + RootfsPathComponent::Normal(name) => { + let candidate = resolved + .iter() + .fold(self.rootfs.clone(), |path, component| path.join(component)) + .join(&name); + match fs::symlink_metadata(&candidate) { + Ok(metadata) if metadata.file_type().is_symlink() => { + symlinks += 1; + if symlinks > MAX_SYMLINKS { + return Err(format!( + "too many symlinks while resolving layer destination {}", + path.display() + )); + } + let target = fs::read_link(&candidate).map_err(|err| { + format!("readlink {}: {err}", candidate.display()) + })?; + if target.is_absolute() { + return Err(format!( + "absolute symlink {} -> {} cannot be traversed while applying a layer", + candidate.display(), + target.display() + )); + } + let mut target_components = rootfs_path_components(&target)?; + target_components.append(&mut pending); + pending = target_components; + } + Ok(_) => resolved.push(name), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => { + resolved.push(name); + } + Err(err) => return Err(format!("stat {}: {err}", candidate.display())), + } + } + } + } + + Ok(resolved + .into_iter() + .fold(self.rootfs.clone(), |path, component| path.join(component))) + } + + #[cfg(not(unix))] + fn copy_symlink(&self, _source_path: &Path, _dest_path: &Path) -> Result<(), String> { + Err("symlink layers are only supported on Unix hosts".to_string()) + } +} + +#[derive(Debug)] +enum RootfsPathComponent { + Parent, + Normal(OsString), +} + +fn rootfs_path_components(path: &Path) -> Result, String> { + path.components() + .filter_map(|component| match component { + Component::CurDir => None, + Component::ParentDir => Some(Ok(RootfsPathComponent::Parent)), + Component::Normal(name) => Some(Ok(RootfsPathComponent::Normal(name.to_os_string()))), + Component::RootDir | Component::Prefix(_) => Some(Err(format!( + "absolute symlink target {} cannot be traversed while applying a layer", + path.display() + ))), + }) + .collect() +} + +#[cfg(all(test, unix))] +mod tests { + use super::{DirectoryMergePlan, LayerApplier, ResolvedDirectory}; + use std::ffi::OsStr; + use std::fs; + use tempfile::tempdir; + + #[test] + fn keeps_using_opened_directory_after_path_replacement() { + let base = tempdir().unwrap(); + let rootfs = base.path().join("rootfs"); + let upper = base.path().join("upper"); + let outside = base.path().join("outside"); + fs::create_dir_all(rootfs.join("pivot")).unwrap(); + fs::create_dir_all(&upper).unwrap(); + fs::create_dir_all(&outside).unwrap(); + fs::write(rootfs.join("pivot/victim"), "inside").unwrap(); + fs::write(outside.join("victim"), "outside").unwrap(); + fs::write(upper.join(".wh..wh..opq"), "").unwrap(); + fs::write(upper.join("payload"), "payload").unwrap(); + + let applier = LayerApplier::new(&rootfs).unwrap(); + let root = ResolvedDirectory { + fd: applier.open_root_dir().unwrap(), + components: Vec::new(), + }; + let DirectoryMergePlan::MergeDirectory(opened) = applier + .plan_directory_merge(&root, OsStr::new("pivot")) + .unwrap() + else { + panic!("expected an existing directory merge"); + }; + + fs::rename(rootfs.join("pivot"), rootfs.join("detached")).unwrap(); + std::os::unix::fs::symlink(&outside, rootfs.join("pivot")).unwrap(); + applier.merge_directory(&upper, &opened).unwrap(); + + assert_eq!( + fs::read_to_string(outside.join("victim")).unwrap(), + "outside" + ); + assert!(!outside.join("payload").exists()); + assert_eq!( + fs::read_to_string(rootfs.join("detached/payload")).unwrap(), + "payload" + ); + assert!(!rootfs.join("detached/victim").exists()); + } +} diff --git a/crates/openshell-driver-vm/src/lib.rs b/crates/openshell-driver-vm/src/lib.rs index 8e3f41a15d..5464dd6449 100644 --- a/crates/openshell-driver-vm/src/lib.rs +++ b/crates/openshell-driver-vm/src/lib.rs @@ -22,6 +22,8 @@ pub mod gpu; #[cfg(feature = "compute-driver")] mod isolation; #[cfg(feature = "compute-driver")] +mod layer_applier; +#[cfg(feature = "compute-driver")] pub mod lifecycle; #[cfg(feature = "compute-driver")] pub mod otel_tracing; diff --git a/crates/openshell-gateway/src/vm.rs b/crates/openshell-gateway/src/vm.rs index 3b7bd6bf31..b0c6958ca0 100644 --- a/crates/openshell-gateway/src/vm.rs +++ b/crates/openshell-gateway/src/vm.rs @@ -173,6 +173,11 @@ impl VmComputeConfig { "grpc_endpoint is required when using the vm compute driver", )); } + if self.bootstrap_image.trim().is_empty() && self.default_image.trim().is_empty() { + return Err(Error::config( + "bootstrap_image or default_image is required when using the vm compute driver; sandbox images cannot be used as VM bootstrap images", + )); + } validate_vm_sandbox_identity(self)?; if self .rootfs_tar_staging_dir @@ -797,6 +802,34 @@ mod tests { ); } + #[test] + fn vm_gateway_requires_a_trusted_bootstrap_image_source() { + let config = VmComputeConfig { + grpc_endpoint: "http://127.0.0.1:50051".to_string(), + default_image: String::new(), + bootstrap_image: String::new(), + ..Default::default() + }; + let error = config + .validate_configuration() + .expect_err("the gateway must reject an empty bootstrap configuration"); + assert!(error.to_string().contains("sandbox images cannot be used")); + + VmComputeConfig { + default_image: "openshell/sandbox:default".to_string(), + ..config.clone() + } + .validate_configuration() + .expect("the operator-controlled default image is a valid fallback"); + + VmComputeConfig { + bootstrap_image: "openshell/sandbox-bootstrap:latest".to_string(), + ..config + } + .validate_configuration() + .expect("an explicit bootstrap image is valid"); + } + #[test] fn vm_driver_command_includes_gateway_otlp_configuration() { let mut command = tokio::process::Command::new("openshell-driver-vm"); diff --git a/crates/openshell-sandbox-backend/src/sandbox_auth.rs b/crates/openshell-sandbox-backend/src/sandbox_auth.rs index e9a72d5fd7..142d50c394 100644 --- a/crates/openshell-sandbox-backend/src/sandbox_auth.rs +++ b/crates/openshell-sandbox-backend/src/sandbox_auth.rs @@ -389,7 +389,7 @@ mod tests { key.serialize_pem().as_bytes(), "current", "test", - DEFAULT_SESSION_TOKEN_TTL, + Some(DEFAULT_SESSION_TOKEN_TTL), clock.clone(), ) .expect("issuer"); diff --git a/crates/openshell-sandbox/src/boundary_server.rs b/crates/openshell-sandbox/src/boundary_server.rs index 3d85d730a7..e6dfba1d92 100644 --- a/crates/openshell-sandbox/src/boundary_server.rs +++ b/crates/openshell-sandbox/src/boundary_server.rs @@ -700,6 +700,10 @@ mod linux { } fn update(&self, expires_at: i64) { + if expires_at == 0 { + self.set_deadline(None); + return; + } let now = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .map_or(0, |duration| duration.as_secs()); @@ -710,11 +714,15 @@ mod linux { } fn update_deadline(&self, deadline: tokio::time::Instant) { + self.set_deadline(Some(deadline)); + } + + fn set_deadline(&self, deadline: Option) { let _ = self.deadline.send_if_modified(|current| { - if *current == Some(deadline) { + if *current == deadline { false } else { - *current = Some(deadline); + *current = deadline; true } }); @@ -3678,7 +3686,7 @@ mod linux { key.serialize_pem().as_bytes(), "test-key", "test-gateway", - DEFAULT_SESSION_TOKEN_TTL, + Some(DEFAULT_SESSION_TOKEN_TTL), Arc::new(SystemJwtClock), ) .expect("test session issuer"); @@ -3991,6 +3999,26 @@ mod linux { .expect("expiry worker must keep the shutdown channel open"); } + #[tokio::test] + async fn non_expiring_connection_has_no_deadline() { + let (shutdown, mut closed) = tokio::sync::watch::channel(()); + let expiry = ConnectionExpiry::new(shutdown); + expiry.update_deadline(tokio::time::Instant::now() + Duration::from_millis(20)); + expiry.update(0); + + assert!( + tokio::time::timeout(Duration::from_millis(80), closed.changed()) + .await + .is_err(), + "non-expiring credentials must clear the connection deadline" + ); + expiry.update_deadline(tokio::time::Instant::now() + Duration::from_millis(20)); + tokio::time::timeout(Duration::from_millis(80), closed.changed()) + .await + .expect("replacement connection deadline must fire") + .expect("expiry worker must keep the shutdown channel open"); + } + #[tokio::test(flavor = "multi_thread")] async fn idle_uds_and_tcp_handshakes_do_not_consume_authenticated_slots() { let directory = tempfile::tempdir().unwrap(); diff --git a/crates/openshell-server/src/auth/sandbox_jwt.rs b/crates/openshell-server/src/auth/sandbox_jwt.rs index d5f1cafa11..a86eca1fdb 100644 --- a/crates/openshell-server/src/auth/sandbox_jwt.rs +++ b/crates/openshell-server/src/auth/sandbox_jwt.rs @@ -121,7 +121,7 @@ impl SandboxSessionJwtAuthority { public_key_pem: &[u8], key_id: String, gateway_id: &str, - ttl: Duration, + ttl: Option, ) -> Result { let clock = Arc::new(SystemJwtClock); let issuer = SessionJwtIssuer::from_ed25519_pem( diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index b578333560..4a38361e1b 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -7828,7 +7828,7 @@ mod tests { material.public_key_pem.as_bytes(), material.kid, "test-gateway", - Duration::from_mins(15), + Some(Duration::from_mins(15)), ) .expect("test session authority") } diff --git a/crates/openshell-server/src/grpc/auth_rpc.rs b/crates/openshell-server/src/grpc/auth_rpc.rs index 90ebd4a30a..83737fc814 100644 --- a/crates/openshell-server/src/grpc/auth_rpc.rs +++ b/crates/openshell-server/src/grpc/auth_rpc.rs @@ -304,15 +304,13 @@ pub async fn handle_refresh_sandbox_token( .sandbox_token .expose_secret() .to_string(), - sandbox_expiration_time: Some( - openshell_core::time::timestamp_from_millis( - authentication - .supervisor - .sandbox_expires_at - .saturating_mul(1000), - ) - .map_err(|error| Status::internal(error.to_string()))?, - ), + sandbox_expiration_time: openshell_core::time::optional_timestamp_from_legacy_millis( + authentication + .supervisor + .sandbox_expires_at + .saturating_mul(1000), + ) + .map_err(|error| Status::internal(error.to_string()))?, session_id: authentication.supervisor.runtime_generation.to_string(), credential_epoch: authentication.supervisor.auth_epoch.get(), })) @@ -447,7 +445,7 @@ mod tests { use std::collections::HashMap; use std::time::Duration; - async fn state_with_issuer() -> Arc { + async fn state_with_ttl(ttl: Option) -> Arc { let mat = generate_jwt_key().expect("jwt key"); let store = Arc::new( Store::connect("sqlite::memory:?cache=shared") @@ -483,7 +481,7 @@ mod tests { mat.public_key_pem.as_bytes(), mat.kid, "test-gateway", - Duration::from_hours(1), + ttl, ) .expect("session authority"), ); @@ -495,6 +493,10 @@ mod tests { state } + async fn state_with_issuer() -> Arc { + state_with_ttl(Some(Duration::from_hours(1))).await + } + async fn insert_sandbox( state: &Arc, sandbox_id: &str, @@ -619,6 +621,25 @@ mod tests { assert!(resp.sandbox_expiration_time.is_some()); } + #[tokio::test] + async fn refresh_propagates_non_expiring_session_credentials() { + let state = state_with_ttl(None).await; + let mut req = Request::new(RefreshSandboxTokenRequest { + extension_service_names: Vec::new(), + }); + req.extensions_mut().insert(sandbox_principal("sandbox-a")); + let _ = authorize_refresh(&state, &mut req).await; + let resp = handle_refresh_sandbox_token(&state, req) + .await + .expect("refresh OK") + .into_inner(); + + assert!(!resp.token.is_empty()); + assert!(!resp.sandbox_token.is_empty()); + assert!(resp.expiration_time.is_none()); + assert!(resp.sandbox_expiration_time.is_none()); + } + #[tokio::test] async fn refresh_replays_one_successor_then_rejects_older_bearers() { let state = state_with_issuer().await; diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index 635acd5257..bc0f1d32e0 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -3030,7 +3030,9 @@ pub(super) async fn handle_create_ssh_session( // Ensure metadata is valid (defense in depth - should always be true for server-constructed metadata) super::validation::validate_object_metadata(session.metadata.as_ref(), "ssh_session")?; - // Use MustCreate to atomically ensure the session token is unique + // `create_relaxed` fails if the token already exists, like MustCreate, but + // skips the per-commit fsync on SQLite. Losing a freshly minted token in a + // crash only makes it invalid; revocation stays on the durable `put_if`. let session_labels = session.object_labels(); let session_labels_json = if session_labels.as_ref().is_none_or(HashMap::is_empty) { None @@ -3042,14 +3044,13 @@ pub(super) async fn handle_create_ssh_session( }; state .store - .put_if( + .create_relaxed( SshSession::object_type(), &token, session.object_name(), session.object_workspace(), &session.encode_to_vec(), session_labels_json.as_deref(), - WriteCondition::MustCreate, ) .await .map_err(|e| Status::internal(format!("persist ssh session failed: {e}")))?; diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index ff5c160be5..a51e17a9ad 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -577,7 +577,7 @@ pub(crate) async fn run_server( &public_pem, kid, &jwt.gateway_id, - jwt.token_ttl(), + jwt.sandbox_token_ttl(), ) .map_err(Error::config)?, ); diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 2dfaa4ede3..8988542b1a 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -2963,7 +2963,7 @@ mod tests { material.public_key_pem.as_bytes(), material.kid.clone(), "test", - Duration::from_mins(15), + Some(Duration::from_mins(15)), ) .expect("session authority"), ); diff --git a/crates/openshell-server/src/persistence/mod.rs b/crates/openshell-server/src/persistence/mod.rs index 0306bdd312..c4f2d54f86 100644 --- a/crates/openshell-server/src/persistence/mod.rs +++ b/crates/openshell-server/src/persistence/mod.rs @@ -384,6 +384,38 @@ impl Store { )) } + /// Create an object that is safe to lose in a crash. + /// + /// Behaves like [`Self::put_if`] with [`WriteCondition::MustCreate`], but + /// the file-backed `SQLite` store commits it with `synchronous=NORMAL`, so + /// a power loss or kernel crash shortly after the call returns may roll + /// the insert back. Use it only for objects whose absence denies access, + /// such as newly minted SSH session tokens. Writes that revoke or tighten + /// anything must use [`Self::put_if`], which is always durable. + #[tracing::instrument( + name = "store", + skip_all, + fields(otel.name = "store.create_relaxed", otel.status_code = tracing::field::Empty, object_type = %object_type, object.id = %id, object.name = %name, workspace = %workspace) + )] + pub async fn create_relaxed( + &self, + object_type: &str, + id: &str, + name: &str, + workspace: &str, + payload: &[u8], + labels: Option<&str>, + ) -> PersistenceResult { + store_dispatch_traced!(self.create_relaxed( + object_type, + id, + name, + workspace, + payload, + labels + )) + } + /// Delete an object by id with compare-and-swap support. /// /// # Arguments diff --git a/crates/openshell-server/src/persistence/postgres.rs b/crates/openshell-server/src/persistence/postgres.rs index d79335b5f5..14bae6dfb1 100644 --- a/crates/openshell-server/src/persistence/postgres.rs +++ b/crates/openshell-server/src/persistence/postgres.rs @@ -180,6 +180,29 @@ ON CONFLICT (object_type, workspace, name) WHERE name IS NOT NULL DO UPDATE SET Ok(()) } + /// Create an object; Postgres commits are always durable, so this is + /// [`Self::put_if`] with [`WriteCondition::MustCreate`]. + pub async fn create_relaxed( + &self, + object_type: &str, + id: &str, + name: &str, + workspace: &str, + payload: &[u8], + labels: Option<&str>, + ) -> PersistenceResult { + self.put_if( + object_type, + id, + name, + workspace, + payload, + labels, + WriteCondition::MustCreate, + ) + .await + } + #[allow(clippy::too_many_arguments)] pub async fn put_if( &self, diff --git a/crates/openshell-server/src/persistence/sqlite.rs b/crates/openshell-server/src/persistence/sqlite.rs index d9f22bcbbe..a2a5e74f60 100644 --- a/crates/openshell-server/src/persistence/sqlite.rs +++ b/crates/openshell-server/src/persistence/sqlite.rs @@ -15,7 +15,9 @@ use openshell_core::SetResourceVersion; use openshell_core::paths::set_file_owner_only; use openshell_core::proto::Sandbox; use prost::Message; -use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection, SqlitePoolOptions}; +use sqlx::sqlite::{ + SqliteConnectOptions, SqliteConnection, SqliteJournalMode, SqlitePoolOptions, SqliteSynchronous, +}; use sqlx::{Connection, QueryBuilder, Row, Sqlite, SqlitePool}; use std::path::{Path, PathBuf}; use std::str::FromStr; @@ -39,6 +41,10 @@ use super::{DELETE_MANY_BATCH_SIZE, DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE} #[derive(Debug, Clone)] pub struct SqliteStore { pool: SqlitePool, + /// Pool for writes whose loss after a crash is harmless; see + /// [`SqliteStore::create_relaxed`]. On-disk stores open it with + /// `synchronous=NORMAL`; in-memory stores share `pool`. + relaxed_pool: SqlitePool, #[cfg_attr(not(any(test, feature = "test-support")), allow(dead_code))] in_memory_keepalive: Option>>>, } @@ -70,6 +76,124 @@ pub(super) async fn replace_pool_connection(store: &SqliteStore) -> PersistenceR connection.close().await.map_err(|e| map_db_error(&e)) } +/// Apply the on-disk journal settings and switch the database file to WAL +/// once, before the pool opens its connections. +/// +/// The gateway's hot paths (SSH-session tokens minted and revoked around every +/// forwarded connection, sandbox status updates) are many small autocommit +/// writes. `SQLite`'s default rollback journal makes each of those commits pay +/// several `fsync` calls and blocks readers while a writer holds the lock, so +/// under a burst of forwarded connections the whole store serializes on disk +/// latency. WAL mode removes the reader/writer exclusion and cuts each commit +/// to a single `fsync` of the WAL file. +/// +/// The main pool keeps `synchronous=FULL` rather than the usual WAL pairing +/// of `NORMAL`. Under `NORMAL` a power loss or kernel crash can roll back +/// transactions that were already acknowledged, and several of those writes +/// tighten authorization: an SSH session revoked just before the crash would +/// come back valid for the rest of its lifetime. `FULL` keeps every +/// acknowledged commit durable. Writes whose loss only ever denies access, +/// such as minting a new SSH session token, go through a separate +/// `synchronous=NORMAL` pool instead ([`SqliteStore::create_relaxed`]). Both +/// pools append to the same WAL file, so the next `FULL` commit's `fsync` also +/// makes every earlier relaxed commit durable, and a crash can never roll back +/// a `FULL` commit. +/// +/// `journal_mode=WAL` is persistent in the database file, but switching into +/// it needs exclusive access: if another connection holds the file open, the +/// switch waits out `busy_timeout` and then fails. Doing it up front on one +/// connection means the pool connections only ever re-apply the pragma to a +/// file that is already in WAL mode, which never blocks, and a failure +/// surfaces as a single clear connect error instead of a pool error later. +/// The first start after upgrading a rollback-journal database therefore needs +/// the file to be otherwise unopened. `synchronous` is a per-connection +/// setting and is applied through the options on every connection. +/// +/// In-memory databases are left on their defaults: WAL is meaningless there +/// and the shared-cache keepalive connection already provides their lifetime +/// guarantees. +async fn configure_on_disk_durability( + options: SqliteConnectOptions, +) -> PersistenceResult { + let options = options + .journal_mode(SqliteJournalMode::Wal) + .synchronous(SqliteSynchronous::Full); + let wal_error = |e: &sqlx::Error| { + PersistenceError::Database(format!( + "failed to switch SQLite database {} to WAL journal mode (the switch needs \ + exclusive access; close other connections to the file and retry): {}", + options.get_filename().display(), + map_db_error(e) + )) + }; + let connection = SqliteConnection::connect_with(&options) + .await + .map_err(|e| wal_error(&e))?; + connection.close().await.map_err(|e| wal_error(&e))?; + Ok(options) +} + +/// Insert a new object at resource version 1, failing if it already exists. +async fn insert_new_object( + pool: &SqlitePool, + object_type: &str, + id: &str, + name: &str, + workspace: &str, + payload: &[u8], + labels: Option<&str>, +) -> PersistenceResult { + let now_ms = current_time_ms(); + sqlx::query( + r#" +INSERT INTO "objects" ("object_type", "id", "name", "workspace", "payload", "created_at_ms", "updated_at_ms", "labels", "resource_version") +VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?6, ?7, 1) +"#, + ) + .bind(object_type) + .bind(id) + .bind(name) + .bind(workspace) + .bind(payload) + .bind(now_ms) + .bind(labels.unwrap_or("{}")) + .execute(pool) + .await + .map_err(|e| map_db_error(&e))?; + + Ok(WriteResult { + resource_version: 1, + created_at_ms: now_ms, + updated_at_ms: now_ms, + }) +} + +#[cfg(test)] +pub(super) async fn journal_settings(store: &SqliteStore) -> PersistenceResult<(String, i64)> { + pool_journal_settings(&store.pool).await +} + +#[cfg(test)] +pub(super) async fn relaxed_journal_settings( + store: &SqliteStore, +) -> PersistenceResult<(String, i64)> { + pool_journal_settings(&store.relaxed_pool).await +} + +#[cfg(test)] +async fn pool_journal_settings(pool: &SqlitePool) -> PersistenceResult<(String, i64)> { + let mut connection = pool.acquire().await.map_err(|e| map_db_error(&e))?; + let journal_mode: String = sqlx::query_scalar("PRAGMA journal_mode") + .fetch_one(&mut *connection) + .await + .map_err(|e| map_db_error(&e))?; + let synchronous: i64 = sqlx::query_scalar("PRAGMA synchronous") + .fetch_one(&mut *connection) + .await + .map_err(|e| map_db_error(&e))?; + Ok((journal_mode, synchronous)) +} + impl SqliteStore { /// Closes the connection pool. #[cfg(test)] @@ -107,6 +231,10 @@ impl SqliteStore { // so we can restrict the permissions after the database is connected. let db_path = (!is_in_memory).then(|| options.get_filename().to_path_buf()); + if !is_in_memory { + options = configure_on_disk_durability(options).await?; + } + let in_memory_keepalive = if is_in_memory { let connection = SqliteConnection::connect_with(&options) .await @@ -116,11 +244,25 @@ impl SqliteStore { None }; + let relaxed_options = + (!is_in_memory).then(|| options.clone().synchronous(SqliteSynchronous::Normal)); + let pool = pool_options .connect_with(options) .await .map_err(|e| map_db_error(&e))?; + // SQLite serializes writers, so one connection is enough for the + // relaxed pool. + let relaxed_pool = match relaxed_options { + Some(relaxed_options) => SqlitePoolOptions::new() + .max_connections(1) + .connect_with(relaxed_options) + .await + .map_err(|e| map_db_error(&e))?, + None => pool.clone(), + }; + // Tighten the permissions of the database file to owner-only access (0o600). if let Some(path) = db_path { restrict_db_file_permissions(&path)?; @@ -128,6 +270,7 @@ impl SqliteStore { Ok(Self { pool, + relaxed_pool, in_memory_keepalive, }) } @@ -182,6 +325,7 @@ impl SqliteStore { /// Do not call from runtime code; this tears down the active pool. #[cfg(any(test, feature = "test-support"))] pub async fn close(&self) { + self.relaxed_pool.close().await; self.pool.close().await; if let Some(keepalive) = &self.in_memory_keepalive { let connection = keepalive.lock().await.take(); @@ -225,6 +369,33 @@ ON CONFLICT ("object_type", "workspace", "name") WHERE "name" IS NOT NULL DO UPD Ok(()) } + /// Create an object with `synchronous=NORMAL` durability. + /// + /// Same semantics as [`Self::put_if`] with [`WriteCondition::MustCreate`], + /// except that a power loss or kernel crash shortly after the call returns + /// may roll the insert back. Use it only for objects whose absence denies + /// access, never for writes that revoke or tighten anything. + pub async fn create_relaxed( + &self, + object_type: &str, + id: &str, + name: &str, + workspace: &str, + payload: &[u8], + labels: Option<&str>, + ) -> PersistenceResult { + insert_new_object( + &self.relaxed_pool, + object_type, + id, + name, + workspace, + payload, + labels, + ) + .await + } + #[allow(clippy::too_many_arguments)] pub async fn put_if( &self, @@ -240,29 +411,16 @@ ON CONFLICT ("object_type", "workspace", "name") WHERE "name" IS NOT NULL DO UPD match condition { WriteCondition::MustCreate => { - // Insert only - fail if object exists - sqlx::query( - r#" -INSERT INTO "objects" ("object_type", "id", "name", "workspace", "payload", "created_at_ms", "updated_at_ms", "labels", "resource_version") -VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?6, ?7, 1) -"#, + insert_new_object( + &self.pool, + object_type, + id, + name, + workspace, + payload, + labels, ) - .bind(object_type) - .bind(id) - .bind(name) - .bind(workspace) - .bind(payload) - .bind(now_ms) - .bind(labels.unwrap_or("{}")) - .execute(&self.pool) .await - .map_err(|e| map_db_error(&e))?; - - Ok(WriteResult { - resource_version: 1, - created_at_ms: now_ms, - updated_at_ms: now_ms, - }) } WriteCondition::MatchResourceVersion(expected_version) => { // Update with version check diff --git a/crates/openshell-server/src/persistence/tests.rs b/crates/openshell-server/src/persistence/tests.rs index 49fa192094..3a27b7b568 100644 --- a/crates/openshell-server/src/persistence/tests.rs +++ b/crates/openshell-server/src/persistence/tests.rs @@ -324,14 +324,265 @@ async fn sqlite_connect_tightens_existing_db_file_permissions() { assert_eq!(mode, 0o600, "expected 0600, got {mode:04o}"); } +fn on_disk_store_url(db_path: &std::path::Path) -> String { + format!("sqlite:{}?mode=rwc", db_path.display()) +} + +async fn connect_on_disk_sqlite(url: &str) -> super::SqliteStore { + match Store::connect(url).await.expect("connect to sqlite") { + Store::Sqlite(store) => store, + Store::Postgres(_) => unreachable!("sqlite URL must select the SQLite store"), + } +} + +async fn file_journal_mode(db_path: &std::path::Path) -> String { + use sqlx::{Connection, SqliteConnection}; + + let mut connection = SqliteConnection::connect(&format!("sqlite:{}", db_path.display())) + .await + .expect("open database file directly"); + let journal_mode: String = sqlx::query_scalar("PRAGMA journal_mode") + .fetch_one(&mut connection) + .await + .expect("read journal_mode"); + connection.close().await.expect("close direct connection"); + journal_mode +} + +#[tokio::test] +async fn sqlite_connect_enables_wal_and_full_synchronous_on_disk() { + let tmp = tempfile::tempdir().expect("tempdir"); + let db_path = tmp.path().join("openshell.db"); + let url = on_disk_store_url(&db_path); + + let store = connect_on_disk_sqlite(&url).await; + + let (journal_mode, synchronous) = super::sqlite::journal_settings(&store) + .await + .expect("read journal settings through the pool"); + assert_eq!(journal_mode, "wal", "on-disk stores must run in WAL mode"); + // FULL (2), not NORMAL (1): acknowledged commits such as SSH session + // revocations must survive a power loss. + assert_eq!( + synchronous, 2, + "on-disk stores must run with synchronous=FULL (2), got {synchronous}" + ); + let (relaxed_journal_mode, relaxed_synchronous) = + super::sqlite::relaxed_journal_settings(&store) + .await + .expect("read journal settings through the relaxed pool"); + assert_eq!(relaxed_journal_mode, "wal"); + assert_eq!( + relaxed_synchronous, 1, + "the relaxed pool must run with synchronous=NORMAL (1), got {relaxed_synchronous}" + ); + + // Force a write so the WAL sidecars exist on disk, then confirm they are + // owner-only like the main file. + store + .put( + "sandbox", + "wal-probe", + "wal-probe", + "default", + b"payload", + None, + ) + .await + .expect("write through the store"); + let [wal_path, shm_path] = super::sqlite::sqlite_sidecar_paths(&db_path); + assert!(wal_path.exists(), "WAL sidecar should exist after a write"); + assert!(shm_path.exists(), "SHM sidecar should exist after a write"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + for path in [&db_path, &wal_path, &shm_path] { + let mode = std::fs::metadata(path) + .expect("sidecar metadata") + .permissions() + .mode() + & 0o777; + assert_eq!( + mode, + 0o600, + "{}: expected 0600, got {mode:04o}", + path.display() + ); + } + } + + store.close().await; + assert_eq!( + file_journal_mode(&db_path).await, + "wal", + "WAL must persist in the database file after the store closes" + ); +} + +#[tokio::test] +async fn sqlite_create_relaxed_is_must_create_visible_to_durable_writes() { + let tmp = tempfile::tempdir().expect("tempdir"); + let db_path = tmp.path().join("openshell.db"); + let store = Store::connect(&on_disk_store_url(&db_path)) + .await + .expect("connect to sqlite"); + + let created = store + .create_relaxed("ssh_session", "tok", "tok", "default", b"issued", None) + .await + .expect("relaxed create"); + assert_eq!(created.resource_version, 1); + + let duplicate = store + .create_relaxed("ssh_session", "tok", "tok", "default", b"again", None) + .await + .expect_err("relaxed create must reject an existing object"); + assert!( + matches!(duplicate, PersistenceError::UniqueViolation { .. }), + "expected UniqueViolation, got {duplicate:?}" + ); + + // The durable pool sees the relaxed insert and can revoke it with CAS. + store + .put_if( + "ssh_session", + "tok", + "tok", + "default", + b"revoked", + None, + super::WriteCondition::MatchResourceVersion(1), + ) + .await + .expect("durable revoke of a relaxed insert"); + let record = store + .get("ssh_session", "tok") + .await + .expect("get") + .expect("record present"); + assert_eq!(record.payload, b"revoked"); + assert_eq!(record.resource_version, 2); +} + +#[tokio::test] +async fn sqlite_connect_switches_existing_rollback_journal_database_to_wal() { + use sqlx::sqlite::SqliteConnectOptions; + use sqlx::{Connection, SqliteConnection}; + use std::str::FromStr; + + let tmp = tempfile::tempdir().expect("tempdir"); + let db_path = tmp.path().join("openshell.db"); + let url = on_disk_store_url(&db_path); + + // A database created by an older gateway, or by any sqlx 0.8 default + // connection, is in rollback-journal (`delete`) mode. + { + let options = SqliteConnectOptions::from_str(&url) + .expect("parse url") + .create_if_missing(true); + let mut connection = SqliteConnection::connect_with(&options) + .await + .expect("create legacy database"); + sqlx::query("CREATE TABLE legacy_probe (x INTEGER)") + .execute(&mut connection) + .await + .expect("write legacy database"); + connection.close().await.expect("close legacy connection"); + } + assert_eq!(file_journal_mode(&db_path).await, "delete"); + + let store = connect_on_disk_sqlite(&url).await; + let (journal_mode, _) = super::sqlite::journal_settings(&store) + .await + .expect("read journal settings"); + assert_eq!( + journal_mode, "wal", + "connect must switch existing files to WAL" + ); + store.close().await; + assert_eq!(file_journal_mode(&db_path).await, "wal"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn sqlite_file_backed_reads_and_writes_proceed_concurrently() { + let tmp = tempfile::tempdir().expect("tempdir"); + let db_path = tmp.path().join("openshell.db"); + let url = on_disk_store_url(&db_path); + let store = std::sync::Arc::new(Store::connect(&url).await.expect("connect to sqlite")); + + store + .put("sandbox", "seed", "seed", "default", b"seed", None) + .await + .expect("seed object"); + + // Mirror the forward-service pattern: many independent autocommit writes + // (a relaxed insert and a durable update per "connection", so the two + // pools contend for the writer lock) while readers keep fetching. Every operation must complete; no caller may observe a + // "database is locked" error even though the writes contend for the + // single SQLite writer. + let writers = (0..8).map(|writer| { + let store = store.clone(); + tokio::spawn(async move { + for index in 0..25 { + let id = format!("session-{writer}-{index}"); + store + .create_relaxed("ssh_session", &id, &id, "default", b"issued", None) + .await + .expect("insert session"); + store + .put_if( + "ssh_session", + &id, + &id, + "default", + b"revoked", + None, + super::WriteCondition::MatchResourceVersion(1), + ) + .await + .expect("revoke session"); + } + }) + }); + let readers = (0..4).map(|_| { + let store = store.clone(); + tokio::spawn(async move { + for _ in 0..100 { + let record = store + .get("sandbox", "seed") + .await + .expect("read while writers are active") + .expect("seed object present"); + assert_eq!(record.payload, b"seed"); + } + }) + }); + + for handle in writers.chain(readers) { + handle.await.expect("task completed"); + } + + let sessions = store + .list("ssh_session", "default", 1000, 0) + .await + .expect("list sessions"); + assert_eq!(sessions.len(), 200, "every session write must be durable"); + assert!( + sessions.iter().all(|record| record.payload == b"revoked"), + "every session must have been revoked by its follow-up write" + ); +} + // The next three tests cover `restrict_db_file_permissions` against the // WAL/SHM sidecars at increasing levels of fidelity: // // 1. `_tightens_main_and_wal_and_shm_files`: synthetic empty files, proves // the chmod loop walks all three paths. -// 2. `_skips_missing_sidecars`: proves the `exists()` guard, which is the -// actual production path today (sqlx 0.8 doesn't default to WAL and -// doesn't accept `journal_mode` as a URL parameter). +// 2. `_skips_missing_sidecars`: proves the `exists()` guard for databases +// that have not been written yet or were created before the adapter +// enabled WAL (sqlx 0.8 doesn't default to WAL and doesn't accept +// `journal_mode` as a URL parameter; `SqliteStore::connect` opts in +// through the builder API). // 3. `_handles_real_sqlite_wal_files`: opens a real sqlx pool with // `SqliteJournalMode::Wal` via the builder API so SQLite materializes // real `-wal` and `-shm` files, then checks the helper tightens them. diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 6f5439a575..f90359832f 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -292,7 +292,7 @@ discovery endpoint or its TLS CA. | server.credentialDrivers.vault.timeoutSecs | string | `""` | HTTP request timeout in seconds. Empty = driver default. | | server.credentialDrivers.vault.tokenPath | string | `""` | Mounted token file path when authMethod is token_file. | | server.credentialStorage.existingSecret | string | `""` | Name of a pre-existing Secret containing the key-encryption key. When set, the chart does NOT generate a new Secret; it references this one instead. The Secret must contain a key named "key-encryption-key" with a base64-encoded 32-byte value. Required for GitOps workflows that render manifests with `helm template` (where `lookup` is unavailable). | -| server.dbUrl | string | `"sqlite:/var/openshell/openshell.db"` | Gateway database URL (used for the default SQLite backend). | +| server.dbUrl | string | `"sqlite:/var/openshell/openshell.db"` | Gateway database URL (used for the default SQLite backend). SQLite runs in WAL mode and needs a local block-backed volume, not NFS or other network filesystems. | | server.defaultRuntimeClassName | string | `""` | Default Kubernetes runtimeClassName for sandbox pods. Applied when a CreateSandbox request does not specify one. Empty (default) = omit the field, using the cluster's default RuntimeClass. Set to a RuntimeClass name (e.g. "kata-containers", "nvidia") to apply it to all sandboxes that don't explicitly override it. | | server.disableTls | bool | `false` | Disable TLS entirely - the server listens on plaintext HTTP. Set to true when a reverse proxy / tunnel terminates TLS at the edge. | | server.drivers.kubernetes.allowDriverConfig | bool | `false` | Allow caller driver JSON; external resources still require approval. | diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index f8d1519bb7..34067eb31d 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -256,7 +256,9 @@ server: # -- Namespace where sandbox pods are created. Defaults to the Helm release # namespace (.Release.Namespace) when left empty. sandboxNamespace: "" - # -- Gateway database URL (used for the default SQLite backend). + # -- Gateway database URL (used for the default SQLite backend). SQLite runs + # in WAL mode and needs a local block-backed volume, not NFS or other network + # filesystems. dbUrl: "sqlite:/var/openshell/openshell.db" # -- Name of a pre-existing Opaque Secret containing a PostgreSQL # connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL diff --git a/deploy/rpm/CONFIGURATION.md b/deploy/rpm/CONFIGURATION.md index ac57e02eba..2030bbac70 100644 --- a/deploy/rpm/CONFIGURATION.md +++ b/deploy/rpm/CONFIGURATION.md @@ -229,6 +229,10 @@ overrides that persist across package upgrades. The database URL is not accepted in TOML. When `OPENSHELL_DB_URL` is unset, the gateway uses `sqlite:$XDG_STATE_HOME/openshell/gateway/openshell.db`. +The SQLite database runs in WAL mode with `synchronous=FULL` (SSH session +issuance alone uses `NORMAL`), so +`openshell.db-wal` and `openshell.db-shm` sit next to it and must be kept +together with it; back it up with `sqlite3 openshell.db ".backup "`. ### Driver TOML settings diff --git a/docs/about/installation.mdx b/docs/about/installation.mdx index 9349ed51ca..45206d5772 100644 --- a/docs/about/installation.mdx +++ b/docs/about/installation.mdx @@ -19,7 +19,11 @@ Install OpenShell with a single command: curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh ``` -The script detects your operating system and installs the OpenShell CLI, standalone policy prover, and gateway with your native package manager. It then starts the local gateway server so you can begin creating sandboxes. +The script detects your operating system and installs the OpenShell CLI, standalone policy prover, and gateway. On Linux, it installs from the Snap Store whenever the `snap` command is available. Otherwise, it uses the native Debian or RPM package. It then starts the local gateway server so you can begin creating sandboxes. + +Snap installs use `latest/stable` by default and `latest/edge` when +`OPENSHELL_VERSION=dev`. Other `OPENSHELL_VERSION` values also use the stable +Snap channel because Snap Store channels do not select an exact GitHub release. You can also download release artifacts directly from the [OpenShell GitHub Releases](https://github.com/NVIDIA/OpenShell/releases) page. @@ -70,9 +74,14 @@ brew services restart openshell ## Linux -On Fedora and RHEL, the install script uses RPM packages. The RPMs install the `openshell` CLI, `openshell-prover`, the `openshell-gateway` daemon, and a systemd user service. +On Linux systems with the `snap` command, the install script uses the OpenShell +snap. Install and start Docker Engine from a system package or Docker's package +repository before running the installer. The Docker snap is not currently +compatible with OpenShell. + +On Fedora and RHEL without `snap`, the install script uses RPM packages. The RPMs install the `openshell` CLI, `openshell-prover`, the `openshell-gateway` daemon, and a systemd user service. -On Debian and Ubuntu, the install script uses a Debian package. The Debian package installs the `openshell` CLI, `openshell-prover`, the `openshell-gateway` daemon, VM sandbox support, and a systemd user service. +On Debian and Ubuntu without `snap`, the install script uses a Debian package. The Debian package installs the `openshell` CLI, `openshell-prover`, the `openshell-gateway` daemon, VM sandbox support, and a systemd user service. Linux packages require glibc 2.28 or newer. The installer checks libc before downloading packages and exits with an error on older glibc versions, Alpine, musl-based distributions, or unknown libc environments. @@ -96,17 +105,54 @@ sudo loginctl enable-linger $USER ## Snap +The OpenShell snap requires a running Docker daemon. Docker can come from the +distribution's Docker package, Docker's package repository, or another +compatible non-Snap installation. The Docker snap is not currently compatible: +its AppArmor confinement prevents OpenShell's hardened containers from starting. +The installer exits without installing OpenShell when Docker is absent or the +Docker snap is installed. + +The snap keeps gateway and CLI state in snap-specific directories and does not +import state from a Debian, RPM, Homebrew, or manual installation. If the +installer detects an existing non-snap OpenShell binary, it stops before +installing the snap. Back up anything you need, clean up existing sandboxes and +runtime resources, stop the non-snap gateway, and remove the old installation. +After completing that cleanup, rerun the installer with +`OPENSHELL_ACK_BREAKING_UPGRADE=1`. + Install the OpenShell snap from the Snap Store: ```shell sudo snap install openshell ``` +The install script installs `latest/stable` by default. Set +`OPENSHELL_VERSION=dev` to install from `latest/edge`: + +```shell +curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | \ + OPENSHELL_VERSION=dev sh +``` + The snap defines two apps: the `openshell` CLI and the `openshell.gateway` -systemd service. The gateway listens on `https://127.0.0.1:17670` and -stores its database at `$SNAP_COMMON/gateway.db` (typically -`/var/snap/openshell/common/gateway.db`). Create `$SNAP_COMMON/gateway.toml` -when you need to override gateway settings. +system service. Unlike the Debian and RPM user services, snapd runs this service +as root. The gateway therefore generates its certificates in root-owned snap +state, where ordinary users running the CLI cannot read the client certificate. +For this reason, the snap disables TLS and listens only on +`http://127.0.0.1:17670`. Its default configuration permits unauthenticated +local CLI access. Sandbox connections still use gateway-minted credentials. + +The gateway stores its database at `$SNAP_COMMON/gateway.db` and its +configuration at `$SNAP_COMMON/gateway.toml`, typically under +`/var/snap/openshell/common/`. Initial snap installation creates the default +configuration only when no configuration already exists. +The install script applies the same default when it installs or refreshes a snap +that has no existing gateway configuration, then restarts the gateway. + +Edit `$SNAP_COMMON/gateway.toml` when you need to override gateway settings, +then restart the service. Do not expose the default plaintext gateway beyond +the local host while unauthenticated user access is enabled. Any process or user +on the same host that can connect to the loopback port can operate the gateway. The snap CLI stores per-user config, data, and state under `$SNAP_USER_COMMON`, typically `~/snap/openshell/common`. Gateway registrations live under @@ -115,48 +161,46 @@ typically `~/snap/openshell/common`. Gateway registrations live under ### Snap store installs -When installing from the Snap Store, snapd automatically connects the `home`, -`network`, and `network-bind` plugs. The `docker` plug still -requires manual connection: - -```shell -sudo snap connect openshell:docker docker:docker-daemon -``` - -The snap declares `default-provider: docker` on the Docker plug so snapd will -offer to install the Docker snap, but the connection itself must be made -manually. +When installing OpenShell from the Snap Store, snapd automatically connects all +required interfaces. The `docker` plug connects to the system `:docker` slot, +which exposes the compatible Docker daemon installed on the host. The +`log-observe` and `system-observe` plugs are also automatically connected +through the OpenShell snap's store assertions. ### Locally built snap packages -When installing a locally built `.snap` file, no plugs are connected by default: +To install a locally-built snap, one can install directly from a `.snap` file +with the `--dangerous` flag to acknowledge that it has not been signed by the +Snap Store. Thus, it lacks the store assertions which tell snapd to auto- +connect the privileged interfaces, so they must be connected manually. ```shell sudo snap install ./openshell_*.snap --dangerous -sudo snap connect openshell:home -sudo snap connect openshell:network -sudo snap connect openshell:network-bind -sudo snap connect openshell:docker docker:docker-daemon sudo snap connect openshell:log-observe sudo snap connect openshell:system-observe +sudo snap connect openshell:docker :docker ``` The `log-observe` and `system-observe` plugs are needed for the gateway service -to read logs and inspect system processes. The `docker` plug requires the -`docker:docker-daemon` slot from the Docker snap and does not work with -system-installed Docker. +to read logs and inspect system processes. The `docker` plug connects to the +same system `:docker` slot used by Store installations. ### Gateway service The gateway runs as a snap daemon with `refresh-mode: endure`, meaning snapd will not restart it during snap refreshes. This prevents the gateway from -killing active sandbox sessions mid-refresh. Restart the service manually after -a snap refresh when you need the updated binary: +killing active sandbox sessions during automatic or direct snap refreshes. +Restart the service manually after a direct `snap refresh` when you need the +updated binary: ```shell sudo systemctl restart snap.openshell.gateway ``` +Rerunning `install.sh` explicitly refreshes the snap and restarts the gateway so +the refreshed binary is active when installation completes. That restart +interrupts active sandbox sessions. + ## Kubernetes Kubernetes deployments use the OpenShell Helm chart. For step-by-step installation, refer to [Kubernetes Setup](/kubernetes/setup). For chart values and packaging details, refer to the [Helm chart README](https://github.com/NVIDIA/OpenShell/blob/main/deploy/helm/openshell/README.md). diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index ec4354aa1a..06404a69de 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -18,6 +18,8 @@ Gateway CLI flag > gateway OPENSHELL_* env var > TOML file > built-in defa `database_url` is env-only. The loader rejects it when it appears in the file. When `OPENSHELL_DB_URL` is unset, the gateway stores its SQLite database under `$XDG_STATE_HOME/openshell/gateway/openshell.db`. +On-disk SQLite databases run in WAL mode with `synchronous=FULL`: the gateway keeps `openshell.db-wal` and `openshell.db-shm` next to the database file, the file must live on a local filesystem, and a backup must use `sqlite3 openshell.db ".backup "` rather than copying `openshell.db` on its own. Every acknowledged write, including SSH session revocations, survives a power loss. The one exception is SSH session issuance, which skips the per-commit sync; a power loss can drop a just-issued session token, and the client must request a new one. + `name` assigns an operator-facing identity to the gateway installation. Set it with `[openshell.gateway].name`, `--name`, or `OPENSHELL_GATEWAY_NAME`. It defaults to `openshell`; the Helm chart defaults it to the chart fullname so all replicas in one installation share a name. Chart fullnames are only unique within their Kubernetes namespace, so set `server.name` explicitly when one collector receives telemetry from multiple namespaces or clusters. This identity is independent of client-side gateway aliases, TLS names, and `gateway_jwt.gateway_id`. ## Package-Managed Locations @@ -90,7 +92,8 @@ future version. To migrate an existing file: 5. Use canonical image pull policies: `always`, `if_not_present`, `never`, or Podman-only `newer`. Kubernetes-style capitalization and Podman's `missing` spelling are rejected. -6. Remove zero sentinels. Omit `gateway_jwt.ttl_secs` to use the 900-second default, +6. Remove zero sentinels. Omit `gateway_jwt.ttl_secs` to use non-expiring local + sandbox session credentials and the 900-second typed extension-token default, omit Docker or Podman `sandbox_pids_limit` to use OpenShell's default limit of 2048, and omit Podman `health_check_interval_secs` to disable health checks. Explicit zero values are invalid. @@ -267,7 +270,7 @@ The client-certificate handshake policy is derived and has no `require_client_au `[openshell.gateway] policy_validation_failure_mode` controls what sandbox supervisors do when a complete candidate policy fails runtime validation. The default, `fail_closed`, deactivates the previous network policy, closes relays pinned to it, and denies new egress until a valid generation loads. `retain_last_valid` leaves the previous valid generation active. Both modes reject the candidate atomically; startup keeps the workload unstarted until the effective policy and matching provider configuration pass admission. A rejected startup exposes `ConfigurationInvalid` and remains available for policy/provider repair in either mode. Gateway mutation paths that can preflight a known effective scope reject invalid candidates before persistence and leave the active policy unchanged regardless of this setting. Changing the value requires restarting the gateway so it can reload `gateway.toml` and distribute the new posture to sandbox supervisors. -`[openshell.gateway.gateway_jwt] ttl_secs` controls the lifetime of gateway-minted, generation-bound sandbox session JWTs and typed extension JWTs. Omit it to use the 900-second default. Explicit `0` is invalid. Helm renders `3600` seconds by default. +`[openshell.gateway.gateway_jwt] ttl_secs` controls generation-bound gateway-facing and Sandbox Protocol credentials minted for a sandbox session, plus typed extension JWTs. Omit it for non-expiring local sandbox session credentials: both session tokens carry `exp = 0`, and refresh responses omit their expiration timestamps. Typed extension JWTs retain a 900-second default when the field is omitted. Use omission only for local single-player Docker, Podman, or VM gateways. Explicit `0` is invalid. Kubernetes and other shared deployments should set a positive TTL; Helm renders `3600` seconds by default, and the gateway logs a warning when a Kubernetes gateway omits the field. `[openshell.gateway.auth] allow_unauthenticated_users = true` is an unsafe local-development and trusted-proxy escape hatch. It accepts user-facing CLI/API calls without OIDC or mTLS credentials while sandbox supervisors still authenticate with gateway-minted sandbox JWTs. Leave it false for shared and production gateways. @@ -928,8 +931,9 @@ image_pull_policy = "if_not_present" # always | if_not_present | never | n grpc_endpoint = "https://127.0.0.1:17670" # The gateway overwrites gateway_port from bind_address at runtime. gateway_port = 17670 -# Omit for the platform default: empty on Linux, 192.168.127.254 on macOS Podman machine. -# Set "" to force Podman's host-gateway resolver. +# Trusted supervisor-side destination returned for host.openshell.internal. +# Omit for the platform default: 127.0.0.1 on native Linux and +# 192.168.127.254 on macOS Podman Machine. # host_gateway_ip = "192.168.127.254" ssh_socket_path = "/run/openshell/ssh.sock" stop_timeout_secs = 45 @@ -1076,7 +1080,9 @@ default_image = "nvcr.io/nvidia/base/ubuntu:24.04" # Optional override. When omitted, the gateway derives # https://host.openshell.internal: for the VM driver. grpc_endpoint = "https://host.openshell.internal:17670" -# Empty falls back to default_image. +# Empty falls back only to default_image. The gateway rejects the VM +# configuration when both fields are empty; sandbox-requested images are never +# bootstrap images. bootstrap_image = "nvcr.io/nvidia/base/ubuntu:24.04" krun_log_level = 1 vcpus = 2 diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 24511c8fae..383f24a329 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -375,7 +375,7 @@ MicroVM-backed sandboxes run inside VM-backed isolation instead of a container b The gateway uses the VM compute driver to create VM-backed sandboxes. MicroVM requires host virtualization support. It uses [libkrun](https://github.com/containers/libkrun) with Apple's [Hypervisor framework](https://developer.apple.com/documentation/hypervisor) on macOS, KVM on Linux, and [QEMU](https://www.qemu.org/) for GPU-backed sandboxes on Linux. -The VM driver boots a cached immutable bootstrap ext4 root disk. When the requested sandbox image differs from the bootstrap image, the driver stages the registry image as an OCI layout, unpacks it inside a bootstrap VM with `umoci`, and caches the prepared image disk by image identity. Each sandbox receives that prepared disk read-only plus its own writable `overlay.ext4` disk for `/`, including `/sandbox` writes and runtime TLS material. The overlay persists for the sandbox lifetime and is deleted with the sandbox state directory. +The VM driver boots a cached immutable bootstrap ext4 root disk. Set `bootstrap_image` explicitly or let it fall back to the operator-configured `default_image`. The gateway rejects the VM configuration when both are empty, and the standalone driver also refuses to start; neither path uses a sandbox-requested image as the VM bootstrap image. When the requested sandbox image differs from the bootstrap image, the driver stages the registry image as an OCI layout, unpacks it inside a bootstrap VM with `umoci`, and caches the prepared image disk by image identity. Each sandbox receives that prepared disk read-only plus its own writable `overlay.ext4` disk for `/`, including `/sandbox` writes and runtime TLS material. The overlay persists for the sandbox lifetime and is deleted with the sandbox state directory. VM sandbox creation follows the same progress model as Kubernetes-backed sandboxes. The gateway accepts the sandbox, then the VM driver publishes watch events while it resolves the image, prepares or reuses the bootstrap and prepared image caches, creates the writable overlay, and starts the VM launcher. diff --git a/docs/sandboxes/manage-sandboxes.mdx b/docs/sandboxes/manage-sandboxes.mdx index a25eed30fc..178ab92630 100644 --- a/docs/sandboxes/manage-sandboxes.mdx +++ b/docs/sandboxes/manage-sandboxes.mdx @@ -312,10 +312,19 @@ attaches to the same process instance and replays up to 1 MiB of recent output. One attachment owns stdin at a time. Use `sandbox exec --tty -- /bin/bash -l` when you want a new independent shell instead. +If an established connection is interrupted, for example when a laptop sleeps +and wakes, the CLI obtains a new SSH session and reattaches to the same main +process. It retries transient transport failures for up to 60 seconds. Initial +authentication failures, sandbox lifecycle changes, and clean SSH exits are +not retried. + Press `Ctrl-P`, then `Ctrl-Q` to disconnect without terminating the main process. `Ctrl-C` retains its normal terminal behavior and interrupts the foreground process. For read-only attachments, `Ctrl-C` only exits the -current viewer. +current viewer. OpenSSH's `~.` escape reports the same status as a broken +transport, so it starts automatic recovery instead of exiting. After `~.`, use +`Ctrl-P`, then `Ctrl-Q` once OpenShell reattaches, or press `Ctrl-C` while the +CLI is between retry attempts to cancel recovery. Launch VS Code or Cursor directly into the sandbox workspace: diff --git a/e2e/parity/test.sh b/e2e/parity/test.sh index f1343f1cfb..9d7f2e3b20 100755 --- a/e2e/parity/test.sh +++ b/e2e/parity/test.sh @@ -57,26 +57,29 @@ source "${ROOT}/e2e/support/gateway-common.sh" # shellcheck source=e2e/support/podman-gateway-config.sh source "${ROOT}/e2e/support/podman-gateway-config.sh" mkdir -p "${WORKDIR}/pki/client" "${WORKDIR}/jwt" -e2e_write_podman_gateway_config "${WORKDIR}/v1.toml" 1 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test '' '' 0 '' -e2e_write_podman_gateway_config "${WORKDIR}/v2.toml" 2 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test '' '' 0 '' -e2e_write_podman_gateway_config "${WORKDIR}/v2-external.toml" 2 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 1 socket network 18181 image:test 15 supervisor:test '' '' 0 '' +e2e_write_podman_gateway_config "${WORKDIR}/v1.toml" 1 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test sandbox:test '' '' 0 '' +e2e_write_podman_gateway_config "${WORKDIR}/v2.toml" 2 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test sandbox:test '' '' 0 '' +e2e_write_podman_gateway_config "${WORKDIR}/v2-external.toml" 2 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 1 socket network 18181 image:test 15 supervisor:test sandbox:test '' '' 0 '' assert_contains "${WORKDIR}/v1.toml" 'version = 1' assert_contains "${WORKDIR}/v1.toml" 'compute_drivers = ["podman"]' assert_contains "${WORKDIR}/v1.toml" 'image_pull_policy = "missing"' assert_contains "${WORKDIR}/v1.toml" 'health_check_interval_secs = 0' +assert_contains "${WORKDIR}/v1.toml" 'sandbox_runtime_image = "sandbox:test"' assert_contains "${WORKDIR}/v1.toml" 'guest_tls_ca = ' assert_contains "${WORKDIR}/v2.toml" 'version = 2' assert_contains "${WORKDIR}/v2.toml" 'compute_driver = "podman"' assert_contains "${WORKDIR}/v2.toml" 'image_pull_policy = "if_not_present"' assert_contains "${WORKDIR}/v2.toml" 'allow_driver_config = true' +assert_contains "${WORKDIR}/v2.toml" 'sandbox_runtime_image = "sandbox:test"' assert_contains "${WORKDIR}/v2.toml" '[openshell.drivers.podman.resource_admission]' assert_not_contains "${WORKDIR}/v2.toml" 'health_check_interval_secs = 0' assert_contains "${WORKDIR}/v2-external.toml" 'socket_path = "socket"' +assert_not_contains "${WORKDIR}/v2-external.toml" 'sandbox_runtime_image = "sandbox:test"' assert_not_contains "${WORKDIR}/v2-external.toml" 'allow_driver_config = true' assert_not_contains "${WORKDIR}/v2-external.toml" '[openshell.drivers.podman.resource_admission]' # V2 guest TLS is emitted before its driver table; V1 is driver-local. -OPENSHELL_E2E_PODMAN_OPTION_PROFILE=podman-options e2e_write_podman_gateway_config "${WORKDIR}/v1-options.toml" 1 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test "" "" 0 "" -OPENSHELL_E2E_PODMAN_OPTION_PROFILE=podman-options e2e_write_podman_gateway_config "${WORKDIR}/v2-options.toml" 2 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test "" "" 0 "" +OPENSHELL_E2E_PODMAN_OPTION_PROFILE=podman-options e2e_write_podman_gateway_config "${WORKDIR}/v1-options.toml" 1 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test sandbox:test "" "" 0 "" +OPENSHELL_E2E_PODMAN_OPTION_PROFILE=podman-options e2e_write_podman_gateway_config "${WORKDIR}/v2-options.toml" 2 "${ROOT}" "${WORKDIR}/pki" "${WORKDIR}/jwt" test-gateway 0 socket network 18181 image:test 15 supervisor:test sandbox:test "" "" 0 "" for config in "${WORKDIR}/v1-options.toml" "${WORKDIR}/v2-options.toml"; do assert_contains "${config}" 'sandbox_pids_limit = 31' assert_contains "${config}" 'health_check_interval_secs = 7' diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index 5bfa0a78f9..102f70c147 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -88,6 +88,11 @@ name = "podman_gateway_start" path = "tests/podman_gateway_start.rs" required-features = ["e2e-podman"] +[[test]] +name = "podman_host_gateway" +path = "tests/podman_host_gateway.rs" +required-features = ["e2e-podman"] + [[test]] name = "podman_corporate_proxy" path = "tests/podman_corporate_proxy.rs" @@ -232,7 +237,7 @@ tonic = { version = "0.14", features = ["transport"] } tonic-prost = "0.14" tower = "0.5" url = "2" -nix = { version = "0.29", features = ["user"] } +nix = { version = "0.29", features = ["process", "signal", "term", "user"] } [dev-dependencies] serial_test = "3" diff --git a/e2e/rust/e2e-podman.sh b/e2e/rust/e2e-podman.sh index 736519458e..d4fa2a8103 100755 --- a/e2e/rust/e2e-podman.sh +++ b/e2e/rust/e2e-podman.sh @@ -10,12 +10,61 @@ set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" E2E_TEST="${OPENSHELL_E2E_PODMAN_TEST:-}" +E2E_TEST_SET="${OPENSHELL_E2E_PODMAN_TEST_SET:-}" E2E_FEATURES="${OPENSHELL_E2E_PODMAN_FEATURES-e2e-podman}" DEFAULT_WORKLOAD_MANIFEST="${ROOT}/e2e/gpu/images/.build/workloads.yaml" RUN_WITH_GATEWAY_COMMAND="__openshell_run_podman_e2e" # shellcheck source=e2e/support/conformance.sh source "${ROOT}/e2e/support/conformance.sh" +# Keep required branch CI on the Podman targets that are known to pass. The +# unfiltered e2e:podman task remains available while the omitted targets are +# stabilized and can be added here. +PODMAN_CI_TESTS=( + bypass_detection + cf_auth_smoke + cli_smoke + core_dump_hardening + credential_gating + default_image + driver_config_volume + edge_tunnel_e2e + forward_proxy_graphql_l7 + forward_proxy_jsonrpc_l7 + forward_proxy_l7_bypass + host_gateway_alias + landlock + live_policy_update + local_driver_token_restart + no_proxy + podman_corporate_proxy + podman_gateway_start + podman_host_gateway + provider_token_exchange +) + +if [ -n "${E2E_TEST}" ] && [ -n "${E2E_TEST_SET}" ]; then + echo "error: OPENSHELL_E2E_PODMAN_TEST and OPENSHELL_E2E_PODMAN_TEST_SET cannot both be set" >&2 + exit 2 +fi + +case "${E2E_TEST_SET}" in + "") + SELECTED_TESTS=() + ;; + ci) + SELECTED_TESTS=("${PODMAN_CI_TESTS[@]}") + ;; + *) + echo "error: unknown Podman E2E test set '${E2E_TEST_SET}' (expected 'ci')" >&2 + exit 2 + ;; +esac + +if [ -n "${E2E_TEST}" ]; then + SELECTED_TESTS=("${E2E_TEST}") +fi + if [ "${E2E_TEST}" = "gpu" ] && [ -z "${OPENSHELL_E2E_WORKLOAD_MANIFEST:-}" ] && [ ! -f "${DEFAULT_WORKLOAD_MANIFEST}" ]; then echo "note: running Podman GPU e2e without a workload manifest; workload validation will log an explicit skip. Build one with 'CONTAINER_ENGINE=podman mise run e2e:workloads:build' or set OPENSHELL_E2E_WORKLOAD_MANIFEST." fi @@ -30,16 +79,16 @@ if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" --features "${E2E_FEATURES}" ) - if [ -n "${E2E_TEST}" ]; then - TEST_ARGS+=(--test "${E2E_TEST}") - fi + for test_target in "${SELECTED_TESTS[@]}"; do + TEST_ARGS+=(--test "${test_target}") + done TEST_ARGS+=(-- --nocapture) "${TEST_ARGS[@]}" exit 0 fi # An empty selector runs the full Podman suite, including provider_token_exchange. -if [ -z "${E2E_TEST}" ] || [ "${E2E_TEST}" = "provider_token_exchange" ]; then +if [ "${#SELECTED_TESTS[@]}" -eq 0 ] || [[ " ${SELECTED_TESTS[*]} " == *" provider_token_exchange "* ]]; then export OPENSHELL_E2E_SPIFFE_FIXTURE="${OPENSHELL_E2E_SPIFFE_FIXTURE:-1}" fi @@ -48,5 +97,9 @@ if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ] && [ -z "${OPENSHELL_BIN:-}" ]; then export OPENSHELL_BIN="${ROOT}/target/debug/openshell" fi +if [ -n "${E2E_FEATURES}" ]; then + CONTAINER_ENGINE=podman bash "${ROOT}/tasks/scripts/e2e-build-workload.sh" +fi + exec "${ROOT}/e2e/with-podman-gateway.sh" \ bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}" diff --git a/e2e/rust/src/harness/container.rs b/e2e/rust/src/harness/container.rs index 5ef7ca9262..a85c7747d0 100644 --- a/e2e/rust/src/harness/container.rs +++ b/e2e/rust/src/harness/container.rs @@ -161,12 +161,12 @@ impl ContainerHttpServer { let engine = ContainerEngine::from_env()?; let host_port = find_free_port(); let network = e2e_network_name(); - // A host-networked Docker supervisor cannot use a Docker network's DNS - // aliases, but it can route directly to containers on the bridge. Use - // the fixture's bridge address instead of overloading the reserved - // host alias, which may point at the CI job container. Podman keeps the - // shared-network alias path. - let use_host_port = network.is_none(); + // Host-networked supervisors cannot use a container network's DNS + // aliases. Docker can route directly to a fixture's bridge address, + // while rootless Podman's host network cannot reliably reach its + // rootless bridge. Publish Podman fixtures on the host and use the + // driver-neutral host alias instead. + let use_host_port = network.is_none() || is_e2e_driver("podman"); let mut host = if use_host_port { "host.openshell.internal".to_string() } else { @@ -288,11 +288,12 @@ pub struct SupportContainer { engine: ContainerEngine, } -/// A TCP fixture published on the test host for Kubernetes sandbox e2e tests. +/// A TCP fixture reachable from host-networked sandbox infrastructure. /// -/// Kubernetes sandboxes reach it through the chart-provided -/// `host.openshell.internal` alias. Unlike [`SupportContainer`], this does not -/// require the Docker e2e network used by local-container driver tests. +/// Kubernetes sandboxes reach published fixtures through the chart-provided +/// `host.openshell.internal` alias. Local Podman supervisors can instead run +/// fixtures in the host network. Unlike [`SupportContainer`], neither mode +/// requires the Docker e2e network used by local-container driver tests. pub struct HostSupportContainer { pub port: u16, container_id: String, @@ -347,6 +348,45 @@ impl HostSupportContainer { Ok(fixture) } + /// Start a Python fixture in the host network on a caller-selected port. + /// + /// Local Podman supervisors also use host networking. Matching that mode + /// avoids a nested rootless port-forward when one host fixture (such as a + /// forward proxy) must dial another by its validated host-gateway address. + pub async fn start_python_on_host_network(script: &str, port: u16) -> Result { + let engine = ContainerEngine::from_env()?; + let output = engine + .command() + .args([ + "run", + "--detach", + "--network", + "host", + "--entrypoint", + "python3", + E2E_WORKLOAD_IMAGE, + "-c", + script, + ]) + .output() + .map_err(|err| format!("start {} host-network fixture: {err}", engine.name()))?; + if !output.status.success() { + return Err(format!( + "{} run failed (exit {:?}):\n{}", + engine.name(), + output.status.code(), + String::from_utf8_lossy(&output.stderr) + )); + } + let fixture = Self { + port, + container_id: String::from_utf8_lossy(&output.stdout).trim().to_string(), + engine, + }; + fixture.wait_until_listening(port).await?; + Ok(fixture) + } + /// Start a Python fixture with several ports published on the test host. /// /// This is useful for a host-networked supervisor when one fixture must diff --git a/e2e/rust/src/harness/sandbox.rs b/e2e/rust/src/harness/sandbox.rs index fb7aff9d93..0d742792e3 100644 --- a/e2e/rust/src/harness/sandbox.rs +++ b/e2e/rust/src/harness/sandbox.rs @@ -21,12 +21,14 @@ use super::output::{extract_field, strip_ansi}; /// /// Product defaults remain on the minimal NVIDIA Ubuntu image. Tests that /// explicitly pass `--from` or `--template` retain their requested workload. -/// Docker setup builds this Noble-based fixture before running the tests. -#[cfg(feature = "e2e-docker")] +/// Docker and Podman setup build this Noble-based fixture before running the +/// tests. +#[cfg(any(feature = "e2e-docker", feature = "e2e-podman"))] pub const E2E_WORKLOAD_IMAGE: &str = "openshell/e2e-python:dev"; -/// Preserve the existing pullable fixture for non-Docker E2E lanes. -#[cfg(not(feature = "e2e-docker"))] +/// Preserve the existing pullable fixture for E2E lanes that cannot build the +/// local container-engine fixture. +#[cfg(not(any(feature = "e2e-docker", feature = "e2e-podman")))] pub const E2E_WORKLOAD_IMAGE: &str = "ghcr.io/astral-sh/uv:0.12.17-python3.12-trixie-slim@sha256:9a59bb7206905ccaae4f7dab222fbac47c125a21e5fc16f43f427cd6c940ade3"; /// Extract the sandbox name from CLI create output. diff --git a/e2e/rust/tests/driver_config_volume.rs b/e2e/rust/tests/driver_config_volume.rs index 054ba3bab5..bcef6a429c 100644 --- a/e2e/rust/tests/driver_config_volume.rs +++ b/e2e/rust/tests/driver_config_volume.rs @@ -17,7 +17,9 @@ use bollard::query_parameters::{ RemoveVolumeOptionsBuilder, StartContainerOptions, WaitContainerOptions, }; use futures_util::TryStreamExt; -use openshell_e2e::harness::container::{ImageGuard, e2e_driver}; +#[cfg(feature = "e2e-docker")] +use openshell_e2e::harness::container::ImageGuard; +use openshell_e2e::harness::container::e2e_driver; use openshell_e2e::harness::sandbox::SandboxGuard; use serde_json::{Map, Value}; diff --git a/e2e/rust/tests/host_gateway_alias.rs b/e2e/rust/tests/host_gateway_alias.rs index 8d355f97fb..e3f8cb1c9f 100644 --- a/e2e/rust/tests/host_gateway_alias.rs +++ b/e2e/rust/tests/host_gateway_alias.rs @@ -7,6 +7,7 @@ use std::io::Write; use std::process::Stdio; use openshell_e2e::harness::binary::openshell_cmd; +use openshell_e2e::harness::container::is_e2e_driver; use openshell_e2e::harness::sandbox::SandboxGuard; use tempfile::{Builder as TempFileBuilder, NamedTempFile}; use tokio::io::AsyncReadExt; @@ -478,8 +479,11 @@ async fn static_provider_credentials_are_bound_to_profile_endpoints() { http_request() {{ local host="$1" path="$2" status_line line HTTP_STATUS= HTTP_BODY= - exec 3<>"/dev/tcp/$host/{0}" || return 1 - printf 'GET %s HTTP/1.1\r\nHost: %s:{0}\r\nAuthorization: Bearer %s\r\nConnection: close\r\n\r\n' "$path" "$host" "$BOUND_TOKEN_A" >&3 + if ! exec 3<>"/dev/tcp/$host/{port}"; then + HTTP_STATUS=connect-denied + return 1 + fi + printf 'GET %s HTTP/1.1\r\nHost: %s:{port}\r\nAuthorization: Bearer %s\r\nConnection: close\r\n\r\n' "$path" "$host" "$BOUND_TOKEN_A" >&3 IFS= read -r status_line <&3 || return 1 status_line="${{status_line%$'\r'}}" HTTP_STATUS="${{status_line#* }}" @@ -492,11 +496,11 @@ http_request() {{ exec 3>&- 3<&- }} http_request host.openshell.internal /allowed/check; allowed="$HTTP_BODY" -http_request host.docker.internal /allowed/check; host_denied="$HTTP_STATUS" +http_request host.docker.internal /allowed/check || true; host_denied="$HTTP_STATUS" http_request host.openshell.internal /other/check; path_denied="$HTTP_STATUS" printf 'ALLOWED=%s HOST_DENIED=%s PATH_DENIED=%s\n' "$allowed" "$host_denied" "$path_denied" "#, - server.port + port = server.port, ); let mut guard = SandboxGuard::create(&[ "--policy", @@ -527,8 +531,13 @@ printf 'ALLOWED=%s HOST_DENIED=%s PATH_DENIED=%s\n' "$allowed" "$host_denied" "$ "credential should resolve at the bound endpoint:\n{}\nlogs:\n{logs}", guard.create_output, ); + let expected_host_denial = if is_e2e_driver("podman") { + "HOST_DENIED=connect-denied" + } else { + "HOST_DENIED=403" + }; assert!( - guard.create_output.contains("HOST_DENIED=403"), + guard.create_output.contains(expected_host_denial), "same placeholder must be denied at an unbound host:\n{}", guard.create_output ); diff --git a/e2e/rust/tests/local_driver_token_restart.rs b/e2e/rust/tests/local_driver_token_restart.rs index 5c9661d89c..55dbf91821 100644 --- a/e2e/rust/tests/local_driver_token_restart.rs +++ b/e2e/rust/tests/local_driver_token_restart.rs @@ -4,15 +4,16 @@ #![cfg(feature = "e2e")] //! Local-driver E2E regression for sandbox bootstrap JWT material and terminal -//! main-process semantics. Local single-player gateways mint non-expiring -//! bootstrap tokens. Stopping a Docker or Podman sandbox container is terminal -//! and must not relaunch its canonical process; restarting the VM gateway still -//! exercises driver recovery without deliberately stopping the sandbox. +//! main-process semantics. Launch-scoped bootstrap tokens must remain valid long +//! enough for local-driver recovery. Stopping a Docker or Podman sandbox +//! workload is terminal and must not relaunch its canonical process; restarting +//! the VM gateway still exercises driver recovery without deliberately stopping +//! the sandbox. use std::fs; use std::path::PathBuf; use std::process::Stdio; -use std::time::{Duration, Instant}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; use base64::Engine as _; use openshell_e2e::harness::cli::{wait_for_healthy, wait_for_sandbox_exec_contains}; @@ -63,16 +64,17 @@ impl LocalDriver { matches!(self, Self::Docker | Self::Podman) } - fn container_filters(self, namespace: &str, sandbox_name: &str) -> Vec { + fn container_filters(self, namespace: &str, sandbox_name: &str, role: &str) -> Vec { match self { Self::Docker => vec![ "label=openshell.ai/managed-by=openshell".to_string(), - "label=openshell.ai/isolation-role=sandbox".to_string(), + format!("label=openshell.ai/isolation-role={role}"), format!("label=openshell.ai/sandbox-namespace={namespace}"), format!("label=openshell.ai/sandbox-name={sandbox_name}"), ], Self::Podman => vec![ "label=openshell.managed=true".to_string(), + format!("label=openshell.io/isolation-role={role}"), format!("label=openshell.ai/sandbox-name={sandbox_name}"), ], Self::Vm => Vec::new(), @@ -102,14 +104,15 @@ fn run_engine(engine: &ContainerEngine, args: &[String]) -> Result Result { let mut args = vec!["ps".to_string(), "-aq".to_string()]; - for filter in driver.container_filters(namespace, sandbox_name) { + for filter in driver.container_filters(namespace, sandbox_name, role) { args.push("--filter".to_string()); args.push(filter); } @@ -123,10 +126,10 @@ fn sandbox_container_id( match ids.as_slice() { [id] => Ok((*id).to_string()), [] => Err(format!( - "no {driver:?} container found for sandbox '{sandbox_name}' in namespace '{namespace}'" + "no {driver:?} {role} container found for sandbox '{sandbox_name}' in namespace '{namespace}'" )), _ => Err(format!( - "multiple {driver:?} containers found for sandbox '{sandbox_name}' in namespace '{namespace}': {ids:?}" + "multiple {driver:?} {role} containers found for sandbox '{sandbox_name}' in namespace '{namespace}': {ids:?}" )), } } @@ -175,16 +178,23 @@ async fn wait_for_container_running( } } -fn read_bootstrap_token(engine: &ContainerEngine, container_id: &str) -> Result { +fn copy_bootstrap_token(engine: &ContainerEngine, container_id: &str) -> Result { + let tempdir = tempfile::tempdir().map_err(|err| format!("create token tempdir: {err}"))?; + let destination = tempdir.path().join("sandbox.jwt"); run_engine( engine, &[ - "exec".to_string(), - container_id.to_string(), - "cat".to_string(), - CONTAINER_TOKEN_MOUNT_PATH.to_string(), + "cp".to_string(), + format!("{container_id}:{CONTAINER_TOKEN_MOUNT_PATH}"), + destination.display().to_string(), ], - ) + )?; + fs::read_to_string(&destination).map_err(|err| { + format!( + "read copied bootstrap token '{}': {err}", + destination.display() + ) + }) } fn read_vm_bootstrap_token(sandbox_name: &str) -> Result { @@ -260,10 +270,19 @@ fn token_exp_claim(token: &str) -> Result { .ok_or_else(|| format!("sandbox JWT missing integer exp claim: {claims}")) } -fn require_non_expiring_token(token: &str, context: &str) -> Result<(), String> { +fn require_unexpired_token(token: &str, context: &str) -> Result<(), String> { let exp = token_exp_claim(token)?; - if exp != 0 { - return Err(format!("{context} should use exp=0, got exp={exp}")); + let now = i64::try_from( + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|err| format!("system clock is before Unix epoch: {err}"))? + .as_secs(), + ) + .map_err(|_| "current Unix timestamp does not fit in i64".to_string())?; + if exp <= now { + return Err(format!( + "{context} should expire in the future, got exp={exp}, now={now}" + )); } Ok(()) } @@ -274,28 +293,30 @@ async fn stop_container_sandbox( namespace: &str, sandbox_name: &str, ) -> Result<(), String> { - let container_id = sandbox_container_id(engine, driver, namespace, sandbox_name)?; - if driver == LocalDriver::Docker { - run_engine( - engine, - &[ - "exec".to_string(), - container_id.clone(), - "sh".to_string(), - "-c".to_string(), - format!("test ! -r {CONTAINER_TOKEN_MOUNT_PATH}"), - ], - ) - .map_err(|error| { - format!("Docker sandbox workload must not be able to read its bootstrap JWT: {error}") - })?; - } else { - let token = read_bootstrap_token(engine, &container_id)?; - require_non_expiring_token(&token, "local-driver bootstrap JWT")?; + let workload_id = container_id_for_role(engine, driver, namespace, sandbox_name, "sandbox")?; + run_engine( + engine, + &[ + "exec".to_string(), + workload_id.clone(), + "/bin/sh".to_string(), + "-c".to_string(), + format!("test ! -r {CONTAINER_TOKEN_MOUNT_PATH}"), + ], + ) + .map_err(|error| { + format!("sandbox workload must not be able to read its bootstrap JWT: {error}") + })?; + + if driver == LocalDriver::Podman { + let supervisor_id = + container_id_for_role(engine, driver, namespace, sandbox_name, "supervisor")?; + let token = copy_bootstrap_token(engine, &supervisor_id)?; + require_unexpired_token(&token, "local-driver bootstrap JWT")?; } - run_engine(engine, &["stop".to_string(), container_id.clone()])?; - wait_for_container_running(engine, &container_id, false, Duration::from_secs(60)).await + run_engine(engine, &["stop".to_string(), workload_id.clone()])?; + wait_for_container_running(engine, &workload_id, false, Duration::from_secs(60)).await } async fn wait_for_sandbox_error(sandbox_name: &str, timeout: Duration) -> Result<(), String> { @@ -327,7 +348,7 @@ async fn wait_for_sandbox_error(sandbox_name: &str, timeout: Duration) -> Result async fn restart_vm_sandbox(gateway: &ManagedGateway, sandbox_name: &str) -> Result<(), String> { let token = read_vm_bootstrap_token(sandbox_name)?; - require_non_expiring_token(&token, "VM bootstrap JWT")?; + require_unexpired_token(&token, "VM bootstrap JWT")?; gateway.stop()?; gateway.start()?; diff --git a/e2e/rust/tests/podman_corporate_proxy.rs b/e2e/rust/tests/podman_corporate_proxy.rs index ef1a08f0b4..59f6c11066 100644 --- a/e2e/rust/tests/podman_corporate_proxy.rs +++ b/e2e/rust/tests/podman_corporate_proxy.rs @@ -23,21 +23,19 @@ //! 4. Deleting the sandbox removes the per-sandbox Podman secret. use std::io::Write as _; +use std::os::unix::fs::PermissionsExt as _; use std::path::PathBuf; use std::time::Duration; use openshell_e2e::harness::cli::wait_for_healthy; -use openshell_e2e::harness::container::{ContainerEngine, SupportContainer}; +use openshell_e2e::harness::container::{ContainerEngine, HostSupportContainer}; use openshell_e2e::harness::gateway::ManagedGateway; +use openshell_e2e::harness::port::find_free_port; use openshell_e2e::harness::sandbox::SandboxGuard; use serial_test::serial; use tempfile::NamedTempFile; -const PROXY_ALIAS: &str = "corp-proxy.openshell.test"; -const PROXY_PORT: u16 = 3128; -const ALLOWED_ALIAS: &str = "tls-upstream.openshell.test"; -const DENIED_ALIAS: &str = "denied-upstream.openshell.test"; -const UPSTREAM_PORT: u16 = 8443; +const HOST_ALIAS: &str = "host.openshell.internal"; const PROXY_USER: &str = "proxyuser"; const PROXY_PASS: &str = "proxypass"; @@ -52,7 +50,7 @@ const SECRET_PREFIX: &str = "openshell-proxy-auth-"; /// /// The log lines are the test's evidence: they record the exact CONNECT target /// (proving validated-IP vs hostname form) and whether credentials arrived. -fn proxy_script() -> String { +fn proxy_script(port: u16) -> String { format!( r#" import base64, select, socket, threading @@ -127,7 +125,7 @@ def handle(conn): server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -server.bind(('0.0.0.0', {PROXY_PORT})) +server.bind(('0.0.0.0', {port})) server.listen(64) log('proxy-listening') while True: @@ -156,7 +154,7 @@ const CA_END: &str = "---PROXY-CA-END---"; /// what a real intercepting proxy does. No Basic auth: this test isolates the /// `https://` proxy + corporate-CA path; credential delivery is covered by the /// plaintext-proxy test. -fn tls_proxy_script() -> String { +fn tls_proxy_script(port: u16) -> String { format!( r" import os, select, socket, ssl, subprocess, tempfile, threading @@ -179,10 +177,10 @@ run('openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', with open(ext, 'w') as fh: fh.write('basicConstraints=critical,CA:FALSE\n' - 'subjectAltName=DNS:{PROXY_ALIAS}\n' + 'subjectAltName=DNS:{HOST_ALIAS}\n' 'extendedKeyUsage=serverAuth\n') run('openssl', 'req', '-newkey', 'rsa:2048', '-nodes', - '-keyout', leaf_key, '-out', leaf_csr, '-subj', '/CN={PROXY_ALIAS}') + '-keyout', leaf_key, '-out', leaf_csr, '-subj', '/CN={HOST_ALIAS}') run('openssl', 'x509', '-req', '-in', leaf_csr, '-CA', ca_crt, '-CAkey', ca_key, '-CAcreateserial', '-out', leaf_crt, '-days', '1', '-extfile', ext) @@ -253,7 +251,7 @@ ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) ctx.load_cert_chain(chain, leaf_key) server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -server.bind(('0.0.0.0', {PROXY_PORT})) +server.bind(('0.0.0.0', {port})) server.listen(64) log('tls-proxy-listening') while True: @@ -287,7 +285,7 @@ fn ca_cert_from_logs(logs: &str) -> Result { /// The sandbox workload uses an unverified TLS context, so the certificate only /// needs to exist — but the TLS handshake itself must be real, because it is /// what proves bytes flowed end to end through the tunnel. -fn tls_upstream_script(common_name: &str, marker: &str) -> String { +fn tls_upstream_script(common_name: &str, marker: &str, port: u16) -> String { format!( r#" import http.server, os, ssl, subprocess, tempfile @@ -320,7 +318,7 @@ class Server(http.server.HTTPServer): ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) ctx.load_cert_chain(crt, key) -server = Server(('0.0.0.0', {UPSTREAM_PORT}), Handler) +server = Server(('0.0.0.0', {port}), Handler) server.socket = ctx.wrap_socket(server.socket, server_side=True) print('tls-upstream-listening', flush=True) server.serve_forever() @@ -330,7 +328,7 @@ server.serve_forever() /// Workload: one approved HTTPS request, one denied, then idle so the test can /// inspect the live sandbox's Podman secret before deleting it. -fn workload_script() -> String { +fn workload_script(allowed_port: u16, denied_port: u16) -> String { format!( r" import json, ssl, time, urllib.request @@ -351,10 +349,10 @@ def fetch(url, retries): # The approved request is retried: policy reload during sandbox startup can # transiently surface as a 403 in the forward proxy. print('ALLOWED_RESULT ' + json.dumps( - fetch('https://{ALLOWED_ALIAS}:{UPSTREAM_PORT}/', 6)), flush=True) + fetch('https://{HOST_ALIAS}:{allowed_port}/', 6)), flush=True) # The denied request must fail, so a single attempt is enough. print('DENIED_RESULT ' + json.dumps( - fetch('https://{DENIED_ALIAS}:{UPSTREAM_PORT}/', 1)), flush=True) + fetch('https://{HOST_ALIAS}:{denied_port}/', 1)), flush=True) print('{READY_MARKER}', flush=True) while True: time.sleep(1) @@ -365,7 +363,7 @@ while True: /// Policy allowing only the approved upstream. `tls: skip` keeps the tunnel raw /// so the workload's TLS session runs end to end to the upstream, which is what /// makes the proxied CONNECT path observable. -fn policy_yaml() -> String { +fn policy_yaml(allowed_port: u16) -> String { format!( r#"version: 1 @@ -395,8 +393,8 @@ network_policies: corporate_proxy_e2e: name: corporate_proxy_e2e endpoints: - - host: {ALLOWED_ALIAS} - port: {UPSTREAM_PORT} + - host: {HOST_ALIAS} + port: {allowed_port} tls: skip enforcement: enforce allowed_ips: @@ -473,19 +471,26 @@ impl GatewayProxyConfig { proxy_config.push_str(&format!("proxy_ca_bundle = \"{ca_bundle}\"\n")); } - // The managed gateway config ends in nested gateway tables. Insert - // proxy fields before the first one, while the active TOML table is - // still [openshell.drivers.podman]. - let insertion = b"\n[openshell.gateway."; - let offset = original - .windows(insertion.len()) - .position(|window| window == insertion) + // Insert the proxy fields at the end of the Podman driver table. The + // generated config may add driver subtables (for example + // `resource_admission`) before the gateway JWT and auth tables, so a + // gateway-table marker is not a reliable boundary. + let podman_table = b"[openshell.drivers.podman]"; + let table_start = original + .windows(podman_table.len()) + .position(|window| window == podman_table) .ok_or_else(|| { format!( - "gateway config '{}' has no nested gateway table insertion point", + "gateway config '{}' has no Podman driver table", config_path.display() ) })?; + let table_body = table_start + podman_table.len(); + let next_table = b"\n["; + let offset = original[table_body..] + .windows(next_table.len()) + .position(|window| window == next_table) + .map_or(original.len(), |relative| table_body + relative); let mut updated = Vec::with_capacity(original.len() + proxy_config.len() + 1); updated.extend_from_slice(&original[..offset]); if !updated.ends_with(b"\n") { @@ -594,7 +599,7 @@ async fn wait_for_secret_removal(secret: &str, timeout: Duration) -> Result<(), /// `output` is the sandbox's stdout; `proxy_logs` is the fake proxy's log, /// which is the only place the CONNECT target form and the credential outcome /// are observable. -fn assert_proxied_egress(output: &str, proxy_logs: &str, allowed_ip: &str, denied_ip: &str) { +fn assert_proxied_egress(output: &str, proxy_logs: &str, allowed_port: u16, denied_port: u16) { // The approved request succeeded and its body came from the upstream, // proving bytes traversed the tunnel rather than the proxy short-circuiting. assert!( @@ -606,18 +611,18 @@ fn assert_proxied_egress(output: &str, proxy_logs: &str, allowed_ip: &str, denie "approved HTTPS request should have returned 200:\n{output}" ); - // The denied destination failed inside the sandbox — and specifically with - // a 403 from the supervisor's own proxy. Asserting the status rather than - // just "it failed" is what rules out a vacuous pass: an unresolvable host - // or a dead upstream would also fail, but neither would prove the request - // was stopped by policy. + // The denied destination failed inside the sandbox. Depending on where the + // isolated Podman path rejects it, this is either the supervisor's HTTP 403 + // or a fail-closed socket permission error. The proxy-log assertion below + // rules out a vacuous failure after upstream contact. let denied_line = output .lines() .find(|line| line.contains("DENIED_RESULT")) .unwrap_or_default(); assert!( - denied_line.contains(r#""status": -1"#) && denied_line.contains("403"), - "denied destination should have been refused by policy with 403:\n{output}" + denied_line.contains(r#""status": -1"#) + && (denied_line.contains("403") || denied_line.contains("Permission denied")), + "denied destination should have been refused by policy:\n{output}" ); assert!( !output.contains(DENIED_MARKER), @@ -627,9 +632,11 @@ fn assert_proxied_egress(output: &str, proxy_logs: &str, allowed_ip: &str, denie // Credentials arrived through the mounted secret: the proxy answers 407 // without them, so a completed CONNECT is proof of delivery. assert!( - proxy_logs.contains(&format!("CONNECT {allowed_ip}:{UPSTREAM_PORT} auth=ok")), + proxy_logs.lines().any(|line| { + line.starts_with("CONNECT ") && line.contains(&format!(":{allowed_port} auth=ok")) + }), "proxy should have seen an authenticated validated-IP CONNECT to \ - {allowed_ip}:{UPSTREAM_PORT}.\nProxy logs:\n{proxy_logs}\nSandbox output:\n{output}" + host port {allowed_port}.\nProxy logs:\n{proxy_logs}\nSandbox output:\n{output}" ); assert!( !proxy_logs.contains("auth=fail"), @@ -640,17 +647,18 @@ fn assert_proxied_egress(output: &str, proxy_logs: &str, allowed_ip: &str, denie // no DNS resolution of its own, so the tunnel stays bound to the address // that passed SSRF and allowed_ips validation. assert!( - !proxy_logs.contains(ALLOWED_ALIAS), + !proxy_logs.contains(HOST_ALIAS), "CONNECT should target a validated IP, not the hostname:\n{proxy_logs}" ); // The denied destination never reached the proxy: policy denial happens - // before any upstream contact. The port is part of the needle so a shorter - // IP cannot match inside a longer one (10.89.0.2 within 10.89.0.20). + // before any upstream contact. Its dedicated host port makes this + // observable even though both fixtures use the driver-neutral host alias. assert!( - !proxy_logs.contains(&format!("{denied_ip}:{UPSTREAM_PORT}")) - && !proxy_logs.contains(DENIED_ALIAS), - "policy-denied destination {denied_ip} ({DENIED_ALIAS}) must never reach the proxy:\n{proxy_logs}" + !proxy_logs + .lines() + .any(|line| line.starts_with("CONNECT ") && line.contains(&format!(":{denied_port} "))), + "policy-denied host port {denied_port} must never reach the proxy:\n{proxy_logs}" ); } @@ -669,33 +677,32 @@ async fn podman_corporate_proxy_routes_approved_tls_egress() { return; } - // ── Fixtures on the shared e2e network ──────────────────────────── - let proxy = SupportContainer::start_python(PROXY_ALIAS, &proxy_script(), PROXY_PORT) - .await - .expect("start fake corporate proxy"); - let allowed = SupportContainer::start_python( - ALLOWED_ALIAS, - &tls_upstream_script(ALLOWED_ALIAS, ALLOWED_MARKER), - UPSTREAM_PORT, + // The split Podman supervisor uses host networking and cannot resolve + // container-network aliases. Run each fixture on a distinct host port and + // reach it through the driver-neutral host alias. + let proxy_port = find_free_port(); + let proxy = + HostSupportContainer::start_python_on_host_network(&proxy_script(proxy_port), proxy_port) + .await + .expect("start fake corporate proxy"); + let allowed_port = find_free_port(); + let allowed = HostSupportContainer::start_python_on_host_network( + &tls_upstream_script(HOST_ALIAS, ALLOWED_MARKER, allowed_port), + allowed_port, ) .await .expect("start approved TLS upstream"); - // A separate container, not another alias on the approved one: CONNECT - // targets are IPs, so a shared IP would make "the proxy never saw the - // denied destination" unprovable. - let denied = SupportContainer::start_python( - DENIED_ALIAS, - &tls_upstream_script(DENIED_ALIAS, DENIED_MARKER), - UPSTREAM_PORT, + let denied_port = find_free_port(); + let denied = HostSupportContainer::start_python_on_host_network( + &tls_upstream_script(HOST_ALIAS, DENIED_MARKER, denied_port), + denied_port, ) .await .expect("start denied TLS upstream"); - let allowed_ip = allowed.ip().expect("resolve approved upstream IP"); - let denied_ip = denied.ip().expect("resolve denied upstream IP"); assert_ne!( - allowed_ip, denied_ip, - "approved and denied upstreams must have distinct IPs" + allowed.port, denied.port, + "approved and denied upstreams must have distinct host ports" ); // ── Point the gateway at the corporate proxy ────────────────────── @@ -711,14 +718,14 @@ async fn podman_corporate_proxy_routes_approved_tls_egress() { let secrets_before = proxy_auth_secret_names().expect("snapshot proxy-auth secrets"); let mut gateway_config = - GatewayProxyConfig::apply(&format!("http://{PROXY_ALIAS}:{PROXY_PORT}"), &auth_path) + GatewayProxyConfig::apply(&format!("http://{HOST_ALIAS}:{}", proxy.port), &auth_path) .await .expect("apply corporate proxy gateway config"); // ── Run the workload ────────────────────────────────────────────── let mut policy = NamedTempFile::new().expect("create policy file"); policy - .write_all(policy_yaml().as_bytes()) + .write_all(policy_yaml(allowed.port).as_bytes()) .expect("write policy file"); policy.flush().expect("flush policy file"); let policy_path = policy @@ -727,7 +734,7 @@ async fn podman_corporate_proxy_routes_approved_tls_egress() { .expect("policy path should be utf-8") .to_string(); - let script = workload_script(); + let script = workload_script(allowed.port, denied.port); let mut sandbox = SandboxGuard::create_keep_with_args( &["--policy", &policy_path], &["python3", "-c", &script], @@ -739,8 +746,8 @@ async fn podman_corporate_proxy_routes_approved_tls_egress() { assert_proxied_egress( &sandbox.create_output, &proxy.logs().expect("read fake proxy logs"), - &allowed_ip, - &denied_ip, + allowed.port, + denied.port, ); // ── Secret lifecycle ────────────────────────────────────────────── @@ -772,7 +779,12 @@ async fn podman_corporate_proxy_routes_approved_tls_egress() { /// Assert the approved destination reached the `https://` proxy over the /// tunnel and the denied one never did. The TLS proxy logs `CONNECT /// : ok` (no auth dimension in this test). -fn assert_https_proxied_egress(output: &str, proxy_logs: &str, allowed_ip: &str, denied_ip: &str) { +fn assert_https_proxied_egress( + output: &str, + proxy_logs: &str, + allowed_port: u16, + denied_port: u16, +) { assert!( output.contains(READY_MARKER), "workload did not finish; output:\n{output}" @@ -784,13 +796,16 @@ fn assert_https_proxied_egress(output: &str, proxy_logs: &str, allowed_ip: &str, "approved upstream body missing — egress did not complete through the https proxy:\n{output}" ); assert!( - proxy_logs.contains(&format!("CONNECT {allowed_ip}:{UPSTREAM_PORT} ok")), - "https proxy should have seen a validated-IP CONNECT to the approved upstream:\n{proxy_logs}" + proxy_logs.lines().any(|line| { + line.starts_with("CONNECT ") && line.contains(&format!(":{allowed_port} ok")) + }), + "https proxy should have seen a validated-IP CONNECT to approved host port {allowed_port}:\n{proxy_logs}" ); assert!( - !proxy_logs.contains(DENIED_ALIAS) - && !proxy_logs.contains(&format!("{denied_ip}:{UPSTREAM_PORT}")), - "policy-denied destination {denied_ip} ({DENIED_ALIAS}) must never reach the https proxy:\n{proxy_logs}" + !proxy_logs + .lines() + .any(|line| line.starts_with("CONNECT ") && line.contains(&format!(":{denied_port} "))), + "policy-denied host port {denied_port} must never reach the https proxy:\n{proxy_logs}" ); } @@ -811,30 +826,31 @@ async fn podman_corporate_proxy_trusts_ca_bundle_for_https_proxy() { return; } - // ── Fixtures on the shared e2e network ──────────────────────────── - let proxy = SupportContainer::start_python(PROXY_ALIAS, &tls_proxy_script(), PROXY_PORT) - .await - .expect("start fake https corporate proxy"); - let allowed = SupportContainer::start_python( - ALLOWED_ALIAS, - &tls_upstream_script(ALLOWED_ALIAS, ALLOWED_MARKER), - UPSTREAM_PORT, + let proxy_port = find_free_port(); + let proxy = HostSupportContainer::start_python_on_host_network( + &tls_proxy_script(proxy_port), + proxy_port, + ) + .await + .expect("start fake https corporate proxy"); + let allowed_port = find_free_port(); + let allowed = HostSupportContainer::start_python_on_host_network( + &tls_upstream_script(HOST_ALIAS, ALLOWED_MARKER, allowed_port), + allowed_port, ) .await .expect("start approved TLS upstream"); - let denied = SupportContainer::start_python( - DENIED_ALIAS, - &tls_upstream_script(DENIED_ALIAS, DENIED_MARKER), - UPSTREAM_PORT, + let denied_port = find_free_port(); + let denied = HostSupportContainer::start_python_on_host_network( + &tls_upstream_script(HOST_ALIAS, DENIED_MARKER, denied_port), + denied_port, ) .await .expect("start denied TLS upstream"); - let allowed_ip = allowed.ip().expect("resolve approved upstream IP"); - let denied_ip = denied.ip().expect("resolve denied upstream IP"); assert_ne!( - allowed_ip, denied_ip, - "approved and denied upstreams must have distinct IPs" + allowed.port, denied.port, + "approved and denied upstreams must have distinct host ports" ); // Recover the proxy's self-signed CA from its logs and hand it back as the @@ -846,6 +862,10 @@ async fn podman_corporate_proxy_trusts_ca_bundle_for_https_proxy() { .write_all(ca_pem.as_bytes()) .expect("write proxy CA bundle"); ca_file.flush().expect("flush proxy CA bundle"); + ca_file + .as_file() + .set_permissions(std::fs::Permissions::from_mode(0o644)) + .expect("make proxy CA bundle readable by the non-root supervisor"); let ca_path = ca_file .path() .to_str() @@ -854,7 +874,7 @@ async fn podman_corporate_proxy_trusts_ca_bundle_for_https_proxy() { // ── Point the gateway at the https:// proxy with the CA bundle ───── let mut gateway_config = GatewayProxyConfig::apply_with( - &format!("https://{PROXY_ALIAS}:{PROXY_PORT}"), + &format!("https://{HOST_ALIAS}:{}", proxy.port), None, Some(&ca_path), ) @@ -864,7 +884,7 @@ async fn podman_corporate_proxy_trusts_ca_bundle_for_https_proxy() { // ── Run the workload ────────────────────────────────────────────── let mut policy = NamedTempFile::new().expect("create policy file"); policy - .write_all(policy_yaml().as_bytes()) + .write_all(policy_yaml(allowed.port).as_bytes()) .expect("write policy file"); policy.flush().expect("flush policy file"); let policy_path = policy @@ -873,7 +893,7 @@ async fn podman_corporate_proxy_trusts_ca_bundle_for_https_proxy() { .expect("policy path should be utf-8") .to_string(); - let script = workload_script(); + let script = workload_script(allowed.port, denied.port); let mut sandbox = SandboxGuard::create_keep_with_args( &["--policy", &policy_path], &["python3", "-c", &script], @@ -885,8 +905,8 @@ async fn podman_corporate_proxy_trusts_ca_bundle_for_https_proxy() { assert_https_proxied_egress( &sandbox.create_output, &proxy.logs().expect("read fake proxy logs"), - &allowed_ip, - &denied_ip, + allowed.port, + denied.port, ); sandbox.cleanup().await; diff --git a/e2e/rust/tests/podman_host_gateway.rs b/e2e/rust/tests/podman_host_gateway.rs new file mode 100644 index 0000000000..8a6dc74064 --- /dev/null +++ b/e2e/rust/tests/podman_host_gateway.rs @@ -0,0 +1,233 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +#![cfg(feature = "e2e-podman")] + +//! Verifies that a Podman sandbox can connect to a service on its host through +//! the driver-neutral `host.openshell.internal` alias. + +use std::fmt::Write as _; +use std::time::Duration; + +use openshell_e2e::harness::cli::{run_cli, wait_for_sandbox_phase}; +use openshell_e2e::harness::container::ContainerEngine; +use openshell_e2e::harness::sandbox::SandboxGuard; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpListener; + +const RESPONSE: &str = "host.openshell.internal is reachable"; +const READY_MARKER: &str = "podman-host-gateway-ready"; + +fn podman_container_diagnostics(sandbox_name: &str) -> String { + let Ok(engine) = ContainerEngine::from_env() else { + return "container engine unavailable".to_string(); + }; + let name_filter = format!("label=openshell.ai/sandbox-name={sandbox_name}"); + let output = match engine + .command() + .args([ + "ps", + "-aq", + "--filter", + "label=openshell.managed=true", + "--filter", + &name_filter, + ]) + .output() + { + Ok(output) => output, + Err(err) => return format!("podman ps failed: {err}"), + }; + let ids = String::from_utf8_lossy(&output.stdout); + let mut diagnostics = String::new(); + for id in ids.lines().map(str::trim).filter(|id| !id.is_empty()) { + let inspect = engine + .command() + .args([ + "inspect", + "--format", + "{{.Name}} role={{index .Config.Labels \"openshell.io/isolation-role\"}} state={{json .State}}", + id, + ]) + .output(); + let logs = engine + .command() + .args(["logs", "--tail", "200", id]) + .output(); + let _ = write!( + diagnostics, + "\n--- container {id} ---\ninspect:\n{}\nlogs:\n{}{}", + inspect + .as_ref() + .map_or_else(std::string::ToString::to_string, |out| { + String::from_utf8_lossy(&out.stdout).into_owned() + }), + logs.as_ref() + .map_or_else(std::string::ToString::to_string, |out| { + String::from_utf8_lossy(&out.stdout).into_owned() + }), + logs.as_ref().map_or_else( + |_| String::new(), + |out| String::from_utf8_lossy(&out.stderr).into_owned() + ) + ); + } + diagnostics +} + +async fn assert_host_gateway_reachable(sandbox: &SandboxGuard, port: u16, stage: &str) { + let url = format!("http://host.openshell.internal:{port}/"); + let probe = async { + let mut failures = Vec::new(); + for attempt in 1..=5 { + match sandbox + .exec(&[ + "/usr/bin/curl", + "--fail", + "--silent", + "--show-error", + "--noproxy", + "*", + "--connect-timeout", + "2", + "--max-time", + "5", + &url, + ]) + .await + { + Ok(output) if output.contains(RESPONSE) => return Ok(()), + Ok(output) => failures.push(format!( + "attempt {attempt} returned an unexpected response:\n{output}" + )), + Err(err) => failures.push(format!("attempt {attempt} failed: {err}")), + } + + if attempt < 5 { + tokio::time::sleep(Duration::from_secs(1)).await; + } + } + Err(failures.join("\n")) + }; + + match tokio::time::timeout(Duration::from_secs(40), probe).await { + Ok(Ok(())) => {} + Ok(Err(err)) => panic!( + "host gateway probe failed during {stage} after bounded retries:\n{err}\n{}", + podman_container_diagnostics(&sandbox.name) + ), + Err(_) => { + let (logs, _) = run_cli(&["logs", &sandbox.name, "-n", "500"]).await; + panic!( + "host gateway probe timed out during {stage}; sandbox logs:\n{logs}\n{}", + podman_container_diagnostics(&sandbox.name) + ); + } + } +} + +#[tokio::test] +async fn podman_sandbox_reaches_host_openshell_internal() { + let listener = TcpListener::bind(("0.0.0.0", 0)) + .await + .expect("bind host-side TCP listener"); + let port = listener.local_addr().expect("read listener address").port(); + let server = tokio::spawn(async move { + loop { + let (mut stream, _) = listener.accept().await.expect("accept sandbox connection"); + tokio::spawn(async move { + let mut request = Vec::new(); + let mut chunk = [0u8; 1024]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + let Ok(read) = stream.read(&mut chunk).await else { + return; + }; + if read == 0 { + return; + } + request.extend_from_slice(&chunk[..read]); + } + let response = format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{RESPONSE}\n", + RESPONSE.len() + 1 + ); + let _ = stream.write_all(response.as_bytes()).await; + }); + } + }); + + let policy = tempfile::NamedTempFile::new().expect("create policy file"); + std::fs::write( + policy.path(), + format!( + r"version: 1 + +filesystem_policy: + include_workdir: true + read_only: [/usr, /bin, /lib, /proc, /dev/urandom, /app, /etc, /var/log] + read_write: [/sandbox, /tmp, /dev/null] + +landlock: + compatibility: best_effort + +network_policies: + host_gateway: + name: host_gateway + endpoints: + - host: host.openshell.internal + port: {port} + protocol: tcp + binaries: + - path: /usr/bin/curl +" + ), + ) + .expect("write policy file"); + + let policy_path = policy.path().to_str().expect("policy path is UTF-8"); + let mut sandbox = SandboxGuard::create_keep_with_args( + &["--policy", policy_path, "--no-tty"], + &[ + "/usr/bin/bash", + "-c", + &format!("echo {READY_MARKER}; exec sleep infinity"), + ], + READY_MARKER, + ) + .await + .expect("create long-running Podman sandbox"); + + let resolv_conf = sandbox + .exec(&["/usr/bin/cat", "/etc/resolv.conf"]) + .await + .expect("read workload resolver configuration"); + assert!( + resolv_conf + .lines() + .any(|line| line.trim() == "nameserver 127.0.0.53"), + "Podman workload did not use the policy-DNS relay:\n{resolv_conf}" + ); + + assert_host_gateway_reachable(&sandbox, port, "initial startup").await; + + let (stop_output, stop_code) = run_cli(&["sandbox", "stop", &sandbox.name]).await; + assert_eq!(stop_code, 0, "sandbox stop should succeed:\n{stop_output}"); + wait_for_sandbox_phase(&sandbox.name, "Stopped", Duration::from_secs(120)) + .await + .expect("Podman sandbox should stop"); + + let (start_output, start_code) = run_cli(&["sandbox", "start", &sandbox.name]).await; + assert_eq!( + start_code, 0, + "sandbox start should succeed:\n{start_output}" + ); + wait_for_sandbox_phase(&sandbox.name, "Ready", Duration::from_secs(120)) + .await + .expect("Podman sandbox should become ready after restart"); + + assert_host_gateway_reachable(&sandbox, port, "restart").await; + + sandbox.cleanup().await; + server.abort(); + let _ = server.await; +} diff --git a/e2e/rust/tests/podman_oci_identity.rs b/e2e/rust/tests/podman_oci_identity.rs index 54a5437397..3405d730c9 100644 --- a/e2e/rust/tests/podman_oci_identity.rs +++ b/e2e/rust/tests/podman_oci_identity.rs @@ -252,23 +252,30 @@ async fn assert_isolated_pair(image: &ImageGuard, sandbox: &SandboxGuard, contai let supervisor_id = container_id_for_role(&image.engine, &sandbox.name, "supervisor") .expect("find separate supervisor companion"); assert_ne!(supervisor_id, container_id); - for id in [container_id, &supervisor_id] { - let user = run_engine( - &image.engine, - &["inspect", "--format", "{{.Config.User}}", id], - ) - .unwrap(); - assert_eq!(user, format!("{OCI_UID}:{OCI_GID}")); - let caps = run_engine( - &image.engine, - &["inspect", "--format", "{{.EffectiveCaps}}", id], - ) - .unwrap(); - assert_eq!( - caps, "[]", - "neither container may have effective capabilities" - ); - } + let workload_user = run_engine( + &image.engine, + &["inspect", "--format", "{{.Config.User}}", container_id], + ) + .unwrap(); + assert_eq!( + workload_user, "0:0", + "the trusted rootless boundary starts as container root before dropping to the OCI identity" + ); + let supervisor_user = run_engine( + &image.engine, + &["inspect", "--format", "{{.Config.User}}", &supervisor_id], + ) + .unwrap(); + assert_eq!(supervisor_user, format!("{OCI_UID}:{OCI_GID}")); + let supervisor_caps = run_engine( + &image.engine, + &["inspect", "--format", "{{.EffectiveCaps}}", &supervisor_id], + ) + .unwrap(); + assert_eq!( + supervisor_caps, "[]", + "the supervisor companion may not have effective capabilities" + ); let network = run_engine( &image.engine, &[ diff --git a/e2e/rust/tests/provider_auto_create.rs b/e2e/rust/tests/provider_auto_create.rs index 43fdc27293..2c8828deaf 100644 --- a/e2e/rust/tests/provider_auto_create.rs +++ b/e2e/rust/tests/provider_auto_create.rs @@ -22,7 +22,7 @@ use std::sync::Mutex; use openshell_e2e::harness::binary::openshell_cmd; use openshell_e2e::harness::output::{extract_field, strip_ansi}; -#[cfg(feature = "e2e-docker")] +#[cfg(feature = "e2e-local-container-driver")] use openshell_e2e::harness::sandbox::E2E_WORKLOAD_IMAGE; const TEST_API_KEY: &str = "sk-e2e-auto-provider-test-key"; @@ -111,7 +111,7 @@ network_policies: {} // --auto-providers skips the interactive prompt. let mut cmd = openshell_cmd(); cmd.arg("sandbox").arg("create"); - #[cfg(feature = "e2e-docker")] + #[cfg(feature = "e2e-local-container-driver")] cmd.arg("--from").arg(E2E_WORKLOAD_IMAGE); cmd.arg("--detach") .arg("--policy") diff --git a/e2e/rust/tests/provider_refresh_handles.rs b/e2e/rust/tests/provider_refresh_handles.rs index b853d35950..1de651415d 100644 --- a/e2e/rust/tests/provider_refresh_handles.rs +++ b/e2e/rust/tests/provider_refresh_handles.rs @@ -16,6 +16,7 @@ use std::time::Duration; use openshell_e2e::harness::binary::openshell_cmd; use openshell_e2e::harness::container::HostSupportContainer; +use openshell_e2e::harness::port::find_free_port; use openshell_e2e::harness::sandbox::SandboxGuard; use tempfile::{Builder as TempFileBuilder, NamedTempFile}; @@ -40,6 +41,7 @@ class Handler(BaseHTTPRequestHandler): return generation += 1 current_token = f"access-token-{generation}" + print(f"issued-token generation={generation}", flush=True) body = json.dumps({ "access_token": current_token, "expires_in": 300, @@ -52,9 +54,11 @@ class Handler(BaseHTTPRequestHandler): self.wfile.write(body) def do_GET(self): + authorized = self.headers.get("Authorization") == f"Bearer {current_token}" + print(f"resource-request path={self.path} authorized={authorized}", flush=True) if self.path == "/": self.send_response(204) - elif self.path == "/probe" and self.headers.get("Authorization") == f"Bearer {current_token}": + elif self.path == "/probe" and authorized: self.send_response(204) else: self.send_response(401) @@ -63,7 +67,7 @@ class Handler(BaseHTTPRequestHandler): def log_message(self, *_args): pass -ThreadingHTTPServer(("0.0.0.0", 8000), Handler).serve_forever() +ThreadingHTTPServer(("0.0.0.0", __PORT__), Handler).serve_forever() "#; async fn run_cli(args: &[&str]) -> Result { @@ -139,7 +143,7 @@ endpoints: - 172.0.0.0/8 - 192.168.0.0/16 binaries: - - /usr/local/bin/python3 + - /** " ); file.write_all(profile.as_bytes()) @@ -181,7 +185,7 @@ network_policies: - 172.0.0.0/8 - 192.168.0.0/16 binaries: - - path: /usr/local/bin/python3 + - path: /** " ); file.write_all(policy.as_bytes()) @@ -295,7 +299,10 @@ async fn wait_for_probe_failure(sandbox: &SandboxGuard) -> Result<(), String> { #[tokio::test] async fn long_running_process_survives_rotations_and_reconfigure_revokes() -> Result<(), String> { delete_provider_resources().await; - let fixture = HostSupportContainer::start_python(FIXTURE_SCRIPT, 8000).await?; + let fixture_port = find_free_port(); + let fixture_script = FIXTURE_SCRIPT.replace("__PORT__", &fixture_port.to_string()); + let fixture = + HostSupportContainer::start_python_on_host_network(&fixture_script, fixture_port).await?; let profile = write_profile(fixture.port, fixture.port)?; let policy = write_policy(fixture.port)?; configure_refresh(&profile).await?; @@ -311,7 +318,7 @@ echo {READY_MARKER} while true; do if [ -f /sandbox/probe-trigger ]; then rm -f /sandbox/probe-trigger - if python3 -c 'import os, urllib.request; request = urllib.request.Request("{resource_url}", headers=dict(Authorization="Bearer " + os.environ["REFRESH_E2E_ACCESS_TOKEN"])); urllib.request.urlopen(request, timeout=5).read()'; then + if /usr/bin/python3 -c 'import os, urllib.request; request = urllib.request.Request("{resource_url}", headers=dict(Authorization="Bearer " + os.environ["REFRESH_E2E_ACCESS_TOKEN"])); urllib.request.urlopen(request, timeout=5).read()'; then echo ok > /sandbox/probe-result else echo failed > /sandbox/probe-result @@ -329,7 +336,10 @@ done"# let result = async { if trigger_probe(&sandbox).await? != "ok" { - return Err("initial long-running credential probe failed".to_string()); + return Err(format!( + "initial long-running credential probe failed; fixture logs:\n{}", + fixture.logs().unwrap_or_else(|error| error) + )); } for _ in 0..12 { @@ -341,7 +351,7 @@ done"# wait_for_probe_failure(&sandbox).await?; let fresh_probe = format!( - r#"python3 -c 'import os, urllib.request; request = urllib.request.Request("{resource_url}", headers=dict(Authorization="Bearer " + os.environ["REFRESH_E2E_ACCESS_TOKEN"])); urllib.request.urlopen(request, timeout=5).read()'"# + r#"/usr/bin/python3 -c 'import os, urllib.request; request = urllib.request.Request("{resource_url}", headers=dict(Authorization="Bearer " + os.environ["REFRESH_E2E_ACCESS_TOKEN"])); urllib.request.urlopen(request, timeout=5).read()'"# ); sandbox.exec(&["sh", "-c", &fresh_probe]).await?; Ok(()) diff --git a/e2e/rust/tests/provider_token_exchange.rs b/e2e/rust/tests/provider_token_exchange.rs index 4d2dd703fc..e845938e70 100644 --- a/e2e/rust/tests/provider_token_exchange.rs +++ b/e2e/rust/tests/provider_token_exchange.rs @@ -10,7 +10,6 @@ use std::io::Write as _; use std::net::{Ipv4Addr, SocketAddr}; use std::os::unix::fs::PermissionsExt as _; use std::path::{Path, PathBuf}; -use std::process::Stdio; use std::sync::Arc; use std::task::{Context, Poll}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; @@ -489,7 +488,7 @@ async fn podman_logs_capture(container_name: &str) -> String { } } -async fn provider_token_debug(sandbox_name: &str, token_port: u16, target_port: u16) -> String { +async fn provider_token_debug(sandbox_name: &str, target_port: u16) -> String { let sandbox_logs = sandbox_logs(sandbox_name).await; let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") else { return format!("Sandbox logs:\n{sandbox_logs}\nOPENSHELL_PODMAN_SOCKET is not set"); @@ -510,17 +509,6 @@ async fn provider_token_debug(sandbox_name: &str, token_port: u16, target_port: ], ) .await; - let token_probe = podman_exec_capture( - &container_name, - &[ - "python3", - "-c", - &format!( - "import socket; s=socket.create_connection(('127.0.0.1', {token_port}), 2); s.sendall(b'POST /token HTTP/1.1\\r\\nHost: 127.0.0.1\\r\\nContent-Length: 0\\r\\nConnection: close\\r\\n\\r\\n'); print(s.recv(4096).decode(errors='replace')); s.close()" - ), - ], - ) - .await; let target_probe = podman_exec_capture( &container_name, &[ @@ -541,7 +529,6 @@ async fn provider_token_debug(sandbox_name: &str, token_port: u16, target_port: --- /etc/hosts ---\n{hosts}\n\ --- ps -ef ---\n{processes}\n\ --- resolve host.openshell.internal ---\n{resolve_host}\n\ - --- token endpoint probe ---\n{token_probe}\n\ --- protected target TCP probe ---\n{target_probe}\n\ --- podman logs ---\n{container_logs}" ) @@ -583,7 +570,6 @@ endpoints: - host: host.openshell.internal port: {target_port} protocol: rest - tls: none access: read-write enforcement: enforce allowed_ips: @@ -592,7 +578,7 @@ endpoints: - 172.0.0.0/8 - 192.168.0.0/16 binaries: - - /usr/local/bin/python3 + - /** " ); file.write_all(profile.as_bytes()) @@ -601,8 +587,7 @@ binaries: file } -fn sandbox_script(token_port: u16) -> String { - let _ = token_port; +fn sandbox_script() -> String { r"set -eu echo token-server-ready while true; do sleep 60; done @@ -610,95 +595,6 @@ while true; do sleep 60; done .to_string() } -fn container_token_endpoint_script() -> String { - format!( - r#" -import json -import sys -from http.server import BaseHTTPRequestHandler, HTTPServer -from urllib.parse import parse_qs - -PORT = int(sys.argv[1]) - -class Handler(BaseHTTPRequestHandler): - def do_POST(self): - if self.path != "/token": - self.send_response(404) - self.end_headers() - return - length = int(self.headers.get("content-length", "0")) - params = parse_qs(self.rfile.read(length).decode()) - subject_token = params.get("subject_token", [""])[0] - client_assertion = params.get("client_assertion", [""])[0] - if subject_token == "{USER_SUBJECT_TOKEN}" and client_assertion: - access_token = "{INTERMEDIATE_TOKEN}" - elif subject_token == "{INTERMEDIATE_TOKEN}" and client_assertion: - access_token = "{FINAL_ACCESS_TOKEN}" - else: - self.send_response(400) - body = json.dumps({{"error": "unexpected_token_exchange"}}).encode() - self.send_header("content-type", "application/json") - self.send_header("content-length", str(len(body))) - self.end_headers() - self.wfile.write(body) - return - body = json.dumps({{ - "access_token": access_token, - "token_type": "Bearer", - "expires_in": 300, - }}).encode() - self.send_response(200) - self.send_header("content-type", "application/json") - self.send_header("content-length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - def log_message(self, fmt, *args): - return - -HTTPServer(("127.0.0.1", PORT), Handler).serve_forever() -"# - ) -} - -async fn start_container_token_endpoint(sandbox_name: &str, token_port: u16) -> Result<(), String> { - let socket = std::env::var("OPENSHELL_PODMAN_SOCKET") - .map_err(|_| "OPENSHELL_PODMAN_SOCKET must be set by e2e-podman.sh".to_string())?; - let container_name = podman_container_name_for_sandbox(&socket, sandbox_name).await?; - let mut cmd = Command::new("podman"); - cmd.arg("--url") - .arg(format!("unix://{socket}")) - .arg("exec") - .arg("-d") - .arg(&container_name) - .arg("python3") - .arg("-c") - .arg(container_token_endpoint_script()) - .arg(token_port.to_string()); - apply_podman_config_env(&mut cmd); - let output = cmd - .output() - .await - .map_err(|err| format!("spawn podman exec token endpoint: {err}"))?; - if !output.status.success() { - return Err(format!( - "podman exec token endpoint failed: {}{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - )); - } - - for _ in 0..20 { - if container_loopback_port_ready(&container_name, token_port).await { - return Ok(()); - } - tokio::time::sleep(Duration::from_millis(250)).await; - } - Err(format!( - "container token endpoint did not become ready on 127.0.0.1:{token_port}" - )) -} - async fn podman_container_name_for_sandbox( socket: &str, sandbox_name: &str, @@ -709,6 +605,8 @@ async fn podman_container_name_for_sandbox( .arg("ps") .arg("--filter") .arg(format!("label=openshell.ai/sandbox-name={sandbox_name}")) + .arg("--filter") + .arg("label=openshell.io/isolation-role=sandbox") .arg("--format") .arg("{{.Names}}"); apply_podman_config_env(&mut cmd); @@ -749,29 +647,6 @@ fn apply_podman_config_env(cmd: &mut Command) { } } -async fn container_loopback_port_ready(container_name: &str, token_port: u16) -> bool { - let Ok(socket) = std::env::var("OPENSHELL_PODMAN_SOCKET") else { - return false; - }; - let probe = format!( - "import socket; s=socket.create_connection(('127.0.0.1', {token_port}), 1); s.close()" - ); - let mut cmd = Command::new("podman"); - cmd.arg("--url") - .arg(format!("unix://{socket}")) - .arg("exec") - .arg(container_name) - .arg("python3") - .arg("-c") - .arg(probe); - apply_podman_config_env(&mut cmd); - cmd.stdout(Stdio::null()).stderr(Stdio::null()); - cmd.status() - .await - .map(|status| status.success()) - .unwrap_or(false) -} - async fn sandbox_exec_http(sandbox_name: &str, target_port: u16) -> Result { let url = format!("http://host.openshell.internal:{target_port}/resource"); let script = format!( @@ -859,7 +734,7 @@ async fn podman_provider_token_exchange_injects_bearer_header() { .await .expect("create provider"); - let script = sandbox_script(token_port); + let script = sandbox_script(); let mut sandbox = SandboxGuard::create_keep_with_args( &["--provider", &provider_name], &["sh", "-lc", &script], @@ -871,13 +746,10 @@ async fn podman_provider_token_exchange_injects_bearer_header() { "sandbox should complete token exchange against {token_endpoint} and protected target port {target_port}:\n{err}" ) }); - start_container_token_endpoint(&sandbox.name, token_port) - .await - .expect("start container token endpoint"); let request_output = match sandbox_exec_http(&sandbox.name, target_port).await { Ok(output) => output, Err(err) => { - let debug = provider_token_debug(&sandbox.name, token_port, target_port).await; + let debug = provider_token_debug(&sandbox.name, target_port).await; panic!("request protected target from kept sandbox: {err}\n{debug}"); } }; diff --git a/e2e/rust/tests/sandbox_lifecycle.rs b/e2e/rust/tests/sandbox_lifecycle.rs index de50bc2d34..3ccd2d73c5 100644 --- a/e2e/rust/tests/sandbox_lifecycle.rs +++ b/e2e/rust/tests/sandbox_lifecycle.rs @@ -3,6 +3,8 @@ #![cfg(feature = "e2e")] +#[cfg(target_os = "linux")] +use std::fs; use std::process::Stdio; use std::time::Duration; @@ -211,6 +213,64 @@ async fn reconnect_with_input_ownership( } } +#[cfg(target_os = "linux")] +fn find_process_with_args(expected_args: &[&str]) -> Option { + for entry in fs::read_dir("/proc").ok()?.filter_map(Result::ok) { + let Ok(pid) = entry.file_name().to_string_lossy().parse::() else { + continue; + }; + let cmdline = fs::read(entry.path().join("cmdline")).unwrap_or_default(); + let args = cmdline.split(|byte| *byte == 0).collect::>(); + if expected_args + .iter() + .all(|expected| args.contains(&expected.as_bytes())) + { + return Some(pid); + } + } + None +} + +#[cfg(target_os = "linux")] +fn find_child_process_with_args(parent_pid: u32, expected_args: &[&str]) -> Option { + for entry in fs::read_dir("/proc").ok()?.filter_map(Result::ok) { + let Ok(pid) = entry.file_name().to_string_lossy().parse::() else { + continue; + }; + let status = fs::read_to_string(entry.path().join("status")).unwrap_or_default(); + let process_parent = status.lines().find_map(|line| { + line.strip_prefix("PPid:") + .and_then(|value| value.trim().parse::().ok()) + }); + if process_parent != Some(parent_pid) { + continue; + } + let cmdline = fs::read(entry.path().join("cmdline")).unwrap_or_default(); + let args = cmdline.split(|byte| *byte == 0).collect::>(); + if expected_args + .iter() + .all(|expected| args.contains(&expected.as_bytes())) + { + return Some(pid); + } + } + None +} + +#[cfg(target_os = "linux")] +async fn wait_for_process_with_args(expected_args: &[&str]) -> u32 { + tokio::time::timeout(Duration::from_secs(10), async { + loop { + if let Some(pid) = find_process_with_args(expected_args) { + return pid; + } + sleep(Duration::from_millis(50)).await; + } + }) + .await + .unwrap_or_else(|_| panic!("process with arguments {expected_args:?} did not start")) +} + #[tokio::test] #[serial(sandbox_lifecycle)] async fn sandbox_stop_start_preserves_workspace() { @@ -722,6 +782,237 @@ async fn canonical_main_disconnect_reconnect_replays_history_for_same_process() sandbox.cleanup().await; } +#[cfg(target_os = "linux")] +#[tokio::test] +#[serial(sandbox_lifecycle)] +async fn canonical_main_connect_recovers_its_ssh_transport() { + let script = r#"trap 'kill "$writer" 2>/dev/null || true' EXIT; (n=1; while true; do printf 'transport_pid=%s sequence=%04d\n' "$$" "$n"; n=$((n + 1)); sleep 0.2; done) & writer=$!; while IFS= read -r line; do printf 'transport_pid=%s input=%s\n' "$$" "$line"; done"#; + let mut sandbox = SandboxGuard::create_detached_main(&["sh", "-lc", script]) + .await + .expect("create retained canonical main process"); + + let mut connect_cmd = openshell_cmd(); + connect_cmd + .args(["sandbox", "connect", &sandbox.name]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let mut connect = connect_cmd.spawn().expect("spawn supervised attachment"); + let mut connect_stdin = connect.stdin.take().expect("connect stdin"); + let connect_stdout = connect.stdout.take().expect("connect stdout"); + let mut connect_lines = BufReader::new(connect_stdout).lines(); + let connect_stderr = connect.stderr.take().expect("connect stderr"); + let mut connect_errors = BufReader::new(connect_stderr).lines(); + + let initial_line = tokio::time::timeout(Duration::from_secs(30), connect_lines.next_line()) + .await + .expect("initial attachment output timeout") + .expect("read initial attachment output") + .expect("initial attachment output should remain open"); + assert!( + initial_line.contains("transport_pid="), + "unexpected initial attachment output: {initial_line}" + ); + + // Recovery intentionally starts only for an established attachment, so + // let the initial SSH process live beyond that setup guard before killing + // only its ProxyCommand transport. + sleep(Duration::from_secs(3)).await; + let proxy_pid = wait_for_process_with_args(&["ssh-proxy", "--sandbox", &sandbox.name]).await; + let kill_status = tokio::process::Command::new("kill") + .args(["-TERM", &proxy_pid.to_string()]) + .status() + .await + .expect("terminate SSH proxy transport"); + assert!(kill_status.success(), "terminate SSH proxy transport"); + + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let line = connect_errors + .next_line() + .await + .expect("read supervised attachment diagnostics") + .expect("supervised attachment diagnostics should remain open"); + if normalize_output(&line).contains("Connection to sandbox lost; reconnecting") { + break; + } + } + }) + .await + .expect("supervised attachment did not enter recovery"); + + let input_token = format!("after-transport-recovery-{:x}", rand::random::()); + connect_stdin + .write_all(format!("{input_token}\n").as_bytes()) + .await + .expect("write input after transport recovery"); + connect_stdin + .flush() + .await + .expect("flush input after transport recovery"); + + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let line = connect_lines + .next_line() + .await + .expect("read recovered attachment output") + .expect("recovered attachment output should remain open"); + if line.contains(&format!("input={input_token}")) { + break; + } + } + }) + .await + .expect("replacement SSH session did not reattach to canonical main"); + assert!( + connect + .try_wait() + .expect("inspect supervised attachment") + .is_none(), + "sandbox connect parent should remain alive after recovery" + ); + + connect + .kill() + .await + .expect("disconnect recovered attachment"); + connect + .wait() + .await + .expect("wait for recovered attachment disconnect"); + sandbox.cleanup().await; +} + +#[cfg(target_os = "linux")] +#[tokio::test] +#[serial(sandbox_lifecycle)] +async fn canonical_main_connect_forwards_pid_targeted_termination_and_reaps_ssh() { + use std::os::fd::OwnedFd; + + let mut sandbox = SandboxGuard::create_detached_main(&["sh", "-lc", "exec sleep infinity"]) + .await + .expect("create retained canonical main process"); + let pty = nix::pty::openpty(None, None).expect("open pseudo-terminal"); + let controller: OwnedFd = pty.master; + let follower: OwnedFd = pty.slave; + + let mut connect_cmd = openshell_cmd(); + connect_cmd + .args(["sandbox", "connect", &sandbox.name]) + .stdin( + follower + .try_clone() + .expect("duplicate PTY follower for stdin"), + ) + .stdout( + follower + .try_clone() + .expect("duplicate PTY follower for stdout"), + ) + .stderr( + follower + .try_clone() + .expect("duplicate PTY follower for stderr"), + ); + let mut connect = connect_cmd + .spawn() + .expect("spawn supervised PTY attachment"); + let connect_pid = connect.id().expect("connect process ID"); + drop(follower); + + let ssh_pid = tokio::time::timeout(Duration::from_secs(30), async { + loop { + if let Some(pid) = + find_child_process_with_args(connect_pid, &["-s", "sandbox", "openshell-main"]) + { + return pid; + } + sleep(Duration::from_millis(50)).await; + } + }) + .await + .expect("supervised SSH child did not start"); + + nix::sys::signal::kill( + nix::unistd::Pid::from_raw(i32::try_from(connect_pid).expect("connect PID fits i32")), + nix::sys::signal::Signal::SIGTERM, + ) + .expect("send SIGTERM to only the OpenShell parent"); + + let status = tokio::time::timeout(Duration::from_secs(10), connect.wait()) + .await + .expect("OpenShell parent did not terminate") + .expect("wait for OpenShell parent"); + assert_eq!(status.code(), Some(143)); + tokio::time::timeout(Duration::from_secs(5), async { + while fs::metadata(format!("/proc/{ssh_pid}")).is_ok() { + sleep(Duration::from_millis(25)).await; + } + }) + .await + .expect("SSH child was not reaped after parent termination"); + + drop(controller); + sandbox.cleanup().await; +} + +#[tokio::test] +#[serial(sandbox_lifecycle)] +async fn canonical_main_exit_255_is_not_retried_as_transport_failure() { + const READY_MARKER: &str = "exit-255-ready"; + const RELEASE_PATH: &str = "/sandbox/.openshell-exit-255-release"; + let script = format!( + "echo {READY_MARKER}; while [ ! -e '{RELEASE_PATH}' ]; do sleep 0.05; done; exit 255" + ); + let mut sandbox = SandboxGuard::create_detached_main(&["sh", "-c", &script]) + .await + .expect("create retained canonical main process"); + + let mut connect_cmd = openshell_cmd(); + connect_cmd + .args(["sandbox", "connect", &sandbox.name]) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let mut connect = connect_cmd.spawn().expect("spawn supervised attachment"); + let connect_stdout = connect.stdout.take().expect("connect stdout"); + let mut connect_lines = BufReader::new(connect_stdout).lines(); + + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let line = connect_lines + .next_line() + .await + .expect("read attachment output") + .expect("attachment output should remain open"); + if line.contains(READY_MARKER) { + break; + } + } + }) + .await + .expect("attachment did not observe the canonical main process"); + + // Keep the attachment alive beyond the setup guard used to distinguish + // initial SSH failures from established transport failures. + sleep(Duration::from_secs(3)).await; + sandbox + .exec(&["touch", RELEASE_PATH]) + .await + .expect("release canonical main process"); + + let status = tokio::time::timeout(Duration::from_secs(10), connect.wait()).await; + if status.is_err() { + connect.kill().await.expect("stop stuck attachment"); + } + sandbox.cleanup().await; + let status = status + .expect("exit status 255 must not enter the transport recovery loop") + .expect("wait for canonical main attachment"); + assert_eq!(status.code(), Some(255)); +} + #[tokio::test] #[serial(sandbox_lifecycle)] async fn sandbox_create_with_no_keep_cleans_up_after_tty_command() { diff --git a/e2e/support/podman-gateway-config.sh b/e2e/support/podman-gateway-config.sh index 9d6ea929e1..bd5a39b1f7 100755 --- a/e2e/support/podman-gateway-config.sh +++ b/e2e/support/podman-gateway-config.sh @@ -68,10 +68,11 @@ e2e_write_podman_gateway_config() { local sandbox_image=${11} local stop_timeout_secs=${12} local supervisor_image=${13} - local provider_spiffe_socket=${14} - local podman_socket=${15} - local oidc_mode=${16} - local oidc_issuer=${17} + local sandbox_runtime_image=${14} + local provider_spiffe_socket=${15} + local podman_socket=${16} + local oidc_mode=${17} + local oidc_issuer=${18} local configured_with_tls option_profile case "${OPENSHELL_E2E_PODMAN_OPTION_PROFILE:-}" in @@ -109,6 +110,7 @@ e2e_write_podman_gateway_config() { fi printf 'stop_timeout_secs = %s\n' "${stop_timeout_secs}" printf 'supervisor_image = %s\n' "$(e2e_podman_toml_string "${supervisor_image}")" + printf 'sandbox_runtime_image = %s\n' "$(e2e_podman_toml_string "${sandbox_runtime_image}")" printf 'guest_tls_ca = %s\n' "$(e2e_podman_toml_string "${pki_dir}/ca.crt")" printf 'guest_tls_cert = %s\n' "$(e2e_podman_toml_string "${pki_dir}/client/tls.crt")" printf 'guest_tls_key = %s\n' "$(e2e_podman_toml_string "${pki_dir}/client/tls.key")" @@ -162,6 +164,7 @@ e2e_write_podman_gateway_config() { fi printf 'stop_timeout_secs = %s\n' "${stop_timeout_secs}" printf 'supervisor_image = %s\n' "$(e2e_podman_toml_string "${supervisor_image}")" + printf 'sandbox_runtime_image = %s\n' "$(e2e_podman_toml_string "${sandbox_runtime_image}")" printf 'enable_bind_mounts = true\n' if [ -n "${provider_spiffe_socket}" ]; then printf 'provider_spiffe_workload_api_socket = %s\n' "$(e2e_podman_toml_string "${provider_spiffe_socket}")" diff --git a/e2e/with-podman-gateway.sh b/e2e/with-podman-gateway.sh index 6e8ec031ad..3b3c5cf46c 100755 --- a/e2e/with-podman-gateway.sh +++ b/e2e/with-podman-gateway.sh @@ -118,7 +118,9 @@ if [ "${OPENSHELL_E2E_SPIFFE_FIXTURE:-0}" = "1" ]; then if [ -z "${OPENSHELL_E2E_PROVIDER_SPIFFE_SOCKET:-}" ]; then OPENSHELL_E2E_PROVIDER_SPIFFE_PORT="$(e2e_pick_port)" export OPENSHELL_E2E_PROVIDER_SPIFFE_LISTEN="0.0.0.0:${OPENSHELL_E2E_PROVIDER_SPIFFE_PORT}" - export OPENSHELL_E2E_PROVIDER_SPIFFE_SOCKET="tcp:169.254.1.2:${OPENSHELL_E2E_PROVIDER_SPIFFE_PORT}" + # Podman supervisors run with host networking, so reach the host-side + # Workload API fixture over loopback rather than the workload bridge. + export OPENSHELL_E2E_PROVIDER_SPIFFE_SOCKET="tcp:127.0.0.1:${OPENSHELL_E2E_PROVIDER_SPIFFE_PORT}" fi fi GATEWAY_BIN="" @@ -208,7 +210,7 @@ cleanup() { podman_cmd volume rm "openshell-channel-${sandbox_id}" >/dev/null 2>&1 || true podman_cmd volume rm -f "openshell-sandbox-${sandbox_id}-workspace" >/dev/null 2>&1 || true local secret_prefix - for secret_prefix in openshell-token openshell-proxy-auth openshell-tls-ca openshell-tls-cert openshell-tls-key; do + for secret_prefix in openshell-token openshell-proxy-auth openshell-resolver openshell-tls-ca openshell-tls-cert openshell-tls-key; do podman_cmd secret rm "${secret_prefix}-${sandbox_id}" >/dev/null 2>&1 || true done fi @@ -729,6 +731,7 @@ e2e_write_podman_gateway_config \ "${SANDBOX_RUNTIME_IMAGE}" \ "${PODMAN_STOP_TIMEOUT_SECS}" \ "${SUPERVISOR_RUNTIME_IMAGE}" \ + "${SANDBOX_BOUNDARY_IMAGE}" \ "${OPENSHELL_E2E_PROVIDER_SPIFFE_SOCKET:-}" \ "${OPENSHELL_PODMAN_SOCKET:-}" \ "${OIDC_MODE}" \ diff --git a/install.sh b/install.sh index 00f8141135..8b8eee90c0 100755 --- a/install.sh +++ b/install.sh @@ -60,14 +60,20 @@ ENVIRONMENT VARIABLES: Prereleases require an authenticated GitHub CLI session. OPENSHELL_ACK_BREAKING_UPGRADE Set to 1 only after backing up and cleaning up a - pre-v0.0.37 installation. + pre-v0.0.37 or non-snap installation. NOTES: When OPENSHELL_VERSION is unset, this resolves the latest tagged release from ${GITHUB_URL}/releases/latest. - Linux installs the Debian package on amd64/arm64 or the RPM packages on - x86_64/aarch64, depending on the host package manager. + On Linux, the installer uses the OpenShell snap whenever the snap command + is available. It installs from latest/edge when OPENSHELL_VERSION=dev and + from latest/stable otherwise. The OpenShell snap requires a running Docker + Engine installed from a system package or Docker's package repository. The + Docker snap is not currently compatible with OpenShell. + + Without snap, Linux installs the Debian package on amd64/arm64 or the RPM + packages on x86_64/aarch64, depending on the host package manager. macOS installs the release Homebrew formula on Apple Silicon and starts a brew services-backed local gateway. EOF @@ -244,12 +250,17 @@ installed_version_needs_breaking_upgrade_notice() { ! semver_at_least "$_version" "$BREAKING_RELEASE_VERSION" } -find_existing_openshell_bin() { +find_existing_native_openshell_bin() { _path="$(command -v openshell 2>/dev/null || true)" - if [ -n "$_path" ] && [ -x "$_path" ]; then - printf '%s\n' "$_path" - return 0 - fi + case "$_path" in + /snap/*) ;; + *) + if [ -n "$_path" ] && [ -x "$_path" ]; then + printf '%s\n' "$_path" + return 0 + fi + ;; + esac for _candidate in \ "${TARGET_HOME:-}/.local/bin/openshell" \ @@ -293,7 +304,7 @@ print_breaking_upgrade_notice() { cat >&2 <&2 <&2 || true + info "OpenShell snap connections:" + as_root snap connections openshell >&2 || true + if has_cmd journalctl; then + info "last ${_lines} lines from the OpenShell snap gateway journal:" + as_root journalctl -b -u snap.openshell.gateway.service --no-pager -n "$_lines" >&2 || true + fi + as_root snap logs openshell.gateway -n="$_lines" >&2 || true +} + dump_homebrew_gateway_diagnostics() { _lines="$1" _brew_prefix="$(as_target_user brew --prefix 2>/dev/null || true)" @@ -985,12 +1065,12 @@ wait_for_local_gateway_status() { error "openshell status did not report connected within ${_timeout}s" } -remove_local_gateway_registration() { +remove_local_gateway_registration_from() { + _config_dir="$1" [ -n "$TARGET_HOME" ] || error "cannot resolve home directory for ${TARGET_USER}" - _config_dir="${TARGET_HOME}/.config/openshell" - # The install-dev gateway is a user service. Replace the CLI registration - # directly instead of asking `gateway destroy` to tear down Docker resources. + # Replace the CLI registration directly instead of asking `gateway destroy` + # to tear down package-managed resources. # shellcheck disable=SC2016 as_target_user sh -c ' config_dir=$1 @@ -1009,6 +1089,15 @@ remove_local_gateway_registration() { ' sh "$_config_dir" } +remove_local_gateway_registration() { + remove_local_gateway_registration_from "${TARGET_HOME}/.config/openshell" +} + +remove_snap_gateway_registration() { + remove_local_gateway_registration_from \ + "${TARGET_HOME}/snap/openshell/common/.config/openshell" +} + register_local_gateway() { _register_bin="${OPENSHELL_REGISTER_BIN:-openshell}" _endpoint="$(local_gateway_endpoint)" @@ -1136,6 +1225,158 @@ install_linux_rpm() { start_user_gateway } +openshell_snap_channel() { + if [ "${OPENSHELL_VERSION:-}" = "dev" ]; then + printf '%s\n' "latest/edge" + else + if [ -n "${OPENSHELL_VERSION:-}" ]; then + warn "OPENSHELL_VERSION=${OPENSHELL_VERSION} is ignored for Snap installs; using latest/stable" + fi + printf '%s\n' "latest/stable" + fi +} + +ensure_snap_gateway_config() { + _config_file="${1:-/var/snap/openshell/common/gateway.toml}" + + as_root sh -c ' + set -eu + config_file=$1 + if [ -e "$config_file" ] || [ -L "$config_file" ]; then + exit 0 + fi + + config_dir=${config_file%/*} + mkdir -p "$config_dir" + umask 077 + temporary_file=$(mktemp "${config_file}.tmp.XXXXXX") + trap '\''rm -f "$temporary_file"'\'' 0 HUP INT TERM + + cat >"$temporary_file" <<'\''EOF'\'' +[openshell] +version = 2 + +[openshell.gateway] + +[openshell.gateway.auth] +allow_unauthenticated_users = true +EOF + + if ! ln "$temporary_file" "$config_file"; then + if [ -e "$config_file" ] || [ -L "$config_file" ]; then + exit 0 + fi + exit 1 + fi + rm -f "$temporary_file" + trap - 0 HUP INT TERM + ' sh "$_config_file" +} + +wait_for_docker_daemon() { + _timeout="${OPENSHELL_INSTALL_DOCKER_TIMEOUT:-30}" + _elapsed=0 + _last_output="" + + info "waiting for Docker daemon to become reachable..." + while [ "$_elapsed" -lt "$_timeout" ]; do + if _last_output="$(as_root docker info 2>&1)"; then + info "Docker daemon is reachable" + return 0 + fi + sleep 1 + _elapsed=$((_elapsed + 1)) + done + + [ -z "$_last_output" ] || printf '%s\n' "$_last_output" >&2 + if snap list docker >/dev/null 2>&1; then + as_root snap services docker >&2 || true + as_root snap changes >&2 || true + fi + error "Docker daemon did not become reachable within ${_timeout}s" +} + +register_snap_gateway() { + _register_bin="${OPENSHELL_REGISTER_BIN:-/snap/bin/openshell}" + _endpoint="http://127.0.0.1:${LOCAL_GATEWAY_PORT}" + + if _add_output="$(as_target_user "$_register_bin" gateway add "$_endpoint" --local --name openshell 2>&1)"; then + [ -z "$_add_output" ] || print_gateway_add_output "$_add_output" + return 0 + else + _add_status=$? + fi + + case "$_add_output" in + *"already exists"*) + info "local gateway already exists; removing and re-adding it..." + remove_snap_gateway_registration + as_target_user "$_register_bin" gateway add "$_endpoint" --local --name openshell + ;; + *) + printf '%s\n' "$_add_output" >&2 + return "$_add_status" + ;; + esac +} + +wait_for_snap_gateway_listener() { + _timeout="${OPENSHELL_INSTALL_GATEWAY_TIMEOUT:-30}" + _elapsed=0 + _last_output="" + _probe_url="http://127.0.0.1:${LOCAL_GATEWAY_PORT}/" + + info "waiting for local gateway listener to become reachable..." + while [ "$_elapsed" -lt "$_timeout" ]; do + if _last_output="$(curl -sS --max-time 2 -o /dev/null "$_probe_url" 2>&1)"; then + info "local gateway listener is reachable" + return 0 + fi + sleep 1 + _elapsed=$((_elapsed + 1)) + done + + [ -z "$_last_output" ] || printf '%s\n' "$_last_output" >&2 + dump_local_gateway_diagnostics + error "local gateway listener did not become reachable at ${_probe_url} within ${_timeout}s" +} + +install_linux_snap() { + require_cmd snap + set_linux_target_runtime_dir + + if snap list docker >/dev/null 2>&1; then + error "the Docker snap is not currently compatible with OpenShell because its AppArmor confinement prevents OpenShell's hardened containers from starting. +Remove the Docker snap and install Docker Engine from a system package or Docker's package repository, then rerun this installer." + fi + if ! has_cmd docker; then + error "Docker is required before installing the OpenShell snap. +Install Docker Engine from a system package or Docker's package repository, then rerun this installer. The Docker snap is not currently compatible with OpenShell." + fi + info "using existing Docker installation" + wait_for_docker_daemon + + _channel="$(openshell_snap_channel)" + if snap list openshell >/dev/null 2>&1; then + info "refreshing OpenShell snap from ${_channel}..." + as_root snap refresh openshell --channel="$_channel" + warn "restarting the OpenShell gateway to use the refreshed snap; active sandbox sessions will be interrupted" + else + info "installing OpenShell snap from ${_channel}..." + as_root snap install openshell --channel="$_channel" + fi + + ensure_snap_gateway_config + as_root snap restart openshell.gateway + + info "installed OpenShell snap from ${_channel}" + info "registering local gateway as ${TARGET_USER}..." + register_snap_gateway + wait_for_snap_gateway_listener + OPENSHELL_REGISTER_BIN="/snap/bin/openshell" + wait_for_local_gateway_status +} + install_macos_homebrew() { check_macos_platform @@ -1206,23 +1447,34 @@ main() { require_cmd curl PLATFORM="$(detect_platform)" - RELEASE_TAG="$(resolve_release_tag)" + if [ "$PLATFORM" = "linux" ]; then + LINUX_INSTALL_METHOD="$(linux_package_method)" + fi TARGET_USER="$(target_user)" TARGET_UID="$(id -u "$TARGET_USER" 2>/dev/null || true)" [ -n "$TARGET_UID" ] || error "cannot resolve uid for ${TARGET_USER}" TARGET_HOME="$(user_home "$TARGET_USER")" - guard_breaking_upgrade + if [ "${LINUX_INSTALL_METHOD:-}" = "snap" ]; then + guard_native_to_snap_transition + else + RELEASE_TAG="$(resolve_release_tag)" + guard_breaking_upgrade + fi case "$PLATFORM" in linux) - require_linux_package_glibc - case "$(linux_package_method)" in + case "$LINUX_INSTALL_METHOD" in + snap) + install_linux_snap + ;; deb) + require_linux_package_glibc install_linux_deb ;; rpm) + require_linux_package_glibc install_linux_rpm ;; *) diff --git a/nix/test-guest/README.md b/nix/test-guest/README.md index fa2ca96052..9a9378abee 100644 --- a/nix/test-guest/README.md +++ b/nix/test-guest/README.md @@ -78,10 +78,10 @@ The root [`flake.nix`](../../flake.nix) exposes this directory as the `test-gues | Fedora 44 | No | Yes | Yes | Yes | `.rpm` | | Rocky Linux 9 | Yes | Yes | No | Yes | `.rpm` | -The `snapd` configuration is available for Ubuntu and prepares snapd for -local Snap lifecycle experiments. It does not install Docker, because the Snap -gateway reproduction uses the Docker **Snap** and its `docker:docker-daemon` -interface rather than the host-package Docker configuration. +The `snapd` configuration is available for Ubuntu and prepares snapd for Snap +Store installation experiments. Combine it with `docker` to reproduce the +system-Docker canary, or use it alone to verify that `install.sh` fails safely +when Docker is absent. `podman-rootless` configures the explicit rootless Podman guest setup used by OpenShell tests. It supports Fedora and Ubuntu 26.04 or later. Ubuntu adds the @@ -296,27 +296,42 @@ nix run .#test-guest -- \ -- openshell --version ``` -## Reproduce Snap gateway startup +## Reproduce Snap installation -The gateway Snap must be native to the guest architecture. Copy an existing -Snap artifact and the reproduction script into a prepared Ubuntu guest, then -run the script as root. It follows the Release Canary ordering exactly: install -the Snap, connect Docker/log/system interfaces, and immediately query the -gateway. On each failure it prints snapd and gateway journals. +Copy the repository installer and reproduction script into a prepared Ubuntu +guest. The script follows both Release Canary Snap flows: it runs `install.sh` +with `OPENSHELL_VERSION=dev`, checks the `latest/edge` channel and Docker +interface, and exercises a sandbox. Repeated attempts also cover the installer's +idempotent Snap refresh path. On each failure the script prints snapd, Docker, +and gateway diagnostics. + +Reuse system Docker and verify that the Docker snap is not installed: + +```shell +nix run .#test-guest -- \ + --distro ubuntu-24-04 \ + --with docker \ + --with snapd \ + --keep \ + --copy ./install.sh:/tmp/install.sh \ + --copy ./nix/test-guest/scripts/snap-gateway-repro.sh:/usr/local/bin/snap-gateway-repro \ + -- /usr/local/bin/snap-gateway-repro /tmp/install.sh system-docker 10 +``` + +Start without Docker and verify that `install.sh` rejects the Snap installation: ```shell nix run .#test-guest -- \ --distro ubuntu-24-04 \ --with snapd \ --keep \ - --copy ./openshell_*.snap:/tmp/openshell.snap \ + --copy ./install.sh:/tmp/install.sh \ --copy ./nix/test-guest/scripts/snap-gateway-repro.sh:/usr/local/bin/snap-gateway-repro \ - -- sudo /usr/local/bin/snap-gateway-repro /tmp/openshell.snap 10 30 + -- /usr/local/bin/snap-gateway-repro /tmp/install.sh missing-docker 10 ``` `--keep` retains the overlay and serial log when diagnosing a failure. The -runner prints their location after shutdown. The final `30` accepts automatic -recovery for up to 30 seconds; omit it to require the canary's immediate check. +runner prints their location after shutdown. The destination must be an absolute guest path. Use bare octal permission bits diff --git a/nix/test-guest/scripts/snap-gateway-repro.sh b/nix/test-guest/scripts/snap-gateway-repro.sh index f3090aece2..d92800b523 100755 --- a/nix/test-guest/scripts/snap-gateway-repro.sh +++ b/nix/test-guest/scripts/snap-gateway-repro.sh @@ -2,106 +2,148 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Reproduce the Release Canary Snap lifecycle: install the OpenShell Snap, -# connect its interfaces after the daemon is started, then immediately use the -# local gateway. Run this as root inside an Ubuntu guest prepared with --with snapd. +# Reproduce the Release Canary Snap lifecycle with install.sh. Run this as the +# test user inside an Ubuntu guest prepared with --with snapd and, for the +# system-docker mode, --with docker. set -uo pipefail usage() { cat <<'EOF' -Usage: snap-gateway-repro.sh SNAP_FILE [ATTEMPTS] [READY_TIMEOUT_SECONDS] +Usage: snap-gateway-repro.sh INSTALL_SCRIPT MODE [ATTEMPTS] -Install SNAP_FILE repeatedly using the Release Canary interface ordering. -ATTEMPTS defaults to 1. READY_TIMEOUT_SECONDS defaults to 0, preserving the -canary's immediate readiness check. Set it to a positive value to wait for -automatic gateway recovery after the immediate check fails. Every failed -attempt prints service, connection, snap-change, journal, gateway-log, and -listener diagnostics. +MODE is system-docker, missing-docker, or docker-snap. The system-docker mode requires an +existing non-Snap Docker daemon, verifies that install.sh does not install the +Docker snap, and exercises a sandbox. The missing-docker mode requires Docker +to be absent and verifies that install.sh fails before installing either snap. +The docker-snap mode requires the Docker snap and verifies that install.sh +rejects it before installing OpenShell. ATTEMPTS defaults to 1; repeated +system-docker attempts exercise idempotent OpenShell Snap refreshes. Every +failed attempt prints service, connection, snap-change, journal, gateway-log, +and listener diagnostics. EOF } -if [ "$#" -lt 1 ] || [ "$#" -gt 3 ]; then +if [ "$#" -lt 2 ] || [ "$#" -gt 3 ]; then usage >&2 exit 2 fi -snap_file=$1 -attempts=${2:-1} -ready_timeout=${3:-0} -if [ ! -f "${snap_file}" ]; then - echo "Snap file does not exist: ${snap_file}" >&2 +install_script=$1 +mode=$2 +attempts=${3:-1} +if [ ! -f "${install_script}" ]; then + echo "Install script does not exist: ${install_script}" >&2 exit 2 fi if [[ ! ${attempts} =~ ^[1-9][0-9]*$ ]]; then echo "ATTEMPTS must be a positive integer: ${attempts}" >&2 exit 2 fi -if [[ ! ${ready_timeout} =~ ^[0-9]+$ ]]; then - echo "READY_TIMEOUT_SECONDS must be a non-negative integer: ${ready_timeout}" >&2 +case "${mode}" in +system-docker) + if ! command -v docker >/dev/null 2>&1 || ! docker info >/dev/null 2>&1; then + echo "system-docker mode requires a running preinstalled Docker daemon" >&2 + exit 2 + fi + if sudo snap list docker >/dev/null 2>&1; then + echo "system-docker mode does not support the Docker snap" >&2 + exit 2 + fi + ;; +missing-docker) + if command -v docker >/dev/null 2>&1; then + echo "missing-docker mode requires Docker to be absent" >&2 + exit 2 + fi + ;; +docker-snap) + if ! sudo snap list docker >/dev/null 2>&1; then + echo "docker-snap mode requires the Docker snap" >&2 + exit 2 + fi + ;; +*) + echo "MODE must be system-docker, missing-docker, or docker-snap: ${mode}" >&2 exit 2 -fi + ;; +esac + +docker_is_ready() { + docker info >/dev/null +} diagnostics() { local attempt=$1 echo "========== Snap diagnostics (attempt ${attempt}) ==========" >&2 - snap services openshell >&2 || true - snap connections openshell >&2 || true - snap changes >&2 || true - systemctl status snap.openshell.gateway.service --no-pager >&2 || true - journalctl -b -u snap.openshell.gateway.service --no-pager -n 300 >&2 || true - journalctl -b -u snapd.service --no-pager -n 300 >&2 || true - snap logs openshell.gateway -n=300 >&2 || true - ss -ltnp '( sport = :17670 )' >&2 || true -} - -gateway_is_ready() { - runuser -u openshell -- /snap/bin/openshell status >/dev/null 2>&1 -} - -wait_for_gateway() { - local deadline=$((SECONDS + ready_timeout)) - while [ "${SECONDS}" -lt "${deadline}" ]; do - if gateway_is_ready; then - return 0 - fi - sleep 1 - done - gateway_is_ready + if sudo snap list docker >/dev/null 2>&1; then + sudo snap services docker >&2 || true + sudo systemctl status snap.docker.dockerd.service --no-pager >&2 || true + sudo journalctl -b -u snap.docker.dockerd.service --no-pager -n 300 >&2 || true + else + sudo systemctl status docker.service --no-pager >&2 || true + sudo journalctl -b -u docker.service --no-pager -n 300 >&2 || true + fi + sudo snap services openshell >&2 || true + sudo snap connections openshell >&2 || true + sudo snap changes >&2 || true + sudo systemctl status snap.openshell.gateway.service --no-pager >&2 || true + sudo journalctl -b -u snap.openshell.gateway.service --no-pager -n 300 >&2 || true + sudo journalctl -b -u snapd.service --no-pager -n 300 >&2 || true + sudo snap logs openshell.gateway -n=300 >&2 || true + sudo ss -ltnp '( sport = :17670 )' >&2 || true } -if ! snap list docker >/dev/null 2>&1; then - echo "==> Installing Docker Snap" - snap install docker -fi - failures=0 for attempt in $(seq 1 "${attempts}"); do - echo "==> Snap gateway reproduction attempt ${attempt}/${attempts}" - snap remove --purge openshell >/dev/null 2>&1 || true - rm -rf /home/openshell/snap/openshell + echo "==> install.sh Snap ${mode} reproduction attempt ${attempt}/${attempts}" + if [ "${mode}" != system-docker ]; then + output=$(mktemp) + if OPENSHELL_VERSION=dev sh "${install_script}" >"${output}" 2>&1; then + echo "install.sh unexpectedly succeeded in ${mode} mode" >&2 + cat "${output}" >&2 + rm -f "${output}" + diagnostics "${attempt}" + failures=$((failures + 1)) + continue + fi + case "${mode}" in + missing-docker) expected="Docker is required before installing the OpenShell snap" ;; + docker-snap) expected="the Docker snap is not currently compatible with OpenShell" ;; + esac + if ! grep -Fq "${expected}" "${output}" || + sudo snap list openshell >/dev/null 2>&1 || + { [ "${mode}" = missing-docker ] && sudo snap list docker >/dev/null 2>&1; }; then + echo "install.sh did not fail safely in ${mode} mode" >&2 + cat "${output}" >&2 + rm -f "${output}" + diagnostics "${attempt}" + failures=$((failures + 1)) + continue + fi + cat "${output}" + rm -f "${output}" + continue + fi - if ! snap install "${snap_file}" --dangerous || - ! snap connect openshell:docker docker:docker-daemon || - ! snap connect openshell:log-observe || - ! snap connect openshell:system-observe; then - echo "OpenShell installation or interface connection failed" >&2 + sandbox="snap-${attempt}-$$" + if ! OPENSHELL_VERSION=dev sh "${install_script}" || + ! sudo snap list openshell >/dev/null || + ! snap info openshell | grep -Eq '^tracking: +latest/edge$' || + ! docker_is_ready || + ! sudo snap connections openshell | grep -Eq '^docker +openshell:docker +:docker +' || + ! /snap/bin/openshell status || + ! /snap/bin/openshell sandbox create --name "${sandbox}" --detach || + ! /snap/bin/openshell sandbox exec --name "${sandbox}" --no-tty -- true || + ! /snap/bin/openshell sandbox delete "${sandbox}"; then + echo "install.sh Snap reproduction failed" >&2 diagnostics "${attempt}" failures=$((failures + 1)) continue fi - # This deliberately does not wait for the listener. It mirrors the canary - # and exposes a daemon that fails or races after late interface connections. - if ! runuser -u openshell -- /snap/bin/openshell gateway add \ - http://127.0.0.1:17670 --local --name snap-docker || - ! runuser -u openshell -- /snap/bin/openshell gateway select snap-docker || - ! gateway_is_ready; then - if [ "${ready_timeout}" -gt 0 ] && wait_for_gateway; then - echo "Gateway recovered automatically within ${ready_timeout}s" - continue - fi - echo "Gateway was not usable immediately after interface connection" >&2 + if sudo snap list docker >/dev/null 2>&1; then + echo "install.sh unexpectedly installed the Docker snap" >&2 diagnostics "${attempt}" failures=$((failures + 1)) fi diff --git a/proto/openshell.proto b/proto/openshell.proto index f90e5335b5..a836e169bf 100644 --- a/proto/openshell.proto +++ b/proto/openshell.proto @@ -859,7 +859,7 @@ message RefreshSandboxTokenResponse { repeated ExtensionServiceCredential extension_credentials = 3; // Fresh Sandbox Protocol bearer token from the same atomic refresh. string sandbox_token = 4 [(openshell.options.v1.secret) = true]; - // Absolute Sandbox Protocol token expiry. Required when sandbox_token is set. + // Absolute Sandbox Protocol token expiry. Absence means the token is non-expiring. google.protobuf.Timestamp sandbox_expiration_time = 105; // Launch generation to which both refreshed credentials are bound. string session_id = 6; diff --git a/sdk/go/proto/openshellv1/openshell.pb.go b/sdk/go/proto/openshellv1/openshell.pb.go index 66c1bb0c11..18d0b9bcde 100644 --- a/sdk/go/proto/openshellv1/openshell.pb.go +++ b/sdk/go/proto/openshellv1/openshell.pb.go @@ -1303,7 +1303,7 @@ type RefreshSandboxTokenResponse struct { ExtensionCredentials []*ExtensionServiceCredential `protobuf:"bytes,3,rep,name=extension_credentials,json=extensionCredentials,proto3" json:"extension_credentials,omitempty"` // Fresh Sandbox Protocol bearer token from the same atomic refresh. SandboxToken string `protobuf:"bytes,4,opt,name=sandbox_token,json=sandboxToken,proto3" json:"sandbox_token,omitempty"` - // Absolute Sandbox Protocol token expiry. Required when sandbox_token is set. + // Absolute Sandbox Protocol token expiry. Absence means the token is non-expiring. SandboxExpirationTime *timestamppb.Timestamp `protobuf:"bytes,105,opt,name=sandbox_expiration_time,json=sandboxExpirationTime,proto3" json:"sandbox_expiration_time,omitempty"` // Launch generation to which both refreshed credentials are bound. SessionId string `protobuf:"bytes,6,opt,name=session_id,json=sessionId,proto3" json:"session_id,omitempty"` diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index d2b2076a9c..406e7209d2 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -335,6 +335,12 @@ Common findings: - On Linux, verify that the host-networked Podman supervisor can reach the gateway's primary loopback endpoint. On macOS, verify Podman Machine's host-loopback forwarding or configure an explicit `grpc_endpoint`. +- If `host.openshell.internal` does not resolve inside a workload, verify its + `/etc/resolv.conf` contains `nameserver 127.0.0.53`. The Podman driver mounts + that file from a per-sandbox secret and supplies the alias destination to the + supervisor. Check `host_gateway_ip` only when the platform default + (`127.0.0.1` on native Linux or `192.168.127.254` on macOS Podman Machine) + does not reach the gateway host. When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`): diff --git a/skills/openshell-cli/SKILL.md b/skills/openshell-cli/SKILL.md index eea2867a23..5ae5703810 100644 --- a/skills/openshell-cli/SKILL.md +++ b/skills/openshell-cli/SKILL.md @@ -352,11 +352,16 @@ openshell sandbox connect my-sandbox --editor vscode Attaches to the sandbox's existing canonical main process. Disconnecting leaves that process running; reconnecting targets the same process instance and replays -recent output. Use `sandbox exec --tty -- /bin/bash -l` for a new shell. Press -`Ctrl-P`, then `Ctrl-Q` to disconnect without terminating main. When you own -stdin, `Ctrl-C` interrupts the foreground process. In a read-only attachment, -`Ctrl-C` exits the viewer and leaves main and other attachments running. -Configure VS Code Remote-SSH with: +recent output. If an established SSH transport is interrupted, such as when a +laptop sleeps and wakes, the CLI retries transient failures for up to 60 seconds +and reattaches to that same process. Use `sandbox exec --tty -- /bin/bash -l` +for a new shell. Press `Ctrl-P`, then `Ctrl-Q` to disconnect without terminating +main. OpenSSH's `~.` escape looks like transport loss and therefore starts +automatic recovery; after it reattaches, use `Ctrl-P`, then `Ctrl-Q` to exit, or +press `Ctrl-C` between retry attempts to cancel recovery. When you own stdin, +`Ctrl-C` interrupts the foreground process. In a read-only attachment, `Ctrl-C` +exits the viewer and leaves main and other attachments running. Configure VS +Code Remote-SSH with: ```bash openshell sandbox ssh-config my-sandbox >> ~/.ssh/config diff --git a/snap/hooks/connect-plug-docker b/snap/hooks/connect-plug-docker index cfd334b74c..6e0e00caa4 100755 --- a/snap/hooks/connect-plug-docker +++ b/snap/hooks/connect-plug-docker @@ -4,8 +4,9 @@ # The gateway daemon can start when this plug is still disconnected. Restart it # after Docker access becomes available so driver auto-detection runs with the -# socket exposed by docker:docker-daemon. This hook does not make normal gateway -# startup conditional on Docker; it runs only after an operator connects Docker. +# socket exposed through the system :docker slot. This hook does not make normal +# gateway startup conditional on Docker; it runs after an automatic or manual +# Docker connection. set -eu diff --git a/snap/hooks/install b/snap/hooks/install new file mode 100755 index 0000000000..c5fb3baf87 --- /dev/null +++ b/snap/hooks/install @@ -0,0 +1,36 @@ +#!/bin/sh +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -eu + +config_file="${SNAP_COMMON}/gateway.toml" +if [ -e "$config_file" ] || [ -L "$config_file" ]; then + exit 0 +fi + +mkdir -p "$SNAP_COMMON" +umask 077 +temporary_file=$(mktemp "${config_file}.tmp.XXXXXX") +trap 'rm -f "$temporary_file"' 0 HUP INT TERM + +cat >"$temporary_file" <<'EOF' +[openshell] +version = 2 + +[openshell.gateway] + +[openshell.gateway.auth] +allow_unauthenticated_users = true +EOF + +# A hard link publishes the config atomically without replacing a path created +# concurrently. SNAP_COMMON and the temporary file are on the same filesystem. +if ! ln "$temporary_file" "$config_file"; then + if [ -e "$config_file" ] || [ -L "$config_file" ]; then + exit 0 + fi + exit 1 +fi +rm -f "$temporary_file" +trap - 0 HUP INT TERM diff --git a/snapcraft.yaml b/snapcraft.yaml index 0e2fc821cb..4c602620cf 100644 --- a/snapcraft.yaml +++ b/snapcraft.yaml @@ -16,19 +16,20 @@ description: | **Setup instructions** - 1. Install the Docker snap: + 1. Install and start Docker using your preferred package source before + installing the OpenShell snap. Use a system package or Docker's package + repository. The Docker snap is not currently compatible with OpenShell. - sudo snap install docker + Snap Store installations automatically connect the required interfaces. - Support for system-installed Docker is coming in snapd 2.76. + If the OpenShell snap was installed before Docker was running, it may + have hit the systemd start limit when trying to repeatedly start and + connect to docker. If this is the case, it can be resolved by doing: - 2. Connect the required interfaces: - - sudo snap connect openshell:docker docker:docker-daemon - sudo snap connect openshell:log-observe - sudo snap connect openshell:system-observe + sudo systemctl reset-failed snap.openshell.gateway.service + sudo snap restart openshell.gateway - 3. Verify the gateway and register it locally: + 2. Verify the gateway and register it locally: snap services openshell.gateway openshell status @@ -44,6 +45,11 @@ description: | base: core24 grade: stable confinement: strict +# The system :docker slot was added in snapd 2.76, so this is the minimum +# required version. Snapd 2.77 allowed manual connection to that slot without +# snap-declaration overrides from the store, such as when installing a locally- +# built snap, but that is not required for installations from the snap store. +assumes: [snapd2.76] license: Apache-2.0 website: https://docs.nvidia.com/openshell/latest/index.html source-code: https://github.com/NVIDIA/OpenShell @@ -86,8 +92,10 @@ apps: # during snap refreshes, which would kill active sandbox sessions. # Operators must manually restart the service after a refresh if needed. refresh-mode: endure - # The wrapper sets OPENSHELL_DISABLE_TLS=true and OPENSHELL_DB_URL to - # use $SNAP_COMMON/gateway.db. Before startup it bootstraps package-managed + # Snapd runs this daemon as root, so ordinary CLI users cannot read the + # client certificate generated in root-owned snap state. The wrapper sets + # OPENSHELL_DISABLE_TLS=true for the loopback-only listener and uses + # $SNAP_COMMON/gateway.db. Before startup it bootstraps package-managed # credentials and validates the selected operator-provided config without # creating or rewriting it. A nonempty OPENSHELL_GATEWAY_CONFIG takes # precedence over gateway.toml. @@ -96,8 +104,6 @@ apps: XDG_RUNTIME_DIR: "$SNAP_COMMON" plugs: - docker - # Docker snap is required because the snap uses the docker:docker-daemon - # interface slot. It does not work with system-installed Docker. - log-observe - network - network-bind diff --git a/tasks/scripts/gateway-podman.sh b/tasks/scripts/gateway-podman.sh index d2b5397fb0..8f3cda7e4d 100644 --- a/tasks/scripts/gateway-podman.sh +++ b/tasks/scripts/gateway-podman.sh @@ -214,7 +214,6 @@ signing_key_path = "${TLS_DIR}/jwt/signing.pem" public_key_path = "${TLS_DIR}/jwt/public.pem" kid_path = "${TLS_DIR}/jwt/kid" gateway_id = "${GATEWAY_NAME}" -ttl_secs = 3600 [openshell.drivers.podman] default_image = "${SANDBOX_IMAGE}" diff --git a/tasks/scripts/test-install-sh.sh b/tasks/scripts/test-install-sh.sh index 220fbb43d4..f6ab4d2990 100755 --- a/tasks/scripts/test-install-sh.sh +++ b/tasks/scripts/test-install-sh.sh @@ -100,6 +100,368 @@ assert_glibc_preflight_fails \ "OpenShell Linux packages require glibc >= 2.28; detected musl or unsupported libc." \ setup_ldd_musl +assert_linux_package_method() { + local name=$1 + local snap_present=$2 + local dpkg_present=$3 + local rpm_present=$4 + local expected=$5 + local actual + + actual="$( + has_cmd() { + case "$1" in + snap) [ "$snap_present" = "1" ] ;; + dpkg) [ "$dpkg_present" = "1" ] ;; + rpm) [ "$rpm_present" = "1" ] ;; + *) return 1 ;; + esac + } + linux_package_method + )" + if [ "$actual" != "$expected" ]; then + echo "FAIL: ${name}: expected ${expected}, got ${actual}" >&2 + exit 1 + fi +} + +assert_linux_package_method "snap takes precedence over deb and rpm" 1 1 1 snap +assert_linux_package_method "deb is selected without snap" 0 1 1 deb +assert_linux_package_method "rpm is selected without snap or deb" 0 0 1 rpm + +if ! ( + find_existing_native_openshell_bin() { return 1; } + guard_native_to_snap_transition +) >"$out" 2>"$err"; then + echo "FAIL: Snap install without an existing native installation should continue" >&2 + cat "$err" >&2 || true + exit 1 +fi + +assert_native_to_snap_blocked() { + local name=$1 + local version=$2 + + if ( + find_existing_native_openshell_bin() { printf '%s\n' /usr/bin/openshell; } + existing_openshell_version() { printf '%s\n' "$version"; } + UPGRADE_NOTICE_ACK="" + guard_native_to_snap_transition + ) >"$out" 2>"$err"; then + echo "FAIL: ${name}: expected native-to-Snap transition to be blocked" >&2 + exit 1 + fi + if ! grep -Fq "detected existing non-snap OpenShell ${version} at /usr/bin/openshell" "$err"; then + echo "FAIL: ${name}: missing native installation warning" >&2 + cat "$err" >&2 || true + exit 1 + fi + if ! grep -Fq "does not import state from a non-snap installation" "$err"; then + echo "FAIL: ${name}: missing isolated Snap state explanation" >&2 + cat "$err" >&2 || true + exit 1 + fi +} + +assert_native_to_snap_blocked "old native install" v0.0.36 +assert_native_to_snap_blocked "current native install" v1.2.3 + +if ! ( + find_existing_native_openshell_bin() { printf '%s\n' /usr/local/bin/openshell; } + existing_openshell_version() { printf '%s\n' v1.2.3; } + UPGRADE_NOTICE_ACK=1 + guard_native_to_snap_transition +) >"$out" 2>"$err"; then + echo "FAIL: acknowledged native-to-Snap transition should continue" >&2 + cat "$err" >&2 || true + exit 1 +fi +if ! grep -Fq "continuing because OPENSHELL_ACK_BREAKING_UPGRADE=1 is set" "$err"; then + echo "FAIL: acknowledged native-to-Snap transition was not reported" >&2 + cat "$err" >&2 || true + exit 1 +fi + +if ( + RELEASE_TAG="" + target_uses_breaking_gateway_model +); then + echo "FAIL: Snap targets must not use the version-boundary upgrade guard" >&2 + exit 1 +fi + +out="$(mktemp)" +err="$(mktemp)" + +if ! OPENSHELL_VERSION=dev openshell_snap_channel >"$out" 2>"$err"; then + echo "FAIL: dev must select the latest/edge Snap channel" >&2 + cat "$err" >&2 || true + exit 1 +fi +if [ "$(cat "$out")" != "latest/edge" ]; then + echo "FAIL: dev must select the latest/edge Snap channel" >&2 + exit 1 +fi +if [ -s "$err" ]; then + echo "FAIL: dev must not warn about an ignored version" >&2 + cat "$err" >&2 || true + exit 1 +fi + +if ! OPENSHELL_VERSION="" openshell_snap_channel >"$out" 2>"$err"; then + echo "FAIL: unset OPENSHELL_VERSION must select the latest/stable Snap channel" >&2 + cat "$err" >&2 || true + exit 1 +fi +if [ "$(cat "$out")" != "latest/stable" ]; then + echo "FAIL: unset OPENSHELL_VERSION must select the latest/stable Snap channel" >&2 + exit 1 +fi +if [ -s "$err" ]; then + echo "FAIL: unset OPENSHELL_VERSION must not warn about an ignored version" >&2 + cat "$err" >&2 || true + exit 1 +fi + +for requested_version in pre v1.2.3; do + if ! OPENSHELL_VERSION="$requested_version" openshell_snap_channel >"$out" 2>"$err"; then + echo "FAIL: '${requested_version}' must select the latest/stable Snap channel" >&2 + cat "$err" >&2 || true + exit 1 + fi + if [ "$(cat "$out")" != "latest/stable" ]; then + echo "FAIL: '${requested_version}' must select the latest/stable Snap channel" >&2 + exit 1 + fi + if ! grep -Fq "OPENSHELL_VERSION=${requested_version} is ignored for Snap installs" "$err"; then + echo "FAIL: '${requested_version}' must warn that the requested version is ignored" >&2 + cat "$err" >&2 || true + exit 1 + fi +done +rm -f "$out" "$err" + +assert_snap_install_flow() { + local name=$1 + local docker_present=$2 + local openshell_present=$3 + local requested_version=$4 + local expected=$5 + local calls + + calls="$( + has_cmd() { + case "$1" in + snap) return 0 ;; + docker) [ "$docker_present" = "1" ] ;; + *) command -v "$1" >/dev/null 2>&1 ;; + esac + } + snap() { + case "${1:-}:${2:-}" in + list:docker) return 1 ;; + list:openshell) [ "$openshell_present" = "1" ] ;; + *) command snap "$@" ;; + esac + } + as_root() { printf 'root:%s\n' "$*"; } + set_linux_target_runtime_dir() { :; } + wait_for_docker_daemon() { printf '%s\n' "wait:docker"; } + ensure_snap_gateway_config() { printf '%s\n' "ensure:gateway-config"; } + register_snap_gateway() { printf '%s\n' "register:gateway"; } + wait_for_snap_gateway_listener() { printf '%s\n' "wait:gateway-listener"; } + wait_for_local_gateway_status() { printf '%s\n' "wait:gateway-status"; } + info() { :; } + export TARGET_USER=test-user + export OPENSHELL_VERSION="$requested_version" + install_linux_snap + )" + if [ "$calls" != "$expected" ]; then + echo "FAIL: ${name}: unexpected command sequence" >&2 + echo "Expected:" >&2 + printf '%s\n' "$expected" >&2 + echo "Actual:" >&2 + printf '%s\n' "$calls" >&2 + exit 1 + fi +} + +assert_snap_install_flow \ + "existing Docker is reused" \ + 1 0 "" \ + "wait:docker +root:snap install openshell --channel=latest/stable +ensure:gateway-config +root:snap restart openshell.gateway +register:gateway +wait:gateway-listener +wait:gateway-status" + +assert_snap_install_flow \ + "existing OpenShell snap is refreshed" \ + 1 1 pre \ + "wait:docker +root:snap refresh openshell --channel=latest/stable +ensure:gateway-config +root:snap restart openshell.gateway +register:gateway +wait:gateway-listener +wait:gateway-status" + +assert_snap_install_rejected() { + local name=$1 + local docker_present=$2 + local docker_snap_present=$3 + local expected=$4 + + if ( + has_cmd() { + case "$1" in + snap) return 0 ;; + docker) [ "$docker_present" = "1" ] ;; + *) command -v "$1" >/dev/null 2>&1 ;; + esac + } + snap() { + if [ "${1:-}:${2:-}" = list:docker ]; then + [ "$docker_snap_present" = "1" ] + else + echo "FAIL: ${name}: unexpected snap command: $*" >&2 + return 99 + fi + } + as_root() { + echo "FAIL: ${name}: installation reached root command: $*" >&2 + return 99 + } + set_linux_target_runtime_dir() { :; } + install_linux_snap + ) >"$out" 2>"$err"; then + echo "FAIL: ${name}: Snap installation should have been rejected" >&2 + exit 1 + fi + if ! grep -Fq "$expected" "$err"; then + echo "FAIL: ${name}: missing rejection message" >&2 + cat "$err" >&2 || true + exit 1 + fi + if grep -Fq "installation reached root command" "$err"; then + cat "$err" >&2 || true + exit 1 + fi +} + +assert_snap_install_rejected \ + "missing Docker" \ + 0 0 \ + "Docker is required before installing the OpenShell snap" + +assert_snap_install_rejected \ + "Docker snap" \ + 1 1 \ + "the Docker snap is not currently compatible with OpenShell" + +snap_config_dir="${tmpdir}/snap-config" +snap_config="${snap_config_dir}/gateway.toml" +if ! (as_root() { "$@"; }; ensure_snap_gateway_config "$snap_config"); then + echo "FAIL: Snap gateway config bootstrap should create a missing config" >&2 + exit 1 +fi +if ! grep -Fq 'allow_unauthenticated_users = true' "$snap_config"; then + echo "FAIL: Snap gateway config must permit the plaintext local CLI" >&2 + exit 1 +fi +if [[ $(stat -c '%a' "$snap_config") != 600 ]]; then + echo "FAIL: Snap gateway config must be mode 0600" >&2 + exit 1 +fi + +printf '\noperator setting = true\n' >>"$snap_config" +cp "$snap_config" "${tmpdir}/snap-config-before" +(as_root() { "$@"; }; ensure_snap_gateway_config "$snap_config") +cmp -s "${tmpdir}/snap-config-before" "$snap_config" + +broken_config="${tmpdir}/broken-gateway.toml" +ln -s "${tmpdir}/missing-gateway.toml" "$broken_config" +(as_root() { "$@"; }; ensure_snap_gateway_config "$broken_config") +if [[ $(readlink "$broken_config") != "${tmpdir}/missing-gateway.toml" ]]; then + echo "FAIL: Snap gateway config bootstrap replaced a broken operator symlink" >&2 + exit 1 +fi + +attempts_file="${tmpdir}/docker-attempts" +root_probes_file="${tmpdir}/docker-root-probes" +printf '0\n' >"$attempts_file" +printf '0\n' >"$root_probes_file" +if ! ( + docker() { + attempts="$(cat "$attempts_file")" + attempts=$((attempts + 1)) + printf '%s\n' "$attempts" >"$attempts_file" + [ "$attempts" -ge 3 ] + } + as_root() { + root_probes="$(cat "$root_probes_file")" + printf '%s\n' "$((root_probes + 1))" >"$root_probes_file" + "$@" + } + sleep() { :; } + info() { :; } + OPENSHELL_INSTALL_DOCKER_TIMEOUT=3 wait_for_docker_daemon +) >"$out" 2>"$err"; then + echo "FAIL: Docker readiness should succeed after retries" >&2 + cat "$err" >&2 || true + exit 1 +fi +if [ "$(cat "$attempts_file")" != "3" ]; then + echo "FAIL: Docker readiness did not retry three times" >&2 + exit 1 +fi +if [ "$(cat "$root_probes_file")" != "3" ]; then + echo "FAIL: Docker readiness probes did not run through as_root" >&2 + exit 1 +fi + +if ( + docker() { return 1; } + as_root() { "$@"; } + snap() { return 1; } + sleep() { :; } + info() { :; } + OPENSHELL_INSTALL_DOCKER_TIMEOUT=2 wait_for_docker_daemon +) >"$out" 2>"$err"; then + echo "FAIL: Docker readiness timeout should fail" >&2 + exit 1 +fi +if ! grep -Fq "Docker daemon did not become reachable within 2s" "$err"; then + echo "FAIL: missing Docker readiness timeout message" >&2 + cat "$err" >&2 || true + exit 1 +fi + +if ! grep -Fq '/snap/bin/openshell' "${ROOT}/install.sh"; then + echo "FAIL: Snap installs must use the Snap CLI explicitly" >&2 + exit 1 +fi + +registration_calls_file="${tmpdir}/registration-calls" +: >"$registration_calls_file" +if ! ( + as_target_user() { printf 'target:%s\n' "$*" >>"$registration_calls_file"; } + print_gateway_add_output() { :; } + register_snap_gateway +) >"$out" 2>"$err"; then + echo "FAIL: Snap gateway registration should succeed" >&2 + cat "$err" >&2 || true + exit 1 +fi +registration_calls="$(cat "$registration_calls_file")" +if [ "$registration_calls" != "target:/snap/bin/openshell gateway add http://127.0.0.1:17670 --local --name openshell" ]; then + echo "FAIL: Snap gateway registration must use the Snap CLI as the target user" >&2 + printf '%s\n' "$registration_calls" >&2 + exit 1 +fi + if [ "$(PLATFORM=darwin local_gateway_endpoint)" != "https://localhost:17670" ]; then echo "FAIL: macOS local gateway endpoint must use a TLS-compatible loopback hostname" >&2 exit 1 diff --git a/tasks/scripts/test-packaging-assets.sh b/tasks/scripts/test-packaging-assets.sh index fb6424ff06..345a0e3964 100755 --- a/tasks/scripts/test-packaging-assets.sh +++ b/tasks/scripts/test-packaging-assets.sh @@ -10,7 +10,7 @@ assert_contains() { local file=$1 local expected=$2 - if ! grep -Fq "$expected" "$file"; then + if ! grep -Fq -- "$expected" "$file"; then echo "FAIL: ${file} is missing expected text:" >&2 echo " ${expected}" >&2 exit 1 @@ -21,7 +21,7 @@ assert_not_contains() { local file=$1 local unexpected=$2 - if grep -Fq "$unexpected" "$file"; then + if grep -Fq -- "$unexpected" "$file"; then echo "FAIL: ${file} contains stale text:" >&2 echo " ${unexpected}" >&2 exit 1 @@ -79,8 +79,20 @@ assert_not_contains "$spec" '%%S/openshell/tls' # Schema-v2 package startup wiring. snap_wrapper="${ROOT}/tasks/scripts/snap-gateway-wrapper.sh" +snapcraft="${ROOT}/snapcraft.yaml" +snap_install_docs="${ROOT}/docs/about/installation.mdx" +snap_canary="${ROOT}/.github/workflows/release-canary.yml" +snap_repro="${ROOT}/nix/test-guest/scripts/snap-gateway-repro.sh" +snap_docker_hook="${ROOT}/snap/hooks/connect-plug-docker" +snap_install_hook="${ROOT}/snap/hooks/install" package_deb="${ROOT}/tasks/scripts/package-deb.sh" assert_file_exists "$snap_wrapper" +assert_file_exists "$snapcraft" +assert_file_exists "$snap_install_docs" +assert_file_exists "$snap_canary" +assert_file_exists "$snap_repro" +assert_file_exists "$snap_docker_hook" +assert_file_exists "$snap_install_hook" assert_file_exists "$package_deb" assert_contains "$service" "ExecStartPre=/usr/bin/openshell-gateway config preflight" assert_contains "$package_deb" "\$src_dir/openshell-gateway.service" @@ -92,6 +104,42 @@ assert_contains \ assert_contains "$snap_wrapper" "config preflight -- --config \"\$CANONICAL_CONFIG_FILE\" \"\$@\"" assert_not_contains "$snap_wrapper" "[ -f \"\$CANONICAL_CONFIG_FILE\" ]" bash "$ROOT/tasks/scripts/test-snap-gateway-wrapper.sh" "$snap_wrapper" + +# Store installs autoconnect all required interfaces and require snapd 2.76 for +# the system Docker slot. Manual connection for locally-built snaps requires +# snapd 2.77. +assert_contains "$snapcraft" "assumes: [snapd2.76]" +for snap_file in \ + "$snapcraft" \ + "$snap_install_docs" \ + "$snap_canary" \ + "$snap_repro" \ + "$snap_docker_hook" \ + "$snap_install_hook"; do + assert_not_contains "$snap_file" "docker:docker-daemon" + assert_not_contains "$snap_file" "default-provider: docker" +done +if [[ ! -x "$snap_install_hook" ]]; then + echo "FAIL: Snap install hook must be executable" >&2 + exit 1 +fi +assert_contains "$snap_install_hook" 'allow_unauthenticated_users = true' +bash "$ROOT/tasks/scripts/test-snap-install-hook.sh" "$snap_install_hook" +assert_not_contains "$snap_install_docs" "snap connect openshell:home" +assert_not_contains "$snap_install_docs" "snap connect openshell:network" +assert_not_contains "$snap_install_docs" "snap connect openshell:network-bind" +assert_contains "$snap_install_docs" "snap connect openshell:docker :docker" +assert_contains "$snap_canary" "install.sh | sh" +assert_contains "$snap_canary" "ubuntu-snap-system-docker:" +assert_contains "$snap_canary" "ubuntu-snap-docker-preflight:" +assert_contains "$snap_repro" 'OPENSHELL_VERSION=dev sh "${install_script}"' +assert_contains "$snap_repro" "system-docker" +assert_contains "$snap_repro" "missing-docker" +assert_contains "$snap_repro" "docker-snap" +assert_not_contains "$snap_canary" "--dangerous" +assert_not_contains "$snap_repro" "--dangerous" +assert_not_contains "$snap_canary" "snap connect openshell:docker" +assert_not_contains "$snap_repro" "snap connect openshell:docker" if ! awk '/config preflight/ { seen = 1 } /generate-certs/ { exit !seen }' "$service"; then echo "FAIL: Debian preflight must precede certificate generation" >&2 exit 1 diff --git a/tasks/scripts/test-snap-install-hook.sh b/tasks/scripts/test-snap-install-hook.sh new file mode 100755 index 0000000000..b2c9c7eab7 --- /dev/null +++ b/tasks/scripts/test-snap-install-hook.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +hook_input=${1:?Usage: test-snap-install-hook.sh } +hook_dir=$(cd "$(dirname "$hook_input")" && pwd) +hook="${hook_dir}/$(basename "$hook_input")" +work=$(mktemp -d "${TMPDIR:-/tmp}/openshell snap install hook.XXXXXX") +trap 'rm -rf "$work"' EXIT + +expected="${work}/expected.toml" +cat >"$expected" <<'EOF' +[openshell] +version = 2 + +[openshell.gateway] + +[openshell.gateway.auth] +allow_unauthenticated_users = true +EOF + +common="${work}/fresh" +SNAP_COMMON="$common" "$hook" +cmp -s "$expected" "$common/gateway.toml" +if [[ $(stat -c '%a' "$common/gateway.toml") != 600 ]]; then + echo "FAIL: install hook config must be mode 0600" >&2 + exit 1 +fi + +printf '\noperator setting = true\n' >>"$common/gateway.toml" +cp "$common/gateway.toml" "${work}/operator-before" +SNAP_COMMON="$common" "$hook" +cmp -s "${work}/operator-before" "$common/gateway.toml" + +common="${work}/broken-link" +mkdir -p "$common" +ln -s "${work}/missing-target" "$common/gateway.toml" +SNAP_COMMON="$common" "$hook" +if [[ $(readlink "$common/gateway.toml") != "${work}/missing-target" ]]; then + echo "FAIL: install hook replaced a broken operator symlink" >&2 + exit 1 +fi + +common="${work}/directory" +mkdir -p "$common/gateway.toml" +SNAP_COMMON="$common" "$hook" +if [[ ! -d "$common/gateway.toml" ]]; then + echo "FAIL: install hook replaced an operator-owned directory" >&2 + exit 1 +fi + +echo "Snap install hook tests passed" diff --git a/tasks/test.toml b/tasks/test.toml index de65f527d6..fd516ae256 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -162,6 +162,11 @@ description = "Run Rust CLI e2e tests against a Podman-backed gateway" depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-podman.sh" +["e2e:podman:ci"] +description = "Run the Podman Rust e2e targets enabled in required branch CI" +depends = ["e2e:conformance:build"] +run = "OPENSHELL_E2E_PODMAN_TEST_SET=ci OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-podman.sh" + ["e2e:oidc-pkce"] description = "Run Linux browser PKCE and RBAC e2e tests against Keycloak and a Podman gateway" run = [