Skip to content

Make the MicroVM sandbox workspace root configurable (currently hard-coded to /sandbox) #5282

Open

Description

@mr-miles

User Story

I am using openshell with images created around a /home/ layout, running in VMs. This conflicts with openshell which forces the writeable area of the image to be /sandbox - hardcoded in multiple places.

Problem Statement

The MicroVM compute driver (openshell-driver-vm) hard-codes the sandbox workspace path /sandbox in several independent places. There is no operator or per-sandbox way to change it. This makes it impossible to run images whose expected working directory / home is somewhere else (e.g. an image built around /home/<user>), and it diverges from the Docker/Podman drivers, which resolve the workspace from the image's OCI WorkingDir.

Current behavior (where /sandbox is embedded)

Three non-test sites must agree, but each hard-codes the literal independently:

  1. Supervisor working directory — crates/openshell-driver-vm/src/driver.rs, in spawn_host_supervisor():

    .arg("--workdir")
    .arg("/sandbox")

    (spawn_host_supervisor has sandbox: &Sandbox in scope; it is the sole caller’s target from provision_sandbox_inner().)

  2. Writable-workspace ownership — crates/openshell-driver-vm/src/rootfs.rs, in normalize_sandbox_owner_in_rootfs_image():

    let sandbox_dir = source.join("sandbox");
    ...
    collect_sandbox_owner_commands(&sandbox_dir, "/sandbox", uid, gid, &mut commands)?
  3. Sandbox account home — crates/openshell-driver-vm/src/rootfs.rs, in ensure_sandbox_guest_user():

    &format!("sandbox:x:{sandbox_uid}:{sandbox_gid}:OpenShell Sandbox:/sandbox:/bin/bash"),

Because the host supervisor derives HOME from the resolved identity's passwd entry (openshell-sandbox process.rs::session_user_and_home), site (3) dictates HOME, site (2) dictates where the sandbox user can write, and site (1) dictates the agent's cwd — and all three are hard-coded separately.

This is intentional today (docs: "Kubernetes and MicroVM always use /sandbox"), so any change should preserve that as the default.

Impact / Why This Matters

  • Without this feature, users must fork the entire vm-driver in order to make a small change.
  • The /sandbox behaviour vs. docker/podman is not obvious - particularly to agents - and only surfaced after some hours of investigation
  • Image-native layouts. Images built to run as /home/<user> (tooling,
    configs, HOME) can't be honored — the agent always runs in /sandbox, and
    anything the image wrote to its home on the read-only rootfs is not the
    writable path. (The only writable location is the chowned /sandbox.)
  • Driver parity. Docker and Podman already resolve the workspace from the
    image's OCI WorkingDir via openshell_core::driver_mounts::resolve_oci_workspace_root
    (crates/openshell-driver-docker/src/lib.rs and
    crates/openshell-driver-podman/src/container.rs), falling back to /sandbox.
    MicroVM is the odd one out.
  • Single source of truth. The three sites agreeing by hand is fragile.

Proposed Design

Add a workspace-root config to VmDriverConfig, defaulting to the existing constant so the default is byte-for-byte unchanged:

// crates/openshell-driver-vm/src/driver.rs — VmDriverConfig
#[serde(default = "default_workspace_root")]
pub workspace_root: String,

fn default_workspace_root() -> String {
    openshell_core::driver_mounts::DEFAULT_WORKSPACE_ROOT.to_string() // "/sandbox"
}

Validate it on construction with the existing helper (openshell_core::driver_mounts::resolve_oci_workspace_root / normalize_absolute_container_path) so only normalized absolute non-reserved paths are accepted.

Then thread workspace_root to the three sites. It rides the same path sandbox_uid/sandbox_gid already take:

VmDriverConfig.workspace_root
  → driver.rs: spawn_host_supervisor() ............. --workdir <workspace_root>
  → prepare_sandbox_rootfs_from_image_root(rootfs, image_identity, sandbox_uid, sandbox_gid, workspace_root)
       → prepare_sandbox_rootfs(...) → ensure_sandbox_guest_user(...) .... passwd home = <workspace_root>
       → create_rootfs_image_from_dir(...) → normalize_sandbox_owner_in_rootfs_image(...) .... chown <workspace_root>

Net diff: one new VmDriverConfig field (+ default/validation) and the three literal substitutions, with the parameter plumbed through the two existing rootfs entry points. Default remains /sandbox; the docs' guarantee holds.

Acceptance Criteria

  • Users of VM driver can set the policy for the mount point in config to "workdir" or "set". Set would be the default and require a path, which would default to "/sandbox"
  • VM driver created sandboxes with default config are created (as today) with a /sandbox mount point for the workload
  • VM driver created sandboxes with non-default set-path config are created with a matching mount point for the workload
  • VM driver created sandboxes with the "workdir" policy would examine the image and use the image's workdir for the workload

Alternatives Considered

Alternative (Option B — parity with Docker/Podman)

Instead of (or in addition to) an operator config, derive the workspace from the image's OCI WorkingDir via resolve_oci_workspace_root(working_dir) — exactly what the Docker/Podman drivers do — and thread the result to the same three sites. This is the most consistent cross-driver behavior, but it's more invasive (the MicroVM path doesn't read Config.WorkingDir today; it builds a rootfs tar rather than inspecting the OCI config) and it changes the documented always-/sandbox default. Option A is proposed as the smaller, backward-compatible step.

Agent Investigation

#4351 (Firecracker compute driver)
#1907 (external compute driver extension point).

Checklist

  • I've reviewed existing issues and the published docs
  • This is a design proposal, not a "please build this" request

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions