feat(driver-podman): add userns config (#2562)

* refactor(driver): extract shared supervisor binary helpers

Move supervisor binary extraction, caching, and validation helpers from
the Docker driver into openshell-core::driver_utils so both Docker and
Podman drivers can reuse them.

Moved helpers: extract_first_tar_entry, write_cache_binary_atomic,
supervisor_cache_path, temp_extract_container_name, and
validate_linux_elf_binary.

The shared extract_first_tar_entry gains entry-type and empty-payload
checks that the Docker-local version lacked.  supervisor_cache_path
takes a driver_subdir parameter so each driver caches under its own
namespace (docker-supervisor vs podman-supervisor).

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

* feat(driver-podman): add userns config

Add a `userns` option to the Podman compute driver that maps to
Podman's user namespace modes.  The mode string is split on the first
colon into the API's `nsmode` and `value` fields so parameterized
values like `auto:size=65536` and `keep-id:uid=1000,gid=1000` are
forwarded correctly.  When the mode is `auto`, the container spec
also sets `idmappings.AutoUserNs = true` as required by the API.

An allowlist validates the mode at startup: `auto` and `keep-id`
accept optional parameters; `host`, `private`, and `nomap` reject
them; everything else is an error.

Podman image volumes use overlay mounts internally and the kernel
does not support idmapped mounts on overlay (`mount_setattr` returns
EINVAL).  When userns is configured (any mode except `host`), the
driver extracts the supervisor binary from the image to a host-side
cache and bind-mounts it instead of using an image volume.

Configurable via TOML `userns = "auto"`, CLI `--userns`, or
environment variable `OPENSHELL_PODMAN_USERNS`.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

---------

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Author: giuseppe

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -222,6 +222,26 @@ Common findings:
   cannot bypass slirp4netns host-loopback isolation. Do not work around
   discovery failures by broadening the primary gateway listener to `0.0.0.0`.
 
+When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`):
+
+- Supervisor delivery uses bind-mount fallback instead of image volumes because
+  overlay mounts do not support `idmapped` mounts. The supervisor binary is
+  extracted from the supervisor image and cached at
+  `$XDG_DATA_HOME/openshell/podman-supervisor/` (typically
+  `~/.local/share/openshell/podman-supervisor/`).
+- Stale cache: if the supervisor image is updated but the cached binary is not
+  refreshed, sandbox creation may fail with an ELF validation error or version
+  mismatch. Remove the cache directory and retry.
+- `auto` mode requires subuid/subgid ranges for the current user in
+  `/etc/subuid` and `/etc/subgid`. If missing, Podman returns a user-namespace
+  mapping error at container creation.
+- `private` mode requires explicit `uidmap` and `gidmap` arrays in the TOML
+  config. Without both, the gateway rejects the config at startup.
+  Rootless Podman uses intermediate IDs (e.g. `uidmap = ["0:0:1", "1:1:65535"]`);
+  rootful Podman uses absolute host IDs (e.g. `uidmap = ["0:1000:1", "1:100000:65536"]`).
+- `nomap` (without hyphen) is accepted as input but canonicalized to `no-map`
+  for Podman's API.
+
 ### Step 6: Check Kubernetes Helm Gateways
 
 ```bash

Cargo.lock

@@ -140,7 +140,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
 | Runtime | Best fit | Sandbox boundary | Notes |
 |---|---|---|---|
 | Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
-| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
+| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). |
 | Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
 | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
 | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
@@ -193,7 +193,7 @@ The supervisor must be available inside each sandbox workload:
 | Runtime | Delivery model |
 |---|---|
 | Docker | Bind-mounted local supervisor binary, or a binary extracted from the configured supervisor image. |
-| Podman | Read-only OCI image volume containing the supervisor binary. |
+| Podman | Read-only OCI image volume by default; host-cached bind mount when `userns` is configured. |
 | Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
 | VM | Embedded in the guest rootfs bundle. |
 | Extension | Defined by the out-of-tree driver. |

architecture/compute-runtimes.md

@@ -30,6 +30,8 @@ ipnet = "2"
 base64 = { workspace = true }
 chrono = { version = "0.4", default-features = false, features = ["clock", "std"], optional = true }
 reqwest = { workspace = true, features = ["blocking", "rustls-tls-native-roots"], optional = true }
+tar = { version = "0.4", optional = true }
+tempfile = { version = "3", optional = true }
 
 [target.'cfg(unix)'.dependencies]
 nix = { workspace = true }
@@ -39,6 +41,7 @@ default = ["telemetry"]
 ## Compile in anonymous telemetry emission support. On by default; disable with
 ## `--no-default-features` (plus any other features you need) for a build that
 ## contains no telemetry endpoint, no HTTP client, and no emission code at all.
+driver-extraction = ["dep:tar", "dep:tempfile"]
 telemetry = ["dep:reqwest", "dep:chrono"]
 
 [build-dependencies]

crates/openshell-core/Cargo.toml

@@ -3,7 +3,7 @@
 
 //! Utility helpers shared across compute-driver crates.
 
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 
 use crate::proto::compute::v1::{DriverSandbox, GetCapabilitiesResponse};
 
@@ -437,6 +437,145 @@ pub fn supervisor_image_should_refresh(image: &str) -> bool {
     matches!(supervisor_image_tag(image), Some("dev" | "latest"))
 }
 
+// ---------------------------------------------------------------------------
+// Supervisor binary extraction helpers (shared by Docker and Podman drivers)
+// ---------------------------------------------------------------------------
+
+#[cfg(feature = "driver-extraction")]
+/// Extract the payload of the first regular-file entry in a tar archive.
+///
+/// Container archive endpoints return a single-file tar when `path` points to
+/// a file, so only the first entry is consumed. Returns an error when the
+/// archive is empty, the first entry is not a regular file, or the payload is
+/// empty.
+pub fn extract_first_tar_entry(tar_bytes: &[u8]) -> Result<Vec<u8>, String> {
+    let mut archive = tar::Archive::new(std::io::Cursor::new(tar_bytes));
+    let mut entries = archive
+        .entries()
+        .map_err(|err| format!("open tar archive: {err}"))?;
+    let mut entry = entries
+        .next()
+        .ok_or_else(|| "tar archive was empty".to_string())?
+        .map_err(|err| format!("read tar entry: {err}"))?;
+    let kind = entry.header().entry_type();
+    if !kind.is_file() {
+        return Err(format!(
+            "expected a regular file in tar archive, got type {kind:?}"
+        ));
+    }
+    let mut bytes = Vec::new();
+    std::io::Read::read_to_end(&mut entry, &mut bytes)
+        .map_err(|err| format!("read tar entry payload: {err}"))?;
+    if bytes.is_empty() {
+        return Err("tar entry payload was empty".to_string());
+    }
+    Ok(bytes)
+}
+
+#[cfg(feature = "driver-extraction")]
+/// Atomically write `bytes` to `final_path` via a sibling temp file.
+///
+/// Creates parent directories as needed. The temp file is synced, `chmod 755`
+/// (on Unix), and renamed into place so concurrent readers never observe a
+/// partial write. Returns a human-readable error string on failure.
+pub fn write_cache_binary_atomic(final_path: &Path, bytes: &[u8]) -> Result<(), String> {
+    let dir = final_path
+        .parent()
+        .ok_or_else(|| format!("cache path '{}' has no parent", final_path.display()))?;
+    std::fs::create_dir_all(dir)
+        .map_err(|err| format!("failed to create cache dir '{}': {err}", dir.display()))?;
+
+    let mut temp = tempfile::Builder::new()
+        .prefix(".openshell-sandbox-")
+        .tempfile_in(dir)
+        .map_err(|err| format!("failed to create temp file in '{}': {err}", dir.display()))?;
+    std::io::Write::write_all(&mut temp, bytes)
+        .map_err(|err| format!("failed to write supervisor binary: {err}"))?;
+    temp.as_file()
+        .sync_all()
+        .map_err(|err| format!("failed to sync supervisor binary: {err}"))?;
+
+    #[cfg(unix)]
+    {
+        use std::os::unix::fs::PermissionsExt;
+        std::fs::set_permissions(temp.path(), std::fs::Permissions::from_mode(0o755))
+            .map_err(|err| format!("failed to chmod supervisor binary: {err}"))?;
+    }
+
+    temp.persist(final_path).map_err(|err| {
+        format!(
+            "failed to persist supervisor binary to '{}': {}",
+            final_path.display(),
+            err.error,
+        )
+    })?;
+    Ok(())
+}
+
+/// Return the host-side cache path for an extracted supervisor binary.
+///
+/// The path is `$XDG_DATA_HOME/openshell/<driver_subdir>/<sanitized-digest>/openshell-sandbox`.
+/// `driver_subdir` distinguishes caches across drivers (e.g. `"docker-supervisor"`,
+/// `"podman-supervisor"`).
+pub fn supervisor_cache_path(driver_subdir: &str, digest: &str) -> Result<PathBuf, String> {
+    let base = crate::paths::xdg_data_dir()
+        .map_err(|err| format!("failed to resolve XDG data dir: {err}"))?;
+    Ok(supervisor_cache_path_with_base(
+        &base,
+        driver_subdir,
+        digest,
+    ))
+}
+
+/// [`supervisor_cache_path`] with an explicit base directory (for testing).
+pub fn supervisor_cache_path_with_base(base: &Path, driver_subdir: &str, digest: &str) -> PathBuf {
+    let sanitized = digest.replace(':', "-");
+    base.join("openshell")
+        .join(driver_subdir)
+        .join(sanitized)
+        .join("openshell-sandbox")
+}
+
+/// Generate a unique container name for supervisor binary extraction.
+///
+/// Uses the process ID and an atomic counter to avoid collisions across
+/// concurrent gateway starts.
+pub fn temp_extract_container_name() -> String {
+    use std::sync::atomic::{AtomicU64, Ordering};
+    static SEQ: AtomicU64 = AtomicU64::new(0);
+    let pid = std::process::id();
+    let seq = SEQ.fetch_add(1, Ordering::Relaxed);
+    format!("openshell-supervisor-extract-{pid}-{seq}")
+}
+
+/// Validate that the file at `path` starts with the ELF magic bytes (`\x7fELF`).
+///
+/// Returns a human-readable error when the file cannot be read or is not a
+/// Linux ELF binary.
+pub fn validate_linux_elf_binary(path: &Path) -> Result<(), String> {
+    use std::io::Read;
+    let mut file = std::fs::File::open(path).map_err(|err| {
+        format!(
+            "failed to open supervisor binary '{}': {err}",
+            path.display()
+        )
+    })?;
+    let mut magic = [0u8; 4];
+    file.read_exact(&mut magic).map_err(|err| {
+        format!(
+            "failed to read supervisor binary '{}': {err}",
+            path.display()
+        )
+    })?;
+    if magic != [0x7f, b'E', b'L', b'F'] {
+        return Err(format!(
+            "supervisor binary '{}' is not a Linux ELF executable",
+            path.display(),
+        ));
+    }
+    Ok(())
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -628,7 +767,7 @@ mod tests {
         let err = read_upstream_proxy_credential_file(dir.path().to_str().unwrap()).unwrap_err();
         assert!(err.contains("regular file"), "{err}");
 
-        if std::path::Path::new("/dev/zero").exists() {
+        if Path::new("/dev/zero").exists() {
             let err = read_upstream_proxy_credential_file("/dev/zero").unwrap_err();
             assert!(err.contains("regular file"), "{err}");
         }

crates/openshell-core/src/driver_utils.rs

@@ -11,7 +11,7 @@ license.workspace = true
 repository.workspace = true
 
 [dependencies]
-openshell-core = { path = "../openshell-core", default-features = false }
+openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] }
 
 tokio = { workspace = true }
 tonic = { workspace = true }
@@ -23,13 +23,13 @@ serde = { workspace = true }
 serde_json = { workspace = true }
 prost-types = { workspace = true }
 bollard = { version = "0.20" }
-tar = "0.4"
-tempfile = "3"
 url = { workspace = true }
 
 [dev-dependencies]
 prost-types = { workspace = true }
+tar = "0.4"
 temp-env = "0.3"
+tempfile = "3"
 
 [lints]
 workspace = true