You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Kubernetes driver never requests CAP_SETPCAP, so combined-topology sandboxes crash-loop with EPERM on privilege drop #2751
On Kubernetes, topology = "combined" sandboxes always crash on startup with EPERM because the supervisor's privilege-drop routine requires CAP_SETPCAP, which the Kubernetes driver never requests in the pod's securityContext.capabilities.add list — under any config.
Root cause
drop_privileges_with_identity (crates/openshell-supervisor-process/src/process.rs) calls drop_capability_bounding_set() before setuid()/setgid(), whenever enforcement_mode.uses_privileged_process_setup() is true (i.e. topology = "combined" → ProcessEnforcementMode::Full, see crates/openshell-sandbox/src/lib.rs:857-858).
drop_capability_bounding_set() (process.rs:277-286) calls capctl::caps::bounding::clear(), which requires CAP_SETPCAP.
The Kubernetes driver's capability list (crates/openshell-driver-kubernetes/src/driver.rs:2692-2699) only ever adds SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYSLOG (plus SETUID, SETGID, DAC_READ_SEARCH when enable_user_namespaces is on) — SETPCAP is never added, in any code path.
Since the driver also sets capabilities: { drop: ["ALL"] } unconditionally, the supervisor (running as root, runAsUser: 0) never actually holds CAP_SETPCAP, so the bounding-set clear fails with EPERM and the pod crash-loops before ever exec'ing the workload.
By contrast, the Podman driver gets this right: crates/openshell-driver-podman/container.rs:1048-1051 explicitly adds SETPCAP to cap_add with a comment referencing exactly this requirement.
Reproduction
Deploy the Kubernetes driver with topology = "combined" (the documented/default topology) on any cluster where sandbox pods run under a restricted PodSecurityStandard / custom SCC that only grants the capabilities the driver actually requests (i.e. not a blanket privileged SCC).
Create a sandbox via CreateSandbox.
Observe the agent container crash-loop with:
Error: × EPERM: Operation not permitted
oc logs <pod> -c agent shows nothing more specific — the error surfaces from validate_capability_bounding_set_clear (process.rs:289-311) with a non-empty remaining bounding set, which does not hit the "already empty" fallback path.
We also confirmed the network-policy portion of sandbox setup (which needs SYS_ADMIN/NET_ADMIN) succeeds and reports ReportPolicyStatus: status=loadedbefore this crash — this is specifically the privilege-drop step, not the network-policy setup step.
Notes
Docs (docs/kubernetes/openshift.mdx:12) note the OpenShift install path is "experimental" and recommend the privileged SCC — but since the driver unconditionally sets capabilities: { drop: ["ALL"], add: [...] } regardless of SCC, granting a more permissive SCC doesn't help; the pod's own requested capability list is still missing SETPCAP.
enable_user_namespaces = true does NOT fix this — it only adds SETUID/SETGID/DAC_READ_SEARCH, not SETPCAP.
Workaround we're using in the meantime: switching to topology = "sidecar" (ProcessEnforcementMode::NetworkOnly), which skips drop_privileges entirely since the container already runs as the target UID.
Suggested fix
Add "SETPCAP" to the capability list built in crates/openshell-driver-kubernetes/src/driver.rs (around line 2692), mirroring the Podman driver's cap_add list, for any topology whose ProcessEnforcementMode::uses_privileged_process_setup() is true.
Environment
OpenShell built from dev branch, commit c4b500a (2026-08-14)
Hello, I tried to reproduce on Kubernetes, but I did not manage to hit the same issue.
Given that you use oc in your steps, could be this is not general Kubernetes issue, but it affects OpenShift specifically?
I am on macOS, so I used a lima-managed VM, like this
❯ limactl start --name k8s --cpus 4 --memory 8 --disk 40 template:k8s
? Creating an instance `k8s` Proceed with the current configuration
INFO[0011] Starting the instance `k8s` with internal VM driver `vz`
INFO[0011] Attempting to download the image arch=aarch64 digest="sha256:7bcf159e29ad0000bfed9c57875908c39268f5ed1257f4958fa6a9f5f60edd54" location="https://cloud-images.ubuntu.com/releases/resolute/release-20260720/ubuntu-26.04-server-cloudimg-arm64.img"
Downloading the image (ubuntu-26.04-server-cloudimg-arm64.img)
...
INFO[0891] READY. Run `limactl shell k8s` to open the shell.
INFO[0891] Message from the instance `k8s`:
To run `kubectl` on the host (assumes kubectl is installed), run the following commands:
------
export KUBECONFIG="/Users/jdanek/.lima/k8s/copied-from-guest/kubeconfig.yaml"
kubectl ...
------
Then I did the necessary setup, I used rancher local-path-provisioner to get a necessary PVC:
(the latest url for agent-standbox did not work, so I put found the latest version string and used that)
And I installed openshell with the combined topology
❯ CHART=0.0.110
❯ helm upgrade --install openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version "$CHART" \
--namespace openshell \
--set supervisor.topology=combined \
--set server.disableTls=true \
--set server.auth.allowUnauthenticatedUsers=true
Release "openshell" does not exist. Installing it now.
Pulled: ghcr.io/nvidia/openshell/helm-chart:0.0.110
Digest: sha256:8bf947365a46184e979a30039a7f22fe57e150827ab699bc688e9260f7c4af12
NAME: openshell
LAST DEPLOYED: Fri Aug 21 17:58:05 2026
NAMESPACE: openshell
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
Which seems to have been applied
❯ kubectl -n openshell get cm -o yaml | grep -n -A2 -E 'topology'
70: topology = "combined"
Then, trying to run a pod, the pod started just fine for me, baring some hiccup with slow network downloading sandbox pod; the cli timeouted, but the sandbox did come up
❯ kubectl -n openshell port-forward svc/openshell 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080
Handling connection for 8080
Handling connection for 8080
❯ openshell gateway add http://127.0.0.1:8080 --local --name k8s
✓ Gateway 'k8s' added and set as active
Endpoint: http://127.0.0.1:8080
Type: local
Auth: plaintext
❯ openshell sandbox create --name cap-setpcap-repro -- true
Created sandbox: cap-setpcap-repro
✗ sandbox provisioning timed out after 300s. Last reported status: DependenciesNotReady: Pod exists with phase: Pending
✓ Sandbox allocated (4s)
✓ Image pulled (13 MB) (6s) Error: × sandbox provisioning timed out after 300s. Last reported status: DependenciesNotReady: Pod exists with phase: Pending
❯ openshell sandbox create --name cap-setpcap-repro -- true
Error: × sandbox 'cap-setpcap-repro' already exists
│
│ hint: delete it first with: openshell sandbox delete <name>
│ or use a different name
❯ kubectl -n openshell get sandboxes
NAME READY REASON AGE
default--cap-setpcap-repro True DependenciesReady 12m
❯ kubectl get pod -n openshell -o yaml default--cap-setpcap-repro
I see no crashloops or any errors besides seemingly innocuous
❯ kubectl logs -n openshell default--cap-setpcap-repro -c agent
2026-08-21T16:13:16.071Z WARN openshell_supervisor_network::opa: Cannot access container filesystem for symlink resolution: path=/usr/local/bin/opencode container_path=/proc/38/root/usr/local/bin/opencode pid=38 error=No such file or directory (os error 2). Binary paths in policy will be matched literally. If this binary is a symlink (e.g., /usr/bin/python3 -> python3.11), use the canonical path instead, or run with CAP_SYS_PTRACE.
2026-08-21T16:13:16.072Z WARN openshell_supervisor_network::opa: Cannot access container filesystem for symlink resolution: path=/usr/local/bin/opencode container_path=/proc/38/root/usr/local/bin/opencode pid=38 error=No such file or directory (os error 2). Binary paths in policy will be matched literally. If this binary is a symlink (e.g., /usr/bin/python3 -> python3.11), use the canonical path instead, or run with CAP_SYS_PTRACE.
2026-08-21T16:13:16.072Z WARN openshell_supervisor_network::opa: Cannot access container filesystem for symlink resolution: path=/usr/bin/wget container_path=/proc/38/root/usr/bin/wget pid=38 error=No such file or directory (os error 2). Binary paths in policy will be matched literally. If this binary is a symlink (e.g., /usr/bin/python3 -> python3.11), use the canonical path instead, or run with CAP_SYS_PTRACE.
2026-08-21T16:13:16.079Z WARN openshell_supervisor_network::opa: Cannot access container filesystem for symlink resolution: path=/usr/bin/wget container_path=/proc/38/root/usr/bin/wget pid=38 error=No such file or directory (os error 2). Binary paths in policy will be matched literally. If this binary is a symlink (e.g., /usr/bin/python3 -> python3.11), use the canonical path instead, or run with CAP_SYS_PTRACE.
2026-08-21T16:13:16.079Z WARN openshell_supervisor_network::opa: Cannot access container filesystem for symlink resolution: path=/app/.venv/bin/python container_path=/proc/38/root/app/.venv/bin/python pid=38 error=No such file or directory (os error 2). Binary paths in policy will be matched literally. If this binary is a symlink (e.g., /usr/bin/python3 -> python3.11), use the canonical path instead, or run with CAP_SYS_PTRACE.
2026-08-21T16:13:16.079Z WARN openshell_supervisor_network::opa: Cannot access container filesystem for symlink resolution: path=/app/.venv/bin/python3 container_path=/proc/38/root/app/.venv/bin/python3 pid=38 error=No such file or directory (os error 2). Binary paths in policy will be matched literally. If this binary is a symlink (e.g., /usr/bin/python3 -> python3.11), use the canonical path instead, or run with CAP_SYS_PTRACE.
2026-08-21T16:13:16.079Z WARN openshell_supervisor_network::opa: Cannot access container filesystem for symlink resolution: path=/app/.venv/bin/pip container_path=/proc/38/root/app/.venv/bin/pip pid=38 error=No such file or directory (os error 2). Binary paths in policy will be matched literally. If this binary is a symlink (e.g., /usr/bin/python3 -> python3.11), use the canonical path instead, or run with CAP_SYS_PTRACE.
Closing as obsolete after RFC 0012. The failure is specific to the retired Kubernetes combined topology, where the privileged supervisor ran in the workload container. Kubernetes now uses separate supervisor and workload pods through #2942/#3144, so adding CAP_SETPCAP to that removed topology would restore privilege to a design OpenShell no longer uses.
Activity
jiridanek commentedon Aug 21, 2026
Hello, I tried to reproduce on Kubernetes, but I did not manage to hit the same issue.
Given that you use
ocin your steps, could be this is not general Kubernetes issue, but it affects OpenShift specifically?I am on macOS, so I used a
lima-managed VM, like thisThen I did the necessary setup, I used rancher local-path-provisioner to get a necessary PVC:
(the
latesturl for agent-standbox did not work, so I put found the latest version string and used that)And I installed openshell with the
combinedtopologyWhich seems to have been applied
Then, trying to run a pod, the pod started just fine for me, baring some hiccup with slow network downloading sandbox pod; the cli timeouted, but the sandbox did come up
❯ kubectl get pod -n openshell -o yaml default--cap-setpcap-repro
The above includes
For the openshell-0 pod
❯ kubectl get pod -n openshell -o yaml openshell-0
That includes
I see no crashloops or any errors besides seemingly innocuous
johntmyers commentedon Sep 17, 2026