bug: Podman driver rejects supplementary groups added by a CDI device at sandbox startup #5305

User Story

I use OpenShell with the Podman driver to run GPU workloads through CDI devices. I directly hit a sandbox startup failure when a CDI device adds a supplementary group, and I need that to work (or to be documented as unsupported) so that CDI-provided device access behaves the same on Podman as on Docker.

Problem Statement

With the Podman driver, a sandbox whose CDI device adds a supplementary group (CDI containerEdits.additionalGids) fails to start. The sandbox rejects the extra group. The Docker driver accepts the same scenario.

It is not clear from the code alone whether the Podman behavior is an omission or intended. Maintainers, please confirm which it is.

Impact / Why This Matters

CDI devices commonly grant access through a device group. On Podman with the default user namespace mode, such a device makes the sandbox refuse to start. The CDI test suite (tests/suites/features/cdi, PR #4461, related issue #4460) therefore cannot cover this scenario on Podman, and the scenario is being removed from PR #4461 until the driver is fixed. Related: PR #4204 makes the sandbox process CDI additional GIDs; its test table says that with #4204 "the generated supplementary GID permits non-root access".

Observed Behavior

Environment: rootful Podman, tmachine environment fedora-podman-rootful-cdi, rpm installer, default Podman userns mode.

openshell sandbox create --gpu 1 \
  --driver-config-json '{"podman":{"cdi_devices":["tmachine.example/cdi-scenario=device-node-with-gid"]}}'

Result: sandbox entered error phase while provisioning: ContainerExited: Container exited with code 1. The container log says:

Error: x sandbox supplementary groups [991] do not match resolved workload []

The same scenario passes on the Docker driver (21 of 21 CDI tests pass on ubuntu-docker-rootful-cdi with the deb installer).

The engine and CDI spec are fine. Plain sudo podman run --rm --device=tmachine.example/cdi-scenario=device-node-with-gid fedora ls -l /dev/tmachine-cdi-group-null prints crw-rw----. 1 root 991 1, 3 ... /dev/tmachine-cdi-group-null, so the node and the group edit work. The failure is inside the OpenShell sandbox check.

Cause

In crates/openshell-sandbox/src/boundary_server.rs, validate_running_identity compares the process's supplementary groups with the workload identity the driver resolved, through supplementary_groups_match(actual, expected, allow_extra). It requires equality unless allows_runtime_supplementary_groups(config) is true. That holds when the resource claim GPU_RESOURCE_CLAIM (openshell.gpu) is "true" or ALLOW_EXTRA_SUPPLEMENTARY_GROUPS_RESOURCE_CLAIM is "true".

• The Docker driver sets the GPU claim whenever a GPU is requested (crates/openshell-driver-docker/src/isolation.rs, provision).
• The Podman driver sets only the extra-groups claim, and only when crate::isolation::userns_preserves_host_groups(self.config.userns.as_deref()) is true (crates/openshell-driver-podman/src/isolation.rs, crates/openshell-driver-podman/src/driver.rs).

With the default userns mode, a CDI-added group therefore makes the Podman sandbox refuse to start.

Proposed Design

The Podman driver needs a defined answer for groups added at runtime by a GPU or CDI device request. Either:

• set the extra-groups claim when a GPU or CDI device request is present, as Docker does; or
• document that runtime-added supplementary groups are unsupported on Podman.

Acceptance Criteria

• A Podman sandbox started with a CDI device that adds a supplementary group either starts, with the group visible to the workload, or fails with a documented, intentional error.
• The device-node-with-gid scenario below is restored to the CDI suite for the Podman driver once the behavior is settled.

Alternatives Considered

Requiring userns=keep-id for CDI groups on Podman. That changes user namespace behavior for all GPU sandboxes and was not evaluated.

Scenario to restore

Original definition in the tmachine cdi_fixtures role spec (vars/scenarios.yaml; also in commit fb39ac14b of PR #4461's old head). The role also creates the group (openshell-cdi-fixture, system group) and creates the node with mknod as a character device with major 1, minor 3. The recorded expectations describe Docker today. On Podman the scenario is currently expected to fail at startup with the error above until the driver gap is closed.

# The device is mode 0660 with a group that only `additionalGids` grants.
- name: device-node-with-gid
  fixtures:
    - {path: group-device, type: chardev, major: 1, minor: 3, mode: "0660", group: openshell-cdi-fixture}
  edits:
    additionalGids: ["{{ cdi_fixture_group.gid | int }}"]
    deviceNodes:
      - path: /dev/tmachine-cdi-group-null
        hostPath: "{{ cdi_scenario_dir }}/group-device"
        permissions: rw
        uid: 0
        gid: "{{ cdi_fixture_group.gid | int }}"
        fileMode: 432
  expect:
    checks:
      - {type: non-root}
      - {type: group-of, path: /dev/tmachine-cdi-group-null}
      - {type: write, path: /dev/tmachine-cdi-group-null, expect: denied}

Related observation

Under rootless Podman, a CDI deviceNodes entry whose container path differs from its hostPath fails with crun: mount '/dev/tmachine-cdi-null' to '/dev/tmachine-cdi-null': No such file or directory, because the rootless runtime appears to bind-mount the host device at the same path. This is probably a runtime limitation, not an OpenShell bug. It is inferred from one observed error and not verified in the runtime's source. It is mentioned only so that anyone adding rootless Podman coverage knows about it.