[pull] main from NVIDIA:main

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -600,6 +600,7 @@ openshell logs <sandbox-name>
 | Symptom | Likely cause | Check |
 |---|---|---|
 | `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs |
+| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart `gateway:docker`, `gateway:podman`, or `gateway:vm` so it re-detects the listener; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` |
 | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs |
 | Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route |
 | Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |

.agents/skills/helm-dev-environment/SKILL.md

@@ -26,6 +26,8 @@ mise run helm:k3s:create
 ```
 
 Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file.
+When the named cluster already exists, the task starts any stopped containers and refreshes
+same-named kubeconfig entries so a recreated load balancer's current API port takes effect.
 Also applies the upstream agent-sandbox CRDs/controller (pinned via `AGENT_SANDBOX_VERSION`
 in `tasks/scripts/helm-k3s-local.sh`, fetched from `github.com/kubernetes-sigs/agent-sandbox`
 releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace
@@ -118,8 +120,9 @@ Kubernetes compute-driver spans under their distinct service names, along with
 Agent Sandbox controller reconciliation spans linked through the Sandbox
 trace-context annotation. The same command exposes OTLP/gRPC on
 `http://127.0.0.1:4317` and, when deployed, the Kubernetes gateway on
-`http://127.0.0.1:8090`; the local `gateway`, `gateway:docker`, and `gateway:vm`
-tasks export to the collector endpoint automatically.
+`http://127.0.0.1:8090`. The local `gateway:docker`, `gateway:podman`, and
+`gateway:vm` tasks detect the collector listener at startup and enable trace
+export only while it is reachable.
 
 **HA test deploy** (two gateway replicas + external PostgreSQL Secret): uncomment
 `#- ci/values-high-availability.yaml` in `deploy/helm/openshell/skaffold.yaml`,

.agents/skills/sbom/SKILL.md

@@ -13,13 +13,28 @@ The OpenShell SBOM tooling produces source-tree CycloneDX JSON SBOMs using Syft,
 
 SBOMs are **release artifacts only** -- they are generated on demand and not committed to the repository. Output lands in `deploy/sbom/output/` (gitignored).
 
-Release Dev and Release Tag image builds separately embed cargo-auditable metadata in the staged gateway and supervisor binaries. This metadata describes the binary's Rust dependency graph and lets Syft discover Cargo packages from the binary itself. It is not a complete image SBOM and is not an OCI SBOM attestation; publishing such an attestation remains separate work.
+Pushed gateway and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, and Release Tag image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
 
 ## Prerequisites
 
 - `mise install` has been run (installs Syft and other tools)
 - The repository is checked out at the root
 
+## Inspecting an Image SBOM
+
+BuildKit uses its default Syft scanner and attaches one SPDX document per platform. Read one without pulling the image:
+
+```bash
+docker buildx imagetools inspect ghcr.io/nvidia/openshell/gateway:latest \
+  --format '{{ json (index .SBOM "linux/amd64").SPDX }}'
+```
+
+Validate the final attestation, requiring a Cargo package for an auditable image:
+
+```bash
+tasks/scripts/verify-image-sbom.sh ghcr.io/nvidia/openshell/gateway:latest --require-cargo
+```
+
 ## Inspecting an Auditable Image Binary
 
 Opt into auditable metadata when staging a local image binary:

.github/actions/build-docker-image/action.yml

@@ -0,0 +1,75 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+name: Build Docker Image
+description: Build and push one architecture of an OpenShell image
+
+inputs:
+  component:
+    description: Docker image and Dockerfile name
+    required: true
+  binary:
+    description: Binary staged in the Docker build context
+    required: true
+  triple:
+    description: Binary artifact target triple
+    required: true
+  arch:
+    description: Docker architecture name
+    required: true
+  platform:
+    description: Docker platform
+    required: true
+  image-tag:
+    description: Docker image tag
+    required: true
+  github-token:
+    description: Token used to push the image
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - uses: ./.github/actions/setup-buildx
+      with:
+        buildkitd-config: /etc/buildkit/buildkitd.toml
+
+    - name: Log in to GHCR
+      shell: bash
+      run: echo "${INPUTS_GITHUB_TOKEN}" | docker login ghcr.io -u "${GITHUB_ACTOR}" --password-stdin
+      env:
+        INPUTS_GITHUB_TOKEN: ${{ inputs.github-token }}
+
+    - name: Download ${{ inputs.binary }}
+      uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+      with:
+        name: ${{ inputs.binary }}-${{ inputs.triple }}
+        path: artifact
+
+    - name: Stage ${{ inputs.binary }}
+      shell: bash
+      run: install -Dm0755 artifact/${INPUTS_BINARY} deploy/docker/.build/prebuilt-binaries/${INPUTS_ARCH}/${INPUTS_BINARY}
+      env:
+        INPUTS_BINARY: ${{ inputs.binary }}
+        INPUTS_ARCH: ${{ inputs.arch }}
+
+    - name: Build ${{ inputs.component }} image
+      shell: bash
+      env:
+        IMAGE_TAG: ${{ inputs.image-tag }}
+        INPUTS_PLATFORM: ${{ inputs.platform }}
+        INPUTS_COMPONENT: ${{ inputs.component }}
+        INPUTS_ARCH: ${{ inputs.arch }}
+      run: |
+        docker buildx build \
+          --builder openshell \
+          --platform ${INPUTS_PLATFORM} \
+          --file deploy/docker/Dockerfile.${INPUTS_COMPONENT} \
+          --target ${INPUTS_COMPONENT} \
+          --tag ghcr.io/nvidia/openshell/${INPUTS_COMPONENT}:${IMAGE_TAG}-${INPUTS_ARCH} \
+          --cache-from type=gha,scope=${INPUTS_COMPONENT}-${INPUTS_ARCH} \
+          --cache-to type=gha,mode=max,scope=${INPUTS_COMPONENT}-${INPUTS_ARCH} \
+          --provenance=mode=min \
+          --attest type=sbom \
+          --output type=image,push=true,oci-mediatypes=true,oci-artifact=true \
+          .

.github/actions/build-rust-binary/action.yml

@@ -0,0 +1,113 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+name: Build Rust Binary
+description: Build, verify, package, and upload an auditable Rust binary
+
+inputs:
+  package:
+    description: Cargo package that owns the binary
+    required: true
+  binary:
+    description: Cargo binary name
+    required: true
+  triple:
+    description: Rust target triple
+    required: true
+  dev-shell:
+    description: Nix development shell used to build the binary
+    required: true
+  cargo-version:
+    description: Cargo package version embedded in the binary
+    required: true
+  image-tag:
+    description: Default supervisor image tag embedded in the binary
+    required: false
+    default: ""
+  artifact-name:
+    description: GitHub artifact name
+    required: false
+    default: ""
+  extra-cargo-flags:
+    description: Additional flags passed to cargo build
+    required: false
+    default: ""
+  interpreter:
+    description: ELF interpreter for a dynamically linked Linux binary
+    required: false
+    default: ""
+
+runs:
+  using: composite
+  steps:
+    - name: Hash development shell
+      id: dev-shell
+      shell: bash
+      env:
+        DEV_SHELL: ${{ inputs.dev-shell }}
+      run: echo "hash=$(nix hash file --type sha256 --base16 "$(nix eval --raw "${DEV_SHELL}.drvPath")")" >> "$GITHUB_OUTPUT"
+
+    - name: Cache Rust artifacts
+      uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
+      with:
+        shared-key: binaries-${{ inputs.binary }}-${{ inputs.triple }}-${{ steps.dev-shell.outputs.hash }}
+        cache-on-failure: "true"
+        cache-workspace-crates: "true"
+        cache-bin: "false"
+        cmd-format: nix develop ${{ inputs.dev-shell }} -c {0}
+
+    - name: Set version
+      shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0}
+      env:
+        INPUTS_CARGO_VERSION: ${{ inputs.cargo-version }}
+      run: sed -i "s/^version = \"0\\.0\\.0\"$/version = \"${INPUTS_CARGO_VERSION}\"/" Cargo.toml
+
+    - name: Build ${{ inputs.binary }}
+      shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0}
+      env:
+        OPENSHELL_IMAGE_TAG: ${{ inputs.image-tag }}
+        INPUTS_PACKAGE: ${{ inputs.package }}
+        INPUTS_BINARY: ${{ inputs.binary }}
+        INPUTS_EXTRA_CARGO_FLAGS: ${{ inputs.extra-cargo-flags }}
+      run: GIT_DIR=/nonexistent cargo auditable build --release --package "${INPUTS_PACKAGE}" --bin "${INPUTS_BINARY}" ${INPUTS_EXTRA_CARGO_FLAGS}
+
+    - name: Verify ${{ inputs.binary }}
+      shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0}
+      env:
+        INPUTS_BINARY: ${{ inputs.binary }}
+      run: |
+        # Confirm the binary runs and reports the expected name.
+        target/release/${INPUTS_BINARY} --version | grep -q "^${INPUTS_BINARY} "
+        # Confirm Syft can decode the embedded cargo-auditable metadata.
+        SYFT_CHECK_FOR_APP_UPDATE=false syft file:target/release/${INPUTS_BINARY} -o cyclonedx-json | grep 'pkg:cargo/' > /dev/null
+
+    - name: Verify static linkage
+      if: endsWith(inputs.triple, '-linux-musl')
+      shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0}
+      env:
+        INPUTS_BINARY: ${{ inputs.binary }}
+      run: tasks/scripts/verify-static-binary.sh target/release/${INPUTS_BINARY}
+
+    - name: Normalize Linux dynamic binary
+      if: endsWith(inputs.triple, '-linux-gnu')
+      shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0}
+      env:
+        INPUTS_BINARY: ${{ inputs.binary }}
+        INPUTS_INTERPRETER: ${{ inputs.interpreter }}
+      run: |
+        binary=target/release/${INPUTS_BINARY}
+        # Remove Nix store paths so the binary can run on other distributions.
+        patchelf --set-interpreter "${INPUTS_INTERPRETER}" --remove-rpath "$binary"
+        # Z3 must be embedded instead of loaded from the target system.
+        test -z "$(patchelf --print-needed "$binary" | grep '^libz3')"
+        # Reject symbols introduced after glibc 2.28.
+        tasks/scripts/verify-glibc-symbols.sh 2.28 "$binary"
+
+    - name: Upload ${{ inputs.binary }}
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+      with:
+        name: ${{ inputs.artifact-name || format('{0}-{1}', inputs.binary, inputs.triple) }}
+        path: target/release/${{ inputs.binary }}
+        compression-level: 0
+        retention-days: 5
+        if-no-files-found: error

.github/actions/check-job-results/action.yml

@@ -0,0 +1,33 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+name: Check job results
+description: Fail when any required upstream job did not succeed
+
+inputs:
+  results:
+    description: JSON-encoded GitHub Actions needs context
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - name: Check required jobs
+      shell: bash
+      env:
+        JOB_RESULTS: ${{ inputs.results }}
+      run: |
+        set -euo pipefail
+        failures="$(
+          jq -r '
+            to_entries[]
+            | select(.value.result != "success")
+            | "\(.key) concluded \(.value.result)"
+          ' <<< "$JOB_RESULTS"
+        )"
+        if [ -n "$failures" ]; then
+          while IFS= read -r failure; do
+            echo "::error::$failure"
+          done <<< "$failures"
+          exit 1
+        fi

.github/actions/setup-e2e-cli/action.yml

@@ -1,29 +1,19 @@
 name: Setup E2E CLI
 description: Download an architecture-matched prebuilt OpenShell CLI for E2E tests
 
-inputs:
-  artifact-prefix:
-    description: Artifact name prefix; linux-<arch> is appended automatically
-    required: true
-
 runs:
   using: composite
   steps:
     - name: Download prebuilt CLI
       uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
       with:
-        name: ${{ format('{0}-linux-{1}', inputs.artifact-prefix, runner.arch == 'X64' && 'amd64' || 'arm64') }}
+        name: ${{ runner.arch == 'X64' && 'openshell-x86_64-unknown-linux-musl' || 'openshell-aarch64-unknown-linux-musl' }}
         path: .e2e/prebuilt-cli
 
     - name: Configure prebuilt CLI
       shell: bash
       run: |
-        set -euo pipefail
         cli="$GITHUB_WORKSPACE/.e2e/prebuilt-cli/openshell"
-        if [[ ! -f "$cli" ]]; then
-          echo "downloaded artifact is missing $cli" >&2
-          exit 1
-        fi
         chmod +x "$cli"
         "$cli" --version
         echo "OPENSHELL_BIN=$cli" >> "$GITHUB_ENV"

.github/actions/setup-e2e-driver/action.yml

@@ -0,0 +1,33 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+name: Setup E2E Driver
+description: Download an architecture-matched standalone compute driver for E2E tests
+
+inputs:
+  binary:
+    description: Compute driver binary name
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - name: Download prebuilt ${{ inputs.binary }}
+      uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+      with:
+        name: ${{ inputs.binary }}-${{ runner.arch == 'X64' && 'x86_64-unknown-linux-gnu' || 'aarch64-unknown-linux-gnu' }}
+        path: .e2e/prebuilt-driver
+
+    - name: Configure prebuilt ${{ inputs.binary }}
+      shell: bash
+      run: | # zizmor: ignore[github-env] validated filename under the trusted workspace path
+        if [[ ! "${INPUTS_BINARY}" =~ ^[a-zA-Z0-9._-]+$ ]]; then
+          echo "invalid driver binary name: ${INPUTS_BINARY}" >&2
+          exit 1
+        fi
+        driver="$GITHUB_WORKSPACE/.e2e/prebuilt-driver/${INPUTS_BINARY}"
+        chmod +x "$driver"
+        "$driver" --version
+        echo "OPENSHELL_EXTERNAL_DRIVER_BIN=$driver" >> "$GITHUB_ENV"
+      env:
+        INPUTS_BINARY: ${{ inputs.binary }}

.github/actions/setup-e2e-gateway/action.yml

@@ -2,28 +2,24 @@ name: Setup E2E Gateway
 description: Download an architecture-matched prebuilt OpenShell gateway for E2E tests
 
 inputs:
-  artifact-prefix:
-    description: Artifact name prefix; linux-<arch> is appended automatically
-    required: true
+  artifact-name:
+    description: GitHub artifact name
+    required: false
+    default: ""
 
 runs:
   using: composite
   steps:
     - name: Download prebuilt gateway
       uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
       with:
-        name: ${{ format('{0}-linux-{1}', inputs.artifact-prefix, runner.arch == 'X64' && 'amd64' || 'arm64') }}
+        name: ${{ inputs.artifact-name || (runner.arch == 'X64' && 'openshell-gateway-x86_64-unknown-linux-gnu' || 'openshell-gateway-aarch64-unknown-linux-gnu') }}
         path: .e2e/prebuilt-gateway
 
     - name: Configure prebuilt gateway
       shell: bash
       run: |
-        set -euo pipefail
         gateway="$GITHUB_WORKSPACE/.e2e/prebuilt-gateway/openshell-gateway"
-        if [[ ! -f "$gateway" ]]; then
-          echo "downloaded artifact is missing $gateway" >&2
-          exit 1
-        fi
         chmod +x "$gateway"
         "$gateway" --version
         echo "OPENSHELL_GATEWAY_BIN=$gateway" >> "$GITHUB_ENV"