diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index e866ec6927..56061385c3 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -266,8 +266,9 @@ task running while using those endpoints. ### Viewing local traces The gateway exports OTLP/gRPC to -`http://openshell-collector.observability.svc.cluster.local:4317` through the -default Skaffold values. Forward OTLP/gRPC and the trace UI to the host: +`http://openshell-collector.observability.svc.cluster.local:4317` through +`ci/values-local-tracing.yaml`, which only `skaffold.yaml` layers. The Kubernetes +E2E wrapper omits it, so E2E clusters export no traces. Forward OTLP/gRPC and the trace UI to the host: ```bash mise run helm:k3s:forward @@ -278,9 +279,10 @@ Kubernetes compute-driver spans under their distinct service names, along with Agent Sandbox controller reconciliation spans linked through the Sandbox trace-context annotation. The same command exposes OTLP/gRPC on `http://127.0.0.1:4317` and, when deployed, the Kubernetes gateway on -`http://127.0.0.1:8090`. The local `gateway:docker`, `gateway:podman`, and -`gateway:vm` tasks detect the collector listener at startup and enable trace -export, OCSF log export, and the agent trace relay only while it is reachable. +`http://127.0.0.1:8090`. The local `gateway`, `gateway:docker`, +`gateway:podman`, and `gateway:vm` tasks enable trace export, OCSF log export, +and the agent trace relay only when `OPENSHELL_DEV_OTLP_ENDPOINT` is set, for +example `OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317`. Spans an agent exports from inside a sandbox arrive through the supervisor's relay with `openshell.telemetry.source=agent` and share `openshell.sandbox.id` with that sandbox's supervisor spans. @@ -642,6 +644,7 @@ for dependencies still declared in `Chart.yaml`. | `deploy/helm/openshell/skaffold.yaml` | Skaffold config — images, Helm releases, values overlays | | `deploy/helm/openshell/values.yaml` | Default Helm values | | `deploy/helm/openshell/ci/values-skaffold.yaml` | Dev overrides (image pull policy, TLS disabled for local Skaffold) | +| `deploy/helm/openshell/ci/values-local-tracing.yaml` | OTLP export to the `helm:k3s:create` collector (Skaffold only) | | `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) | | `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay | | `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) | diff --git a/.agents/skills/release-captain/SKILL.md b/.agents/skills/release-captain/SKILL.md index 5950a9b08c..6395ccc759 100644 --- a/.agents/skills/release-captain/SKILL.md +++ b/.agents/skills/release-captain/SKILL.md @@ -169,8 +169,7 @@ git worktree remove --force "$wt" ``` 4. **Watch and verify.** Check that `Select Qualified Prerelease` names the - approved candidate, the GitHub Release has assets, no job failed, and the - wheel reached PyPI: + approved candidate, the GitHub Release has assets and no job failed: ```shell gh run watch --exit-status diff --git a/.agents/skills/release-openshell/SKILL.md b/.agents/skills/release-openshell/SKILL.md index aca2a98f63..303e9d5c4b 100644 --- a/.agents/skills/release-openshell/SKILL.md +++ b/.agents/skills/release-openshell/SKILL.md @@ -33,8 +33,8 @@ Release creates `vX.Y.Z` at that exact commit, then rebuilds and publishes all stable artifacts without rerunning qualification. Embedded versions change, so stable artifacts are not byte-identical to the qualified prerelease. An existing stable tag may be retried only if it already points to the selected commit. A retry -publishes only missing artifacts; published images, charts, release assets, npm, -and PyPI versions are kept, while the snap is uploaded again. +publishes only missing artifacts; published images, charts, release assets, and +npm versions are kept, while the snap is uploaded again. Stable images are saved as OCI archives during the build and pushed under the stable version tags during publication. Their SBOMs are verified before updating `latest`; the qualified prerelease's commit-SHA image tags remain unchanged. @@ -84,17 +84,6 @@ immutable version and advance latest only for a newer version. Watch the stable release run and report its result. -## PyPI publishing +## Python SDK -Stable releases publish the Python wheel to PyPI from `wheel-publish.yml`, which the `publish-pypi` job in `release-publish.yml` calls after the GitHub Release is created. The workflow downloads the wheel from the stable GitHub Release, not a run's build artifact, and checks the stable tag's source commit and the wheel's release checksum. It uses Trusted Publishing with the `pypi` GitHub environment. PyPI matches the workflow file that contains the publishing job, so configure the `openshell` PyPI project's GitHub publisher with owner `NVIDIA`, repository `OpenShell`, workflow `wheel-publish.yml`, and environment `pypi`. No PyPI API token or GitLab trigger credentials are required. - -The environment must allow the branch used to dispatch Release or `wheel-publish.yml`. For a support release, explicitly allow its support branch before dispatching. Required reviewers, if configured, must approve the PyPI job. An existing wheel on PyPI is skipped; PyPI does not allow replacing a published wheel. - -If a stable release did not reach PyPI, publish the wheel without rerunning Release: - -```shell -gh workflow run wheel-publish.yml -R NVIDIA/OpenShell --ref main \ - -f release-tag=vX.Y.Z -f source-sha= -f python-version=X.Y.Z -``` - -Development and prerelease wheels remain GitHub Release assets and do not publish to PyPI. +Core no longer builds or publishes the Python SDK for releases. The stable, prerelease, and rolling `dev` GitHub Releases carry no wheel, and nothing in core publishes to PyPI. `openshell` `0.1.7` is the last core-built wheel. From `0.1.8` on, [ai-openshell/sdk-python](https://github.com/ai-openshell/sdk-python) publishes `openshell` on the same day as each stable core release, once its update PR for the core tag merges. diff --git a/.github/workflows/branch-checks.yml b/.github/workflows/branch-checks.yml index db22a74913..004e181b2c 100644 --- a/.github/workflows/branch-checks.yml +++ b/.github/workflows/branch-checks.yml @@ -598,6 +598,9 @@ jobs: - name: Test run: mise run test:python + - name: Test Python scripts + run: mise run test:python-scripts + - name: Test local gateway configuration helpers run: | bash tasks/scripts/test-gateway-pull-policy.sh @@ -638,6 +641,41 @@ jobs: echo "$changed" exit 1 + python-sdk-freeze: + # The Python SDK moved to ai-openshell/sdk-python (#5287), which imported it + # at v0.1.7. Its files stay only until core removes them, so PRs must not + # add or change them. Deleting files is allowed (the removal PR). The paths + # are the ones the import moved: all of python/ plus the two SDK build + # scripts. The shared root pyproject.toml and uv.lock stay editable. + # Merge queue runs skip this job; the PR already passed it on its mirror. + name: Python SDK Freeze + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' && github.event_name == 'push' + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Reject changes to the Python SDK + env: + BASE_SHA: ${{ needs.pr_metadata.outputs.base_sha }} + run: | + set -euo pipefail + changed=$(git diff --name-only --diff-filter=d "${BASE_SHA}...HEAD" -- \ + python \ + tasks/scripts/generate_python_proto.py) + if [ -z "$changed" ]; then + echo "No added or changed Python SDK files." + exit 0 + fi + echo "::error title=Python SDK moved::The Python SDK moved to https://github.com/ai-openshell/sdk-python. Drop the Python SDK changes from this PR and open SDK changes there. Regenerated bindings are not needed here; sdk-python regenerates them from core releases." + echo "Changed Python SDK files:" + echo "$changed" + exit 1 + markdown: name: Markdown needs: pr_metadata @@ -687,3 +725,19 @@ jobs: run: | OPENSHELL_NPM_VERSION="$(uv run python tasks/scripts/release.py get-version --npm)" \ mise run sdk:ts:publish + + install-script: + name: install.sh + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + # The RPM lookup test uses gawk when present; gawk warns on bad escapes. + - name: Install gawk + run: sudo apt-get install -y --no-install-recommends gawk + + - name: Test install.sh + run: tasks/scripts/test-install-sh.sh diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 0f59ab2a81..ceb62b860d 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -101,11 +101,21 @@ jobs: echo "rpm_release=$rpm_release" } >> "$GITHUB_OUTPUT" - build-binaries: + build-binaries-linux: needs: version permissions: contents: read - uses: ./.github/workflows/build-binaries.yml + uses: ./.github/workflows/build-binaries-linux.yml + with: + cargo-version: ${{ needs.version.outputs.cargo }} + secrets: + CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} + + build-binaries-macos: + needs: version + permissions: + contents: read + uses: ./.github/workflows/build-binaries-macos.yml with: cargo-version: ${{ needs.version.outputs.cargo }} secrets: @@ -146,7 +156,7 @@ jobs: # The conformance program, built once per architecture for the host lanes. # It lives in its own workspace and is not released, so it is not part of - # build-binaries.yml. + # build-binaries-linux.yml. build-conformance: name: openshell-conformance (${{ matrix.triple }}) needs: [pr_metadata] @@ -198,6 +208,62 @@ jobs: retention-days: 5 if-no-files-found: error + # The test bundles of the tmachine E2E suites rows that don't run + # openshell-conformance, built once per run instead of in every row (see + # e2e-suites.yml). The rows only run on amd64, so only the x86_64 guest + # target is built. Each bundle holds its whole family; a row picks its + # package at run time. Rust Cache (rust-cache.yml) saves the cache on main; + # keep the env and build commands aligned with its e2e-bundles job. + build-e2e-bundles: + name: E2E suite bundles (x86_64-unknown-linux-musl) + needs: [pr_metadata] + if: needs.pr_metadata.outputs.run_integration == 'true' + permissions: + contents: read + runs-on: linux-amd64-cpu8 + timeout-minutes: 60 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + ref: ${{ github.sha }} + - uses: ./.github/actions/setup-nix + with: + prepare-shell: "true" + cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }} + - uses: ./.github/actions/setup-rust + with: + cache-key: e2e-bundles-v1-x86_64-unknown-linux-musl + - name: Build the bundles + env: + # One target directory for every family, so later families reuse + # testkit-ready and the dependencies the first one built. It is the + # root target directory, which setup-rust caches. + CARGO_TARGET_DIR: ${{ github.workspace }}/target + # Every row downloads the bundles. Line tables keep backtraces + # readable at a third of the size of full debuginfo. + CARGO_PROFILE_DEV_DEBUG: line-tables-only + run: | + set -euo pipefail + # Retry dependency preparation, then build once. + nix build --no-link .#openshell-test-runner || nix build --no-link .#openshell-test-runner + for family in features drivers/podman; do + nix run .#openshell-test-runner -- build \ + --family "$family" --out "artifacts/e2e-bundles/${family//\//-}" + done + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: openshell-e2e-bundle-features-x86_64-unknown-linux-musl + path: artifacts/e2e-bundles/features + retention-days: 5 + if-no-files-found: error + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: openshell-e2e-bundle-drivers-podman-x86_64-unknown-linux-musl + path: artifacts/e2e-bundles/drivers-podman + retention-days: 5 + if-no-files-found: error + build-external-drivers: name: ${{ matrix.binary }} (${{ matrix.triple }}) needs: [pr_metadata, version] @@ -237,19 +303,30 @@ jobs: triple: ${{ matrix.triple }} cargo-version: ${{ needs.version.outputs.cargo }} - build-vm-driver: - needs: [pr_metadata, version, build-binaries] + build-vm-driver-linux: + needs: [pr_metadata, version, build-binaries-linux] if: needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: contents: read - uses: ./.github/workflows/build-vm-driver.yml + uses: ./.github/workflows/build-vm-driver-linux.yml + with: + cargo-version: ${{ needs.version.outputs.cargo }} + secrets: + CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} + + build-vm-driver-macos: + needs: [pr_metadata, version, build-binaries-linux, build-binaries-macos] + if: needs.pr_metadata.outputs.run_core_e2e == 'true' + permissions: + contents: read + uses: ./.github/workflows/build-vm-driver-macos.yml with: cargo-version: ${{ needs.version.outputs.cargo }} secrets: CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} build-images: - needs: [pr_metadata, build-binaries] + needs: [pr_metadata, build-binaries-linux] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true' permissions: contents: read @@ -258,7 +335,7 @@ jobs: build-deb: name: Build Debian packages - needs: [pr_metadata, version, build-binaries, build-vm-driver] + needs: [pr_metadata, version, build-binaries-linux, build-vm-driver-linux] if: needs.pr_metadata.outputs.run_integration == 'true' permissions: contents: read @@ -270,7 +347,7 @@ jobs: build-rpm: name: Build RPM packages - needs: [pr_metadata, version, build-binaries] + needs: [pr_metadata, version, build-binaries-linux] if: needs.pr_metadata.outputs.run_integration == 'true' permissions: actions: read @@ -282,8 +359,48 @@ jobs: rpm-release: ${{ needs.version.outputs.rpm_release }} cargo-version: ${{ needs.version.outputs.cargo }} + # Test inputs don't depend on the packages, so build them while the packages + # build instead of in prepare-integration. + build-tmachine-test-inputs: + name: Build tmachine test inputs + needs: [pr_metadata] + if: needs.pr_metadata.outputs.run_integration == 'true' + permissions: + contents: read + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + ref: ${{ github.sha }} + - uses: ./.github/actions/setup-nix + - name: Build test archives + run: | + # Retry dependency preparation, then generate artifacts once. + nix build --no-link .#build-artifacts-test-archives || nix build --no-link .#build-artifacts-test-archives + nix run .#build-artifacts-test-archives + - name: Build test workload images + run: | + # Retry dependency preparation, then generate artifacts once. + nix build --no-link .#build-artifacts-test-images || nix build --no-link .#build-artifacts-test-images + nix run .#build-artifacts-test-images + - name: Package Helm chart + run: | + # Retry dependency preparation, then generate artifacts once. + nix build --no-link .#build-artifacts-helm || nix build --no-link .#build-artifacts-helm + nix run .#build-artifacts-helm + - name: Upload tmachine test inputs + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: openshell-tmachine-test-inputs + path: artifacts + compression-level: 0 + retention-days: 5 + if-no-files-found: error + prepare-integration: - needs: [pr_metadata, build-binaries, build-images, build-deb, build-rpm] + needs: [pr_metadata, build-binaries-linux, build-images, build-deb, build-rpm, build-tmachine-test-inputs] if: needs.pr_metadata.outputs.run_integration == 'true' permissions: actions: read @@ -293,10 +410,11 @@ jobs: with: deb-artifact-name: deb-linux-amd64 rpm-artifact-name: rpm-linux-x86_64 + test-inputs-artifact-name: openshell-tmachine-test-inputs # Run driver-independent conformance tests. conformance-integration: - needs: prepare-integration + needs: [prepare-integration, build-conformance] permissions: actions: read contents: read @@ -304,6 +422,7 @@ jobs: uses: ./.github/workflows/e2e-suites.yml with: family: conformance + prebuilt: true source-sha: ${{ needs.prepare-integration.outputs.source_sha }} integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }} secrets: @@ -312,7 +431,7 @@ jobs: # Run the testkit feature suites (tests/suites/features) on tmachine. The # environments and suites are listed in e2e-suites.yml. feature-specific-integration: - needs: prepare-integration + needs: [prepare-integration, build-e2e-bundles] permissions: actions: read contents: read @@ -320,6 +439,7 @@ jobs: uses: ./.github/workflows/e2e-suites.yml with: family: features + prebuilt: true source-sha: ${{ needs.prepare-integration.outputs.source_sha }} integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }} secrets: @@ -328,7 +448,7 @@ jobs: # Run the testkit driver suites (tests/suites/drivers) on tmachine. The # environments and suites are listed in e2e-suites.yml. driver-specific-integration: - needs: prepare-integration + needs: [prepare-integration, build-e2e-bundles] permissions: actions: read contents: read @@ -336,13 +456,13 @@ jobs: uses: ./.github/workflows/e2e-suites.yml with: family: drivers + prebuilt: true source-sha: ${{ needs.prepare-integration.outputs.source_sha }} integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }} secrets: CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} - # Run the Podman e2e suite on tmachine. It needs the binary installer's CLI - # path and local HTTP gateway. + # Run the Podman e2e suite on tmachine against the packaged gateway. podman-e2e-integration: needs: prepare-integration permissions: @@ -356,11 +476,11 @@ jobs: integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }} test-matrix: >- [ - {"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"} + {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"e2e-podman"} ] docker-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -373,7 +493,7 @@ jobs: runner: linux-arm64-cpu8 podman-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -387,7 +507,7 @@ jobs: [{"suite":"ci","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:ci"}] podman-additional-ca-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -401,7 +521,7 @@ jobs: [{"suite":"additional-ca","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"OPENSHELL_E2E_PODMAN_TEST=additional_ca mise run --no-deps --skip-deps e2e:podman"}] vm-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-vm-driver] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-vm-driver-linux] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -412,7 +532,7 @@ jobs: prebuilt-conformance: true vm-additional-ca-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-vm-driver] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-vm-driver-linux] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -425,7 +545,7 @@ jobs: test-name: additional_ca docker-external-driver-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-gateway-plain, build-external-drivers, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-gateway-plain, build-external-drivers, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -442,7 +562,7 @@ jobs: [{"suite":"external-driver","cmd":"mise run --no-deps --skip-deps e2e:docker:external-driver","apt_packages":"openssh-client","python_proto":false,"mcp":false}] podman-external-driver-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-gateway-plain, build-external-drivers, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-gateway-plain, build-external-drivers, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -458,7 +578,7 @@ jobs: [{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver"}] vm-external-driver-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-gateway-plain, build-vm-driver] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-gateway-plain, build-vm-driver-linux] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -472,7 +592,7 @@ jobs: e2e-task: e2e:vm:external-driver gpu-e2e: - needs: [pr_metadata, build-binaries, build-images] + needs: [pr_metadata, build-binaries-linux, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true' permissions: actions: read @@ -483,7 +603,7 @@ jobs: image-tag: ${{ github.sha }} kubernetes-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' strategy: fail-fast: false @@ -514,7 +634,7 @@ jobs: test-name: ${{ matrix.test_name }} kubernetes-app-listener-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' strategy: fail-fast: false @@ -552,7 +672,7 @@ jobs: listener-takeover: ${{ matrix.takeover }} kubernetes-workspace-managed-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -566,7 +686,7 @@ jobs: e2e-task: e2e:kubernetes:workspace-managed kubernetes-external-driver-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-gateway-plain, build-external-drivers, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-gateway-plain, build-external-drivers, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -583,7 +703,7 @@ jobs: cluster-images: sandbox supervisor kubernetes-workspace-operator-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -597,7 +717,7 @@ jobs: e2e-task: e2e:kubernetes:workspace-operator kubernetes-ha-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true' permissions: actions: read @@ -615,7 +735,7 @@ jobs: use-envoy-gateway: true kubernetes-credential-drivers-e2e: - needs: [pr_metadata, build-binaries, build-conformance, build-images] + needs: [pr_metadata, build-binaries-linux, build-conformance, build-images] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true' permissions: actions: read @@ -632,6 +752,8 @@ jobs: name: Core E2E result needs: - pr_metadata + - build-binaries-macos + - build-vm-driver-macos - conformance-integration - feature-specific-integration - driver-specific-integration @@ -661,7 +783,7 @@ jobs: gpu-e2e-result: name: GPU E2E result - needs: [pr_metadata, gpu-e2e] + needs: [pr_metadata, build-binaries-macos, gpu-e2e] if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true' runs-on: ubuntu-latest steps: @@ -674,7 +796,7 @@ jobs: kubernetes-ha-e2e-result: name: Kubernetes HA E2E result - needs: [pr_metadata, kubernetes-ha-e2e] + needs: [pr_metadata, build-binaries-macos, kubernetes-ha-e2e] if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true' runs-on: ubuntu-latest steps: @@ -687,7 +809,7 @@ jobs: kubernetes-credential-drivers-e2e-result: name: Kubernetes Credential Drivers E2E result - needs: [pr_metadata, kubernetes-credential-drivers-e2e] + needs: [pr_metadata, build-binaries-macos, kubernetes-credential-drivers-e2e] if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true' runs-on: ubuntu-latest steps: diff --git a/.github/workflows/build-binaries.yml b/.github/workflows/build-binaries-linux.yml similarity index 90% rename from .github/workflows/build-binaries.yml rename to .github/workflows/build-binaries-linux.yml index 1289968d14..1adf9be0e2 100644 --- a/.github/workflows/build-binaries.yml +++ b/.github/workflows/build-binaries-linux.yml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -name: Build Binaries +name: Build Linux Binaries on: workflow_call: @@ -40,8 +40,6 @@ jobs: runner: linux-amd64-cpu8 - triple: aarch64-unknown-linux-musl runner: linux-arm64-cpu8 - - triple: aarch64-apple-darwin - runner: macos-15-xlarge runs-on: ${{ matrix.runner }} timeout-minutes: 60 steps: &build_steps @@ -74,8 +72,6 @@ jobs: runner: linux-amd64-cpu8 - triple: aarch64-unknown-linux-musl runner: linux-arm64-cpu8 - - triple: aarch64-apple-darwin - runner: macos-15-xlarge runs-on: ${{ matrix.runner }} timeout-minutes: 60 steps: *build_steps @@ -92,8 +88,6 @@ jobs: runner: linux-amd64-cpu8 - triple: aarch64-unknown-linux-gnu runner: linux-arm64-cpu8 - - triple: aarch64-apple-darwin - runner: macos-15-xlarge runs-on: ${{ matrix.runner }} timeout-minutes: 60 steps: *build_steps @@ -122,10 +116,6 @@ jobs: binary: openshell-supervisor triple: aarch64-unknown-linux-gnu runner: linux-arm64-cpu8 - - package: openshell-supervisor - binary: openshell-supervisor - triple: aarch64-apple-darwin - runner: macos-15-xlarge runs-on: ${{ matrix.runner }} timeout-minutes: 60 steps: *build_steps diff --git a/.github/workflows/build-binaries-macos.yml b/.github/workflows/build-binaries-macos.yml new file mode 100644 index 0000000000..e5f7d09648 --- /dev/null +++ b/.github/workflows/build-binaries-macos.yml @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Build macOS Binaries + +on: + workflow_call: + inputs: + cargo-version: + required: true + type: string + supervisor-image-tag: + required: false + type: string + default: "" + checkout-ref: + required: false + type: string + default: "" + secrets: + CACHIX_AUTH_TOKEN: + required: true + +permissions: + contents: read + +jobs: + build: + name: ${{ matrix.binary }} (${{ matrix.triple }}) + strategy: + fail-fast: false + matrix: + include: + - package: openshell-cli + binary: openshell + triple: aarch64-apple-darwin + - package: openshell-prover-cli + binary: openshell-prover + triple: aarch64-apple-darwin + - package: openshell-gateway + binary: openshell-gateway + triple: aarch64-apple-darwin + - package: openshell-supervisor + binary: openshell-supervisor + triple: aarch64-apple-darwin + runs-on: macos-15-xlarge + timeout-minutes: 60 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + ref: ${{ inputs.checkout-ref || github.sha }} + - uses: ./.github/actions/setup-nix + with: + prepare-shell: "true" + cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }} + - uses: ./.github/actions/build-rust-binary + with: + package: ${{ matrix.package }} + binary: ${{ matrix.binary }} + triple: ${{ matrix.triple }} + cargo-version: ${{ inputs.cargo-version }} + supervisor-image-tag: ${{ inputs.supervisor-image-tag }} diff --git a/.github/workflows/build-vm-driver.yml b/.github/workflows/build-vm-driver-linux.yml similarity index 77% rename from .github/workflows/build-vm-driver.yml rename to .github/workflows/build-vm-driver-linux.yml index a93f6928da..bdca89ce02 100644 --- a/.github/workflows/build-vm-driver.yml +++ b/.github/workflows/build-vm-driver-linux.yml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -name: Build VM Driver +name: Build Linux VM Driver on: workflow_call: @@ -36,10 +36,6 @@ jobs: - arch: aarch64 triple: aarch64-unknown-linux-gnu runner: linux-arm64-cpu8 - - arch: aarch64 - triple: aarch64-apple-darwin - runner: macos-15-xlarge - entitlements-file: crates/openshell-driver-vm/entitlements.plist runs-on: ${{ matrix.runner }} timeout-minutes: 60 defaults: @@ -63,19 +59,11 @@ jobs: path: sandbox - name: Download openshell-supervisor - if: endsWith(matrix.triple, '-unknown-linux-gnu') uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: openshell-supervisor-${{ matrix.arch }}-unknown-linux-gnu path: supervisor - - name: Download native macOS host supervisor - if: endsWith(matrix.triple, '-apple-darwin') - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: openshell-supervisor-aarch64-apple-darwin - path: host-supervisor - - name: Download VM guest init uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -90,14 +78,10 @@ jobs: - name: Assemble compressed VM runtime run: | compressed_dir="${RUNNER_TEMP}/vm-runtime-compressed" - host_supervisor="supervisor/openshell-supervisor" - if [[ "${{ matrix.triple }}" == *-apple-darwin ]]; then - host_supervisor="host-supervisor/openshell-supervisor" - fi install -d "$compressed_dir" cp result/compressed/*.zst "$compressed_dir/" zstd -19 -T1 sandbox/openshell-sandbox -o "$compressed_dir/openshell-sandbox.zst" - zstd -19 -T1 "$host_supervisor" -o "$compressed_dir/openshell-supervisor.zst" + zstd -19 -T1 supervisor/openshell-supervisor -o "$compressed_dir/openshell-supervisor.zst" zstd -19 -T1 vm-init/openshell-vm-init -o "$compressed_dir/openshell-vm-init.zst" - name: Build openshell-driver-vm @@ -110,4 +94,3 @@ jobs: triple: ${{ matrix.triple }} cargo-version: ${{ inputs['cargo-version'] }} supervisor-image-tag: ${{ inputs['supervisor-image-tag'] }} - entitlements-file: ${{ matrix.entitlements-file }} diff --git a/.github/workflows/build-vm-driver-macos.yml b/.github/workflows/build-vm-driver-macos.yml new file mode 100644 index 0000000000..b210014eb0 --- /dev/null +++ b/.github/workflows/build-vm-driver-macos.yml @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Build macOS VM Driver + +on: + workflow_call: + inputs: + cargo-version: + required: true + type: string + supervisor-image-tag: + required: false + type: string + default: "" + checkout-ref: + required: false + type: string + default: "" + secrets: + CACHIX_AUTH_TOKEN: + required: true + +permissions: + contents: read + +jobs: + build: + name: openshell-driver-vm (${{ matrix.triple }}) + strategy: + matrix: + include: + - arch: aarch64 + triple: aarch64-apple-darwin + runs-on: macos-15-xlarge + timeout-minutes: 60 + defaults: + run: + shell: nix develop -c bash -euo pipefail {0} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + ref: ${{ inputs['checkout-ref'] || github.sha }} + + - uses: ./.github/actions/setup-nix + with: + prepare-shell: "true" + cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }} + + - name: Download openshell-sandbox + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-sandbox-${{ matrix.arch }}-unknown-linux-musl + path: sandbox + + - name: Download openshell-supervisor + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-supervisor-${{ matrix.arch }}-apple-darwin + path: supervisor + + - name: Download VM guest init + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-vm-init-${{ matrix.arch }}-unknown-linux-musl + path: vm-init + + - name: Build VM runtime + run: | + # HTTP 416 is not retried by Nix; restart the build once on failure. + nix build .#vm-runtime || nix build .#vm-runtime + + - name: Assemble compressed VM runtime + run: | + compressed_dir="${RUNNER_TEMP}/vm-runtime-compressed" + install -d "$compressed_dir" + cp result/compressed/*.zst "$compressed_dir/" + zstd -19 -T1 sandbox/openshell-sandbox -o "$compressed_dir/openshell-sandbox.zst" + zstd -19 -T1 supervisor/openshell-supervisor -o "$compressed_dir/openshell-supervisor.zst" + zstd -19 -T1 vm-init/openshell-vm-init -o "$compressed_dir/openshell-vm-init.zst" + + - name: Build openshell-driver-vm + uses: ./.github/actions/build-rust-binary + env: + OPENSHELL_VM_RUNTIME_COMPRESSED_DIR: ${{ runner.temp }}/vm-runtime-compressed + with: + package: openshell-driver-vm + binary: openshell-driver-vm + triple: ${{ matrix.triple }} + cargo-version: ${{ inputs['cargo-version'] }} + supervisor-image-tag: ${{ inputs['supervisor-image-tag'] }} + entitlements-file: crates/openshell-driver-vm/entitlements.plist diff --git a/.github/workflows/e2e-suites.yml b/.github/workflows/e2e-suites.yml index 791761befb..7c995a198b 100644 --- a/.github/workflows/e2e-suites.yml +++ b/.github/workflows/e2e-suites.yml @@ -7,9 +7,11 @@ # optionally, the package in it, the gateway name the installer registers, and # the runner label for the environment's architecture. `openshell-test-runner` # builds the tests, boots the environment, runs them inside it and checks the -# results. The `family` input selects the rows of one family, including the -# workspaces under it (drivers selects drivers/podman); leave it empty to run -# every row. +# results. With `prebuilt`, the rows run what the caller built once for the +# run instead: conformance rows the static `openshell-conformance`, the other +# rows their family's test bundle. The `family` input selects the rows of one +# family, including the workspaces under it (drivers selects drivers/podman); +# leave it empty to run every row. name: E2E Suites @@ -29,6 +31,11 @@ on: description: Immutable ID of the prepared integration inputs artifact required: true type: string + prebuilt: + description: Run the rows with this run's prebuilt tests instead of building them in every job; conformance rows take the openshell-conformance-x86_64-unknown-linux-musl artifact, other rows their family's openshell-e2e-bundle--x86_64-unknown-linux-musl artifact + required: false + type: boolean + default: false secrets: CACHIX_AUTH_TOKEN: description: Token used to push Nix builds, such as the test runner, to Cachix @@ -84,14 +91,14 @@ jobs: {"environment":"ubuntu-k3s","installer":"k3s-ha","family":"features","package":"host-gateway","gateway":"tmachine","runner":"ubuntu-24.04"}, {"environment":"fedora-podman-rootful-keycloak","installer":"rpm","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"fedora-podman-rootless-keycloak","installer":"rpm","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootful-userns-default","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootful-userns-auto","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootful-userns-keep-id","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootful-userns-private","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootless-userns-default","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootless-userns-auto","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootless-userns-keep-id","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootless-userns-private","installer":"binaries","family":"drivers/podman","gateway":"tmachine","runner":"ubuntu-24.04"} + {"environment":"fedora-podman-rootful-userns-default","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootful-userns-auto","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootful-userns-keep-id","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootful-userns-private","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootless-userns-default","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootless-userns-auto","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootless-userns-keep-id","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootless-userns-private","installer":"rpm","family":"drivers/podman","gateway":"openshell","runner":"ubuntu-24.04"} ]' # `suite` names the row's tests in the job and artifact names. matrix=$(jq -c --arg family "${FAMILY}" '{include: map( @@ -99,6 +106,7 @@ jobs: | .package //= "" | .suite = (if .package == "" then .family else "\(.family)/\(.package)" end) | .artifact = ([.suite, .environment, .installer] | join("-") | gsub("/"; "-")) + | .bundle = "openshell-e2e-bundle-\(.family | gsub("/"; "-"))-x86_64-unknown-linux-musl" )}' <<< "$rows") if [ "$(jq '.include | length' <<< "$matrix")" -eq 0 ]; then echo "::error::No E2E suites match family '${FAMILY}'" @@ -161,26 +169,58 @@ jobs: artifact-ids: ${{ inputs['integration-inputs-artifact-id'] }} path: artifacts + # The program and the bundles are built once per run (see + # build-conformance and build-e2e-bundles in branch-e2e.yml), so the rows + # need not each build the suites. + - name: Download prebuilt openshell-conformance + if: inputs.prebuilt && matrix.family == 'conformance' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-conformance-x86_64-unknown-linux-musl + path: artifacts/prebuilt-conformance + + - name: Download prebuilt test bundle + if: inputs.prebuilt && matrix.family != 'conformance' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: ${{ matrix.bundle }} + path: artifacts/prebuilt-bundle + - name: Run the suites in this environment env: + PREBUILT: ${{ inputs.prebuilt }} ENVIRONMENT: ${{ matrix.environment }} INSTALLER: ${{ matrix.installer }} FAMILY: ${{ matrix.family }} PACKAGE: ${{ matrix.package }} GATEWAY: ${{ matrix.gateway }} run: | - # Retry dependency preparation, then run the suites once. The runner - # builds the suites for the guest, so it needs no prepared archives. + # Retry dependency preparation, then run the suites once. Without + # prebuilt tests, the runner builds the suites for the guest, so it + # needs no prepared archives. nix build --no-link .#openshell-test-runner || nix build --no-link .#openshell-test-runner - packages=() - if [ -n "${PACKAGE}" ]; then - packages=(--package "${PACKAGE}") + # A prebuilt program replaces the build, and so the package choice. + # A prebuilt bundle holds the whole family; the package selects the + # tests to run from it. + selection=() + if [ "${PREBUILT}" = "true" ] && [ "${FAMILY}" = "conformance" ]; then + program="artifacts/prebuilt-conformance/openshell-conformance" + chmod +x "${program}" + "${program}" list > /dev/null + selection=(--program "${program}") + else + if [ "${PREBUILT}" = "true" ]; then + selection=(--bundle artifacts/prebuilt-bundle) + fi + if [ -n "${PACKAGE}" ]; then + selection+=(--package "${PACKAGE}") + fi fi - nix run .#openshell-test-runner -- \ + nix run .#openshell-test-runner -- run \ --environment "${ENVIRONMENT}" \ --installer "${INSTALLER}" \ --family "${FAMILY}" \ - "${packages[@]}" \ + "${selection[@]}" \ --gateway "${GATEWAY}" \ --results artifacts/test-results diff --git a/.github/workflows/integration-runner.yml b/.github/workflows/integration-runner.yml index 906a60e0cb..f84811bdd9 100644 --- a/.github/workflows/integration-runner.yml +++ b/.github/workflows/integration-runner.yml @@ -24,7 +24,7 @@ on: type: string default: >- [ - {"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"} + {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"e2e-podman"} ] permissions: diff --git a/.github/workflows/integration-test.yml b/.github/workflows/integration-test.yml index 5efe0bd4e1..961e7b3c22 100644 --- a/.github/workflows/integration-test.yml +++ b/.github/workflows/integration-test.yml @@ -27,7 +27,7 @@ on: type: string default: >- [ - {"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"e2e-podman"} + {"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"e2e-podman"} ] permissions: diff --git a/.github/workflows/prepare-integration-inputs.yml b/.github/workflows/prepare-integration-inputs.yml index 7b2bf62ab1..9a8784285e 100644 --- a/.github/workflows/prepare-integration-inputs.yml +++ b/.github/workflows/prepare-integration-inputs.yml @@ -26,6 +26,11 @@ on: required: false type: string default: "" + test-inputs-artifact-name: + description: Prebuilt test archives, test workload images, and Helm chart; built here when empty + required: false + type: string + default: "" outputs: source_sha: description: Source revision of the candidate artifacts @@ -66,6 +71,7 @@ jobs: persist-credentials: false - uses: ./.github/actions/setup-nix + if: inputs['test-inputs-artifact-name'] == '' - name: Download CLI artifact uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 @@ -151,19 +157,31 @@ jobs: docker tag "ghcr.io/nvidia/openshell/supervisor:${IMAGE_TAG}" openshell/supervisor:tmachine docker save --output artifacts/images/openshell-supervisor-tmachine.tar openshell/supervisor:tmachine + - name: Download test inputs + if: inputs['test-inputs-artifact-name'] != '' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: ${{ inputs['test-inputs-artifact-name'] }} + path: artifacts + github-token: ${{ github.token }} + run-id: ${{ inputs['artifact-run-id'] || github.run_id }} + - name: Build test archives + if: inputs['test-inputs-artifact-name'] == '' run: | # Retry dependency preparation, then generate artifacts once. nix build --no-link .#build-artifacts-test-archives || nix build --no-link .#build-artifacts-test-archives nix run .#build-artifacts-test-archives - name: Build test workload images + if: inputs['test-inputs-artifact-name'] == '' run: | # Retry dependency preparation, then generate artifacts once. nix build --no-link .#build-artifacts-test-images || nix build --no-link .#build-artifacts-test-images nix run .#build-artifacts-test-images - name: Package Helm chart + if: inputs['test-inputs-artifact-name'] == '' run: | # Retry dependency preparation, then generate artifacts once. nix build --no-link .#build-artifacts-helm || nix build --no-link .#build-artifacts-helm diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml index c85d29d2fc..ec94c3fe5a 100644 --- a/.github/workflows/release-build.yml +++ b/.github/workflows/release-build.yml @@ -20,9 +20,6 @@ on: required: true type: boolean outputs: - python_version: - description: Python package version - value: ${{ jobs.compute-versions.outputs.python_version }} npm_version: description: TypeScript package version value: ${{ jobs.compute-versions.outputs.npm_version }} @@ -56,7 +53,6 @@ jobs: run: shell: nix develop -c bash -euo pipefail {0} outputs: - python_version: ${{ steps.v.outputs.python }} cargo_version: ${{ steps.v.outputs.cargo }} npm_version: ${{ steps.v.outputs.npm }} deb_version: ${{ steps.v.outputs.deb }} @@ -106,7 +102,6 @@ jobs: fi { echo "is_prerelease=${is_prerelease}" - echo "python=$(uv run --no-project python tasks/scripts/release.py get-version --python)" echo "cargo=$(uv run --no-project python tasks/scripts/release.py get-version --cargo)" echo "npm=$(uv run --no-project python tasks/scripts/release.py get-version --npm)" echo "deb=$(uv run --no-project python tasks/scripts/release.py get-version --deb)" @@ -116,11 +111,11 @@ jobs: echo "source_sha=$(git rev-parse HEAD)" } >> "$GITHUB_OUTPUT" - build-binaries: + build-binaries-linux: needs: compute-versions permissions: contents: read - uses: ./.github/workflows/build-binaries.yml + uses: ./.github/workflows/build-binaries-linux.yml with: cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} supervisor-image-tag: ${{ needs.compute-versions.outputs.semver }} @@ -128,11 +123,35 @@ jobs: secrets: CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} - build-vm-driver: - needs: [compute-versions, build-binaries] + build-binaries-macos: + needs: compute-versions + permissions: + contents: read + uses: ./.github/workflows/build-binaries-macos.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + supervisor-image-tag: ${{ needs.compute-versions.outputs.semver }} + checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} + secrets: + CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} + + build-vm-driver-linux: + needs: [compute-versions, build-binaries-linux] + permissions: + contents: read + uses: ./.github/workflows/build-vm-driver-linux.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + supervisor-image-tag: ${{ needs.compute-versions.outputs.semver }} + checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} + secrets: + CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} + + build-vm-driver-macos: + needs: [compute-versions, build-binaries-linux, build-binaries-macos] permissions: contents: read - uses: ./.github/workflows/build-vm-driver.yml + uses: ./.github/workflows/build-vm-driver-macos.yml with: cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} supervisor-image-tag: ${{ needs.compute-versions.outputs.semver }} @@ -141,7 +160,7 @@ jobs: CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} package-binaries: - needs: [compute-versions, build-binaries, build-vm-driver] + needs: [compute-versions, build-binaries-linux, build-binaries-macos, build-vm-driver-linux, build-vm-driver-macos] permissions: actions: read contents: read @@ -150,7 +169,7 @@ jobs: checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} build-images: - needs: [compute-versions, build-binaries] + needs: [compute-versions, build-binaries-linux, build-binaries-macos] permissions: contents: read packages: write @@ -160,49 +179,9 @@ jobs: push-images: ${{ inputs.is-prerelease }} checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} - build-python-wheel: - name: Build Python Wheel - needs: [compute-versions] - runs-on: linux-amd64-cpu8 - timeout-minutes: 20 - defaults: - run: - shell: nix develop -c bash -euo pipefail {0} - env: - SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL: ${{ needs.compute-versions.outputs.python_version }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - fetch-depth: 0 - persist-credentials: false - ref: ${{ needs.compute-versions.outputs.source_sha }} - - - uses: ./.github/actions/setup-nix - with: - prepare-shell: "true" - - - name: Sync Python dependencies - run: uv sync --frozen - - - name: Build Python wheel - run: | - set -euo pipefail - uv run --frozen python tasks/scripts/generate_python_proto.py - uv build --wheel --out-dir target/wheels - uv run --no-project python tasks/scripts/verify-python-wheel.py target/wheels \ - --expected-version "${SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL}" - ls -la target/wheels/*.whl - - - name: Upload wheel artifact - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: python-wheel - path: target/wheels/*.whl - retention-days: 5 - build-deb: name: Build Debian Packages - needs: [compute-versions, build-binaries, build-vm-driver] + needs: [compute-versions, build-binaries-linux, build-binaries-macos, build-vm-driver-linux, build-vm-driver-macos] uses: ./.github/workflows/deb-package.yml with: deb-version: ${{ needs.compute-versions.outputs.deb_version }} @@ -210,14 +189,14 @@ jobs: build-snap: name: Build Snap - needs: [compute-versions, build-binaries] + needs: [compute-versions, build-binaries-linux, build-binaries-macos] uses: ./.github/workflows/snap-package.yml with: checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} build-rpm: name: Build RPM Packages - needs: [compute-versions, build-binaries] + needs: [compute-versions, build-binaries-linux, build-binaries-macos] uses: ./.github/workflows/rpm-package.yml with: checkout-ref: ${{ needs.compute-versions.outputs.source_sha }} diff --git a/.github/workflows/release-dev.yml b/.github/workflows/release-dev.yml index 71db45ffb1..b4add4d101 100644 --- a/.github/workflows/release-dev.yml +++ b/.github/workflows/release-dev.yml @@ -59,11 +59,11 @@ jobs: echo "rpm_release=$(uv run python tasks/scripts/release.py get-version --dev --rpm-release)" } >> "$GITHUB_OUTPUT" - build-binaries: + build-binaries-linux: needs: compute-versions permissions: contents: read - uses: ./.github/workflows/build-binaries.yml + uses: ./.github/workflows/build-binaries-linux.yml with: cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} supervisor-image-tag: dev @@ -71,11 +71,35 @@ jobs: secrets: CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} - build-vm-driver: - needs: [compute-versions, build-binaries] + build-binaries-macos: + needs: compute-versions permissions: contents: read - uses: ./.github/workflows/build-vm-driver.yml + uses: ./.github/workflows/build-binaries-macos.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + supervisor-image-tag: dev + checkout-ref: ${{ github.sha }} + secrets: + CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} + + build-vm-driver-linux: + needs: [compute-versions, build-binaries-linux] + permissions: + contents: read + uses: ./.github/workflows/build-vm-driver-linux.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + supervisor-image-tag: dev + checkout-ref: ${{ github.sha }} + secrets: + CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} + + build-vm-driver-macos: + needs: [compute-versions, build-binaries-linux, build-binaries-macos] + permissions: + contents: read + uses: ./.github/workflows/build-vm-driver-macos.yml with: cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} supervisor-image-tag: dev @@ -84,7 +108,7 @@ jobs: CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} package-binaries: - needs: [build-binaries, build-vm-driver] + needs: [build-binaries-linux, build-binaries-macos, build-vm-driver-linux, build-vm-driver-macos] permissions: actions: read contents: read @@ -93,7 +117,7 @@ jobs: checkout-ref: ${{ github.sha }} build-images: - needs: build-binaries + needs: [build-binaries-linux, build-binaries-macos] permissions: contents: read packages: write @@ -102,7 +126,7 @@ jobs: checkout-ref: ${{ github.sha }} prepare-integration: - needs: [build-binaries, build-deb, build-images, build-rpm] + needs: [build-binaries-linux, build-binaries-macos, build-deb, build-images, build-rpm] permissions: actions: read contents: read @@ -141,7 +165,7 @@ jobs: CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }} docker-e2e: - needs: [build-binaries, build-images] + needs: [build-binaries-linux, build-binaries-macos, build-images] permissions: actions: read contents: read @@ -153,7 +177,7 @@ jobs: runner: linux-arm64-cpu8 vm-e2e: - needs: [build-binaries, build-vm-driver] + needs: [build-binaries-linux, build-binaries-macos, build-vm-driver-linux, build-vm-driver-macos] permissions: actions: read contents: read @@ -185,49 +209,9 @@ jobs: "${REGISTRY}/${component}:${{ github.sha }}" done - build-python-wheel: - name: Build Python Wheel - needs: [compute-versions] - runs-on: linux-amd64-cpu8 - timeout-minutes: 20 - container: - image: ghcr.io/nvidia/openshell/ci:9cb72baa2e61a1b5f12407e6e82da7fdba0aa722@sha256:67a9a0c32cb99825e6d3e9d9eec45d67149ea1ff7c11a1b1e1b3480d2d3df684 - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - env: - MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL: ${{ needs.compute-versions.outputs.python_version }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - fetch-depth: 0 - persist-credentials: false - ref: ${{ github.sha }} - - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" - - - name: Sync Python dependencies - run: uv sync - - - name: Build Python wheel - run: | - set -euo pipefail - mise run python:build - ls -la target/wheels/*.whl - - - name: Upload wheel artifact - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: python-wheel - path: target/wheels/*.whl - retention-days: 5 - - build-deb: name: Build Debian Packages - needs: [compute-versions, build-binaries, build-vm-driver] + needs: [compute-versions, build-binaries-linux, build-binaries-macos, build-vm-driver-linux, build-vm-driver-macos] uses: ./.github/workflows/deb-package.yml with: deb-version: ${{ needs.compute-versions.outputs.deb_version }} @@ -235,14 +219,14 @@ jobs: build-snap: name: Build snap - needs: [compute-versions, build-binaries] + needs: [compute-versions, build-binaries-linux, build-binaries-macos] uses: ./.github/workflows/snap-package.yml with: checkout-ref: ${{ github.sha }} build-rpm: name: Build RPM Packages - needs: [compute-versions, build-binaries] + needs: [compute-versions, build-binaries-linux, build-binaries-macos] uses: ./.github/workflows/rpm-package.yml with: checkout-ref: ${{ github.sha }} @@ -264,14 +248,13 @@ jobs: SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }} # --------------------------------------------------------------------------- - # Create / update the dev GitHub Release with CLI, gateway, driver, and wheels + # Create / update the dev GitHub Release with CLI, gateway, and driver packages # --------------------------------------------------------------------------- release-dev: name: Release Dev needs: - compute-versions - package-binaries - - build-python-wheel - conformance-integration - feature-specific-integration - docker-e2e @@ -337,12 +320,6 @@ jobs: name: prover-checksums path: release/ - - name: Download wheel artifact - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: python-wheel - path: release/ - - name: Download Debian package artifacts uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -389,16 +366,6 @@ jobs: ls -la release/ - - name: Verify wheel count - run: | - set -euo pipefail - shopt -s nullglob - wheels=(release/*.whl) - if [ "${#wheels[@]}" -ne 1 ]; then - echo "expected exactly one Python wheel, found ${#wheels[@]}" >&2 - exit 1 - fi - - name: Generate checksums run: | set -euo pipefail @@ -411,8 +378,7 @@ jobs: openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz \ openshell-driver-vm-aarch64-apple-darwin.tar.gz \ *.deb \ - openshell-*.rpm \ - *.whl > openshell-checksums-sha256.txt + openshell-*.rpm > openshell-checksums-sha256.txt cat openshell-checksums-sha256.txt sha256sum -- \ openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \ @@ -540,7 +506,6 @@ jobs: release/openshell-prover-x86_64-unknown-linux-musl.tar.gz release/openshell-prover-aarch64-unknown-linux-musl.tar.gz release/openshell-prover-aarch64-apple-darwin.tar.gz - release/*.whl release/openshell.rb release/openshell-checksums-sha256.txt release/openshell-gateway-checksums-sha256.txt diff --git a/.github/workflows/release-prerelease.yml b/.github/workflows/release-prerelease.yml index 2d54d02b23..6dc3b51378 100644 --- a/.github/workflows/release-prerelease.yml +++ b/.github/workflows/release-prerelease.yml @@ -124,7 +124,6 @@ jobs: release-tag: ${{ needs.resolve.outputs.tag }} source-sha: ${{ needs.build.outputs.source_sha }} semver: ${{ needs.build.outputs.semver }} - python-version: ${{ needs.build.outputs.python_version }} npm-version: ${{ needs.build.outputs.npm_version }} is-prerelease: true secrets: diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml index ed6e63827c..c4e9331587 100644 --- a/.github/workflows/release-publish.yml +++ b/.github/workflows/release-publish.yml @@ -27,10 +27,6 @@ on: description: Release version without the v prefix required: true type: string - python-version: - description: Python package version - required: true - type: string npm-version: description: TypeScript package version required: true @@ -116,12 +112,6 @@ jobs: name: prover-checksums path: release/ - - name: Download wheel artifact - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: python-wheel - path: release/ - - name: Download Debian package artifacts uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -143,16 +133,6 @@ jobs: path: release/ merge-multiple: true - - name: Verify wheel count - run: | - set -euo pipefail - shopt -s nullglob - wheels=(release/*.whl) - if [ "${#wheels[@]}" -ne 1 ]; then - echo "expected exactly one Python wheel, found ${#wheels[@]}" >&2 - exit 1 - fi - - name: Generate checksums run: | set -euo pipefail @@ -165,8 +145,7 @@ jobs: openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz \ openshell-driver-vm-aarch64-apple-darwin.tar.gz \ openshell_*.deb \ - openshell-*.rpm \ - *.whl > openshell-checksums-sha256.txt + openshell-*.rpm > openshell-checksums-sha256.txt cat openshell-checksums-sha256.txt sha256sum -- \ openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \ @@ -199,7 +178,6 @@ jobs: release/*.tar.gz release/*.deb release/*.rpm - release/*.whl - name: Upload assembled release assets uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 @@ -428,7 +406,6 @@ jobs: release/openshell-prover-x86_64-unknown-linux-musl.tar.gz release/openshell-prover-aarch64-unknown-linux-musl.tar.gz release/openshell-prover-aarch64-apple-darwin.tar.gz - release/*.whl release/openshell.rb release/openshell-checksums-sha256.txt release/openshell-gateway-checksums-sha256.txt @@ -522,19 +499,6 @@ jobs: buf push proto --label "${RELEASE_TAG}" \ --source-control-url "${SOURCE_CONTROL_URL}" - publish-pypi: - name: Publish Python Wheel - needs: release - if: ${{ !inputs.is-prerelease }} - permissions: - contents: read - id-token: write - uses: ./.github/workflows/wheel-publish.yml - with: - release-tag: ${{ inputs.release-tag }} - source-sha: ${{ inputs.source-sha }} - python-version: ${{ inputs.python-version }} - publish-snap: name: Publish Snap needs: release @@ -600,7 +564,6 @@ jobs: - tag-ghcr-release - release-helm - release - - publish-pypi - publish-snap - publish-prerelease-snap - publish-sdk-typescript @@ -620,7 +583,7 @@ jobs: if [[ "${IS_PRERELEASE}" == "true" ]]; then expected+=(publish-prerelease-snap) else - expected+=(publish-pypi publish-sdk-typescript tag-sdk-go) + expected+=(publish-sdk-typescript tag-sdk-go) if [[ "${MAKE_LATEST}" == "true" ]]; then expected+=(publish-snap) fi diff --git a/.github/workflows/release-vm-kernel.yml b/.github/workflows/release-vm-kernel.yml index e1a95bfb98..f5de59c5fb 100644 --- a/.github/workflows/release-vm-kernel.yml +++ b/.github/workflows/release-vm-kernel.yml @@ -79,6 +79,7 @@ jobs: path: | target/libkrun-build/kernel.c target/libkrun-build/ABI_VERSION + target/libkrun-build/provenance.json retention-days: 1 # --------------------------------------------------------------------------- @@ -149,6 +150,8 @@ jobs: run: tasks/scripts/vm/build-libkrun-macos.sh --kernel-dir target/kernel-artifact - name: Package runtime tarball + env: + CUSTOM_PROVENANCE_DIR: target/kernel-artifact run: | tasks/scripts/vm/package-vm-runtime.sh \ --platform darwin-aarch64 \ diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6ec1e1914f..5f64d393a8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -153,7 +153,6 @@ jobs: release-tag: ${{ needs.promote.outputs.tag }} source-sha: ${{ needs.build.outputs.source_sha }} semver: ${{ needs.build.outputs.semver }} - python-version: ${{ needs.build.outputs.python_version }} npm-version: ${{ needs.build.outputs.npm_version }} is-prerelease: false make-latest: ${{ needs.promote.outputs.make-latest == 'true' }} diff --git a/.github/workflows/rust-cache.yml b/.github/workflows/rust-cache.yml index d9ec30c631..dcf7c8027d 100644 --- a/.github/workflows/rust-cache.yml +++ b/.github/workflows/rust-cache.yml @@ -3,9 +3,10 @@ name: Rust Cache -# Branch Checks never runs on main, so this saves the rust-checks cache that PR -# mirrors and merge groups restore. Keep env and build commands aligned with -# the Branch Checks rust-test job so the cache keys and contents match. +# Branch Checks and Branch E2E never run on main, so this saves the Rust +# caches that PR mirrors and merge groups restore. Keep each job's env and +# build commands aligned with the job that restores its cache, so the cache +# keys and contents match. on: push: branches: @@ -65,3 +66,37 @@ jobs: cargo nextest run --locked --profile ci --workspace --features openshell-server/test-support --no-run cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --no-run cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/testkit/Cargo.toml --no-run + + # Restored by build-e2e-bundles in branch-e2e.yml. + e2e-bundles: + name: E2E suite bundles cache (x86_64-unknown-linux-musl) + runs-on: linux-amd64-cpu8 + timeout-minutes: 60 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - uses: ./.github/actions/setup-nix + with: + prepare-shell: "true" + cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }} + + - id: rust + uses: ./.github/actions/setup-rust + with: + cache-key: e2e-bundles-v1-x86_64-unknown-linux-musl + save-cache: "true" + + - name: Build the bundles + if: steps.rust.outputs.cache-hit != 'true' + env: + CARGO_TARGET_DIR: ${{ github.workspace }}/target + CARGO_PROFILE_DEV_DEBUG: line-tables-only + run: | + set -euo pipefail + nix build --no-link .#openshell-test-runner || nix build --no-link .#openshell-test-runner + for family in features drivers/podman; do + nix run .#openshell-test-runner -- build \ + --family "$family" --out "$RUNNER_TEMP/e2e-bundles/${family//\//-}" + done diff --git a/.github/workflows/wheel-publish.yml b/.github/workflows/wheel-publish.yml deleted file mode 100644 index 15512ee639..0000000000 --- a/.github/workflows/wheel-publish.yml +++ /dev/null @@ -1,92 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -name: Publish Wheel - -# PyPI Trusted Publishing matches the `job_workflow_ref` claim, which names this -# file for both calls and dispatches. The `openshell` PyPI project trusts -# workflow `wheel-publish.yml` with environment `pypi`. -on: - workflow_call: - inputs: - release-tag: - required: true - type: string - source-sha: - required: true - type: string - python-version: - required: true - type: string - workflow_dispatch: - inputs: - release-tag: - description: Stable tag whose GitHub Release holds the wheel, e.g. v0.1.7 - required: true - type: string - source-sha: - description: Commit the tag points to - required: true - type: string - python-version: - description: Python package version, e.g. 0.1.7 - required: true - type: string - -permissions: - contents: read - -jobs: - publish: - name: Publish Python Wheel to PyPI - runs-on: ubuntu-latest - timeout-minutes: 10 - environment: - name: pypi - url: https://pypi.org/project/openshell/ - permissions: - contents: read - id-token: write - steps: - # Publish the wheel attached to the GitHub Release rather than a run's - # build artifact, so a retry or dispatch publishes the same bytes. - - name: Download and verify the released wheel - shell: bash - env: - GH_TOKEN: ${{ github.token }} - RELEASE_TAG: ${{ inputs.release-tag }} - SOURCE_SHA: ${{ inputs.source-sha }} - PYTHON_VERSION: ${{ inputs.python-version }} - run: | - set -euo pipefail - tag_sha=$(gh api "repos/${GITHUB_REPOSITORY}/commits/${RELEASE_TAG}" --jq .sha) - if [[ "${tag_sha}" != "${SOURCE_SHA}" ]]; then - echo "${RELEASE_TAG} points to ${tag_sha}, not ${SOURCE_SHA}" >&2 - exit 1 - fi - gh release download "${RELEASE_TAG}" -R "${GITHUB_REPOSITORY}" \ - --pattern '*.whl' --pattern openshell-checksums-sha256.txt --dir release - shopt -s nullglob - wheels=(release/*.whl) - if [[ "${#wheels[@]}" -ne 1 ]]; then - echo "Expected exactly one Python wheel on ${RELEASE_TAG}, found ${#wheels[@]}" >&2 - exit 1 - fi - wheel=$(basename "${wheels[0]}") - if [[ "${wheel}" != openshell-"${PYTHON_VERSION}"-*.whl ]]; then - echo "${wheel} is not an openshell ${PYTHON_VERSION} wheel" >&2 - exit 1 - fi - checksum=$(awk -v wheel="${wheel}" '$2 == wheel { print $1 " " $2 }' release/openshell-checksums-sha256.txt) - if [[ -z "${checksum}" ]]; then - echo "${wheel} is missing from the release checksums" >&2 - exit 1 - fi - (cd release && sha256sum -c - <<<"${checksum}") - mkdir dist - mv "release/${wheel}" dist/ - - - name: Publish wheel to PyPI - uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086 # v1.14.3 - with: - skip-existing: true diff --git a/AGENTS.md b/AGENTS.md index 8866ead489..4760164e97 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,7 @@ Do not rely on this file for a full inventory. The detailed public and contribut | `crates/openshell-supervisor-network/` | Network supervisor | Proxying, L7 enforcement, policy evaluation, and provider credential injection | | `crates/openshell-supervisor-process/` | Process supervisor | Process lifecycle, namespace, and bypass monitoring | | `crates/openshell-vfio/` | VFIO support | PCI and GPU passthrough preparation and lifecycle | -| `python/openshell/` | Python SDK | Python bindings and CLI packaging | +| `python/openshell/` | Python SDK (frozen) | Moved to ai-openshell/sdk-python; kept until removal | | `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types | | `proto/` | Protobuf definitions | gRPC service contracts | | `tests/testkit/` | E2E testkit | Support library and `#[testkit::test]` attribute for SDK-driven E2E suites | @@ -170,6 +170,10 @@ ocsf_emit!(event); `openshell_core::net::set_tcp_nodelay_best_effort` on an accepted or already-connected stream, or `openshell_core::net::connect_tcp_nodelay_best_effort` when dialing. +- Clippy rejects raw `TcpStream::connect` (`clippy.toml`). Dial with + `openshell_core::net::connect_tcp_nodelay_best_effort`, or + `connect_std_tcp_nodelay_best_effort` on blocking threads. The lint does not + cover accepted sockets, so keep setting nodelay on those by hand. - This applies to loopback/localhost TCP too — the delayed-ACK stall is a timer behavior, not wire latency. - You should skip it for unix domain sockets (no Nagle). It's not critical for @@ -197,6 +201,11 @@ Follow `CONTRIBUTING.md` and `create-github-pr`. Authors without target-reposito - The Go SDK moved to [ai-openshell/sdk-go](https://github.com/ai-openshell/sdk-go) (module `github.com/ai-openshell/sdk-go`). Open SDK changes there. - `sdk/go/` is frozen until it is removed; Branch Checks fails PRs that change it. Proto changes do not need regenerated Go bindings. +## Python SDK (`python/openshell/`) + +- The Python SDK moved to [ai-openshell/sdk-python](https://github.com/ai-openshell/sdk-python), which publishes `openshell` to PyPI. Open SDK changes there. +- `python/` and the SDK build script `tasks/scripts/generate_python_proto.py` are frozen until core removes them; Branch Checks fails PRs that change them. Put tests for core Python scripts next to the scripts, not under `python/`. + ## TypeScript SDK (`sdk/typescript/`) - Run `mise run sdk:ts:ci` for codegen, proto lint, Biome lint, type checking, unit tests, coverage, and build validation. diff --git a/CI.md b/CI.md index a1a1b8bc77..6dac0a4bf5 100644 --- a/CI.md +++ b/CI.md @@ -166,10 +166,11 @@ selected. DEB and RPM packages build only when integration qualification is sele the DEB on Ubuntu with Docker and installs the CLI and gateway RPMs on Fedora with rootful and rootless Podman. These lanes run conformance using the matching runtime images. Release Dev, Release, and Pre-release use the same package installers. -The feature suites (`tests/suites/features`) run the same way, on the -package installers. The Podman driver suite (`tests/suites/drivers/podman`) and the -`e2e-podman` lane retain the binary installer because they configure its system -service, local HTTP gateway, and CLI path. The driver suite and `e2e-podman` run +The feature suites (`tests/suites/features`) and the Podman driver suite +(`tests/suites/drivers/podman`) run the same way, on the package installers; the +driver suite's environments merge their Podman settings into the packaged +gateway's configuration (see `TESTING.md`). The `e2e-podman` lane runs on the +RPM installer as well. The driver suite and `e2e-podman` run only in `Branch E2E Checks`. The manual Integration Tests workflow runs the remaining tmachine testsuites, such as `e2e-podman`, and downloads the packages selected by its matrix. @@ -215,7 +216,7 @@ candidate inputs (see [TESTING.md](TESTING.md)) and narrow the tests with a nextest filterset: ```shell -nix run .#openshell-test-runner -- \ +nix run .#openshell-test-runner -- run \ --environment ubuntu-docker-rootful --installer deb \ --family conformance --package conformance \ --filter 'test(/^policy_advisor::/)' @@ -440,7 +441,7 @@ without rerunning qualification. These artifacts have different embedded version from the qualified prerelease. Existing stable tags are never moved; a retry must resolve to the same qualified candidate. A retry rebuilds, then publishes only what is missing: it keeps published GHCR version tags, Helm charts, GitHub Release assets, -npm versions, and PyPI versions. Snap uploads are not skipped. +and npm versions. Snap uploads are not skipped. Stable image builds save multi-platform OCI archives as workflow artifacts without pushing to GHCR. Publication copies those archives, including their attestations, @@ -691,7 +692,7 @@ The bot's full administrator documentation is internal to NVIDIA. The only comma |---|---| | `.github/workflows/branch-checks.yml` | Required non-E2E checks. Triggers on `push: pull-request/[0-9]+` for PR mirrors and `merge_group` for queued merges. | | `.github/workflows/branch-e2e.yml` | Standard, GPU, Kubernetes HA, and Kubernetes credential-driver E2E. PR mirror pushes use `test:e2e`, `test:e2e-gpu`, and `test:e2e-kubernetes` labels; merge groups do not run E2E. | -| `.github/workflows/build-binaries.yml`, `build-vm-driver.yml` | Shared binary matrices used by branch and release workflows. The VM driver remains separate because its build consumes the runtime binaries. | +| `.github/workflows/build-binaries-{linux,macos}.yml`, `build-vm-driver-{linux,macos}.yml` | Shared per-platform binary matrices used by branch and release workflows. Each platform is its own workflow, so a consumer depends only on the platforms it needs. The VM driver remains separate because its build consumes the runtime binaries. Its build steps stay inline rather than in a local action because release builds check out the release tag, which may predate the action. | | `.github/workflows/build-images.yml` | Builds and pushes multi-platform images, then uploads the same OCI images as workflow artifacts. | | `.github/workflows/package-release-binaries.yml` | Packages raw build artifacts into release tarballs without rebuilding them. | | `.github/workflows/e2e-docker-test.yml`, `e2e-podman-test.yml`, `e2e-vm-test.yml`, `e2e-kubernetes-test.yml` | Reusable runtime lanes called directly by branch and release workflows. Callers select suites and declare only the artifacts each runtime consumes. | @@ -720,19 +721,18 @@ Each step that publishes to an external registry, store, or site lives in its ow | File | Role | |---|---| -| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | +| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries and packages, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | | `.github/workflows/release.yml` | Manually promote the latest qualified prerelease for a version and branch, create its stable tag, then rebuild and publish. Support releases preserve shared latest channels. Publishes the prerelease's notes preview, retagged, or generates notes when it is unavailable. The notes-only mode generates a release notes preview artifact. Published artifacts are never overwritten: a rerun fills in what is missing, and a bad published artifact is fixed by releasing a new patch version. | | `.github/workflows/release-auto-tag.yml` | Weekdays at 14:00 Europe/Zurich, tags the next `-pre.N` on `main` when commits landed since the latest one, then dispatches Pre-release. It skips when the train has no `-pre.1` or its version is already stable, so push each train's `-pre.1` tag by hand. | | `.github/workflows/release-prerelease.yml` | Pre-release orchestrator, triggered by prerelease tag pushes or dispatched by Release Auto-Tag or manually. A tagless dispatch creates the next candidate on its branch; explicit tags select existing candidates. Builds, then publishes and qualifies in parallel. Qualification failure fails the run and sends the failure notification; published artifacts remain available. | | `.github/workflows/release-build.yml` | Shared version computation, binary and image builds, and package builds. Validates that the tag matches the caller's release kind. | -| `.github/workflows/release-publish.yml` | Shared artifact assembly, OCI image tags and Helm chart publication, and package publication. Stable releases also publish release-labeled public schemas to Buf (`BUF_TOKEN` required) and GitHub Releases, and call the PyPI, snap, npm, and Go SDK publishers. | +| `.github/workflows/release-publish.yml` | Shared artifact assembly, OCI image tags and Helm chart publication, and package publication. Stable releases also publish release-labeled public schemas to Buf (`BUF_TOKEN` required) and GitHub Releases, and call the snap, npm, and Go SDK publishers. The Python SDK is released from [ai-openshell/sdk-python](https://github.com/ai-openshell/sdk-python), so no GitHub Release carries a wheel. | | `.github/workflows/release-notes.yml` | Shared prompt-driven release notes generation through Inference Hub, used by prerelease previews, notes-only previews, and stable releases. Stable releases reuse the promoted prerelease's preview when available. | | `.github/workflows/release-qualification.yml` | Reusable qualification suites and result aggregation. Runs security scans, conformance and feature integration, Docker and VM E2E, protobuf compatibility, and advisory compatibility review. | | `.github/workflows/snap-package.yml` | Builds snap and component artifacts for Release Dev, Release, and Pre-release without Store credentials or publication. | | `.github/workflows/snap-publish.yml` | Uploads existing snap and component artifacts directly to a Store channel without rebuilding. Release Dev uses `latest/edge`, Pre-release uses `latest/beta`, and Release uses `latest/stable` for the stable rebuild. Uploads queue per channel and architecture, so tagged beta and stable jobs recheck repository tags after earlier publication finishes and before updating their floating channels. Manual dispatch uploads the `snap-linux-*` artifacts from `artifact-run-id`. | | `.github/workflows/sdk-typescript-publish.yml` | Publishes the TypeScript SDK to GitHub Packages. Called by stable Release; manual dispatch publishes a given tag and npm version. | | `.github/workflows/sdk-go-tag.yml` | Creates the `sdk/go/vX.Y.Z` tag and asks the Go module proxy to fetch it. Called by stable Release; manual dispatch tags a given version and commit. | -| `.github/workflows/wheel-publish.yml` | Publishes the wheel attached to a stable GitHub Release to PyPI with Trusted Publishing in the `pypi` environment. Called by stable Release; manual dispatch publishes a given tag. | | `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts in the `macos`, `ubuntu-deb`, `ubuntu-snap-system-docker`, `fedora`, and `kubernetes` (kind + Helm) jobs. Each job reaches its gateway and creates, exercises, and deletes a sandbox. The `ubuntu-snap-system-docker` job verifies a compatible system Docker lifecycle, while `ubuntu-snap-docker-preflight` tests fail-fast behavior when Docker is absent or supplied by the Docker snap. The positive snap lane also runs a local policy containment check with the packaged prover. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref `). See the `test-release-canary` skill for the playbook and local kind reproduction. | ## Required status contexts diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3e800debdf..e238089b18 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -382,7 +382,7 @@ These are the primary `mise` tasks for day-to-day development: | `crates/` | Rust crates | | `crates/openshell-policy-schema/` | Canonical authored policy DTOs and bounded YAML/JSON parser | | `crates/openshell-prover-cli/` | Standalone local policy boundary checker | -| `python/` | Python SDK and bindings | +| `python/` | Python SDK (frozen, moved to ai-openshell/sdk-python) | | `sdk/go/` | Go SDK (types, gRPC clients, converters) | | `sdk/typescript/` | TypeScript SDK (Connect client and generated protobuf bindings) | | `proto/` | Protocol buffer definitions and [public API conventions](proto/README.md) | @@ -390,7 +390,7 @@ These are the primary `mise` tasks for day-to-day development: | `tests/testkit/` | Support library for SDK-driven end-to-end suites | | `tests/suites/` | End-to-end suites, grouped by test family | | `tests/tmachine/` | QEMU test machines provisioned with Ansible, and the `Session` API that drives them | -| `tests/runner/` | `openshell-test-runner`: runs a test family in a tmachine environment and checks the results | +| `tests/runner/` | `openshell-test-runner`: builds a test family for tmachine guests, runs it in a tmachine environment and checks the results | | `deploy/` | Dockerfiles, Helm chart, Kubernetes manifests | | `docs/` | Published Fern docs source, navigation, and content assets | | `fern/` | Fern site config, components, and theme assets | diff --git a/Cargo.lock b/Cargo.lock index 94cc7b3df6..d8b2f36221 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4388,6 +4388,7 @@ dependencies = [ "openshell-bootstrap", "openshell-core", "openshell-crypto", + "openshell-otel", "openshell-policy", "openshell-providers", "openshell-sdk", @@ -4411,6 +4412,7 @@ dependencies = [ "tracing", "tracing-subscriber", "url", + "wiremock", ] [[package]] @@ -6711,7 +6713,6 @@ dependencies = [ "rand 0.9.4", "serde", "serde_json", - "serde_yaml", "spin", "thiserror 2.0.20", ] diff --git a/README.md b/README.md index cdbe456239..072e83f13b 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,11 @@ [![Security Policy](https://img.shields.io/badge/Security-Report%20a%20Vulnerability-red)](SECURITY.md) [![Documentation](https://img.shields.io/badge/docs-latest-brightgreen)](https://docs.nvidia.com/openshell/latest/index.html) +> [!NOTE] +> 🚧 OpenShell is now part of the CNCF! We’re in the process of moving development to [ai-openshell](https://github.com/ai-openshell) and governance to the CNCF. Until the move is complete, please continue to use the [NVIDIA/OpenShell](https://github.com/NVIDIA/OpenShell) repo for code, issues, and pull requests. + + + > [!IMPORTANT] > **New in OpenShell 0.1.x:** a stable release cadence, new isolation primitives, an expanded extension surface, and new APIs. [Read the 0.1.0 upgrade guide](https://docs.nvidia.com/openshell/latest/upgrade/0-1-0). diff --git a/TESTING.md b/TESTING.md index a3936ad649..2250118e6e 100644 --- a/TESTING.md +++ b/TESTING.md @@ -224,7 +224,7 @@ Adapted from the handover notes in NVIDIA/OpenShell#4629 by Evan Lezar. ### Running suites in a tmachine environment (`tests/runner/`) -`openshell-test-runner` runs a test family inside a tmachine VM, against the +`openshell-test-runner run` runs a test family inside a tmachine VM, against the OpenShell that the VM's installer set up. `--family` is the path of the workspace under `tests/suites/`, such as `conformance` or `drivers/podman`, and `--package` narrows it to some of its packages. It must name a workspace, so @@ -233,7 +233,7 @@ selects every row whose family is that name or lies under it, so `drivers` selects all the driver suites. ```shell -nix run .#openshell-test-runner -- \ +nix run .#openshell-test-runner -- run \ --environment ubuntu-docker-rootful --installer deb \ --family conformance --package conformance ``` @@ -248,7 +248,21 @@ complete, for example because the environment didn't come up. It writes `artifacts/test-results/` (`--results`), even when the run timed out. `--filter ` narrows the tests with a nextest filterset, and -`--keep` leaves the VM running until you press Enter. Two runs on one host +`--keep` leaves the VM running until you press Enter. `--program ` runs a +prebuilt testkit program, such as a static `openshell-conformance` for the +guest's architecture, instead of building the family's tests; it can't be +combined with `--package`. With it, `--filter` (repeatable) takes parts of test +names, as the program does, such as `--filter policy_advisor::`, rather than a +nextest filterset. The runner runs the program's `ready` +and `run` commands in the guest and still writes `junit.xml` and `nextest.log` +(the program's output) to the results directory. Branch E2E builds +`openshell-conformance` once and passes it to every conformance row this way. + +`openshell-test-runner build --family --out ` builds a family's +test bundle for the guest without booting a VM, and `run --bundle ` runs +it instead of building one. `--package` then selects the tests to run from the +bundle, which must hold those packages. Branch E2E builds the `features` and +`drivers/podman` bundles once and passes them to their rows this way. Two runs on one host need different `TMACHINE_SSH_PORT` values (default 2222). The installer inputs, such as the Debian package and the images, come from `artifacts/`, built by `nix run .#build-artifacts` or downloaded by CI. @@ -312,7 +326,7 @@ The Keycloak fixture is `tests/support/keycloak`. Run the suite that uses it with: ```shell -nix run .#openshell-test-runner -- \ +nix run .#openshell-test-runner -- run \ --environment ubuntu-docker-rootful-keycloak --installer deb \ --family features --package keycloak-provider-refresh ``` @@ -424,11 +438,14 @@ Suites: - Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests. - Conformance (`tests/suites/conformance`) - the testkit `conformance` crate: - lifecycle, the CLI, persistent workspaces, file transfer, and the policy - advisor. Driver E2E runs it with `openshell-conformance` against the lane's - gateway and writes a JUnit report under `results/`; set - `OPENSHELL_CONFORMANCE_BIN` to use a prebuilt program, as Branch E2E does - with the static build it makes once per architecture. The + lifecycle, the CLI, persistent workspaces, file transfer, the policy + advisor, and credential gating. Driver E2E runs it with + `openshell-conformance` against the lane's gateway and writes a JUnit report + under `results/`; set `OPENSHELL_CONFORMANCE_BIN` to use a prebuilt program, + as Branch E2E does with the static build it makes once per architecture. + The Docker and Podman runners build `openshell/e2e-python:dev` for it; the + Kubernetes and VM runners set `OPENSHELL_TEST_PYTHON_IMAGE` to a pullable + Python image unless it is already set. The installed-artifact conformance lanes run the same crate through `openshell-test-runner` (see above). - Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or @@ -515,7 +532,7 @@ rebuilds only the RPMs. Run conformance against the installed RPMs: ```shell nix run .#build-artifacts -nix run .#openshell-test-runner -- \ +nix run .#openshell-test-runner -- run \ --environment fedora-podman-rootful --installer rpm \ --family conformance --package conformance ``` @@ -524,7 +541,7 @@ Run the portable subset in a disposable rootless Podman guest: ```shell nix run .#build-artifacts -nix run .#tmachine -- test fedora-podman-rootless binaries e2e-podman +nix run .#tmachine -- test fedora-podman-rootless rpm e2e-podman ``` Print the exact tmachine archive selection as a shell `PODMAN_CI_TESTS` array: @@ -549,14 +566,40 @@ matching lines in `branch-checks.yml`, and `ALIGNED_WORKSPACES`. The Podman driver suite (`tests/suites/drivers/podman`) compares OpenShell and direct Podman user-namespace mappings. Each `fedora-podman-{rootful,rootless}-userns-` environment configures -the gateway with one mode (`default`, `auto`, `keep-id` or `private`); only the -`binaries` installer applies it: +the gateway with one mode (`default`, `auto`, `keep-id` or `private`) and works +with every installer; CI runs it with `rpm`: ```shell -nix run .#openshell-test-runner -- \ - --environment fedora-podman-rootless-userns-keep-id --installer binaries \ - --gateway tmachine --family drivers/podman -``` +nix run .#openshell-test-runner -- run \ + --environment fedora-podman-rootless-userns-keep-id --installer rpm \ + --family drivers/podman +``` + +#### Gateway overrides in an environment + +An environment overrides gateway settings by writing the tables to change to +`/etc/openshell-test/gateway-overrides.toml` in a playbook that runs before the +installer. The file holds whole tables, such as `[openshell.gateway]` or +`[openshell.drivers.podman]`, and is not a complete configuration. Each +installer applies it to the gateway's own file: + +- `binaries` applies it to the `/etc/openshell/gateway.toml` it generates. +- `deb` and `rpm` apply it to the service user's + `~/.config/openshell/gateway.toml`, the file the packaged service reads. The + file starts as a copy of the rpm's + `/usr/share/openshell-gateway/gateway.toml.default`, where the service would + seed it on first start, or as a minimal `version = 2` file when the package + has no default. No environment replaces the packaged file, and an environment + with no overrides leaves it untouched. +- `k3s` installers ignore the file. + +`tests/tmachine/ansible/roles/openshell_gateway/files/merge-gateway-toml.py` +does the work. Each key in the overrides replaces the same key in the table, or +is added to it, and a table the file lacks is appended. The rest of the file, +comments included, stays. The script parses the result and fails the setup, +leaving the file unchanged, when it isn't valid TOML. It can't remove a key. +Settings that the packaged service reads from `gateway.env` instead of the +TOML are not covered. The `oci-image` feature suite (`tests/suites/features/oci-image`) checks OCI image identity and working-directory behavior shared by the Docker and Podman @@ -564,7 +607,7 @@ drivers against installed deb packages on Ubuntu and rpm packages on Fedora. CI runs it on Docker rootful, Podman rootful, and Podman rootless guests: ```shell -nix run .#openshell-test-runner -- \ +nix run .#openshell-test-runner -- run \ --environment ubuntu-docker-rootful --installer deb \ --family features --package oci-image ``` @@ -656,6 +699,10 @@ explains the isolation backend separation behind the capability-free runtime. On a **remote** cluster, drop the `e2e-host-gateway` feature. Those tests rely on the sandbox-side `host.openshell.internal` alias reaching the machine running the tests, which is unreachable from pods on a remote cluster, so they fail. +The conformance suite's `credential_gating` tests need the alias to reach that +machine too, and the feature selection doesn't skip them, so they fail on a +remote cluster. [#5409](https://github.com/NVIDIA/OpenShell/issues/5409) +tracks handling that. Left enabled, the `host_gateway_alias` suite fails because `host.openshell.internal` does not resolve inside the pod, so the gateway SSRF-denies the request (`DNS resolution failed` / `ssrf_denied`) — a networking diff --git a/clippy.toml b/clippy.toml new file mode 100644 index 0000000000..b447a1eafb --- /dev/null +++ b/clippy.toml @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +disallowed-methods = [ + { path = "tokio::net::TcpStream::connect", reason = "use openshell_core::net::connect_tcp_nodelay_best_effort so Nagle is disabled on the dialed stream" }, + { path = "std::net::TcpStream::connect", reason = "use openshell_core::net::connect_std_tcp_nodelay_best_effort so Nagle is disabled on the dialed stream" }, +] diff --git a/crates/openshell-binary-identity/src/lib.rs b/crates/openshell-binary-identity/src/lib.rs index c5dc5696f3..245185bceb 100644 --- a/crates/openshell-binary-identity/src/lib.rs +++ b/crates/openshell-binary-identity/src/lib.rs @@ -13,13 +13,20 @@ use openshell_isolation_interface::contract::{BinaryIdentity, ResolveError}; #[cfg(target_os = "linux")] use openshell_isolation_interface::contract::{ExecutableIdentity, Sha256Digest}; #[cfg(target_os = "linux")] -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; #[cfg(target_os = "linux")] use std::sync::{Arc, Mutex}; #[cfg(target_os = "linux")] const EXECUTABLE_DIGEST_CACHE_CAPACITY: usize = 1_024; +/// Bounds the total cmdline bytes read from `/proc//cmdline`. This is a +/// diagnostic-read cap on the whole command line, not a per-argument limit: +/// the kernel allows a single argument larger than this cap, so one +/// maximum-size argument plus argv[0] already exceeds it and is truncated. +#[cfg(target_os = "linux")] +const CMDLINE_READ_LIMIT: usize = 128 * 1024; + /// Resolves executable identity from a Linux procfs process identifier. /// /// The configured scope bounds ancestry and cmdline collection to the observed @@ -106,20 +113,13 @@ fn resolve_linux_process( .map(|(snapshot, _)| snapshot.binary_path.clone()) .collect::>(); excluded_paths.push(snapshot.binary_path.clone()); - let cmdline_paths = cmdline_absolute_paths(&snapshot.cmdline) - .into_iter() - .chain( - ancestor_processes - .iter() - .flat_map(|(snapshot, _)| cmdline_absolute_paths(&snapshot.cmdline)), - ) - .filter(|path| !excluded_paths.contains(path)) - .fold(Vec::new(), |mut paths, path| { - if !paths.contains(&path) { - paths.push(path); - } - paths - }); + let cmdline_paths = collect_cmdline_paths( + &snapshot.cmdline, + ancestor_processes + .iter() + .map(|(snapshot, _)| snapshot.cmdline.as_slice()), + &excluded_paths, + ); // Digests computed during this call. They reach the shared cache only after // every snapshot validates, but later processes in the chain reuse them. @@ -243,7 +243,11 @@ fn cache_executable_digest( .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); if cache.len() >= EXECUTABLE_DIGEST_CACHE_CAPACITY { - cache.clear(); + // Evict a single arbitrary entry: clearing the whole cache would + // force a rehash of every live executable after one capacity crossing. + if let Some(evict) = cache.keys().next().copied() { + cache.remove(&evict); + } } cache.insert(key, digest); } @@ -335,8 +339,39 @@ fn process_stat(pid: u32) -> Result<(u32, u64), ResolveError> { #[cfg(target_os = "linux")] fn read_process_cmdline(pid: u32) -> Result, ResolveError> { + use std::io::Read as _; + let path = format!("/proc/{pid}/cmdline"); - std::fs::read(&path).map_err(|error| ResolveError::Failed(format!("read {path}: {error}"))) + let file = std::fs::File::open(&path) + // Keep the message prefix std::fs::read produced, so denial-path + // strings downstream stay byte-identical. + .map_err(|error| ResolveError::Failed(format!("read {path}: {error}")))?; + // Reading one byte past the limit makes the truncation predicate exact: + // a cmdline of exactly CMDLINE_READ_LIMIT bytes is kept whole. + let mut buffer = Vec::new(); + file.take(CMDLINE_READ_LIMIT as u64 + 1) + .read_to_end(&mut buffer) + .map_err(|error| ResolveError::Failed(format!("read {path}: {error}")))?; + Ok(cap_cmdline(buffer)) +} + +#[cfg(target_os = "linux")] +fn cap_cmdline(mut buffer: Vec) -> Vec { + if buffer.len() <= CMDLINE_READ_LIMIT { + return buffer; + } + buffer.truncate(CMDLINE_READ_LIMIT); + // Drop the last argument whether it is partial or happens to end exactly + // at the cap: a truncated fragment must not reach cmdline_absolute_paths, + // and diagnostic-only data does not justify boundary-case recovery. + if buffer.last().is_some_and(|byte| *byte == 0) { + buffer.pop(); + } + match buffer.iter().rposition(|byte| *byte == 0) { + Some(index) => buffer.truncate(index + 1), + None => buffer.clear(), + } + buffer } #[cfg(target_os = "linux")] @@ -363,6 +398,11 @@ fn executable_path(pid: u32) -> Result { } #[cfg(target_os = "linux")] +// Stack buffer by design: no heap allocation per cache miss. The array is +// zeroed on each call, a negligible cost next to hashing the executable. +// The 64 KiB size exceeds clippy's 16 KiB large_stack_arrays default, which +// is far below every thread stack in this call path. +#[allow(clippy::large_stack_arrays)] fn hash_executable(pid: u32, executable: &mut std::fs::File) -> Result { use std::fmt::Write as _; use std::io::Read as _; @@ -370,7 +410,7 @@ fn hash_executable(pid: u32, executable: &mut std::fs::File) -> Result Option { - std::fs::read_to_string(format!("/proc/{pid}/status")) - .ok()? - .lines() - .find_map(|line| line.strip_prefix("PPid:"))? - .trim() - .parse() - .ok() +#[derive(Debug, Default, PartialEq, Eq)] +struct ProcessStatus { + ppid: Option, + innermost_nspid: Option, +} + +#[cfg(target_os = "linux")] +fn read_process_status(pid: u32) -> Option { + let path = format!("/proc/{pid}/status"); + Some(parse_process_status(&std::fs::read_to_string(path).ok()?)) +} + +#[cfg(target_os = "linux")] +fn parse_process_status(status: &str) -> ProcessStatus { + let mut parsed = ProcessStatus::default(); + for line in status.lines() { + match line.split_once(':') { + Some(("PPid", value)) => parsed.ppid = value.trim().parse().ok(), + Some(("NSpid", value)) => { + parsed.innermost_nspid = value + .split_whitespace() + .next_back() + .and_then(|nspid| nspid.parse().ok()); + } + _ => {} + } + } + parsed } #[cfg(target_os = "linux")] @@ -440,28 +500,21 @@ fn nested_pid_namespace_init(pid: u32) -> Option { let mut current = pid; for _ in 0..MAX_DEPTH { - if namespace_pid(current) == Some(1) { + // One status read per hop carries both the namespace PID check and + // the parent PID for the next hop. + let status = read_process_status(current)?; + if status.innermost_nspid == Some(1) { // Host PID 1 is outside every workload. A nested namespace init // has a distinct host PID and is a valid workload ancestry root. return (current != 1).then_some(current); } - current = parent_pid(current).filter(|parent| *parent > 0 && *parent != current)?; + current = status + .ppid + .filter(|parent| *parent > 0 && *parent != current)?; } None } -#[cfg(target_os = "linux")] -fn namespace_pid(pid: u32) -> Option { - std::fs::read_to_string(format!("/proc/{pid}/status")) - .ok()? - .lines() - .find_map(|line| line.strip_prefix("NSpid:"))? - .split_whitespace() - .next_back()? - .parse() - .ok() -} - #[cfg(target_os = "linux")] fn cmdline_absolute_paths(cmdline: &[u8]) -> Vec { cmdline @@ -471,6 +524,32 @@ fn cmdline_absolute_paths(cmdline: &[u8]) -> Vec { .collect() } +#[cfg(target_os = "linux")] +fn collect_cmdline_paths<'a>( + own_cmdline: &'a [u8], + ancestor_cmdlines: impl Iterator, + excluded_paths: &[std::path::PathBuf], +) -> Vec { + // Insertion order is load-bearing: consumers compare the collected + // vectors wholesale, so the own cmdline comes first and ancestors follow + // nearest-first. The excluded-paths scan stays a bounded linear check + // over binary paths only. Candidates stream one cmdline at a time, and a + // PathBuf is cloned only for unique survivors, never for duplicates or + // excluded paths. + let mut paths = Vec::new(); + let mut seen_paths: HashSet = HashSet::new(); + for cmdline in std::iter::once(own_cmdline).chain(ancestor_cmdlines) { + for path in cmdline_absolute_paths(cmdline) { + if excluded_paths.contains(&path) || seen_paths.contains(&path) { + continue; + } + seen_paths.insert(path.clone()); + paths.push(path); + } + } + paths +} + #[cfg(all(test, target_os = "linux"))] mod tests { use super::*; @@ -523,15 +602,16 @@ mod tests { let _ = std::io::Read::read(&mut std::io::stdin(), &mut [0_u8]); } - #[test] - fn resolves_child_and_hashes_ancestor_chain() { - let parent_pid = std::process::id(); + /// Spawns `identity_helper_process` with optional extra argv entries and + /// waits for its readiness signal. Only the spawn itself reports errors: + /// callers decide how to react (the long-cmdline test skips on E2BIG). + fn spawn_identity_helper(extra_args: &[String]) -> std::io::Result { let mut child = std::process::Command::new(std::env::current_exe().unwrap()) .args(["--ignored", "--exact", "tests::identity_helper_process"]) + .args(extra_args) .stdin(std::process::Stdio::piped()) .stdout(std::process::Stdio::piped()) - .spawn() - .expect("spawn child"); + .spawn()?; let mut child_stdout = std::io::BufReader::new(child.stdout.take().expect("child stdout")); let mut line = String::new(); // Wait until the helper has entered the test before taking the @@ -545,7 +625,13 @@ mod tests { break; } } - let child = ChildGuard(child); + Ok(ChildGuard(child)) + } + + #[test] + fn resolves_child_and_hashes_ancestor_chain() { + let parent_pid = std::process::id(); + let child = spawn_identity_helper(&[]).expect("spawn child"); let identity = ProcfsIdentityResolver::for_process_tree(parent_pid) .resolve(child.0.id()) @@ -579,6 +665,41 @@ mod tests { assert!(cache.lock().unwrap().is_empty()); } + #[test] + fn hash_executable_matches_one_shot_sha256_over_the_same_file() { + let current_exe = std::env::current_exe().unwrap(); + let mut executable = std::fs::File::open(¤t_exe).unwrap(); + + let streamed = hash_executable(std::process::id(), &mut executable).unwrap(); + let one_shot = openshell_crypto::sha256(&std::fs::read(¤t_exe).unwrap()).unwrap(); + + assert_eq!(streamed.as_bytes(), &one_shot); + } + + #[test] + fn cache_executable_digest_evicts_one_entry_at_capacity() { + let sentinel: Sha256Digest = "ab".repeat(32).parse().unwrap(); + let cache = Mutex::new(HashMap::new()); + for index in 0..EXECUTABLE_DIGEST_CACHE_CAPACITY as u64 { + cache_executable_digest(&cache, synthetic_cache_key(index), sentinel); + } + assert_eq!( + cache.lock().unwrap().len(), + EXECUTABLE_DIGEST_CACHE_CAPACITY + ); + + let newest_key = synthetic_cache_key(EXECUTABLE_DIGEST_CACHE_CAPACITY as u64); + let newest_sentinel: Sha256Digest = "cd".repeat(32).parse().unwrap(); + cache_executable_digest(&cache, newest_key, newest_sentinel); + + // Crossing the capacity evicts exactly one entry, not the whole cache. + // Which entry drops is arbitrary, so only the size and the newest + // mapping are asserted. + let cache = cache.lock().unwrap(); + assert_eq!(cache.len(), EXECUTABLE_DIGEST_CACHE_CAPACITY); + assert_eq!(cache.get(&newest_key), Some(&newest_sentinel)); + } + #[test] fn process_tree_root_does_not_escape_into_host_ancestry() { let pid = std::process::id(); @@ -588,4 +709,198 @@ mod tests { assert!(identity.ancestors.is_empty()); } + + fn synthetic_cache_key(index: u64) -> ExecutableCacheKey { + ExecutableCacheKey { + device: 1, + inode: index, + size: 1, + mtime: 0, + mtime_nsec: 0, + ctime: 0, + ctime_nsec: 0, + } + } + + #[test] + fn resolves_child_with_cmdline_beyond_the_read_limit() { + // The kernel caps a single argument at a size that already reaches + // the whole-cmdline cap, so one oversized argument fails spawn with + // E2BIG before the child ever runs. The cmdline is instead pushed + // past CMDLINE_READ_LIMIT with several arguments that each stay under + // the per-argument limit. Separately, the whole argv+env list is + // capped at RLIMIT_STACK/4, so a host with a small stack limit can + // still reject the filler list with E2BIG; that skips the test + // instead of failing it. + let filler = "x".repeat(64 * 1024); + let fillers = [filler.clone(), filler.clone(), filler]; + let parent_pid = std::process::id(); + let child = match spawn_identity_helper(&fillers) { + Ok(child) => child, + Err(error) if error.kind() == std::io::ErrorKind::ArgumentListTooLong => { + eprintln!( + "skipping: host argv+env limit (RLIMIT_STACK/4) too small for the filler arguments" + ); + return; + } + Err(error) => panic!("spawn child: {error}"), + }; + + let identity = ProcfsIdentityResolver::for_process_tree(parent_pid) + .resolve(child.0.id()) + .expect("resolve child identity"); + assert!(identity.executable.digest.is_some()); + + let cmdline = read_process_cmdline(child.0.id()).expect("read capped cmdline"); + assert!(cmdline.len() <= CMDLINE_READ_LIMIT); + // The capped cmdline ends on an argument boundary, not mid-argument. + assert_eq!(cmdline.last(), Some(&0)); + // Every surviving filler argument is complete: the argv prefix + // (executable path plus harness flags) is far below one filler, so + // the cut lands inside the second filler and only the first one, + // which ends before the cap, survives whole. + let complete_fillers = cmdline + .split(|byte| *byte == 0) + .filter(|argument| !argument.is_empty() && argument.iter().all(|byte| *byte == b'x')) + .count(); + assert_eq!(complete_fillers, 1); + } + + #[test] + fn cap_cmdline_keeps_cmdline_at_the_limit_whole() { + let mut buffer = b"/bin/one\0".to_vec(); + buffer.extend(vec![b'a'; CMDLINE_READ_LIMIT - buffer.len() - 1]); + buffer.push(0); + assert_eq!(buffer.len(), CMDLINE_READ_LIMIT); + + assert_eq!(cap_cmdline(buffer.clone()), buffer); + } + + #[test] + fn cap_cmdline_truncates_over_limit_cmdline_and_drops_last_argument() { + let mut buffer = b"/bin/one\0".to_vec(); + buffer.extend(vec![b'a'; CMDLINE_READ_LIMIT - buffer.len() - 1]); + buffer.push(0); + buffer.push(b'x'); + assert_eq!(buffer.len(), CMDLINE_READ_LIMIT + 1); + + assert_eq!(cap_cmdline(buffer), b"/bin/one\0"); + } + + #[test] + fn cap_cmdline_drops_argument_terminated_at_the_cut_point() { + // The NUL terminating the second argument is the last byte of the + // cut region: the argument is complete, but the cap deliberately + // drops it rather than recover boundary-case arguments. + let mut buffer = b"/bin/one\0".to_vec(); + buffer.extend(vec![b'a'; CMDLINE_READ_LIMIT - buffer.len() - 1]); + buffer.push(0); + buffer.extend(b"/bin/two\0"); + assert!(buffer.len() > CMDLINE_READ_LIMIT); + + assert_eq!(cap_cmdline(buffer), b"/bin/one\0"); + } + + #[test] + fn cap_cmdline_drops_sole_argument_over_the_limit_entirely() { + let buffer = vec![b'a'; CMDLINE_READ_LIMIT + 1]; + + assert_eq!(cap_cmdline(buffer), Vec::::new()); + } + + #[test] + fn cap_cmdline_passes_short_cmdlines_through() { + let without_trailing_nul = b"/bin/one\0/bin/two".to_vec(); + assert_eq!( + cap_cmdline(without_trailing_nul.clone()), + without_trailing_nul + ); + + assert_eq!(cap_cmdline(Vec::new()), Vec::::new()); + } + + #[test] + fn read_process_cmdline_maps_a_missing_process_to_a_read_error() { + // u32::MAX - 1 is beyond the kernel PID ceiling, so reading + // /proc//cmdline fails. The message prefix must match what + // std::fs::read produced, since the string reaches identity-source + // denial paths downstream. + let pid = u32::MAX - 1; + match read_process_cmdline(pid) { + Err(ResolveError::Failed(message)) => { + let expected = format!("read /proc/{pid}/cmdline: "); + assert!( + message.starts_with(&expected), + "unexpected error message: {message}" + ); + } + other => panic!("expected a failed cmdline read, got {other:?}"), + } + } + + #[test] + fn parse_process_status_reads_ppid_and_innermost_nspid() { + let status = "Name:\tbash\nUmask:\t0022\nState:\tS\nPPid:\t1234\n\ + NSpid:\t5678\t9\n"; + let parsed = parse_process_status(status); + assert_eq!(parsed.ppid, Some(1234)); + // The last NSpid field is the innermost namespace PID. + assert_eq!(parsed.innermost_nspid, Some(9)); + } + + #[test] + fn parse_process_status_keeps_a_zero_ppid_distinct_from_a_missing_one() { + // Some(0) vs None is load-bearing: the ancestry walk's `parent > 0` + // termination filter relies on a present-but-zero PPid being Some. + let parsed = parse_process_status("Name:\tkthreadd\nPPid:\t0\nNSpid:\t2\n"); + assert_eq!(parsed.ppid, Some(0)); + assert_eq!(parsed.innermost_nspid, Some(2)); + } + + #[test] + fn parse_process_status_reports_missing_fields_as_none() { + let parsed = parse_process_status("Name:\tbash\nState:\tS\n"); + assert_eq!(parsed.ppid, None); + assert_eq!(parsed.innermost_nspid, None); + } + + #[test] + fn parse_process_status_reports_garbage_as_none() { + let parsed = parse_process_status(""); + assert_eq!(parsed.ppid, None); + assert_eq!(parsed.innermost_nspid, None); + + let parsed = parse_process_status("PPid:\tnot-a-pid\nNSpid:\talso not pids\n"); + assert_eq!(parsed.ppid, None); + assert_eq!(parsed.innermost_nspid, None); + } + + #[test] + fn read_process_status_returns_none_for_a_missing_process() { + // u32::MAX - 1 is beyond the kernel PID ceiling, so the status file + // cannot be read and the namespace walk terminates on this hop. + // parse_process_status alone never yields None: only the failed read + // does, which keeps a garbage status distinct from an absent one. + assert_eq!(read_process_status(u32::MAX - 1), None); + } + + #[test] + fn collect_cmdline_paths_preserves_order_and_deduplicates() { + let own = b"/own/exe\0/usr/bin/dupe\0relative\0".to_vec(); + let ancestors = [ + b"/anc/one\0/usr/bin/dupe\0".to_vec(), + b"/anc/one\0/anc/two\0".to_vec(), + ]; + let excluded_paths = vec![std::path::PathBuf::from("/own/exe")]; + + let paths = + collect_cmdline_paths(&own, ancestors.iter().map(Vec::as_slice), &excluded_paths); + + let expected = [ + std::path::PathBuf::from("/usr/bin/dupe"), + std::path::PathBuf::from("/anc/one"), + std::path::PathBuf::from("/anc/two"), + ]; + assert_eq!(paths, expected); + } } diff --git a/crates/openshell-bootstrap/src/oidc_token.rs b/crates/openshell-bootstrap/src/oidc_token.rs index 85dd29d791..9677d1e720 100644 --- a/crates/openshell-bootstrap/src/oidc_token.rs +++ b/crates/openshell-bootstrap/src/oidc_token.rs @@ -32,6 +32,11 @@ pub struct OidcTokenBundle { /// OIDC client ID used to obtain the token. pub client_id: String, + + /// Whether the token came from the `client_credentials` grant, so it can + /// be renewed by repeating that grant. Absent in older bundles. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub client_credentials: bool, } /// Path to the stored OIDC token bundle for a gateway. @@ -162,6 +167,7 @@ mod tests { expires_at: None, issuer: "https://issuer.example.com".to_string(), client_id: "openshell-cli".to_string(), + client_credentials: false, }; assert!(store_oidc_token("../escape", &bundle).is_err()); assert!(load_oidc_token("../escape").is_none()); @@ -169,6 +175,31 @@ mod tests { }); } + #[test] + fn client_credentials_marker_is_optional_on_disk() { + let old: OidcTokenBundle = serde_json::from_str( + r#"{"access_token":"t","issuer":"https://issuer","client_id":"c"}"#, + ) + .unwrap(); + assert!(!old.client_credentials); + assert!( + !serde_json::to_string(&old) + .unwrap() + .contains("client_credentials") + ); + + let marked = OidcTokenBundle { + client_credentials: true, + ..old + }; + let json = serde_json::to_string(&marked).unwrap(); + assert!( + serde_json::from_str::(&json) + .unwrap() + .client_credentials + ); + } + #[test] fn expiry_checks_distinguish_refresh_window_from_actual_expiry() { let now = std::time::SystemTime::now() @@ -181,6 +212,7 @@ mod tests { expires_at: Some(now + 10), issuer: "https://issuer.example.com".to_string(), client_id: "openshell-cli".to_string(), + client_credentials: false, }; assert!(is_token_expired(&bundle)); @@ -222,6 +254,7 @@ mod tests { expires_at: None, issuer: "https://issuer.example.com".to_string(), client_id: "openshell-cli".to_string(), + client_credentials: false, }; store_oidc_token("alpha", &bundle).unwrap(); diff --git a/crates/openshell-cli/Cargo.toml b/crates/openshell-cli/Cargo.toml index 0cdd32ade0..43678b8aa7 100644 --- a/crates/openshell-cli/Cargo.toml +++ b/crates/openshell-cli/Cargo.toml @@ -18,6 +18,7 @@ path = "src/main.rs" openshell-crypto = { workspace = true } openshell-bootstrap = { path = "../openshell-bootstrap" } openshell-core = { path = "../openshell-core", default-features = false } +openshell-otel = { path = "../openshell-otel", default-features = false } openshell-policy = { path = "../openshell-policy" } openshell-providers = { path = "../openshell-providers" } openshell-sdk = { path = "../openshell-sdk" } @@ -97,3 +98,4 @@ serde_json = { workspace = true } temp-env = "0.3" tokio-stream = { workspace = true } url = { workspace = true } +wiremock = "0.6" diff --git a/crates/openshell-cli/src/auth.rs b/crates/openshell-cli/src/auth.rs index f84e203d7c..44b4c50ead 100644 --- a/crates/openshell-cli/src/auth.rs +++ b/crates/openshell-cli/src/auth.rs @@ -540,7 +540,9 @@ mod tests { )); // Simulate a browser XHR POST. - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let body = r#"{"token":"test-jwt-123","code":"ABC-1234"}"#; let request = format!( "POST /callback HTTP/1.1\r\n\ @@ -571,7 +573,9 @@ mod tests { TEST_GATEWAY.to_string(), )); - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let body = r#"{"token":"jwt","code":"ABC-1234"}"#; let request = format!( "POST /callback HTTP/1.1\r\n\ @@ -616,7 +620,9 @@ mod tests { TEST_GATEWAY.to_string(), )); - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let body = r#"{"token":"jwt","code":"ABC-1234"}"#; let request = format!( "POST /callback HTTP/1.1\r\n\ @@ -663,7 +669,9 @@ mod tests { )); // POST without Origin header. - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let body = r#"{"token":"jwt","code":"ABC-1234"}"#; let request = format!( "POST /callback HTTP/1.1\r\n\ @@ -707,7 +715,9 @@ mod tests { TEST_GATEWAY.to_string(), )); - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let body = r#"{"token":"test-jwt","code":"WRONG-CODE"}"#; let request = format!( "POST /callback HTTP/1.1\r\n\ @@ -755,7 +765,9 @@ mod tests { )); // POST with no body. - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let request = format!( "POST /callback HTTP/1.1\r\n\ Host: 127.0.0.1\r\n\ @@ -792,7 +804,9 @@ mod tests { )); // Send OPTIONS preflight with correct origin. - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let preflight = format!( "OPTIONS /callback HTTP/1.1\r\n\ Host: 127.0.0.1\r\n\ @@ -816,7 +830,9 @@ mod tests { ); // Now send the actual POST — the server should still be listening. - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let body = r#"{"token":"jwt-after-preflight","code":"ABC-1234"}"#; let request = format!( "POST /callback HTTP/1.1\r\n\ @@ -847,7 +863,9 @@ mod tests { TEST_GATEWAY.to_string(), )); - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let body = r#"{"token":"fragmented-jwt","code":"ABC-1234"}"#; let headers = format!( "POST /callback HTTP/1.1\r\n\ @@ -880,7 +898,9 @@ mod tests { TEST_GATEWAY.to_string(), )); - let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let request = format!( "GET /callback?token=jwt&code=ABC-1234 HTTP/1.1\r\n\ Host: 127.0.0.1\r\n\ diff --git a/crates/openshell-cli/src/callback.rs b/crates/openshell-cli/src/callback.rs index 1045c66269..96fd34b46b 100644 --- a/crates/openshell-cli/src/callback.rs +++ b/crates/openshell-cli/src/callback.rs @@ -236,13 +236,13 @@ mod tests { IpAddr::V6("::1".parse::().expect("v6")), addr.port(), ); - let stream = tokio::net::TcpStream::connect(v6) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(v6) .await .expect("[::1] callback must be reachable"); drop(stream); let v4 = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), addr.port()); - let refused = tokio::net::TcpStream::connect(v4).await; + let refused = openshell_core::net::connect_tcp_nodelay_best_effort(v4).await; assert!(refused.is_err(), "IPv4 loopback must not reach a ::1 bind"); }) .await; diff --git a/crates/openshell-cli/src/completers.rs b/crates/openshell-cli/src/completers.rs index 2b88046fb9..36c9aa08f2 100644 --- a/crates/openshell-cli/src/completers.rs +++ b/crates/openshell-cli/src/completers.rs @@ -11,7 +11,7 @@ use openshell_bootstrap::{list_gateways, load_active_gateway, load_gateway_metad use openshell_core::ObjectName; use openshell_core::proto::{ListProvidersRequest, ListSandboxesRequest, ListWorkspacesRequest}; -use crate::oidc_auth::oidc_refresh_token; +use crate::oidc_auth::oidc_renew_token; use crate::tls::{GrpcClient, TlsOptions, grpc_client}; /// Complete gateway names from local metadata files (no network call). @@ -136,8 +136,9 @@ async fn completion_grpc_client(server: &str, gateway_name: &str) -> Option { if let Some(bundle) = load_oidc_token(gateway_name) { if is_token_expired(&bundle) { - match oidc_refresh_token( + match oidc_renew_token( &bundle, + meta.oidc_audience.as_deref(), meta.oidc_scopes.as_deref(), tls_opts.gateway_insecure, ) diff --git a/crates/openshell-cli/src/main.rs b/crates/openshell-cli/src/main.rs index 7411629483..4fa79b9df4 100644 --- a/crates/openshell-cli/src/main.rs +++ b/crates/openshell-cli/src/main.rs @@ -193,8 +193,9 @@ fn apply_auth_with_status(tls: &mut TlsOptions, gateway_name: &str) -> Option, /// Add or merge an endpoint: host:port[:access[:protocol[:enforcement[:options]]]]. - /// Options include allowed-ip=..., credential rewrite flags, and - /// allow-uninspected-credentials. + /// Options include allowed-ip=..., allow-encoded-slash, credential + /// rewrite flags, and allow-uninspected-credentials. #[arg(long = "add-endpoint")] add_endpoints: Vec, @@ -2775,7 +2776,12 @@ async fn run_async() -> Result<()> { // Top-level status // ----------------------------------------------------------- Some(Commands::Status { output }) => { - if let Ok(ctx) = resolve_gateway(&cli.gateway, &cli.gateway_endpoint) { + // A gateway named on the command line must be registered. + let resolved = match resolve_gateway(&cli.gateway, &cli.gateway_endpoint) { + Err(err) if cli.gateway.is_some() => return Err(err), + resolved => resolved, + }; + if let Ok(ctx) = resolved { let mut tls = tls.with_gateway_name(&ctx.name); let auth_error = apply_auth_with_status(&mut tls, &ctx.name); run::gateway_status( diff --git a/crates/openshell-cli/src/oidc_auth.rs b/crates/openshell-cli/src/oidc_auth.rs index fa46f67086..33edc36169 100644 --- a/crates/openshell-cli/src/oidc_auth.rs +++ b/crates/openshell-cli/src/oidc_auth.rs @@ -30,6 +30,7 @@ use tokio::sync::oneshot; use tracing::debug; const AUTH_TIMEOUT: Duration = Duration::from_mins(2); +const CLIENT_SECRET_ENV: &str = "OPENSHELL_OIDC_CLIENT_SECRET"; /// OIDC discovery document (subset of fields we need). #[derive(Debug, Deserialize)] @@ -277,7 +278,8 @@ pub async fn oidc_browser_auth_flow( /// Run the OIDC Client Credentials flow (for CI/automation). /// -/// Reads `OPENSHELL_OIDC_CLIENT_SECRET` from the environment. +/// Reads `OPENSHELL_OIDC_CLIENT_SECRET` from the environment. The secret is +/// never stored; the returned bundle is marked so it can be renewed later. pub async fn oidc_client_credentials_flow( issuer: &str, client_id: &str, @@ -285,9 +287,9 @@ pub async fn oidc_client_credentials_flow( scopes: Option<&str>, insecure: bool, ) -> Result { - let client_secret = std::env::var("OPENSHELL_OIDC_CLIENT_SECRET").map_err(|_| { + let client_secret = std::env::var(CLIENT_SECRET_ENV).map_err(|_| { miette::miette!( - "OPENSHELL_OIDC_CLIENT_SECRET environment variable is required for client credentials flow" + "{CLIENT_SECRET_ENV} environment variable is required for client credentials flow" ) })?; @@ -312,11 +314,10 @@ pub async fn oidc_client_credentials_flow( .await .map_err(|e| miette::miette!("client credentials token exchange failed: {e}"))?; - Ok(bundle_from_oauth2_response( - &token_response, - issuer, - client_id, - )) + Ok(OidcTokenBundle { + client_credentials: true, + ..bundle_from_oauth2_response(&token_response, issuer, client_id) + }) } /// Run the OIDC Device Authorization Grant flow (RFC 8628). @@ -482,7 +483,7 @@ pub async fn oidc_device_code_flow( /// /// Preserves the existing refresh token if the server does not return a new /// one (per OAuth 2.0 spec, the refresh response may omit `refresh_token`). -pub async fn oidc_refresh_token( +async fn oidc_refresh_token( bundle: &OidcTokenBundle, scopes: Option<&str>, insecure: bool, @@ -523,9 +524,39 @@ fn bundle_from_refresh_output( expires_at, issuer: previous.issuer.clone(), client_id: previous.client_id.clone(), + client_credentials: previous.client_credentials, } } +/// Renew a stored OIDC token. +/// +/// Uses the refresh token when there is one. A client credentials token has +/// none, so it is renewed by repeating that grant, which needs +/// `OPENSHELL_OIDC_CLIENT_SECRET` in the environment. +pub async fn oidc_renew_token( + bundle: &OidcTokenBundle, + audience: Option<&str>, + scopes: Option<&str>, + insecure: bool, +) -> Result { + if bundle.refresh_token.is_some() || !bundle.client_credentials { + return oidc_refresh_token(bundle, scopes, insecure).await; + } + if std::env::var_os(CLIENT_SECRET_ENV).is_none() { + return Err(miette::miette!( + "token was issued via client credentials; set {CLIENT_SECRET_ENV} to renew it, or run: openshell gateway login" + )); + } + oidc_client_credentials_flow( + &bundle.issuer, + &bundle.client_id, + audience, + scopes, + insecure, + ) + .await +} + /// Ensure we have a valid OIDC token bundle for the given gateway, refreshing if needed. pub async fn ensure_valid_oidc_token_bundle( gateway_name: &str, @@ -547,10 +578,14 @@ pub async fn ensure_valid_oidc_token_bundle( gateway = gateway_name, "OIDC token expired, attempting refresh" ); - let scopes = openshell_bootstrap::load_gateway_metadata(gateway_name) - .ok() - .and_then(|metadata| metadata.oidc_scopes); - let refreshed = oidc_refresh_token(&bundle, scopes.as_deref(), insecure).await?; + let metadata = openshell_bootstrap::load_gateway_metadata(gateway_name).ok(); + let refreshed = oidc_renew_token( + &bundle, + metadata.as_ref().and_then(|m| m.oidc_audience.as_deref()), + metadata.as_ref().and_then(|m| m.oidc_scopes.as_deref()), + insecure, + ) + .await?; openshell_bootstrap::oidc_token::store_oidc_token(gateway_name, &refreshed)?; Ok(refreshed) } @@ -582,6 +617,7 @@ fn bundle_from_oauth2_response( expires_at: resp.expires_in().map(|ei| now + ei.as_secs()), issuer: issuer.to_string(), client_id: client_id.to_string(), + client_credentials: false, } } @@ -743,6 +779,103 @@ fn html_response(status: StatusCode, message: &str) -> Response> { #[cfg(test)] mod tests { use super::*; + use wiremock::matchers::{body_string_contains, method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + + fn expired_bundle(issuer: &str, client_credentials: bool) -> OidcTokenBundle { + OidcTokenBundle { + access_token: "expired-access".to_string(), + refresh_token: None, + expires_at: Some(0), + issuer: issuer.to_string(), + client_id: "client-id".to_string(), + client_credentials, + } + } + + #[test] + fn expired_client_credentials_bundle_renews_without_refresh_token() { + let rt = tokio::runtime::Runtime::new().unwrap(); + let renewed = temp_env::with_var(CLIENT_SECRET_ENV, Some("s3cret"), || { + rt.block_on(async { + let server = MockServer::start().await; + let issuer = server.uri(); + Mock::given(method("GET")) + .and(path("/.well-known/openid-configuration")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "issuer": issuer, + "authorization_endpoint": format!("{issuer}/authorize"), + "token_endpoint": format!("{issuer}/token"), + }))) + .mount(&server) + .await; + Mock::given(method("POST")) + .and(path("/token")) + .and(body_string_contains("grant_type=client_credentials")) + .and(body_string_contains("client_id=client-id")) + .and(body_string_contains("client_secret=s3cret")) + .and(body_string_contains("audience=openshell-api")) + .and(body_string_contains("scope=custom")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "access_token": "renewed-access", + "token_type": "bearer", + "expires_in": 300, + }))) + .expect(1) + .mount(&server) + .await; + + oidc_renew_token( + &expired_bundle(&issuer, true), + Some("openshell-api"), + Some("custom"), + false, + ) + .await + .unwrap() + }) + }); + + assert_eq!(renewed.access_token, "renewed-access"); + assert!(renewed.client_credentials); + assert!(renewed.refresh_token.is_none()); + assert!(renewed.expires_at.is_some_and(|at| at > 0)); + assert!(!serde_json::to_string(&renewed).unwrap().contains("s3cret")); + } + + #[test] + fn client_credentials_renewal_without_secret_names_the_secret() { + let rt = tokio::runtime::Runtime::new().unwrap(); + let err = temp_env::with_var_unset(CLIENT_SECRET_ENV, || { + rt.block_on(oidc_renew_token( + &expired_bundle("https://issuer.example", true), + None, + None, + false, + )) + .unwrap_err() + }); + + let message = err.to_string(); + assert!(message.contains(CLIENT_SECRET_ENV), "{message}"); + assert!(!message.contains("no refresh token"), "{message}"); + } + + #[test] + fn interactive_bundle_without_refresh_token_keeps_refresh_error() { + let rt = tokio::runtime::Runtime::new().unwrap(); + let err = temp_env::with_var(CLIENT_SECRET_ENV, Some("s3cret"), || { + rt.block_on(oidc_renew_token( + &expired_bundle("https://issuer.example", false), + None, + None, + false, + )) + .unwrap_err() + }); + + assert!(err.to_string().contains("no refresh token"), "{err}"); + } #[test] fn http_client_secure_rejects_self_signed() { @@ -910,6 +1043,7 @@ mod tests { expires_at: Some(0), issuer: "https://issuer.example".to_string(), client_id: "client-id".to_string(), + client_credentials: false, }; let refreshed = diff --git a/crates/openshell-cli/src/policy_update.rs b/crates/openshell-cli/src/policy_update.rs index 2907afaaed..6aac515e14 100644 --- a/crates/openshell-cli/src/policy_update.rs +++ b/crates/openshell-cli/src/policy_update.rs @@ -268,11 +268,14 @@ fn l7_target_to_proto(target: &L7RuleTarget) -> ProtoL7RuleTarget { } } -fn ensure_websocket_credential_rewrite_protocol( +/// Require the endpoint protocol segment to be one of `allowed` for `option`. +fn ensure_option_protocol( spec: &str, endpoint: &NetworkEndpoint, + option: &str, + allowed: &[&str], ) -> Result<()> { - if matches!(endpoint.protocol.as_str(), "rest" | "websocket") { + if allowed.contains(&endpoint.protocol.as_str()) { return Ok(()); } let protocol = if endpoint.protocol.is_empty() { @@ -280,25 +283,13 @@ fn ensure_websocket_credential_rewrite_protocol( } else { endpoint.protocol.as_str() }; + let allowed = allowed + .iter() + .map(|protocol| format!("'{protocol}'")) + .collect::>() + .join(" or "); Err(miette!( - "websocket-credential-rewrite endpoint option requires --add-endpoint protocol segment to be 'rest' or 'websocket'; got '{protocol}' in '{spec}'" - )) -} - -fn ensure_request_body_credential_rewrite_protocol( - spec: &str, - endpoint: &NetworkEndpoint, -) -> Result<()> { - if endpoint.protocol == "rest" { - return Ok(()); - } - let protocol = if endpoint.protocol.is_empty() { - "" - } else { - endpoint.protocol.as_str() - }; - Err(miette!( - "request-body-credential-rewrite endpoint option requires --add-endpoint protocol segment to be 'rest'; got '{protocol}' in '{spec}'" + "{option} endpoint option requires --add-endpoint protocol segment to be {allowed}; got '{protocol}' in '{spec}'" )) } @@ -504,12 +495,16 @@ fn apply_add_endpoint_options( "allow-uninspected-credentials" => { endpoint.allow_uninspected_credentials = true; } + "allow-encoded-slash" => { + ensure_option_protocol(spec, endpoint, option, &["rest", "websocket"])?; + endpoint.allow_encoded_slash = true; + } "websocket-credential-rewrite" => { - ensure_websocket_credential_rewrite_protocol(spec, endpoint)?; + ensure_option_protocol(spec, endpoint, option, &["rest", "websocket"])?; endpoint.websocket_credential_rewrite = true; } "request-body-credential-rewrite" => { - ensure_request_body_credential_rewrite_protocol(spec, endpoint)?; + ensure_option_protocol(spec, endpoint, option, &["rest"])?; endpoint.request_body_credential_rewrite = true; } _ if option.starts_with(ALLOWED_IP_OPTION_PREFIX) => { @@ -521,7 +516,7 @@ fn apply_add_endpoint_options( } _ => { return Err(miette!( - "--add-endpoint options segment supports only 'allow-uninspected-credentials', 'websocket-credential-rewrite', 'request-body-credential-rewrite', and 'allowed-ip='; got '{option}' in '{spec}'" + "--add-endpoint options segment supports only 'allow-uninspected-credentials', 'allow-encoded-slash', 'websocket-credential-rewrite', 'request-body-credential-rewrite', and 'allowed-ip='; got '{option}' in '{spec}'" )); } } @@ -831,6 +826,45 @@ mod tests { assert!(endpoint.request_body_credential_rewrite); } + #[test] + fn parse_add_endpoint_enables_allow_encoded_slash_on_rest_and_websocket() { + for protocol in ["rest", "websocket"] { + let plan = build_policy_update_plan( + &[format!( + "registry.npmjs.org:443:full:{protocol}:enforce:allow-encoded-slash" + )], + &[], + &[], + &[], + &[], + &[], + None, + ) + .expect("plan should build"); + + let PolicyMergeOp::AddRule { rule, .. } = &plan.preview_operations[0] else { + panic!("expected add-rule preview"); + }; + assert!(rule.endpoints[0].allow_encoded_slash, "{protocol}"); + } + } + + #[test] + fn allow_encoded_slash_rejects_endpoint_without_http_protocol() { + for spec in [ + "registry.npmjs.org:443::::allow-encoded-slash", + "registry.npmjs.org:443::tcp::allow-encoded-slash", + ] { + let error = + build_policy_update_plan(&[spec.to_string()], &[], &[], &[], &[], &[], None) + .expect_err("plan should fail"); + assert!( + error.to_string().contains("to be 'rest' or 'websocket'"), + "{error}" + ); + } + } + #[test] fn parse_add_endpoint_enables_allow_uninspected_credentials() { let plan = build_policy_update_plan( diff --git a/crates/openshell-cli/src/ssh.rs b/crates/openshell-cli/src/ssh.rs index d161a4c165..5eb6738ddd 100644 --- a/crates/openshell-cli/src/ssh.rs +++ b/crates/openshell-cli/src/ssh.rs @@ -34,7 +34,6 @@ use std::process::{Command, ExitStatus, Stdio}; use std::sync::OnceLock; use std::time::{Duration, Instant}; use tokio::io::AsyncWriteExt; -use tokio::net::TcpStream; use tokio::process::{Child, Command as TokioCommand}; use tokio_stream::wrappers::ReceiverStream; use tonic::Code; @@ -920,7 +919,7 @@ where async fn probe_forward_listener(spec: &ForwardSpec) -> std::result::Result<(), String> { match tokio::time::timeout( FORWARD_LISTENER_CONNECT_TIMEOUT, - TcpStream::connect((forward_probe_host(spec), spec.port)), + openshell_core::net::connect_tcp_nodelay_best_effort((forward_probe_host(spec), spec.port)), ) .await { @@ -1408,6 +1407,19 @@ fn validate_sandbox_source_path(workspace_root: &str, path: &str) -> Result String { + let operand = if sandbox_path.starts_with('/') { + sandbox_path.to_string() + } else { + format!("./{sandbox_path}") + }; + let path = shell_escape(&operand); + format!("pwd -P && if [ -e {path} ]; then realpath {path}; fi") +} + /// Discover the workspace root and resolve every symlink in `sandbox_path` in /// one SSH probe, then refuse the result if it lands outside the workspace. /// @@ -1422,10 +1434,7 @@ async fn resolve_sandbox_source_path( session: &SshSessionConfig, sandbox_path: &str, ) -> Result { - let resolve_cmd = format!( - "pwd -P && realpath -e -- {path}", - path = shell_escape(sandbox_path) - ); + let resolve_cmd = resolve_probe_command(sandbox_path); let output = ssh_run_capture_stdout(session, &resolve_cmd) .await .wrap_err_with(|| format!("failed to resolve sandbox source path '{sandbox_path}'"))?; @@ -3080,6 +3089,30 @@ mod tests { assert_eq!(lexical_clean_absolute_path("./a"), None); } + #[test] + #[cfg(unix)] + fn resolve_probe_command_is_posix_and_resolves_symlinks() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().canonicalize().unwrap(); + fs::write(root.join("-f"), "x").unwrap(); + std::os::unix::fs::symlink(root.join("-f"), root.join("link")).unwrap(); + let run = |path: &str| { + let output = Command::new("sh") + .arg("-c") + .arg(resolve_probe_command(path)) + .current_dir(&root) + .output() + .unwrap(); + assert!(output.status.success()); + String::from_utf8(output.stdout).unwrap() + }; + let cwd = format!("{}\n", root.display()); + assert!(!resolve_probe_command("a").contains(" -e --")); + assert_eq!(run("link"), format!("{cwd}{}/-f\n", root.display())); + assert_eq!(run("-f"), format!("{cwd}{}/-f\n", root.display())); + assert_eq!(run("missing"), cwd); + } + #[test] fn validate_sandbox_source_path_accepts_workspace_paths() { let workspace_root = "/workspace/project"; diff --git a/crates/openshell-cli/src/tls.rs b/crates/openshell-cli/src/tls.rs index 41381f9a40..fd138aae5b 100644 --- a/crates/openshell-cli/src/tls.rs +++ b/crates/openshell-cli/src/tls.rs @@ -5,6 +5,7 @@ use miette::{IntoDiagnostic, Result, WrapErr}; pub use openshell_core::replica_routing::{ owner_replica, retry_unrouted, routed_to, with_owner_hint, }; +use openshell_otel::EnvTraceContextInterceptor; use openshell_sdk::{AuthConfig, ClientConfig, ClientIdentity, OpenShellClient}; use rustls::{ RootCertStore, @@ -16,7 +17,32 @@ use std::path::PathBuf; use tonic::transport::Channel; /// Concrete gRPC client type used by all commands. -pub type GrpcClient = openshell_sdk::raw::AuthedGrpcClient; +pub type GrpcClient = openshell_sdk::raw::GrpcClient< + tonic::service::interceptor::InterceptedService, +>; + +/// Interceptor stack applied to gateway requests from clients built by +/// [`grpc_client`]. +/// +/// Runs authentication header injection first, then passive W3C trace-context +/// forwarding so a CI pipeline's `TRACEPARENT` extends into the gateway's +/// spans. tonic allows a single interceptor per channel, so the two concerns +/// are composed here. +#[derive(Clone)] +pub struct GatewayInterceptor { + auth: openshell_sdk::EdgeAuthInterceptor, + trace: EnvTraceContextInterceptor, +} + +impl tonic::service::Interceptor for GatewayInterceptor { + fn call( + &mut self, + request: tonic::Request<()>, + ) -> std::result::Result, tonic::Status> { + let request = self.auth.call(request)?; + self.trace.call(request) + } +} #[derive(Clone, Debug, Default)] pub struct TlsOptions { @@ -313,9 +339,10 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result { /// Build the SDK client and return its raw gRPC escape hatch. /// -/// When `tls.edge_token` is set, the returned client is wrapped with an -/// interceptor that injects authentication headers on every request. -/// Otherwise, standard mTLS is used (interceptor is a no-op). +/// The returned client carries a [`GatewayInterceptor`] that injects +/// authentication headers (when a token is set) and passively forwards the +/// `TRACEPARENT`/`TRACESTATE` trace context from the environment on every +/// request. With no token and no trace context, the interceptor is a no-op. pub async fn grpc_client(server: &str, tls: &TlsOptions) -> Result { let config = sdk_client_config(server, tls)?; // Keep the SDK's transport state machine out of every command future. @@ -323,14 +350,71 @@ pub async fn grpc_client(server: &str, tls: &TlsOptions) -> Result { let client = Box::pin(OpenShellClient::connect(config)) .await .into_diagnostic()?; - Ok(client.raw_grpc()) + // tonic allows a single interceptor per channel: compose the SDK's auth + // interceptor (sharing its live bearer slot) with the CLI's passive + // trace-context forwarding. + let interceptor = GatewayInterceptor { + auth: client.auth_interceptor(), + trace: EnvTraceContextInterceptor::from_env(), + }; + Ok(openshell_sdk::raw::GrpcClient::with_interceptor( + client.channel(), + interceptor, + )) } #[cfg(test)] mod tests { - use super::{TlsOptions, sdk_client_config}; - use openshell_sdk::AuthConfig; + use super::{GatewayInterceptor, TlsOptions, sdk_client_config}; + use crate::TEST_ENV_LOCK; + use openshell_otel::EnvTraceContextInterceptor; + use openshell_sdk::{AuthConfig, EdgeAuthInterceptor}; use std::io::Write; + use tonic::service::Interceptor as _; + + #[test] + fn gateway_interceptor_composes_auth_and_trace_forwarding() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let traceparent = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + temp_env::with_vars( + [ + ("TRACEPARENT", Some(traceparent)), + ("TRACESTATE", Some("vendor=value")), + ], + || { + let mut interceptor = GatewayInterceptor { + auth: EdgeAuthInterceptor::new(Some("test-token"), None).unwrap(), + trace: EnvTraceContextInterceptor::from_env(), + }; + + let request = interceptor.call(tonic::Request::new(())).unwrap(); + + assert_eq!( + request + .metadata() + .get("authorization") + .and_then(|value| value.to_str().ok()), + Some("Bearer test-token") + ); + assert_eq!( + request + .metadata() + .get("traceparent") + .and_then(|value| value.to_str().ok()), + Some(traceparent) + ); + assert_eq!( + request + .metadata() + .get("tracestate") + .and_then(|value| value.to_str().ok()), + Some("vendor=value") + ); + }, + ); + } #[test] fn sdk_config_carries_cli_mtls_materials() { diff --git a/crates/openshell-cli/tests/common/mod.rs b/crates/openshell-cli/tests/common/mod.rs index b86741d8b6..c7cab6a33d 100644 --- a/crates/openshell-cli/tests/common/mod.rs +++ b/crates/openshell-cli/tests/common/mod.rs @@ -12,6 +12,15 @@ pub struct CliOutput { } pub fn run(config_dir: &Path, system_dir: &Path, args: &[&str]) -> CliOutput { + run_with_env(config_dir, system_dir, args, &[]) +} + +pub fn run_with_env( + config_dir: &Path, + system_dir: &Path, + args: &[&str], + env: &[(&str, &str)], +) -> CliOutput { let output = Command::new(env!("CARGO_BIN_EXE_openshell")) .args(args) .env("XDG_CONFIG_HOME", config_dir) @@ -20,6 +29,7 @@ pub fn run(config_dir: &Path, system_dir: &Path, args: &[&str]) -> CliOutput { .env("OPENSHELL_NO_BROWSER", "1") .env_remove("OPENSHELL_GATEWAY") .env_remove("OPENSHELL_GATEWAY_ENDPOINT") + .envs(env.iter().copied()) .stdin(Stdio::null()) .output() .expect("run openshell"); diff --git a/crates/openshell-cli/tests/gateway_registration_integration.rs b/crates/openshell-cli/tests/gateway_registration_integration.rs index 28688a4ea5..706ec1dee9 100644 --- a/crates/openshell-cli/tests/gateway_registration_integration.rs +++ b/crates/openshell-cli/tests/gateway_registration_integration.rs @@ -7,7 +7,7 @@ mod common; use std::path::Path; -use common::{run, run_isolated}; +use common::{run, run_isolated, run_with_env}; fn write_gateway_metadata( root: &Path, @@ -93,6 +93,96 @@ fn status_without_gateway_prints_registration_hint() { ); } +#[test] +fn status_with_unregistered_named_gateway_fails() { + let output = run_isolated(&["--gateway", "org-a", "status"]); + assert_ne!(output.code, 0, "{}", output.combined); + assert!( + output.combined.contains("Unknown gateway 'org-a'"), + "{}", + output.combined + ); +} + +#[test] +fn status_with_unregistered_environment_gateway_fails() { + let config_dir = tempfile::tempdir().expect("create user config dir"); + let system_dir = tempfile::tempdir().expect("create system config dir"); + let output = run_with_env( + config_dir.path(), + system_dir.path(), + &["status"], + &[("OPENSHELL_GATEWAY", "org-a")], + ); + assert_ne!(output.code, 0, "{}", output.combined); + assert!( + output.combined.contains("Unknown gateway 'org-a'"), + "{}", + output.combined + ); +} + +#[test] +fn status_without_gateway_json_succeeds() { + let output = run_isolated(&["status", "--output", "json"]); + assert_eq!(output.code, 0, "{}", output.combined); + let status: serde_json::Value = + serde_json::from_str(&output.stdout).expect("parse status JSON"); + assert_eq!(status["status"], "not_configured"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn status_with_unhealthy_registered_gateway_json_succeeds() { + let config_dir = tempfile::tempdir().expect("create user config dir"); + let system_dir = tempfile::tempdir().expect("create system config dir"); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind health endpoint"); + let port = listener.local_addr().expect("local address").port(); + let endpoint = format!("http://127.0.0.1:{port}"); + let server = tokio::spawn(async move { + loop { + let (stream, _) = listener.accept().await.expect("accept health connection"); + openshell_core::net::set_tcp_nodelay_best_effort(&stream); + tokio::spawn(async move { + let service = hyper::service::service_fn(|_| async { + Ok::<_, std::convert::Infallible>( + hyper::Response::builder() + .status(hyper::StatusCode::SERVICE_UNAVAILABLE) + .body(http_body_util::Full::new(bytes::Bytes::new())) + .expect("health response"), + ) + }); + let _ = hyper::server::conn::http1::Builder::new() + .serve_connection(hyper_util::rt::TokioIo::new(stream), service) + .await; + }); + } + }); + write_user_gateway_metadata( + config_dir.path(), + "offline", + &endpoint, + port, + false, + "plaintext", + ); + + let output = run( + config_dir.path(), + system_dir.path(), + &["--gateway", "offline", "status", "--output", "json"], + ); + server.abort(); + assert_eq!(output.code, 0, "{}", output.combined); + let status: serde_json::Value = + serde_json::from_str(&output.stdout).expect("parse status JSON"); + assert!( + matches!(status["status"].as_str(), Some("disconnected" | "error")), + "{status}" + ); +} + #[test] fn gateway_list_table_shows_user_and_system_sources() { let config_dir = tempfile::tempdir().expect("create user config dir"); diff --git a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs index 587910fbaf..8e7c0b190f 100644 --- a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs +++ b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs @@ -3810,6 +3810,7 @@ fn write_oidc_test_credentials( expires_at: Some(expires_at), issuer: issuer.to_string(), client_id: "openshell-cli".to_string(), + client_credentials: false, }) .unwrap(), ) @@ -4117,7 +4118,7 @@ async fn echo_through_local_forward( !forward.is_finished(), "service forward exited before the local listener opened" ); - match tokio::net::TcpStream::connect(("127.0.0.1", port)).await { + match openshell_core::net::connect_tcp_nodelay_best_effort(("127.0.0.1", port)).await { Ok(stream) => break stream, Err(err) => { assert!( diff --git a/crates/openshell-core/src/config.rs b/crates/openshell-core/src/config.rs index 20f6a334b1..25ce4a1c1d 100644 --- a/crates/openshell-core/src/config.rs +++ b/crates/openshell-core/src/config.rs @@ -46,6 +46,9 @@ pub const DEFAULT_DOCKER_NETWORK_NAME: &str = "openshell-docker"; /// Default domain used for browser-facing sandbox service URLs. pub const DEFAULT_SERVICE_ROUTING_DOMAIN: &str = "openshell.localhost"; +/// Default per-sandbox cap for each relay budget (SSH and service forwards). +pub const DEFAULT_MAX_CONNECTIONS_PER_SANDBOX: u32 = 20; + /// Gateway delivery path for supervisor configuration. /// /// `Push` sends configuration over the supervisor session to supervisors that @@ -324,10 +327,23 @@ pub struct Config { /// TTL for SSH session tokens, in seconds. 0 disables expiry. pub ssh_session_ttl_secs: u64, + /// Maximum concurrent SSH relays per sandbox. Must be at least 1. + pub max_ssh_connections_per_sandbox: u32, + + /// Maximum concurrent service forward relays per sandbox. Must be at + /// least 1. Kept apart from the SSH budget so open forwards cannot block + /// SSH access. + pub max_service_connections_per_sandbox: u32, + /// Absolute image preparation and initial supervisor startup budget for new /// sandbox attempts, in seconds. Must be between 1 and 86400, inclusive. pub image_preparation_timeout_seconds: u32, + /// Provisioning repair window, in seconds. The first authenticated + /// supervisor configuration report starts admission repair, which stays + /// armed until Ready. Must be between 1 and 86400, inclusive. + pub provisioning_repair_window_seconds: i64, + /// Maximum gRPC requests allowed per rate-limit window. /// /// When paired with [`Self::grpc_rate_limit_window_secs`], positive values @@ -1047,7 +1063,10 @@ impl Config { credential_drivers: Vec::new(), default_credential_driver: None, ssh_session_ttl_secs: default_ssh_session_ttl_secs(), + max_ssh_connections_per_sandbox: DEFAULT_MAX_CONNECTIONS_PER_SANDBOX, + max_service_connections_per_sandbox: DEFAULT_MAX_CONNECTIONS_PER_SANDBOX, image_preparation_timeout_seconds: 1800, + provisioning_repair_window_seconds: 300, grpc_rate_limit_requests: None, grpc_rate_limit_window_secs: None, service_routing: ServiceRoutingConfig::default(), diff --git a/crates/openshell-core/src/net.rs b/crates/openshell-core/src/net.rs index a9bbc23217..e41beb9c73 100644 --- a/crates/openshell-core/src/net.rs +++ b/crates/openshell-core/src/net.rs @@ -16,7 +16,7 @@ //! use ipnet::{IpNet, Ipv4Net, Ipv6Net}; -use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; use tokio::net::TcpStream; /// Check if a hostname is a known cloud metadata hostname that resolves to an @@ -296,16 +296,35 @@ pub fn set_tcp_nodelay_best_effort(stream: &TcpStream) { } } -/// Connect to `addrs`, then enable `TCP_NODELAY` on a best-effort basis, propagating +/// Connect to `addr`, then enable `TCP_NODELAY` on a best-effort basis, propagating /// any errors from `TcpStream::connect`. /// /// The returned stream is not *guaranteed* to have `TCP_NODELAY` set. -pub async fn connect_tcp_nodelay_best_effort(addrs: &[SocketAddr]) -> std::io::Result { - let stream = TcpStream::connect(addrs).await?; +pub async fn connect_tcp_nodelay_best_effort( + addr: impl tokio::net::ToSocketAddrs, +) -> std::io::Result { + #[allow(clippy::disallowed_methods, reason = "the one sanctioned async dial")] + let stream = TcpStream::connect(addr).await?; set_tcp_nodelay_best_effort(&stream); Ok(stream) } +/// Blocking counterpart of [`connect_tcp_nodelay_best_effort`] for threads that +/// cannot use the async helper, such as a `setns` worker. +pub fn connect_std_tcp_nodelay_best_effort( + addr: impl std::net::ToSocketAddrs, +) -> std::io::Result { + #[allow( + clippy::disallowed_methods, + reason = "the one sanctioned blocking dial" + )] + let stream = std::net::TcpStream::connect(addr)?; + if let Err(e) = stream.set_nodelay(true) { + tracing::debug!(error = %e, "failed to set TCP_NODELAY"); + } + Ok(stream) +} + #[cfg(test)] mod tests { use super::*; @@ -735,6 +754,7 @@ mod tests { .await .expect("bind"); let addr = listener.local_addr().expect("local addr"); + #[allow(clippy::disallowed_methods, reason = "needs a stream without nodelay")] let stream = TcpStream::connect(addr).await.expect("connect"); set_tcp_nodelay_best_effort(&stream); @@ -748,7 +768,7 @@ mod tests { .expect("bind"); let addr = listener.local_addr().expect("local addr"); - let stream = connect_tcp_nodelay_best_effort(&[addr]) + let stream = connect_tcp_nodelay_best_effort(addr) .await .expect("connect"); assert!(stream.nodelay().expect("query TCP_NODELAY")); diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index aa1d7de706..e5421431f8 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -14,7 +14,7 @@ use bollard::models::{ ContainerCreateBody, ContainerState, ContainerStateStatusEnum, ContainerSummary, ContainerSummaryStateEnum, ContainerWaitResponse, CreateImageInfo, DeviceRequest, HealthConfig, HealthStatusEnum, HostConfig, Mount, MountTmpfsOptions, MountTypeEnum, MountVolumeOptions, - ProgressDetail, SystemInfo, VolumeCreateRequest, + ProgressDetail, SystemInfo, SystemVersion, VolumeCreateRequest, }; use bollard::query_parameters::{ CreateContainerOptionsBuilder, CreateImageOptions, DownloadFromContainerOptionsBuilder, @@ -917,9 +917,12 @@ impl DockerComputeDriver { .map_err(|err| { Error::execution(format!("failed to create Docker client: {err}")) })?; - docker.version().await.map_err(|err| { + let version = docker.version().await.map_err(|err| { Error::execution(format!("failed to query Docker daemon version: {err}")) })?; + if let Some(warning) = unsupported_docker_engine_warning(&version) { + warn!("{warning}"); + } let info = docker.info().await.map_err(|err| { Error::execution(format!("failed to query Docker daemon info: {err}")) })?; @@ -6033,6 +6036,31 @@ fn docker_cdi_gpu_inventory(info: &SystemInfo) -> CdiGpuInventory { ) } +/// Oldest Docker Engine major version the driver is supported on. +const MIN_DOCKER_ENGINE_MAJOR: u64 = 28; + +/// Warn text when the daemon is a Docker Engine older than the supported +/// minimum. Older engines fail sandbox bundle staging with opaque errors. +fn unsupported_docker_engine_warning(version: &SystemVersion) -> Option { + let engine = version + .components + .iter() + .flatten() + .find(|component| component.name == "Engine")?; + let major: u64 = engine + .version + .split(|c: char| !c.is_ascii_digit()) + .next()? + .parse() + .ok()?; + (major < MIN_DOCKER_ENGINE_MAJOR).then(|| { + format!( + "Docker Engine {} is older than the supported minimum {MIN_DOCKER_ENGINE_MAJOR}.0; sandbox creation may fail. See the support matrix in the OpenShell docs.", + engine.version + ) + }) +} + fn docker_info_reports_wsl2(info: &SystemInfo) -> bool { [ info.kernel_version.as_deref(), diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index 70fc34f256..c022adc923 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -3274,6 +3274,38 @@ fn driver_default_gpu_selection_consumes_distinct_devices_for_creates() { ); } +fn engine_version(name: &str, version: &str) -> SystemVersion { + SystemVersion { + components: Some(vec![bollard::models::SystemVersionComponents { + name: name.to_string(), + version: version.to_string(), + details: None, + }]), + ..Default::default() + } +} + +#[test] +fn warns_on_docker_engine_older_than_minimum() { + let warning = unsupported_docker_engine_warning(&engine_version("Engine", "26.1.5+dfsg1")) + .expect("old engine warns"); + assert!(warning.contains("26.1.5+dfsg1"), "{warning}"); + assert!(warning.contains("28.0"), "{warning}"); +} + +#[test] +fn no_warning_for_supported_unknown_or_other_engines() { + for version in [ + engine_version("Engine", "28.0.0"), + engine_version("Engine", "29.9.3"), + engine_version("Engine", "garbage"), + engine_version("Other Engine", "5.8.2"), + SystemVersion::default(), + ] { + assert!(unsupported_docker_engine_warning(&version).is_none()); + } +} + #[test] fn docker_info_reports_wsl2_from_kernel_version() { let info = SystemInfo { diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 49dc465362..d1db98a0d1 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -262,8 +262,13 @@ fails closed. and the routing preface only selects which generation-pinned TLS identity the supervisor demands. -The topology does not support upstream proxies or provider SPIFFE identity, and -the shared supervisor is a single point of failure for the sandboxes it hosts. +The topology does not support upstream proxies or provider SPIFFE identity. +With a live replacement supervisor and a takeover-capable runtime, the gateway +can move a paused sandbox generation to the replacement using a signed takeover +grant. Recovery preserves workload processes and memory, but proxied connections +and interactive sessions must reconnect. If recovery exceeds the configured +deadline, the runtime terminates the workload. It cannot recover a lost workload +Pod or node. See [supervisor recovery](../../docs/kubernetes/high-availability.mdx#recover-from-a-supervisor-failure). Shared sandbox instances export no supervisor OTLP traces or OCSF events and do not relay agent traces; the driver warns at startup when the gateway configures any of them. diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index 28d3161f42..a18b51e1de 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -346,6 +346,17 @@ pub struct KubernetesComputeConfig { pub supervisor_image_pull_policy: Option, /// Cross-pod sandbox/supervisor settings. pub sandbox_runtime: KubernetesSandboxRuntimeConfig, + /// Bound on how long a crash-interrupted, fail-closed sandbox runtime + /// bootstrap may remain stranded before the driver reclaims it, in seconds. Slow + /// VM-backed runtimes (CoCo/peer-pods) need longer than the default. + pub runtime_bootstrap_grace_seconds: u64, + /// Supervisor attach/policy-discover request timeout, in seconds. + /// When set, the driver copies the value into the supervisor pod + /// environment (`OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS`) so slow VM-backed + /// runtimes have time to boot the remote boundary before the supervisor + /// gives up. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub sandbox_attach_request_timeout_seconds: Option, /// Corporate HTTP forward proxy used by the network supervisor for /// policy-approved TLS CONNECT egress. pub https_proxy: Option, @@ -409,6 +420,10 @@ pub struct KubernetesComputeConfig { /// Applied when a `CreateSandbox` request does not specify one. /// Empty string (default) = omit the field, using the cluster default. pub default_runtime_class_name: String, + /// Kubernetes `runtimeClassName` for dedicated supervisor pods. Set only + /// by the gateway administrator; sandbox creators cannot change it, and + /// it never inherits the workload class. Empty (default) = omit the field. + pub supervisor_runtime_class_name: String, /// Pod annotation keys that sandbox templates may set. Each entry is an /// exact key (`example.com/team`) or a prefix wildcard (`example.com/*`). /// Empty (default) rejects every caller-supplied pod annotation. @@ -509,6 +524,8 @@ impl Default for KubernetesComputeConfig { supervisor_image: config::default_supervisor_image(), supervisor_image_pull_policy: None, sandbox_runtime: KubernetesSandboxRuntimeConfig::default(), + runtime_bootstrap_grace_seconds: 300, + sandbox_attach_request_timeout_seconds: None, https_proxy: None, no_proxy: None, proxy_auth_secret_name: None, @@ -526,6 +543,7 @@ impl Default for KubernetesComputeConfig { workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(), workspace_storage_class: String::new(), default_runtime_class_name: String::new(), + supervisor_runtime_class_name: String::new(), allowed_pod_annotations: Vec::new(), allowed_pod_labels: Vec::new(), sa_token_ttl_secs: 3600, @@ -539,6 +557,7 @@ impl Default for KubernetesComputeConfig { impl KubernetesComputeConfig { /// Validate startup configuration without connecting to Kubernetes. pub fn validate_configuration(&self) -> Result<(), String> { + self.validate_bootstrap_windows()?; self.validate_supervisor_topology()?; self.validate_workspace_mode()?; self.validate_provider_spiffe_workload_api_socket_path()?; @@ -549,6 +568,24 @@ impl KubernetesComputeConfig { self.validate_allowed_pod_labels() } + fn validate_bootstrap_windows(&self) -> Result<(), String> { + for (name, value) in [ + ( + "runtime_bootstrap_grace_seconds", + Some(self.runtime_bootstrap_grace_seconds), + ), + ( + "sandbox_attach_request_timeout_seconds", + self.sandbox_attach_request_timeout_seconds, + ), + ] { + if value.is_some_and(|value| !(1..=86_400).contains(&value)) { + return Err(format!("{name} must be between 1 and 86400 seconds")); + } + } + Ok(()) + } + /// The shared supervisor serves only sandbox instances it can isolate: it /// holds no per-sandbox workload identity, upstream proxy credential, or /// proxy CA. @@ -1244,6 +1281,43 @@ mod tests { } } + #[test] + fn bootstrap_grace_defaults_and_parses_from_toml() { + assert_eq!( + KubernetesComputeConfig::default().runtime_bootstrap_grace_seconds, + 300 + ); + assert!( + KubernetesComputeConfig::default() + .sandbox_attach_request_timeout_seconds + .is_none() + ); + let config: KubernetesComputeConfig = toml::from_str( + "runtime_bootstrap_grace_seconds = 900\nsandbox_attach_request_timeout_seconds = 900\n", + ) + .unwrap(); + assert_eq!(config.runtime_bootstrap_grace_seconds, 900); + assert_eq!(config.sandbox_attach_request_timeout_seconds, Some(900)); + assert!(config.validate_configuration().is_ok()); + let partial: KubernetesComputeConfig = toml::from_str("namespace = 'custom'").unwrap(); + assert_eq!(partial.runtime_bootstrap_grace_seconds, 300); + for field in [ + "runtime_bootstrap_grace_seconds", + "sandbox_attach_request_timeout_seconds", + ] { + for value in [0, 86_401] { + let config: KubernetesComputeConfig = + toml::from_str(&format!("{field} = {value}")).unwrap(); + assert!(config.validate_configuration().unwrap_err().contains(field)); + } + for value in [1, 86_400] { + let config: KubernetesComputeConfig = + toml::from_str(&format!("{field} = {value}")).unwrap(); + assert!(config.validate_configuration().is_ok()); + } + } + } + #[test] fn published_kubernetes_example_is_valid_toml() { let docs = include_str!("../../../docs/how-it-works/gateways/configuration.mdx"); @@ -1431,6 +1505,17 @@ mod tests { fn default_runtime_class_name_is_empty() { let cfg = KubernetesComputeConfig::default(); assert!(cfg.default_runtime_class_name.is_empty()); + assert!(cfg.supervisor_runtime_class_name.is_empty()); + } + + #[test] + fn serde_override_supervisor_runtime_class_name() { + let json = serde_json::json!({ + "supervisor_runtime_class_name": "kata-supervisor" + }); + let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap(); + assert_eq!(cfg.supervisor_runtime_class_name, "kata-supervisor"); + assert!(cfg.default_runtime_class_name.is_empty()); } #[test] diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index c08a567eed..e1a4226187 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -612,10 +612,6 @@ fn admission_error(error: tonic::Status) -> KubernetesDriverError { } } const SANDBOX_RUNTIME_RECONCILE_INTERVAL: Duration = Duration::from_secs(30); -/// Unsupervised shared sandbox instances removed at once by one reconcile pass. -/// Bound how long a crash-interrupted, fail-closed bootstrap may remain stranded. -const SANDBOX_RUNTIME_BOOTSTRAP_GRACE: Duration = Duration::from_mins(5); - fn random_sandbox_runtime_token() -> Result { use std::fmt::Write as _; @@ -3544,9 +3540,8 @@ impl KubernetesComputeDriver { .ensure_network_additional_ca_config_map(namespace) .await?; let pods: Api = Api::namespaced(self.client.clone(), namespace); - pods.create( - &PostParams::default(), - &crate::app_listener::configure_supervisor_pod( + let pod = crate::sandbox_runtime::with_runtime_class( + crate::app_listener::configure_supervisor_pod( supervisor_pod( namespace, names, @@ -3583,16 +3578,19 @@ impl KubernetesComputeDriver { .map(|generation| (generation.name.as_str(), generation.digest.as_str())), owner, &self.supervisor_otlp_environment(), + self.config.sandbox_attach_request_timeout_seconds, ) .map_err(KubernetesDriverError::Message)?, &self.config.sandbox_runtime.app_listener, ), - ) - .await - .map_err(KubernetesDriverError::from_kube)? - .metadata - .uid - .ok_or_else(|| KubernetesDriverError::Message("supervisor Pod has no UID".to_string())) + &self.config.supervisor_runtime_class_name, + ); + pods.create(&PostParams::default(), &pod) + .await + .map_err(KubernetesDriverError::from_kube)? + .metadata + .uid + .ok_or_else(|| KubernetesDriverError::Message("supervisor Pod has no UID".to_string())) } /// Read the gateway's shared supervisor Service: its address and the @@ -4102,6 +4100,7 @@ impl KubernetesComputeDriver { sandbox_api.api.clone(), cr_name.to_string(), Some(cr_uid.to_string()), + Duration::from_secs(self.config.runtime_bootstrap_grace_seconds), ); Ok(()) } @@ -5543,7 +5542,7 @@ impl KubernetesComputeDriver { if !sandbox_runtime_bootstrap_is_stale( object, SystemTime::now(), - SANDBOX_RUNTIME_BOOTSTRAP_GRACE, + Duration::from_secs(self.config.runtime_bootstrap_grace_seconds), ) { return; } @@ -5986,9 +5985,10 @@ fn spawn_sandbox_runtime_bootstrap_completion( sandboxes: Api, sandbox_name: String, expected_sandbox_uid: Option, + bootstrap_grace: Duration, ) { tokio::spawn(async move { - let deadline = tokio::time::Instant::now() + SANDBOX_RUNTIME_BOOTSTRAP_GRACE; + let deadline = tokio::time::Instant::now() + bootstrap_grace; match tokio::time::timeout_at(deadline, control.wait_available()).await { Ok(true) => {} Ok(false) | Err(_) => return, @@ -10513,6 +10513,7 @@ mod tests { #[test] fn sandbox_runtime_bootstrap_marker_and_age_gate_rollback() { let started = Duration::from_hours(490_896); + let bootstrap_grace = Duration::from_mins(5); let mut object: DynamicObject = serde_json::from_value(serde_json::json!({ "apiVersion": "agents.x-k8s.io/v1beta1", "kind": "Sandbox", @@ -10535,13 +10536,13 @@ mod tests { let started_at = SystemTime::UNIX_EPOCH + started; assert!(!sandbox_runtime_bootstrap_is_stale( &object, - started_at + SANDBOX_RUNTIME_BOOTSTRAP_GRACE - Duration::from_secs(1), - SANDBOX_RUNTIME_BOOTSTRAP_GRACE + started_at + bootstrap_grace - Duration::from_secs(1), + bootstrap_grace )); assert!(sandbox_runtime_bootstrap_is_stale( &object, - started_at + SANDBOX_RUNTIME_BOOTSTRAP_GRACE, - SANDBOX_RUNTIME_BOOTSTRAP_GRACE + started_at + bootstrap_grace, + bootstrap_grace )); object .metadata @@ -10552,7 +10553,7 @@ mod tests { assert!(sandbox_runtime_bootstrap_is_stale( &object, started_at, - SANDBOX_RUNTIME_BOOTSTRAP_GRACE + bootstrap_grace )); object .metadata diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index 69ffc69ccb..15c15030db 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -29,6 +29,14 @@ struct Args { default_value = "{}" )] admission_config_json: openshell_core::resource_admission::DriverAdmissionConfig, + /// Runtime bootstrap budget in seconds. + #[arg(long, env = "OPENSHELL_RUNTIME_BOOTSTRAP_GRACE_SECONDS", default_value = "300", + value_parser = clap::value_parser!(u64).range(1..=86_400))] + runtime_bootstrap_grace_seconds: u64, + /// Initial attachment budget passed to dedicated supervisor Pods. + #[arg(long, env = "OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS", + value_parser = clap::value_parser!(u64).range(1..=86_400))] + sandbox_attach_request_timeout_seconds: Option, /// Public compute-driver Unix socket used by an external gateway. #[arg(long, env = "OPENSHELL_COMPUTE_DRIVER_SOCKET")] bind_socket: Option, @@ -286,6 +294,8 @@ async fn main() -> Result<()> { openshell_driver_kubernetes::config::KubernetesAppListenerConfig::default(), boundary_port: args.sandbox_runtime_boundary_port, }, + runtime_bootstrap_grace_seconds: args.runtime_bootstrap_grace_seconds, + sandbox_attach_request_timeout_seconds: args.sandbox_attach_request_timeout_seconds, https_proxy: args.https_proxy, no_proxy: args.no_proxy, proxy_auth_secret_name: args.proxy_auth_secret_name, @@ -310,6 +320,10 @@ async fn main() -> Result<()> { .unwrap_or_default(), default_runtime_class_name: std::env::var("OPENSHELL_K8S_DEFAULT_RUNTIME_CLASS_NAME") .unwrap_or_default(), + supervisor_runtime_class_name: std::env::var( + "OPENSHELL_K8S_SUPERVISOR_RUNTIME_CLASS_NAME", + ) + .unwrap_or_default(), allowed_pod_annotations: args.allowed_pod_annotations, allowed_pod_labels: args.allowed_pod_labels, sa_token_ttl_secs: args.sa_token_ttl_secs, @@ -365,6 +379,33 @@ async fn main() -> Result<()> { #[cfg(test)] mod tests { + #[test] + fn bootstrap_timeout_arguments_reject_invalid_values() { + let defaults = Args::try_parse_from(["openshell-driver-kubernetes"]).unwrap(); + assert_eq!(defaults.runtime_bootstrap_grace_seconds, 300); + assert!(defaults.sandbox_attach_request_timeout_seconds.is_none()); + for flag in [ + "--runtime-bootstrap-grace-seconds", + "--sandbox-attach-request-timeout-seconds", + ] { + for value in ["0", "86401", "invalid"] { + assert!( + Args::try_parse_from(["openshell-driver-kubernetes", flag, value]).is_err() + ); + } + for value in ["1", "900", "86400"] { + let args = + Args::try_parse_from(["openshell-driver-kubernetes", flag, value]).unwrap(); + let seconds = value.parse::().unwrap(); + if flag == "--runtime-bootstrap-grace-seconds" { + assert_eq!(args.runtime_bootstrap_grace_seconds, seconds); + } else { + assert_eq!(args.sandbox_attach_request_timeout_seconds, Some(seconds)); + } + } + } + } + use super::*; #[test] diff --git a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs index 39876b3d84..9ed4ce71e9 100644 --- a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs +++ b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs @@ -322,6 +322,16 @@ pub fn shared_control_policy( policy } +/// Set the dedicated supervisor Pod's `runtimeClassName`. An empty class +/// leaves it unset. +#[must_use] +pub fn with_runtime_class(mut pod: Pod, class: &str) -> Pod { + if let Some(spec) = pod.spec.as_mut().filter(|_| !class.is_empty()) { + spec.runtime_class_name = Some(class.to_string()); + } + pod +} + #[allow(clippy::too_many_arguments, clippy::similar_names)] pub fn supervisor_pod( namespace: &str, @@ -353,6 +363,7 @@ pub fn supervisor_pod( network_additional_ca: Option<(&str, &str)>, owner: OwnerReference, tracing_environment: &[(&str, String)], + attach_request_timeout_seconds: Option, ) -> Result { let labels = control_labels(sandbox_id, gateway_id); let mut environment = vec![ @@ -394,6 +405,12 @@ pub fn supervisor_pod( .iter() .map(|(name, value)| env_var(name, value)), ); + if let Some(timeout_seconds) = attach_request_timeout_seconds { + environment.push(env_var( + "OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS", + &timeout_seconds.to_string(), + )); + } if let Some(server_name) = gateway_tls_server_name(grpc_endpoint) { environment.push(env_var( openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME, @@ -1045,6 +1062,7 @@ mod tests { None, owner(), &[], + None, ) .expect("render supervisor Pod") } @@ -1074,6 +1092,25 @@ mod tests { (env, volumes) } + #[test] + fn supervisor_runtime_class_is_set_only_when_configured() { + let pod = Pod { + spec: Some(PodSpec::default()), + ..Default::default() + }; + let class = |pod: Pod, name: &str| { + with_runtime_class(pod, name) + .spec + .unwrap() + .runtime_class_name + }; + assert_eq!(class(pod.clone(), ""), None); + assert_eq!( + class(pod, "kata-supervisor").as_deref(), + Some("kata-supervisor") + ); + } + #[test] fn supervisor_pod_carries_tracing_environment() { const TRACEPARENT: &str = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; @@ -1111,6 +1148,7 @@ mod tests { ), ("TRACEPARENT", TRACEPARENT.to_string()), ], + None, ) .expect("render supervisor Pod"); let environment = pod.spec.expect("Pod spec").containers[0] @@ -1127,6 +1165,56 @@ mod tests { assert_eq!(environment["TRACEPARENT"], TRACEPARENT); } + #[test] + fn supervisor_pod_sets_attach_request_timeout_only_when_configured() { + let env = |timeout: Option| { + supervisor_pod( + "sandbox", + &SandboxRuntimeNames::new("pair"), + "pair", + "demo", + "gateway", + "supervisor:latest", + None, + "sandbox-sa", + 1000, + 1000, + &[], + "https://gateway:8080", + SupervisorClientTls::Disabled, + "{}", + "info", + 600, + None, + None, + None, + false, + false, + false, + None, + None, + owner(), + &[], + timeout, + ) + .unwrap() + .spec + .expect("Pod spec") + .containers[0] + .env + .clone() + .unwrap_or_default() + .into_iter() + .map(|variable| (variable.name, variable.value.unwrap_or_default())) + .collect::>() + }; + assert!(!env(None).contains_key("OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS")); + assert_eq!( + env(Some(900))["OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS"], + "900" + ); + } + #[test] fn supervisor_pod_reads_client_tls_from_the_bootstrap_secret() { let (env, volumes) = tls_env_and_volumes(&supervisor_pod_with_client_tls( @@ -1232,6 +1320,7 @@ mod tests { None, owner(), &[], + None, ) .expect("render supervisor Pod"); let pod_spec = pod.spec.as_ref().expect("Pod spec"); @@ -1401,6 +1490,7 @@ mod tests { Some(("openshell-additional-ca-gateway-generation", digest)), owner(), &[], + None, ) .expect("render supervisor Pod with destination trust"); let pod = crate::app_listener::configure_supervisor_pod( @@ -1593,6 +1683,7 @@ mod tests { None, owner(), &[], + None, ) .expect("render supervisor Pod") } diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index a0371bacfb..cb9433d08e 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -70,7 +70,7 @@ By default `mise run gateway:vm`: - Persists the gateway SQLite DB under `.cache/gateway-vm/gateway.db`. - Places the VM driver state (per-sandbox `overlay.ext4`, image cache, and `run/compute-driver.sock`) under `/tmp/openshell-vm-driver-$USER-vm-dev/` so the AF_UNIX socket path stays under macOS `SUN_LEN`. - Writes `.cache/gateway-vm/gateway.toml` with `[openshell.drivers.vm].driver_dir = "$PWD/target/debug"` so the freshly built `openshell-driver-vm` is used instead of an older installed copy from `~/.local/libexec/openshell`, `/usr/libexec/openshell`, or `/usr/local/libexec`. -- Enables OTLP trace export, OCSF log export, and the agent trace relay to `http://127.0.0.1:4317` only when a local collector is listening there. Otherwise, it omits the OTLP configuration to avoid repeated export failures. +- Enables OTLP trace export, OCSF log export, and the agent trace relay only when `OPENSHELL_DEV_OTLP_ENDPOINT` names a collector, for example `http://127.0.0.1:4317`. Otherwise, it omits the OTLP configuration. For GPU passthrough (VFIO), pass `-- --gpu` and run with root privileges: diff --git a/crates/openshell-driver-vm/runtime/README.md b/crates/openshell-driver-vm/runtime/README.md index bb02232f92..c229ca81b2 100644 --- a/crates/openshell-driver-vm/runtime/README.md +++ b/crates/openshell-driver-vm/runtime/README.md @@ -65,9 +65,9 @@ publish the driver binary next to `openshell-gateway`. ## Provenance `package-vm-runtime.sh` writes `provenance.json` into each runtime tarball with -the platform, libkrunfw commit, kernel version, and umoci version, -GitHub SHA, and build time. The driver logs this metadata when it extracts and -loads a runtime bundle. +the platform, libkrunfw commit, kernel version, final kernel config SHA-256, +umoci version, GitHub SHA, and build time. The driver logs this metadata when it +extracts and loads a runtime bundle. The release workflow also publishes GitHub artifact attestations for each runtime tarball. Verify a downloaded runtime with: diff --git a/crates/openshell-gateway-interceptors/src/routes.rs b/crates/openshell-gateway-interceptors/src/routes.rs index 30e053e323..43afa43467 100644 --- a/crates/openshell-gateway-interceptors/src/routes.rs +++ b/crates/openshell-gateway-interceptors/src/routes.rs @@ -23,6 +23,10 @@ pub const INTERCEPTABLE_METHODS: &[&str] = &[ "AttachSandboxProvider", "DetachSandboxProvider", "DeleteSandbox", + "StopSandbox", + "StartSandbox", + "SuspendSandbox", + "ResumeSandbox", "CreateSshSession", "ExposeService", "DeleteService", @@ -339,6 +343,10 @@ mod tests { .input_type(SERVICE_OPEN_SHELL, "UpdateSandbox") .is_none() ); + assert!(index.is_interceptable("openshell.v1.OpenShell", "StopSandbox")); + assert!(index.is_interceptable("openshell.v1.OpenShell", "StartSandbox")); + assert!(index.is_interceptable("openshell.v1.OpenShell", "SuspendSandbox")); + assert!(index.is_interceptable("openshell.v1.OpenShell", "ResumeSandbox")); assert!(index.is_interceptable("openshell.v1.OpenShell", "UpdateConfig")); assert!(index.is_interceptable("openshell.v1.OpenShell", "SubmitPolicyAnalysis")); assert!(!index.is_interceptable("openshell.v1.OpenShell", "Health")); diff --git a/crates/openshell-gateway/src/lib.rs b/crates/openshell-gateway/src/lib.rs index a4b618cbc9..6f05eeaef5 100644 --- a/crates/openshell-gateway/src/lib.rs +++ b/crates/openshell-gateway/src/lib.rs @@ -73,8 +73,7 @@ pub fn install_default_compute_drivers() -> ComputeDriverRegistry { fn install_mxc_compute_driver(registry: &mut ComputeDriverRegistry) { let registration = ComputeDriverRegistration::new("mxc", u16::MAX, None, MxcFactory) .expect("first-party driver name is valid") - .with_telemetry_category(TelemetryComputeDriver::anonymous_category("mxc")) - .with_local_singleplayer(); + .with_telemetry_category(TelemetryComputeDriver::anonymous_category("mxc")); registry .install(registration) .expect("first-party driver names are unique"); @@ -204,7 +203,6 @@ fn install_in_tree_compute_drivers(registry: &mut ComputeDriverRegistry) { .map(|registration| { registration .with_telemetry_category(TelemetryComputeDriver::anonymous_category("podman")) - .with_local_singleplayer() .with_in_process_tracing(openshell_driver_podman::otel_tracing::TRACING) }), #[cfg(feature = "compute-driver-docker")] @@ -217,14 +215,11 @@ fn install_in_tree_compute_drivers(registry: &mut ComputeDriverRegistry) { .map(|registration| { registration .with_telemetry_category(TelemetryComputeDriver::anonymous_category("docker")) - .with_local_singleplayer() .with_in_process_tracing(openshell_driver_docker::otel_tracing::TRACING) }), #[cfg(feature = "compute-driver-vm")] ComputeDriverRegistration::new("vm", u16::MAX, None, VmFactory).map(|registration| { - registration - .with_telemetry_category(TelemetryComputeDriver::anonymous_category("vm")) - .with_local_singleplayer() + registration.with_telemetry_category(TelemetryComputeDriver::anonymous_category("vm")) }), ] { registry @@ -342,6 +337,32 @@ fn kubernetes_config( config.supervisor_image, std::env::var(openshell_core::config::SUPERVISOR_IMAGE_ENV).ok(), ); + if let Some(raw) = std::env::var_os("OPENSHELL_RUNTIME_BOOTSTRAP_GRACE_SECONDS") { + let raw = raw.to_str().ok_or_else(|| { + openshell_core::Error::config( + "OPENSHELL_RUNTIME_BOOTSTRAP_GRACE_SECONDS must be an integer number of seconds", + ) + })?; + let grace = raw.parse::().map_err(|_| { + openshell_core::Error::config( + "OPENSHELL_RUNTIME_BOOTSTRAP_GRACE_SECONDS must be an integer number of seconds", + ) + })?; + config.runtime_bootstrap_grace_seconds = grace; + } + if let Some(raw) = std::env::var_os("OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS") { + let raw = raw.to_str().ok_or_else(|| { + openshell_core::Error::config( + "OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS must be an integer number of seconds", + ) + })?; + let timeout = raw.parse::().map_err(|_| { + openshell_core::Error::config( + "OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS must be an integer number of seconds", + ) + })?; + config.sandbox_attach_request_timeout_seconds = Some(timeout); + } if let Ok(size) = std::env::var("OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE") { config.workspace_default_storage_size = size; } diff --git a/crates/openshell-isolation-interface/src/contract.rs b/crates/openshell-isolation-interface/src/contract.rs index de2a01975f..bd328fc0a1 100644 --- a/crates/openshell-isolation-interface/src/contract.rs +++ b/crates/openshell-isolation-interface/src/contract.rs @@ -141,7 +141,13 @@ pub struct BackendDescriptor { } /// Exact non-root identity selected before the immutable workload is created. +/// +/// Validation and normalization run in [`ResolvedWorkloadIdentity::new`] and +/// again when a value is decoded from a serialized descriptor: deserialization +/// routes through the same constructor, so an invalid identity fails to decode +/// instead of bypassing the contract. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "RawResolvedWorkloadIdentity")] pub struct ResolvedWorkloadIdentity { /// Effective and real user ID used by sandbox and all workload children. pub uid: u32, @@ -187,6 +193,32 @@ impl ResolvedWorkloadIdentity { } } +/// Unvalidated wire shape of [`ResolvedWorkloadIdentity`], existing only so +/// serde decoding can route through [`ResolvedWorkloadIdentity::new`]. +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct RawResolvedWorkloadIdentity { + uid: u32, + gid: u32, + supplementary_gids: Vec, + source: String, + resource_digest: String, +} + +impl TryFrom for ResolvedWorkloadIdentity { + type Error = BackendError; + + fn try_from(raw: RawResolvedWorkloadIdentity) -> Result { + Self::new( + raw.uid, + raw.gid, + raw.supplementary_gids, + raw.source, + raw.resource_digest, + ) + } +} + /// The trusted sandbox context, constructed by trusted common code after the /// control plane assigns the resource to the admitted sandbox. /// diff --git a/crates/openshell-isolation-interface/tests/backend_conformance.rs b/crates/openshell-isolation-interface/tests/backend_conformance.rs index e34e6215bd..9928b02073 100644 --- a/crates/openshell-isolation-interface/tests/backend_conformance.rs +++ b/crates/openshell-isolation-interface/tests/backend_conformance.rs @@ -77,6 +77,115 @@ fn workload_identity_rejects_root_and_normalizes_groups() { assert_eq!(identity.supplementary_gids, vec![1002, 1003]); } +#[test] +fn workload_identity_deserialization_enforces_new_invariants() { + fn payload(uid: u32, gid: u32, gids: &[u32], source: &str, digest: &str) -> serde_json::Value { + serde_json::json!({ + "uid": uid, + "gid": gid, + "supplementary_gids": gids, + "source": source, + "resource_digest": digest, + }) + } + + assert!( + serde_json::from_value::(payload( + 0, + 1000, + &[], + "policy", + "sha256:test" + )) + .is_err() + ); + assert!( + serde_json::from_value::(payload( + 1000, + 0, + &[], + "policy", + "sha256:test" + )) + .is_err() + ); + assert!( + serde_json::from_value::(payload( + 1000, + 1000, + &[0], + "policy", + "sha256:test" + )) + .is_err() + ); + assert!( + serde_json::from_value::(payload( + 1000, + 1000, + &[], + "", + "sha256:test" + )) + .is_err() + ); + assert!( + serde_json::from_value::(payload(1000, 1000, &[], "policy", "")) + .is_err() + ); + assert!( + serde_json::from_value::(payload( + 1000, + 1000, + &[], + " ", + "sha256:test" + )) + .is_err() + ); + assert!( + serde_json::from_value::(payload(1000, 1000, &[], "policy", " ")) + .is_err() + ); + + let identity = workload_identity(); + let encoded = serde_json::to_value(&identity).unwrap(); + assert_eq!( + serde_json::from_value::(encoded).unwrap(), + identity + ); + + let decoded = serde_json::from_value::(payload( + 1000, + 1000, + &[1003, 1001, 1002, 1003], + "policy", + "sha256:test", + )) + .unwrap(); + assert_eq!(decoded.supplementary_gids, vec![1001, 1002, 1003]); + + let decoded = serde_json::from_value::(payload( + 1000, + 1000, + &[1000, 1001], + "policy", + "sha256:test", + )) + .unwrap(); + assert_eq!(decoded.supplementary_gids, vec![1001]); +} + +#[test] +fn boundary_confirmation_rejects_invalid_embedded_identity() { + let encoded = serde_json::to_value(confirmation()).unwrap(); + serde_json::from_value::(encoded.clone()).unwrap(); + + let mut invalid = encoded; + invalid["identity"]["uid"] = serde_json::json!(0); + assert!(serde_json::from_value::(invalid).is_err()); +} + #[test] fn confirmation_rejects_missing_generation_or_identity_drift() { let expected = workload_identity(); diff --git a/crates/openshell-ocsf/src/events/mod.rs b/crates/openshell-ocsf/src/events/mod.rs index 66038cd9cb..22ecbb6dfc 100644 --- a/crates/openshell-ocsf/src/events/mod.rs +++ b/crates/openshell-ocsf/src/events/mod.rs @@ -169,6 +169,11 @@ impl OcsfEvent { } } + /// Associate this event with other events produced for the same operation. + pub fn set_correlation_uid(&mut self, uid: impl Into) { + self.base_mut().metadata.correlation_uid = Some(uid.into()); + } + /// Returns mutable base data common to all event classes. pub(crate) fn base_mut(&mut self) -> &mut BaseEventData { match self { @@ -281,6 +286,29 @@ mod tests { assert_eq!(deserialized.class_uid(), 2004); } + #[test] + fn correlation_uid_can_be_set_on_any_event_class() { + let ctx = test_sandbox_context(); + let mut event = AuthenticationBuilder::new( + &ctx, + User::other("user-1", "User"), + Service::new("gateway"), + ) + .status(StatusId::Success) + .build(); + + event.set_correlation_uid("request-123"); + + assert_eq!( + event.base().metadata.correlation_uid.as_deref(), + Some("request-123") + ); + assert_eq!( + serde_json::to_value(&event).unwrap()["metadata"]["correlation_uid"], + "request-123" + ); + } + #[test] fn test_roundtrip_application_lifecycle() { let ctx = test_sandbox_context(); diff --git a/crates/openshell-otel/Cargo.toml b/crates/openshell-otel/Cargo.toml index c6cd0111c9..a857018b91 100644 --- a/crates/openshell-otel/Cargo.toml +++ b/crates/openshell-otel/Cargo.toml @@ -11,31 +11,31 @@ license.workspace = true repository.workspace = true [dependencies] -async-trait = "0.1" -bytes = { workspace = true } -futures = { workspace = true } +async-trait = { version = "0.1", optional = true } +bytes = { workspace = true, optional = true } +futures = { workspace = true, optional = true } http = { workspace = true } -http-body-util = { workspace = true } -hyper-rustls = { version = "0.27", default-features = false, features = ["http1", "logging", "tls12"] } -hyper-util = { workspace = true, features = ["client-legacy", "http1", "tokio"] } -metrics = { workspace = true } -openshell-crypto = { workspace = true } -openshell-ocsf = { path = "../openshell-ocsf" } +http-body-util = { workspace = true, optional = true } +hyper-rustls = { version = "0.27", default-features = false, optional = true, features = ["http1", "logging", "tls12"] } +hyper-util = { workspace = true, optional = true, features = ["client-legacy", "http1", "tokio"] } +metrics = { workspace = true, optional = true } +openshell-crypto = { workspace = true, optional = true } +openshell-ocsf = { path = "../openshell-ocsf", optional = true } opentelemetry = { workspace = true } -opentelemetry_sdk = { workspace = true, features = ["logs"] } -opentelemetry-http = { workspace = true } -opentelemetry-otlp = { workspace = true, features = ["gzip-http", "http-proto", "logs"] } -rustls = { workspace = true } -rustls-native-certs = { workspace = true } -serde_json = { workspace = true } -thiserror = { workspace = true } -tokio = { workspace = true } +opentelemetry_sdk = { workspace = true } +opentelemetry-http = { workspace = true, optional = true } +opentelemetry-otlp = { workspace = true, optional = true, features = ["gzip-http", "http-proto", "logs"] } +rustls = { workspace = true, optional = true } +rustls-native-certs = { workspace = true, optional = true } +serde_json = { workspace = true, optional = true } +thiserror = { workspace = true, optional = true } +tokio = { workspace = true, optional = true } tracing = { workspace = true } tracing-opentelemetry = { workspace = true } -tracing-subscriber = { workspace = true } +tracing-subscriber = { workspace = true, optional = true } tonic = { workspace = true } -tower-http = { workspace = true } -zeroize = { workspace = true } +tower-http = { workspace = true, optional = true } +zeroize = { workspace = true, optional = true } [dev-dependencies] openshell-otel-test-support = { path = "../openshell-otel-test-support" } @@ -44,6 +44,37 @@ opentelemetry_sdk = { workspace = true, features = ["testing"] } prost = { workspace = true } tempfile = "3" tokio = { workspace = true, features = ["test-util"] } +tracing-subscriber = { workspace = true } + +[features] +default = ["export"] +# OTLP/gRPC export: tracer providers, tracing layers, compute-driver and +# gRPC tracing machinery, and the OCSF log bridge. Services that report to +# a collector keep the default; clients that only forward trace context +# (e.g. the CLI) opt out with `default-features = false` so the exporter +# stack stays out of their binaries. +export = [ + "dep:async-trait", + "dep:bytes", + "dep:futures", + "dep:http-body-util", + "dep:hyper-rustls", + "dep:hyper-util", + "dep:metrics", + "dep:openshell-crypto", + "dep:openshell-ocsf", + "dep:opentelemetry-http", + "dep:opentelemetry-otlp", + "dep:rustls", + "dep:rustls-native-certs", + "dep:serde_json", + "dep:thiserror", + "dep:tokio", + "dep:tower-http", + "dep:tracing-subscriber", + "dep:zeroize", + "opentelemetry_sdk/logs", +] [lints] workspace = true diff --git a/crates/openshell-otel/src/lib.rs b/crates/openshell-otel/src/lib.rs index 1f9de6bf17..32077a7f93 100644 --- a/crates/openshell-otel/src/lib.rs +++ b/crates/openshell-otel/src/lib.rs @@ -2,573 +2,79 @@ // SPDX-License-Identifier: Apache-2.0 //! Shared OpenTelemetry trace and log export support for `OpenShell` services. +//! +//! The crate has two tiers: +//! +//! - Trace-context propagation, always compiled in: W3C `traceparent` / +//! `tracestate` extraction and injection for HTTP headers, tonic metadata, +//! and process environments. +//! - The `export` feature (on by default): OTLP tracer providers, tracing +//! layers, compute-driver and gRPC tracing machinery, and the OCSF log +//! bridge. Services that report to a collector keep the default; clients +//! that only forward trace context (e.g. the CLI) opt out with +//! `default-features = false` to keep the exporter stack out of their +//! binaries. -mod correlation; -pub use correlation::current_ocsf_trace_correlation; +mod propagation; + +pub use propagation::{ + EnvTraceContextInterceptor, HeaderMapExtractor, MetadataMapInjector, TRACEPARENT_ENV, + TRACESTATE_ENV, TraceContextInterceptor, current_trace_context_carrier, + current_trace_context_environment, set_parent_from_environment, set_parent_from_trace_context, + trace_context_environment, +}; +#[cfg(feature = "export")] +mod correlation; +#[cfg(feature = "export")] mod driver; +#[cfg(feature = "export")] mod grpc; +#[cfg(feature = "export")] mod headers; +#[cfg(feature = "export")] mod logs; +#[cfg(feature = "export")] mod ocsf; -mod propagation; +#[cfg(feature = "export")] +mod setup; +#[cfg(feature = "export")] mod tls; +#[cfg(feature = "export")] mod transport; #[cfg(test)] #[path = "../../../tests/support/environment.rs"] mod test_environment; +#[cfg(feature = "export")] +pub use correlation::current_ocsf_trace_correlation; + +#[cfg(feature = "export")] pub use driver::{ BoxGrpcStream, ComputeDriverTracing, DriverTracingConfig, DriverTracingHandle, IN_PROCESS_COMPUTE_DRIVER_TARGET, InProcessRpcTracer, install_driver_tracing, }; - +#[cfg(feature = "export")] pub use grpc::{ COMPUTE_DRIVER_RPC_SERVICE, ComputeDriverRpc, ComputeDriverRpcSpan, RecordGrpcFailure, RecordGrpcStatus, TracedGrpcStream, compute_driver_rpc_layer, compute_driver_rpc_operation, grpc_status_code_name, record_grpc_status, rpc, }; +#[cfg(feature = "export")] pub use headers::{encode_otlp_headers, merge_otlp_headers, parse_otlp_headers}; +#[cfg(feature = "export")] pub use ocsf::{OcsfOtlp, OcsfOtlpLayer}; -pub use propagation::{ - HeaderMapExtractor, MetadataMapInjector, TRACEPARENT_ENV, TRACESTATE_ENV, - TraceContextInterceptor, current_trace_context_carrier, current_trace_context_environment, - set_parent_from_environment, set_parent_from_trace_context, trace_context_environment, -}; +#[cfg(feature = "export")] pub use tls::{MAX_TLS_FILE_BYTES, OtlpTls, TlsError, TlsFileError}; - +#[cfg(feature = "export")] pub use transport::{ CONNECT_TIMEOUT, OtlpHttpClient, TCP_KEEPALIVE_IDLE, TCP_KEEPALIVE_INTERVAL, TCP_KEEPALIVE_RETRIES, signal_url, }; -use opentelemetry::KeyValue; -use opentelemetry::trace::TracerProvider as _; -pub use opentelemetry_otlp::LogExporter; -use opentelemetry_otlp::tonic_types::metadata::MetadataMap; -use opentelemetry_otlp::{SpanExporter, WithExportConfig, WithHttpConfig, WithTonicConfig}; -use opentelemetry_sdk::Resource; -use opentelemetry_sdk::trace::SdkTracer; -pub use opentelemetry_sdk::trace::SdkTracerProvider; -use tracing::Subscriber; -use tracing_opentelemetry::OpenTelemetryLayer; -use tracing_subscriber::Layer as _; -use tracing_subscriber::layer::{Context, Filter}; -use tracing_subscriber::registry::LookupSpan; - -/// Export supervisor spans at INFO by default, or DEBUG and TRACE when enabled. -/// Dependency spans remain at INFO. -#[must_use] -pub fn supervisor_span_filter(log_level: &str) -> tracing_subscriber::EnvFilter { - use tracing_subscriber::filter::LevelFilter; - let level = log_level - .parse::() - .unwrap_or(LevelFilter::INFO) - .max(LevelFilter::INFO); - tracing_subscriber::EnvFilter::new(format!("info,openshell={level}")) -} - -const SDK_UNKNOWN_SERVICE_PREFIX: &str = "unknown_service"; - -/// Build the resource attributes shared by gateway and compute-driver providers. -pub fn gateway_resource_attributes( - gateway_name: Option<&str>, - compute_driver: Option<&str>, -) -> Vec { - let mut attributes = Vec::new(); - if let Some(name) = gateway_name.map(str::trim).filter(|name| !name.is_empty()) { - attributes.push(KeyValue::new("openshell.gateway.name", name.to_string())); - } - if let Some(driver) = compute_driver - .map(str::trim) - .filter(|driver| !driver.is_empty()) - { - attributes.push(KeyValue::new( - "openshell.gateway.compute_driver", - driver.to_string(), - )); - } - attributes -} - -/// Mark `span` as failed. -/// -/// The field must be declared on the span at creation because `tracing` drops -/// records for fields a span does not have. -pub fn mark_error(span: &tracing::Span) { - span.record("otel.status_code", "ERROR"); -} - -/// Marks the current span when an instrumented operation returns an error. -pub fn record_error_result(result: Result) -> Result { - if result.is_err() { - mark_error(&tracing::Span::current()); - } - result -} - -/// Marks an instrumented function's span when it exits before returning success. -/// -/// Create the guard at the start of the instrumented function and return the -/// successful result through [`Self::finish`]. An early `?` or error return -/// drops the unfinished guard and marks the captured span as failed. -#[must_use] -pub struct ErrorStatusGuard { - span: tracing::Span, - finished: bool, -} - -impl ErrorStatusGuard { - /// Captures the current instrumented span. - pub fn current() -> Self { - Self { - span: tracing::Span::current(), - finished: false, - } - } - - /// Returns `result`, marking this guard complete when it is successful. - pub fn finish(mut self, result: Result) -> Result { - self.finished = result.is_ok(); - result - } -} - -impl Drop for ErrorStatusGuard { - fn drop(&mut self) { - if !self.finished { - mark_error(&self.span); - } - } -} - -/// How a process chooses its OpenTelemetry `service.name`. -#[derive(Debug, Clone, Copy)] -pub enum ServiceName<'a> { - /// Always use this name, overriding `OTEL_SERVICE_NAME`. - Fixed(&'a str), - /// Use `OTEL_SERVICE_NAME` when set, otherwise use this default. - EnvironmentOr(&'a str), -} - -/// OTLP transport to a collector. -#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] -pub enum Protocol { - #[default] - Grpc, - HttpProtobuf, -} - -/// Inputs for OTLP export. -#[derive(Debug, Clone)] -pub struct OtlpExportConfig<'a> { - pub endpoint: &'a str, - pub service_name: ServiceName<'a>, - pub service_version: Option<&'a str>, - pub resource_attributes: Vec, -} - -/// How exports reach a collector: the transport, request headers, and TLS. -#[derive(Clone, Default, PartialEq, Eq)] -pub struct CollectorTransport { - pub protocol: Protocol, - pub headers: http::HeaderMap, - pub tls: OtlpTls, -} - -impl std::fmt::Debug for CollectorTransport { - fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - formatter - .debug_struct("CollectorTransport") - .field("protocol", &self.protocol) - .field("header_names", &self.headers.keys().collect::>()) - .field("tls", &self.tls) - .finish() - } -} - -/// Failure to set up OTLP export. -#[derive(Debug, thiserror::Error)] -pub enum SetupError { - #[error("OTLP endpoint is empty")] - EmptyEndpoint, - - #[error("invalid OTLP endpoint {endpoint:?}: {source}")] - InvalidEndpoint { - endpoint: String, - source: http::uri::InvalidUri, - }, - - #[error("failed to build the OTLP exporter: {0}")] - Exporter(#[from] opentelemetry_otlp::ExporterBuildError), - - #[error("invalid OTLP endpoint {endpoint:?}: {source}")] - InvalidSignalUrl { - endpoint: String, - source: http::Error, - }, - - #[error("OTLP TLS settings need an https:// endpoint, not {endpoint:?}")] - TlsWithoutHttps { endpoint: String }, - - #[error("invalid OTLP TLS settings: {0}")] - Tls(#[from] TlsError), - - #[error("invalid OTLP TLS files: {0}")] - TlsFiles(#[from] TlsFileError), - - #[error("OTLP/HTTP export needs a tokio runtime")] - NoRuntime, -} - -fn resource_attributes(config: &OtlpExportConfig<'_>) -> Vec { - let mut attributes = config.resource_attributes.clone(); - if let Some(version) = config - .service_version - .map(str::trim) - .filter(|version| !version.is_empty()) - { - attributes.push(KeyValue::new("service.version", version.to_string())); - } - attributes -} - -/// Build the OpenTelemetry resource for an export configuration. -#[must_use] -pub fn resource_for(config: &OtlpExportConfig<'_>) -> Resource { - let attributes = resource_attributes(config); - match config.service_name { - ServiceName::Fixed(name) => Resource::builder() - .with_service_name(name.trim().to_string()) - .with_attributes(attributes) - .build(), - ServiceName::EnvironmentOr(default) => { - let detected = Resource::builder() - .with_attributes(attributes.clone()) - .build(); - if detected - .get(&opentelemetry::Key::from_static_str("service.name")) - .is_some_and(|value| !value.to_string().starts_with(SDK_UNKNOWN_SERVICE_PREFIX)) - { - detected - } else { - Resource::builder() - .with_service_name(default.trim().to_string()) - .with_attributes(attributes) - .build() - } - } - } -} - -/// Trim and validate an OTLP endpoint before handing it to an exporter. -/// -/// Every exporter that reads an operator-supplied endpoint uses this, so a -/// value accepted for one signal is accepted for all of them. -pub fn validated_endpoint(endpoint: &str) -> Result<(&str, http::Uri), SetupError> { - let endpoint = endpoint.trim(); - if endpoint.is_empty() { - return Err(SetupError::EmptyEndpoint); - } - let uri = endpoint - .parse::() - .map_err(|source| SetupError::InvalidEndpoint { - endpoint: endpoint.to_string(), - source, - })?; - Ok((endpoint, uri)) -} - -/// A validated OTLP collector endpoint and the resource that every signal -/// exported to it describes. -#[derive(Clone)] -pub struct OtlpTarget { - endpoint: String, - tls: bool, - protocol: Protocol, - headers: http::HeaderMap, - tls_settings: OtlpTls, - resource: Resource, -} - -impl std::fmt::Debug for OtlpTarget { - fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - formatter - .debug_struct("OtlpTarget") - .field("endpoint", &self.endpoint) - .field("tls", &self.tls) - .field("protocol", &self.protocol) - .field("header_names", &self.headers.keys().collect::>()) - .field("tls_settings", &self.tls_settings) - .field("resource", &self.resource) - .finish() - } -} - -impl OtlpTarget { - /// Validate the endpoint and build the resource. - pub fn new(config: &OtlpExportConfig<'_>) -> Result { - let (endpoint, uri) = validated_endpoint(config.endpoint)?; - Ok(Self { - endpoint: endpoint.to_string(), - tls: uri.scheme() == Some(&http::uri::Scheme::HTTPS), - protocol: Protocol::Grpc, - headers: http::HeaderMap::new(), - tls_settings: OtlpTls::default(), - resource: resource_for(config), - }) - } - - /// Export over `protocol` instead of gRPC. - #[must_use] - pub fn with_protocol(mut self, protocol: Protocol) -> Self { - self.protocol = protocol; - self - } - - /// Send `headers` with every export request. Headers from the process - /// environment (`OTEL_EXPORTER_OTLP_*HEADERS`) still apply and replace - /// any of the same name. - #[must_use] - pub fn with_headers(mut self, headers: http::HeaderMap) -> Self { - self.headers = headers; - self - } - - /// Reach the collector the way `transport` describes. - #[must_use] - pub fn with_transport(self, transport: &CollectorTransport) -> Self { - self.with_protocol(transport.protocol) - .with_headers(transport.headers.clone()) - .with_tls(transport.tls.clone()) - } - - /// Verify an `https://` collector and authenticate to it with `tls`. - #[must_use] - pub fn with_tls(mut self, tls: OtlpTls) -> Self { - self.tls_settings = tls; - self - } - - /// The URL of `signal_path` under the endpoint, for OTLP/HTTP export. - pub(crate) fn http_signal_url(&self, signal_path: &str) -> Result { - signal_url(&self.endpoint, signal_path) - .map(|url| url.to_string()) - .map_err(|source| SetupError::InvalidSignalUrl { - endpoint: self.endpoint.clone(), - source, - }) - } - - /// Point an exporter builder at this target, over TLS for `https://`. - pub(crate) fn configure(&self, builder: B) -> B - where - B: WithExportConfig + WithTonicConfig, - { - let builder = builder - .with_endpoint(self.endpoint.clone()) - .with_metadata(MetadataMap::from_headers(self.headers.clone())); - if self.tls { - builder.with_tls_config(self.tls_settings.grpc_config()) - } else { - builder - } - } - - /// Point an OTLP/HTTP exporter builder at `signal_path` under this - /// target. - pub(crate) fn configure_http( - &self, - builder: B, - signal_path: &str, - timeout_var: &str, - ) -> Result - where - B: WithExportConfig + WithHttpConfig, - { - // The SDK percent-decodes these values again. - let headers = self - .headers - .iter() - .filter_map(|(name, value)| { - Some((name.to_string(), value.to_str().ok()?.replace('%', "%25"))) - }) - .collect(); - Ok(builder - .with_protocol(opentelemetry_otlp::Protocol::HttpBinary) - .with_endpoint(self.http_signal_url(signal_path)?) - .with_http_client(OtlpHttpClient::new( - self.tls.then_some(&self.tls_settings), - transport::export_timeout(timeout_var), - )?) - .with_headers(headers)) - } - - #[must_use] - pub fn endpoint(&self) -> &str { - &self.endpoint - } - - #[must_use] - pub fn resource(&self) -> &Resource { - &self.resource - } - - /// Fails when TLS settings are given for a plaintext endpoint. - pub(crate) fn check_tls(&self) -> Result<(), SetupError> { - if self.tls || self.tls_settings.is_default() { - Ok(()) - } else { - Err(SetupError::TlsWithoutHttps { - endpoint: self.endpoint.clone(), - }) - } - } - - /// Build a trace provider that exports spans to this target in batches. - pub fn tracer_provider(&self) -> Result { - self.check_tls()?; - let exporter = match self.protocol { - Protocol::Grpc => self - .configure(SpanExporter::builder().with_tonic()) - .build()?, - Protocol::HttpProtobuf => self - .configure_http( - SpanExporter::builder().with_http(), - "v1/traces", - "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT", - )? - .build()?, - }; - Ok(SdkTracerProvider::builder() - .with_batch_exporter(exporter) - .with_resource(self.resource.clone()) - .build()) - } -} - -/// Build an OTLP trace provider. -pub fn build_provider(config: &OtlpExportConfig<'_>) -> Result { - OtlpTarget::new(config)?.tracer_provider() -} - -/// Build the provider for an optional OTLP configuration. -/// -/// Telemetry setup failures disable export and remain available for the caller -/// to report after its tracing subscriber is installed. -#[must_use] -pub fn provider_for( - config: Option>, -) -> (Option, Option) { - match config.as_ref().map(build_provider) { - None => (None, None), - Some(Ok(provider)) => (Some(provider), None), - Some(Err(error)) => (None, Some(error)), - } -} - -/// Filtered OpenTelemetry layer returned by [`layer`]. -pub type OtlpLayer = tracing_subscriber::filter::Filtered< - OpenTelemetryLayer, - tracing_subscriber::filter::FilterFn, - S, ->; - -pub type TargetOtlpLayer = - tracing_subscriber::filter::Filtered, TargetPrefixFilter, S>; - -#[derive(Debug, Clone)] -pub struct TargetPrefixFilter { - prefixes: Vec<&'static str>, - include: bool, -} - -impl Filter for TargetPrefixFilter { - fn enabled(&self, metadata: &tracing::Metadata<'_>, _ctx: &Context<'_, S>) -> bool { - let matches_prefix = self - .prefixes - .iter() - .any(|prefix| metadata.target().starts_with(prefix)); - metadata.is_span() - && !metadata.target().starts_with("opentelemetry") - && (matches_prefix == self.include) - } -} - -/// Build a tracing layer that exports spans and excludes exporter callsites. -pub fn layer(provider: &SdkTracerProvider, instrumentation_scope: &'static str) -> OtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - tracing_opentelemetry::layer() - .with_tracer(provider.tracer(instrumentation_scope)) - .with_filter(tracing_subscriber::filter::filter_fn(|metadata| { - metadata.is_span() && !metadata.target().starts_with("opentelemetry") - })) -} - -/// Build a tracing layer that exports only spans from `target_prefix`. -pub fn layer_for_target_prefix( - provider: &SdkTracerProvider, - instrumentation_scope: &'static str, - target_prefix: &'static str, -) -> TargetOtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - layer_for_target_prefixes(provider, instrumentation_scope, [target_prefix]) -} - -/// Build a tracing layer that exports only spans from `target_prefixes`. -pub fn layer_for_target_prefixes( - provider: &SdkTracerProvider, - instrumentation_scope: &'static str, - target_prefixes: impl IntoIterator, -) -> TargetOtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - tracing_opentelemetry::layer() - .with_tracer(provider.tracer(instrumentation_scope)) - .with_filter(TargetPrefixFilter { - prefixes: target_prefixes.into_iter().collect(), - include: true, - }) -} - -/// Build a tracing layer that excludes spans from `target_prefix`. -/// -/// `None` excludes no application spans. -pub fn layer_excluding_target_prefix( - provider: &SdkTracerProvider, - instrumentation_scope: &'static str, - target_prefix: Option<&'static str>, -) -> TargetOtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - layer_excluding_target_prefixes(provider, instrumentation_scope, target_prefix) -} - -/// Build a tracing layer that excludes spans from `target_prefixes`. -/// -/// An empty iterator excludes no application spans. -pub fn layer_excluding_target_prefixes( - provider: &SdkTracerProvider, - instrumentation_scope: &'static str, - target_prefixes: impl IntoIterator, -) -> TargetOtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - tracing_opentelemetry::layer() - .with_tracer(provider.tracer(instrumentation_scope)) - .with_filter(TargetPrefixFilter { - prefixes: target_prefixes.into_iter().collect(), - include: false, - }) -} +#[cfg(feature = "export")] +pub use setup::*; #[cfg(test)] pub(crate) fn test_lock() -> std::sync::MutexGuard<'static, ()> { @@ -576,265 +82,3 @@ pub(crate) fn test_lock() -> std::sync::MutexGuard<'static, ()> { LOCK.lock() .unwrap_or_else(std::sync::PoisonError::into_inner) } - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn supervisor_span_filter_raises_only_openshell_targets() { - let directives = |log_level: &str| supervisor_span_filter(log_level).to_string(); - assert_eq!(directives("warn"), "openshell=info,info"); - assert_eq!(directives("info"), "openshell=info,info"); - assert_eq!(directives("debug"), "openshell=debug,info"); - assert_eq!(directives("info,h2=debug"), "openshell=info,info"); - } - - #[test] - fn error_status_guard_marks_only_unfinished_results() { - use tracing_subscriber::layer::SubscriberExt as _; - - let _tracing_lock = test_lock(); - let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); - let provider = SdkTracerProvider::builder() - .with_simple_exporter(exporter.clone()) - .build(); - let subscriber = tracing_subscriber::registry().with(layer(&provider, "guard-test")); - - tracing::subscriber::with_default(subscriber, || { - let failed = - tracing::info_span!("failed-operation", otel.status_code = tracing::field::Empty); - { - let _entered = failed.enter(); - drop(ErrorStatusGuard::current()); - } - drop(failed); - - let succeeded = tracing::info_span!( - "successful-operation", - otel.status_code = tracing::field::Empty - ); - { - let _entered = succeeded.enter(); - ErrorStatusGuard::current().finish(Ok::<_, ()>(())).unwrap(); - } - drop(succeeded); - }); - - let spans = exporter.get_finished_spans().unwrap(); - let failed = spans - .iter() - .find(|span| span.name == "failed-operation") - .unwrap(); - assert!(matches!( - failed.status, - opentelemetry::trace::Status::Error { .. } - )); - let succeeded = spans - .iter() - .find(|span| span.name == "successful-operation") - .unwrap(); - assert_eq!(succeeded.status, opentelemetry::trace::Status::Unset); - } - - #[test] - fn target_scoped_layers_partition_in_process_driver_spans() { - use tracing_subscriber::layer::SubscriberExt as _; - - let _tracing_lock = test_lock(); - let gateway_exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); - let gateway_provider = SdkTracerProvider::builder() - .with_simple_exporter(gateway_exporter.clone()) - .build(); - let driver_exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); - let driver_provider = SdkTracerProvider::builder() - .with_simple_exporter(driver_exporter.clone()) - .build(); - let subscriber = tracing_subscriber::registry() - .with(layer_excluding_target_prefix( - &gateway_provider, - "gateway", - Some("driver_target"), - )) - .with(layer_for_target_prefix( - &driver_provider, - "driver", - "driver_target", - )); - - tracing::subscriber::with_default(subscriber, || { - let gateway = tracing::info_span!(target: "gateway_target", "gateway.span"); - let _entered = gateway.enter(); - drop(tracing::info_span!(target: "driver_target::operation", "driver.span")); - }); - - gateway_provider.force_flush().unwrap(); - driver_provider.force_flush().unwrap(); - let gateway_spans = gateway_exporter.get_finished_spans().unwrap(); - let driver_spans = driver_exporter.get_finished_spans().unwrap(); - assert_eq!( - gateway_spans - .iter() - .map(|span| span.name.as_ref()) - .collect::>(), - ["gateway.span"] - ); - assert_eq!( - driver_spans - .iter() - .map(|span| span.name.as_ref()) - .collect::>(), - ["driver.span"] - ); - assert_eq!( - driver_spans[0].span_context.trace_id(), - gateway_spans[0].span_context.trace_id() - ); - assert_eq!( - driver_spans[0].parent_span_id, - gateway_spans[0].span_context.span_id() - ); - } - - #[test] - fn excluding_no_target_prefix_exports_all_application_spans() { - use tracing_subscriber::layer::SubscriberExt as _; - - let _tracing_lock = test_lock(); - let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); - let provider = SdkTracerProvider::builder() - .with_simple_exporter(exporter.clone()) - .build(); - let subscriber = tracing_subscriber::registry() - .with(layer_excluding_target_prefix(&provider, "gateway", None)); - - tracing::subscriber::with_default(subscriber, || { - drop(tracing::info_span!(target: "\0driver_target", "nul-prefixed.span")); - drop(tracing::info_span!(target: "gateway_target", "gateway.span")); - }); - - provider.force_flush().unwrap(); - let spans = exporter.get_finished_spans().unwrap(); - assert_eq!(spans.len(), 2); - assert!(spans.iter().any(|span| span.name == "nul-prefixed.span")); - assert!(spans.iter().any(|span| span.name == "gateway.span")); - } - - #[test] - fn resource_uses_fixed_service_identity_and_custom_attributes() { - let resource = resource_for(&OtlpExportConfig { - endpoint: "http://127.0.0.1:4317", - service_name: ServiceName::Fixed("openshell-driver-vm"), - service_version: Some("1.2.3"), - resource_attributes: vec![KeyValue::new("openshell.gateway.name", "vm-dev")], - }); - - assert_eq!( - resource - .get(&opentelemetry::Key::from_static_str("service.name")) - .map(|value| value.to_string()), - Some("openshell-driver-vm".to_string()) - ); - assert_eq!( - resource - .get(&opentelemetry::Key::from_static_str("service.version")) - .map(|value| value.to_string()), - Some("1.2.3".to_string()) - ); - assert_eq!( - resource - .get(&opentelemetry::Key::from_static_str( - "openshell.gateway.name", - )) - .map(|value| value.to_string()), - Some("vm-dev".to_string()) - ); - } - - #[tokio::test] - async fn tls_settings_need_an_https_endpoint() { - let pki = openshell_otel_test_support::TestPki::generate(); - let tls = OtlpTls::from_pem(Some(pki.ca_pem.as_bytes()), None).unwrap(); - for protocol in [Protocol::Grpc, Protocol::HttpProtobuf] { - let target = |endpoint| { - OtlpTarget::new(&OtlpExportConfig { - endpoint, - service_name: ServiceName::Fixed("test-service"), - service_version: None, - resource_attributes: Vec::new(), - }) - .unwrap() - .with_protocol(protocol) - .with_tls(tls.clone()) - }; - assert!(matches!( - target("http://collector:4317").tracer_provider(), - Err(SetupError::TlsWithoutHttps { .. }) - )); - assert!(matches!( - target("http://collector:4317").log_exporter(), - Err(SetupError::TlsWithoutHttps { .. }) - )); - assert!(target("https://collector:4317").tracer_provider().is_ok()); - } - } - - #[test] - fn debug_output_names_headers_without_their_values() { - let target = OtlpTarget::new(&OtlpExportConfig { - endpoint: "http://collector:4317", - service_name: ServiceName::Fixed("test-service"), - service_version: None, - resource_attributes: Vec::new(), - }) - .unwrap() - .with_headers(parse_otlp_headers("authorization=Bearer%20secret-token")); - - let debug = format!("{target:?}"); - assert!(debug.contains("authorization"), "{debug}"); - assert!(!debug.contains("secret-token"), "{debug}"); - } - - #[tokio::test(flavor = "multi_thread")] - async fn trace_provider_sends_headers_as_grpc_metadata() { - test_environment::Environment::new() - .remove("OTEL_EXPORTER_OTLP_HEADERS") - .remove("OTEL_EXPORTER_OTLP_TRACES_HEADERS") - .remove("OTEL_EXPORTER_OTLP_LOGS_HEADERS") - .run_async(async { - use opentelemetry::trace::Tracer as _; - - let collector = openshell_otel_test_support::OtlpTestServer::start().await; - let provider = OtlpTarget::new(&OtlpExportConfig { - endpoint: collector.endpoint(), - service_name: ServiceName::Fixed("test-service"), - service_version: None, - resource_attributes: Vec::new(), - }) - .unwrap() - .with_headers(parse_otlp_headers("authorization=Bearer%20token")) - .tracer_provider() - .unwrap(); - provider.tracer("test").in_span("grpc-span", |_| {}); - let flush = tokio::task::spawn_blocking(move || provider.force_flush()); - - collector.wait_for_export().await; - flush.await.unwrap().expect("flush succeeds"); - assert_eq!(collector.received().trace_authorizations, ["Bearer token"]); - }) - .await; - } - - #[tokio::test] - async fn malformed_endpoint_disables_export_with_a_reportable_error() { - let (provider, error) = provider_for(Some(OtlpExportConfig { - endpoint: "definitely not a url", - service_name: ServiceName::Fixed("test-service"), - service_version: None, - resource_attributes: Vec::new(), - })); - - assert!(provider.is_none()); - assert!(matches!(error, Some(SetupError::InvalidEndpoint { .. }))); - } -} diff --git a/crates/openshell-otel/src/ocsf.rs b/crates/openshell-otel/src/ocsf.rs index 63489fde1d..86ff38de6a 100644 --- a/crates/openshell-otel/src/ocsf.rs +++ b/crates/openshell-otel/src/ocsf.rs @@ -34,6 +34,7 @@ const MAX_EXPORT_ATTEMPTS: usize = 4; const INITIAL_BACKOFF: Duration = Duration::from_millis(250); const MAX_BACKOFF: Duration = Duration::from_secs(2); const GAP_EVENT: &str = "openshell.ocsf.gap"; +const CORRELATION_UID_ATTRIBUTE: &str = "ocsf.metadata.correlation_uid"; /// Exports OCSF events to an OTLP collector as log records. pub struct OcsfOtlp { @@ -391,6 +392,9 @@ fn log_record( if let Some(uid) = &base.metadata.uid { record.add_attribute("ocsf.metadata.uid", uid.clone()); } + if let Some(uid) = &base.metadata.correlation_uid { + record.add_attribute(CORRELATION_UID_ATTRIBUTE, uid.clone()); + } if let Some(sandbox_id) = event.sandbox_id() { record.add_attribute("openshell.sandbox.id", sandbox_id.to_string()); } @@ -629,6 +633,16 @@ mod tests { assert_eq!(native(&event)["resources"][0]["uid"], "requested-sandbox"); } + #[test] + fn correlation_uid_is_a_stable_otlp_attribute() { + let mut event = event("sandbox-1", SeverityId::Informational); + event.set_correlation_uid("request-123"); + assert_eq!( + attribute(&record(&event), CORRELATION_UID_ATTRIBUTE), + Some(AnyValue::String("request-123".into())) + ); + } + #[test] fn events_without_a_sandbox_carry_no_sandbox_attribute() { let record = record(&event("", SeverityId::Informational)); @@ -788,12 +802,23 @@ mod tests { .with(otlp.layer()); let span_context = tracing::subscriber::with_default(subscriber, || { let span = tracing::info_span!("decision"); - span.in_scope(|| ocsf_emit!(event("sandbox-1", SeverityId::Medium))); + let mut event = event("sandbox-1", SeverityId::Medium); + event.set_correlation_uid("request-with-or-without-sampling"); + span.in_scope(|| ocsf_emit!(event)); span.context().span().span_context().clone() }); otlp.shutdown().await; let records = exporter.exported(); + assert_eq!(records.len(), 1); + assert_eq!( + attribute(&records[0].record, CORRELATION_UID_ATTRIBUTE), + Some(AnyValue::String("request-with-or-without-sampling".into())) + ); + assert_eq!( + body(&records[0].record)["metadata"]["correlation_uid"], + "request-with-or-without-sampling" + ); let trace = records[0].record.trace_context(); if span_context.is_sampled() { let trace = trace.expect("sampled span context is attached"); diff --git a/crates/openshell-otel/src/propagation.rs b/crates/openshell-otel/src/propagation.rs index 02f2bdb120..5bb7ba5ecc 100644 --- a/crates/openshell-otel/src/propagation.rs +++ b/crates/openshell-otel/src/propagation.rs @@ -6,6 +6,7 @@ use std::collections::BTreeMap; use http::HeaderMap; +use opentelemetry::Context; use opentelemetry::propagation::{Extractor, Injector, TextMapPropagator}; use opentelemetry_sdk::propagation::TraceContextPropagator; use tracing_opentelemetry::OpenTelemetrySpanExt as _; @@ -112,12 +113,16 @@ pub fn current_trace_context_environment() -> Vec<(&'static str, String)> { /// Parent `span` under the trace context in `TRACEPARENT` and `TRACESTATE`. /// +/// Reads the environment the same way [`EnvTraceContextInterceptor`] does: +/// both spellings are accepted, whitespace is trimmed, and blank values are +/// treated as unset. +/// /// Leaves `span` unchanged when the environment carries no valid context. pub fn set_parent_from_environment(span: &tracing::Span) { set_parent_from_trace_context( span, - std::env::var(TRACEPARENT_ENV).ok().as_deref(), - std::env::var(TRACESTATE_ENV).ok().as_deref(), + read_env(TRACEPARENT_ENV).as_deref(), + read_env(TRACESTATE_ENV).as_deref(), ); } @@ -177,22 +182,142 @@ impl tonic::service::Interceptor for TraceContextInterceptor { } } +/// Reads W3C trace-context propagation fields from process environment +/// variables (`TRACEPARENT` and `TRACESTATE`). +/// +/// OpenTelemetry integrations in CI systems (for example the Jenkins +/// OpenTelemetry plugin or an OpenTelemetry GitHub Action) export these +/// variables to propagate the pipeline trace to child processes. +/// Both upper-case and lower-case spellings are accepted; blank values are +/// treated as unset. +#[derive(Debug, Clone, Default)] +struct EnvTraceContext { + traceparent: Option, + tracestate: Option, +} + +impl EnvTraceContext { + fn from_env() -> Self { + Self { + traceparent: read_env(TRACEPARENT_ENV), + tracestate: read_env(TRACESTATE_ENV), + } + } +} + +/// Read one environment variable, trimming whitespace and discarding blanks. +fn read_one(name: &str) -> Option { + std::env::var(name) + .ok() + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) +} + +/// Read an environment variable by its upper-case name, falling back to the +/// lower-case spelling, trimming whitespace and discarding blank values. +/// +/// Each spelling is normalized before the fallback, so a blank upper-case +/// value never shadows a valid lower-case one. +fn read_env(upper: &str) -> Option { + read_one(upper).or_else(|| read_one(&upper.to_ascii_lowercase())) +} + +impl Extractor for EnvTraceContext { + fn get(&self, key: &str) -> Option<&str> { + match key { + "traceparent" => self.traceparent.as_deref(), + "tracestate" => self.tracestate.as_deref(), + _ => None, + } + } + + fn keys(&self) -> Vec<&str> { + let mut keys = Vec::new(); + if self.traceparent.is_some() { + keys.push("traceparent"); + } + if self.tracestate.is_some() { + keys.push("tracestate"); + } + keys + } +} + +/// Passively forwards a W3C trace context captured from the process +/// environment onto outbound tonic requests. +/// +/// Unlike [`TraceContextInterceptor`], this does not require an active +/// OpenTelemetry span or an `SdkTracerProvider`; it reads `TRACEPARENT` +/// (and optionally `TRACESTATE`) once at construction and injects them as +/// gRPC metadata. This lets short-lived clients (e.g. the CLI on a CI runner) +/// extend a pipeline trace to the gateway without any collector configuration. +/// +/// When `TRACEPARENT` is unset or invalid the captured context carries no +/// valid span, so injection is a no-op. +#[derive(Debug, Clone)] +pub struct EnvTraceContextInterceptor { + context: Context, +} + +impl EnvTraceContextInterceptor { + /// Capture the W3C trace context from the process environment. + #[must_use] + pub fn from_env() -> Self { + let env = EnvTraceContext::from_env(); + let context = context_from(&env); + if env.traceparent.is_some() && !span_context_is_valid(&context) { + tracing::debug!("TRACEPARENT is set but invalid; not forwarding trace context"); + } + Self { context } + } +} + +/// Extract a [`Context`] from an [`Extractor`] using a fresh base context, so +/// the result depends only on the extractor and not on any ambient context. +fn context_from(extractor: &impl Extractor) -> Context { + TraceContextPropagator::new().extract_with_context(&Context::new(), extractor) +} + +fn span_context_is_valid(context: &Context) -> bool { + use opentelemetry::trace::TraceContextExt as _; + + context.span().span_context().is_valid() +} + +impl tonic::service::Interceptor for EnvTraceContextInterceptor { + fn call( + &mut self, + mut request: tonic::Request<()>, + ) -> Result, tonic::Status> { + TraceContextPropagator::new().inject_context( + &self.context, + &mut MetadataMapInjector::new(request.metadata_mut()), + ); + Ok(request) + } +} + #[cfg(test)] mod tests { use super::*; + use tonic::service::Interceptor as _; fn env_test_exporter() -> ( opentelemetry_sdk::trace::SdkTracerProvider, opentelemetry_sdk::trace::InMemorySpanExporter, impl tracing::Subscriber + Send + Sync, ) { + use opentelemetry::trace::TracerProvider as _; use tracing_subscriber::layer::SubscriberExt as _; let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "env-test")); + // Built directly from `tracing_opentelemetry` so the test runs even + // without the `export` feature. + let subscriber = tracing_subscriber::registry() + .with(tracing_opentelemetry::layer().with_tracer(provider.tracer("env-test"))); (provider, exporter, subscriber) } @@ -263,6 +388,7 @@ mod tests { #[test] fn invalid_environment_carrier_leaves_the_span_unparented() { + use opentelemetry::trace::TracerProvider as _; use tracing_subscriber::layer::SubscriberExt as _; let _tracing_lock = crate::test_lock(); @@ -270,7 +396,8 @@ mod tests { let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "env-test")); + let subscriber = tracing_subscriber::registry() + .with(tracing_opentelemetry::layer().with_tracer(provider.tracer("env-test"))); tracing::subscriber::with_default(subscriber, || { let span = tracing::info_span!("orphan"); @@ -285,12 +412,14 @@ mod tests { #[test] fn interceptor_adds_traceparent_only_inside_an_exported_span() { + use opentelemetry::trace::TracerProvider as _; use tonic::service::Interceptor as _; use tracing_subscriber::layer::SubscriberExt as _; let _tracing_lock = crate::test_lock(); let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder().build(); - let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "rpc-test")); + let subscriber = tracing_subscriber::registry() + .with(tracing_opentelemetry::layer().with_tracer(provider.tracer("rpc-test"))); let (outside, inside) = tracing::subscriber::with_default(subscriber, || { let outside = TraceContextInterceptor @@ -330,4 +459,147 @@ mod tests { Some("value") ); } + + fn interceptor_with(extractor: EnvTraceContext) -> EnvTraceContextInterceptor { + EnvTraceContextInterceptor { + context: context_from(&extractor), + } + } + + /// Restores snapshotted environment variables when dropped, so a failed + /// assertion cannot leak them into later tests sharing the process. + struct EnvRestore(Vec<(&'static str, Option)>); + + impl EnvRestore { + fn snapshot(names: &[&'static str]) -> Self { + Self( + names + .iter() + .map(|name| (*name, std::env::var(name).ok())) + .collect(), + ) + } + } + + impl Drop for EnvRestore { + #[allow(unsafe_code)] + fn drop(&mut self) { + for (name, value) in &self.0 { + unsafe { + match value { + Some(value) => std::env::set_var(name, value), + None => std::env::remove_var(name), + } + } + } + } + } + + #[test] + #[allow(unsafe_code)] + fn env_trace_context_from_env_trims_and_skips_blanks() { + let _tracing_lock = crate::test_lock(); + let _restore = EnvRestore::snapshot(&[TRACEPARENT_ENV, TRACESTATE_ENV]); + let traceparent = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + + // Upper-case spelling with surrounding whitespace. + unsafe { + std::env::set_var(TRACEPARENT_ENV, format!(" {traceparent} ")); + std::env::set_var(TRACESTATE_ENV, " vendor=value "); + } + let padded = EnvTraceContext::from_env(); + assert_eq!(padded.traceparent.as_deref(), Some(traceparent)); + assert_eq!(padded.tracestate.as_deref(), Some("vendor=value")); + + // Blank values are treated as unset. + unsafe { + std::env::set_var(TRACEPARENT_ENV, " "); + std::env::set_var(TRACESTATE_ENV, ""); + } + let blank = EnvTraceContext::from_env(); + assert_eq!(blank.traceparent, None); + assert_eq!(blank.tracestate, None); + } + + /// Windows treats environment names case-insensitively, so the lower-case + /// fallback cannot be exercised there: `TRACEPARENT` and `traceparent` + /// are the same variable. + #[test] + #[cfg(not(windows))] + #[allow(unsafe_code)] + fn env_trace_context_from_env_reads_lower_case_spelling() { + let _tracing_lock = crate::test_lock(); + let _restore = + EnvRestore::snapshot(&[TRACEPARENT_ENV, "traceparent", TRACESTATE_ENV, "tracestate"]); + let traceparent = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + + // Lower-case-only spelling. + unsafe { + std::env::remove_var(TRACEPARENT_ENV); + std::env::remove_var(TRACESTATE_ENV); + std::env::set_var("traceparent", traceparent); + std::env::set_var("tracestate", "vendor=value"); + } + let lower = EnvTraceContext::from_env(); + assert_eq!(lower.traceparent.as_deref(), Some(traceparent)); + assert_eq!(lower.tracestate.as_deref(), Some("vendor=value")); + + // A blank upper-case value must not shadow a valid lower-case one. + unsafe { + std::env::set_var(TRACEPARENT_ENV, " "); + std::env::remove_var("tracestate"); + } + let shadowed = EnvTraceContext::from_env(); + assert_eq!(shadowed.traceparent.as_deref(), Some(traceparent)); + assert_eq!(shadowed.tracestate, None); + } + + #[test] + fn env_interceptor_forwards_valid_traceparent() { + let traceparent = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + let mut interceptor = interceptor_with(EnvTraceContext { + traceparent: Some(traceparent.to_string()), + tracestate: Some("vendor=value".to_string()), + }); + + let request = interceptor.call(tonic::Request::new(())).unwrap(); + + assert_eq!( + request + .metadata() + .get("traceparent") + .and_then(|value| value.to_str().ok()), + Some(traceparent) + ); + assert_eq!( + request + .metadata() + .get("tracestate") + .and_then(|value| value.to_str().ok()), + Some("vendor=value") + ); + } + + #[test] + fn env_interceptor_is_noop_without_traceparent() { + let mut interceptor = interceptor_with(EnvTraceContext::default()); + + let request = interceptor.call(tonic::Request::new(())).unwrap(); + + assert!(request.metadata().get("traceparent").is_none()); + assert!(request.metadata().get("tracestate").is_none()); + } + + #[test] + fn env_interceptor_is_noop_for_invalid_traceparent() { + let mut interceptor = interceptor_with(EnvTraceContext { + traceparent: Some("not-a-valid-traceparent".to_string()), + tracestate: Some("vendor=value".to_string()), + }); + + let request = interceptor.call(tonic::Request::new(())).unwrap(); + + assert!(request.metadata().get("traceparent").is_none()); + assert!(request.metadata().get("tracestate").is_none()); + } } diff --git a/crates/openshell-otel/src/setup.rs b/crates/openshell-otel/src/setup.rs new file mode 100644 index 0000000000..1bf8407e59 --- /dev/null +++ b/crates/openshell-otel/src/setup.rs @@ -0,0 +1,798 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OTLP export setup: resources, providers, and tracing layers. +//! +//! Compiled only with the `export` feature; see the crate-level documentation. + +use crate::tls::{OtlpTls, TlsError, TlsFileError}; +use crate::transport::{self, OtlpHttpClient, signal_url}; +use opentelemetry::KeyValue; +use opentelemetry::trace::TracerProvider as _; +pub use opentelemetry_otlp::LogExporter; +use opentelemetry_otlp::tonic_types::metadata::MetadataMap; +use opentelemetry_otlp::{SpanExporter, WithExportConfig, WithHttpConfig, WithTonicConfig}; +use opentelemetry_sdk::Resource; +use opentelemetry_sdk::trace::SdkTracer; +pub use opentelemetry_sdk::trace::SdkTracerProvider; +use tracing::Subscriber; +use tracing_opentelemetry::OpenTelemetryLayer; +use tracing_subscriber::Layer as _; +use tracing_subscriber::layer::{Context, Filter}; +use tracing_subscriber::registry::LookupSpan; + +/// Export supervisor spans at INFO by default, or DEBUG and TRACE when enabled. +/// Dependency spans remain at INFO. +#[must_use] +pub fn supervisor_span_filter(log_level: &str) -> tracing_subscriber::EnvFilter { + use tracing_subscriber::filter::LevelFilter; + let level = log_level + .parse::() + .unwrap_or(LevelFilter::INFO) + .max(LevelFilter::INFO); + tracing_subscriber::EnvFilter::new(format!("info,openshell={level}")) +} + +const SDK_UNKNOWN_SERVICE_PREFIX: &str = "unknown_service"; + +/// Build the resource attributes shared by gateway and compute-driver providers. +pub fn gateway_resource_attributes( + gateway_name: Option<&str>, + compute_driver: Option<&str>, +) -> Vec { + let mut attributes = Vec::new(); + if let Some(name) = gateway_name.map(str::trim).filter(|name| !name.is_empty()) { + attributes.push(KeyValue::new("openshell.gateway.name", name.to_string())); + } + if let Some(driver) = compute_driver + .map(str::trim) + .filter(|driver| !driver.is_empty()) + { + attributes.push(KeyValue::new( + "openshell.gateway.compute_driver", + driver.to_string(), + )); + } + attributes +} + +/// Mark `span` as failed. +/// +/// The field must be declared on the span at creation because `tracing` drops +/// records for fields a span does not have. +pub fn mark_error(span: &tracing::Span) { + span.record("otel.status_code", "ERROR"); +} + +/// Marks the current span when an instrumented operation returns an error. +pub fn record_error_result(result: Result) -> Result { + if result.is_err() { + mark_error(&tracing::Span::current()); + } + result +} + +/// Marks an instrumented function's span when it exits before returning success. +/// +/// Create the guard at the start of the instrumented function and return the +/// successful result through [`Self::finish`]. An early `?` or error return +/// drops the unfinished guard and marks the captured span as failed. +#[must_use] +pub struct ErrorStatusGuard { + span: tracing::Span, + finished: bool, +} + +impl ErrorStatusGuard { + /// Captures the current instrumented span. + pub fn current() -> Self { + Self { + span: tracing::Span::current(), + finished: false, + } + } + + /// Returns `result`, marking this guard complete when it is successful. + pub fn finish(mut self, result: Result) -> Result { + self.finished = result.is_ok(); + result + } +} + +impl Drop for ErrorStatusGuard { + fn drop(&mut self) { + if !self.finished { + mark_error(&self.span); + } + } +} + +/// How a process chooses its OpenTelemetry `service.name`. +#[derive(Debug, Clone, Copy)] +pub enum ServiceName<'a> { + /// Always use this name, overriding `OTEL_SERVICE_NAME`. + Fixed(&'a str), + /// Use `OTEL_SERVICE_NAME` when set, otherwise use this default. + EnvironmentOr(&'a str), +} + +/// OTLP transport to a collector. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum Protocol { + #[default] + Grpc, + HttpProtobuf, +} + +/// Inputs for OTLP export. +#[derive(Debug, Clone)] +pub struct OtlpExportConfig<'a> { + pub endpoint: &'a str, + pub service_name: ServiceName<'a>, + pub service_version: Option<&'a str>, + pub resource_attributes: Vec, +} + +/// How exports reach a collector: the transport, request headers, and TLS. +#[derive(Clone, Default, PartialEq, Eq)] +pub struct CollectorTransport { + pub protocol: Protocol, + pub headers: http::HeaderMap, + pub tls: OtlpTls, +} + +impl std::fmt::Debug for CollectorTransport { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("CollectorTransport") + .field("protocol", &self.protocol) + .field("header_names", &self.headers.keys().collect::>()) + .field("tls", &self.tls) + .finish() + } +} + +/// Failure to set up OTLP export. +#[derive(Debug, thiserror::Error)] +pub enum SetupError { + #[error("OTLP endpoint is empty")] + EmptyEndpoint, + + #[error("invalid OTLP endpoint {endpoint:?}: {source}")] + InvalidEndpoint { + endpoint: String, + source: http::uri::InvalidUri, + }, + + #[error("failed to build the OTLP exporter: {0}")] + Exporter(#[from] opentelemetry_otlp::ExporterBuildError), + + #[error("invalid OTLP endpoint {endpoint:?}: {source}")] + InvalidSignalUrl { + endpoint: String, + source: http::Error, + }, + + #[error("OTLP TLS settings need an https:// endpoint, not {endpoint:?}")] + TlsWithoutHttps { endpoint: String }, + + #[error("invalid OTLP TLS settings: {0}")] + Tls(#[from] TlsError), + + #[error("invalid OTLP TLS files: {0}")] + TlsFiles(#[from] TlsFileError), + + #[error("OTLP/HTTP export needs a tokio runtime")] + NoRuntime, +} + +fn resource_attributes(config: &OtlpExportConfig<'_>) -> Vec { + let mut attributes = config.resource_attributes.clone(); + if let Some(version) = config + .service_version + .map(str::trim) + .filter(|version| !version.is_empty()) + { + attributes.push(KeyValue::new("service.version", version.to_string())); + } + attributes +} + +/// Build the OpenTelemetry resource for an export configuration. +#[must_use] +pub fn resource_for(config: &OtlpExportConfig<'_>) -> Resource { + let attributes = resource_attributes(config); + match config.service_name { + ServiceName::Fixed(name) => Resource::builder() + .with_service_name(name.trim().to_string()) + .with_attributes(attributes) + .build(), + ServiceName::EnvironmentOr(default) => { + let detected = Resource::builder() + .with_attributes(attributes.clone()) + .build(); + if detected + .get(&opentelemetry::Key::from_static_str("service.name")) + .is_some_and(|value| !value.to_string().starts_with(SDK_UNKNOWN_SERVICE_PREFIX)) + { + detected + } else { + Resource::builder() + .with_service_name(default.trim().to_string()) + .with_attributes(attributes) + .build() + } + } + } +} + +/// Trim and validate an OTLP endpoint before handing it to an exporter. +/// +/// Every exporter that reads an operator-supplied endpoint uses this, so a +/// value accepted for one signal is accepted for all of them. +pub fn validated_endpoint(endpoint: &str) -> Result<(&str, http::Uri), SetupError> { + let endpoint = endpoint.trim(); + if endpoint.is_empty() { + return Err(SetupError::EmptyEndpoint); + } + let uri = endpoint + .parse::() + .map_err(|source| SetupError::InvalidEndpoint { + endpoint: endpoint.to_string(), + source, + })?; + Ok((endpoint, uri)) +} + +/// A validated OTLP collector endpoint and the resource that every signal +/// exported to it describes. +#[derive(Clone)] +pub struct OtlpTarget { + pub(crate) endpoint: String, + pub(crate) tls: bool, + pub(crate) protocol: Protocol, + pub(crate) headers: http::HeaderMap, + pub(crate) tls_settings: OtlpTls, + pub(crate) resource: Resource, +} + +impl std::fmt::Debug for OtlpTarget { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("OtlpTarget") + .field("endpoint", &self.endpoint) + .field("tls", &self.tls) + .field("protocol", &self.protocol) + .field("header_names", &self.headers.keys().collect::>()) + .field("tls_settings", &self.tls_settings) + .field("resource", &self.resource) + .finish() + } +} + +impl OtlpTarget { + /// Validate the endpoint and build the resource. + pub fn new(config: &OtlpExportConfig<'_>) -> Result { + let (endpoint, uri) = validated_endpoint(config.endpoint)?; + Ok(Self { + endpoint: endpoint.to_string(), + tls: uri.scheme() == Some(&http::uri::Scheme::HTTPS), + protocol: Protocol::Grpc, + headers: http::HeaderMap::new(), + tls_settings: OtlpTls::default(), + resource: resource_for(config), + }) + } + + /// Export over `protocol` instead of gRPC. + #[must_use] + pub fn with_protocol(mut self, protocol: Protocol) -> Self { + self.protocol = protocol; + self + } + + /// Send `headers` with every export request. Headers from the process + /// environment (`OTEL_EXPORTER_OTLP_*HEADERS`) still apply and replace + /// any of the same name. + #[must_use] + pub fn with_headers(mut self, headers: http::HeaderMap) -> Self { + self.headers = headers; + self + } + + /// Reach the collector the way `transport` describes. + #[must_use] + pub fn with_transport(self, transport: &CollectorTransport) -> Self { + self.with_protocol(transport.protocol) + .with_headers(transport.headers.clone()) + .with_tls(transport.tls.clone()) + } + + /// Verify an `https://` collector and authenticate to it with `tls`. + #[must_use] + pub fn with_tls(mut self, tls: OtlpTls) -> Self { + self.tls_settings = tls; + self + } + + /// The URL of `signal_path` under the endpoint, for OTLP/HTTP export. + pub(crate) fn http_signal_url(&self, signal_path: &str) -> Result { + signal_url(&self.endpoint, signal_path) + .map(|url| url.to_string()) + .map_err(|source| SetupError::InvalidSignalUrl { + endpoint: self.endpoint.clone(), + source, + }) + } + + /// Point an exporter builder at this target, over TLS for `https://`. + pub(crate) fn configure(&self, builder: B) -> B + where + B: WithExportConfig + WithTonicConfig, + { + let builder = builder + .with_endpoint(self.endpoint.clone()) + .with_metadata(MetadataMap::from_headers(self.headers.clone())); + if self.tls { + builder.with_tls_config(self.tls_settings.grpc_config()) + } else { + builder + } + } + + /// Point an OTLP/HTTP exporter builder at `signal_path` under this + /// target. + pub(crate) fn configure_http( + &self, + builder: B, + signal_path: &str, + timeout_var: &str, + ) -> Result + where + B: WithExportConfig + WithHttpConfig, + { + // The SDK percent-decodes these values again. + let headers = self + .headers + .iter() + .filter_map(|(name, value)| { + Some((name.to_string(), value.to_str().ok()?.replace('%', "%25"))) + }) + .collect(); + Ok(builder + .with_protocol(opentelemetry_otlp::Protocol::HttpBinary) + .with_endpoint(self.http_signal_url(signal_path)?) + .with_http_client(OtlpHttpClient::new( + self.tls.then_some(&self.tls_settings), + transport::export_timeout(timeout_var), + )?) + .with_headers(headers)) + } + + #[must_use] + pub fn endpoint(&self) -> &str { + &self.endpoint + } + + #[must_use] + pub fn resource(&self) -> &Resource { + &self.resource + } + + /// Fails when TLS settings are given for a plaintext endpoint. + pub(crate) fn check_tls(&self) -> Result<(), SetupError> { + if self.tls || self.tls_settings.is_default() { + Ok(()) + } else { + Err(SetupError::TlsWithoutHttps { + endpoint: self.endpoint.clone(), + }) + } + } + + /// Build a trace provider that exports spans to this target in batches. + pub fn tracer_provider(&self) -> Result { + self.check_tls()?; + let exporter = match self.protocol { + Protocol::Grpc => self + .configure(SpanExporter::builder().with_tonic()) + .build()?, + Protocol::HttpProtobuf => self + .configure_http( + SpanExporter::builder().with_http(), + "v1/traces", + "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT", + )? + .build()?, + }; + Ok(SdkTracerProvider::builder() + .with_batch_exporter(exporter) + .with_resource(self.resource.clone()) + .build()) + } +} + +/// Build an OTLP trace provider. +pub fn build_provider(config: &OtlpExportConfig<'_>) -> Result { + OtlpTarget::new(config)?.tracer_provider() +} + +/// Build the provider for an optional OTLP configuration. +/// +/// Telemetry setup failures disable export and remain available for the caller +/// to report after its tracing subscriber is installed. +#[must_use] +pub fn provider_for( + config: Option>, +) -> (Option, Option) { + match config.as_ref().map(build_provider) { + None => (None, None), + Some(Ok(provider)) => (Some(provider), None), + Some(Err(error)) => (None, Some(error)), + } +} + +/// Filtered OpenTelemetry layer returned by [`layer`]. +pub type OtlpLayer = tracing_subscriber::filter::Filtered< + OpenTelemetryLayer, + tracing_subscriber::filter::FilterFn, + S, +>; + +pub type TargetOtlpLayer = + tracing_subscriber::filter::Filtered, TargetPrefixFilter, S>; + +#[derive(Debug, Clone)] +pub struct TargetPrefixFilter { + prefixes: Vec<&'static str>, + include: bool, +} + +impl Filter for TargetPrefixFilter { + fn enabled(&self, metadata: &tracing::Metadata<'_>, _ctx: &Context<'_, S>) -> bool { + let matches_prefix = self + .prefixes + .iter() + .any(|prefix| metadata.target().starts_with(prefix)); + metadata.is_span() + && !metadata.target().starts_with("opentelemetry") + && (matches_prefix == self.include) + } +} + +/// Build a tracing layer that exports spans and excludes exporter callsites. +pub fn layer(provider: &SdkTracerProvider, instrumentation_scope: &'static str) -> OtlpLayer +where + S: Subscriber + for<'span> LookupSpan<'span>, +{ + tracing_opentelemetry::layer() + .with_tracer(provider.tracer(instrumentation_scope)) + .with_filter(tracing_subscriber::filter::filter_fn(|metadata| { + metadata.is_span() && !metadata.target().starts_with("opentelemetry") + })) +} + +/// Build a tracing layer that exports only spans from `target_prefix`. +pub fn layer_for_target_prefix( + provider: &SdkTracerProvider, + instrumentation_scope: &'static str, + target_prefix: &'static str, +) -> TargetOtlpLayer +where + S: Subscriber + for<'span> LookupSpan<'span>, +{ + layer_for_target_prefixes(provider, instrumentation_scope, [target_prefix]) +} + +/// Build a tracing layer that exports only spans from `target_prefixes`. +pub fn layer_for_target_prefixes( + provider: &SdkTracerProvider, + instrumentation_scope: &'static str, + target_prefixes: impl IntoIterator, +) -> TargetOtlpLayer +where + S: Subscriber + for<'span> LookupSpan<'span>, +{ + tracing_opentelemetry::layer() + .with_tracer(provider.tracer(instrumentation_scope)) + .with_filter(TargetPrefixFilter { + prefixes: target_prefixes.into_iter().collect(), + include: true, + }) +} + +/// Build a tracing layer that excludes spans from `target_prefix`. +/// +/// `None` excludes no application spans. +pub fn layer_excluding_target_prefix( + provider: &SdkTracerProvider, + instrumentation_scope: &'static str, + target_prefix: Option<&'static str>, +) -> TargetOtlpLayer +where + S: Subscriber + for<'span> LookupSpan<'span>, +{ + layer_excluding_target_prefixes(provider, instrumentation_scope, target_prefix) +} + +/// Build a tracing layer that excludes spans from `target_prefixes`. +/// +/// An empty iterator excludes no application spans. +pub fn layer_excluding_target_prefixes( + provider: &SdkTracerProvider, + instrumentation_scope: &'static str, + target_prefixes: impl IntoIterator, +) -> TargetOtlpLayer +where + S: Subscriber + for<'span> LookupSpan<'span>, +{ + tracing_opentelemetry::layer() + .with_tracer(provider.tracer(instrumentation_scope)) + .with_filter(TargetPrefixFilter { + prefixes: target_prefixes.into_iter().collect(), + include: false, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::parse_otlp_headers; + + #[test] + fn supervisor_span_filter_raises_only_openshell_targets() { + let directives = |log_level: &str| supervisor_span_filter(log_level).to_string(); + assert_eq!(directives("warn"), "openshell=info,info"); + assert_eq!(directives("info"), "openshell=info,info"); + assert_eq!(directives("debug"), "openshell=debug,info"); + assert_eq!(directives("info,h2=debug"), "openshell=info,info"); + } + + #[test] + fn error_status_guard_marks_only_unfinished_results() { + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = crate::test_lock(); + let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry().with(layer(&provider, "guard-test")); + + tracing::subscriber::with_default(subscriber, || { + let failed = + tracing::info_span!("failed-operation", otel.status_code = tracing::field::Empty); + { + let _entered = failed.enter(); + drop(ErrorStatusGuard::current()); + } + drop(failed); + + let succeeded = tracing::info_span!( + "successful-operation", + otel.status_code = tracing::field::Empty + ); + { + let _entered = succeeded.enter(); + ErrorStatusGuard::current().finish(Ok::<_, ()>(())).unwrap(); + } + drop(succeeded); + }); + + let spans = exporter.get_finished_spans().unwrap(); + let failed = spans + .iter() + .find(|span| span.name == "failed-operation") + .unwrap(); + assert!(matches!( + failed.status, + opentelemetry::trace::Status::Error { .. } + )); + let succeeded = spans + .iter() + .find(|span| span.name == "successful-operation") + .unwrap(); + assert_eq!(succeeded.status, opentelemetry::trace::Status::Unset); + } + + #[test] + fn target_scoped_layers_partition_in_process_driver_spans() { + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = crate::test_lock(); + let gateway_exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); + let gateway_provider = SdkTracerProvider::builder() + .with_simple_exporter(gateway_exporter.clone()) + .build(); + let driver_exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); + let driver_provider = SdkTracerProvider::builder() + .with_simple_exporter(driver_exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry() + .with(layer_excluding_target_prefix( + &gateway_provider, + "gateway", + Some("driver_target"), + )) + .with(layer_for_target_prefix( + &driver_provider, + "driver", + "driver_target", + )); + + tracing::subscriber::with_default(subscriber, || { + let gateway = tracing::info_span!(target: "gateway_target", "gateway.span"); + let _entered = gateway.enter(); + drop(tracing::info_span!(target: "driver_target::operation", "driver.span")); + }); + + gateway_provider.force_flush().unwrap(); + driver_provider.force_flush().unwrap(); + let gateway_spans = gateway_exporter.get_finished_spans().unwrap(); + let driver_spans = driver_exporter.get_finished_spans().unwrap(); + assert_eq!( + gateway_spans + .iter() + .map(|span| span.name.as_ref()) + .collect::>(), + ["gateway.span"] + ); + assert_eq!( + driver_spans + .iter() + .map(|span| span.name.as_ref()) + .collect::>(), + ["driver.span"] + ); + assert_eq!( + driver_spans[0].span_context.trace_id(), + gateway_spans[0].span_context.trace_id() + ); + assert_eq!( + driver_spans[0].parent_span_id, + gateway_spans[0].span_context.span_id() + ); + } + + #[test] + fn excluding_no_target_prefix_exports_all_application_spans() { + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = crate::test_lock(); + let exporter = opentelemetry_sdk::trace::InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry() + .with(layer_excluding_target_prefix(&provider, "gateway", None)); + + tracing::subscriber::with_default(subscriber, || { + drop(tracing::info_span!(target: "\0driver_target", "nul-prefixed.span")); + drop(tracing::info_span!(target: "gateway_target", "gateway.span")); + }); + + provider.force_flush().unwrap(); + let spans = exporter.get_finished_spans().unwrap(); + assert_eq!(spans.len(), 2); + assert!(spans.iter().any(|span| span.name == "nul-prefixed.span")); + assert!(spans.iter().any(|span| span.name == "gateway.span")); + } + + #[test] + fn resource_uses_fixed_service_identity_and_custom_attributes() { + let resource = resource_for(&OtlpExportConfig { + endpoint: "http://127.0.0.1:4317", + service_name: ServiceName::Fixed("openshell-driver-vm"), + service_version: Some("1.2.3"), + resource_attributes: vec![KeyValue::new("openshell.gateway.name", "vm-dev")], + }); + + assert_eq!( + resource + .get(&opentelemetry::Key::from_static_str("service.name")) + .map(|value| value.to_string()), + Some("openshell-driver-vm".to_string()) + ); + assert_eq!( + resource + .get(&opentelemetry::Key::from_static_str("service.version")) + .map(|value| value.to_string()), + Some("1.2.3".to_string()) + ); + assert_eq!( + resource + .get(&opentelemetry::Key::from_static_str( + "openshell.gateway.name", + )) + .map(|value| value.to_string()), + Some("vm-dev".to_string()) + ); + } + + #[tokio::test] + async fn tls_settings_need_an_https_endpoint() { + let pki = openshell_otel_test_support::TestPki::generate(); + let tls = OtlpTls::from_pem(Some(pki.ca_pem.as_bytes()), None).unwrap(); + for protocol in [Protocol::Grpc, Protocol::HttpProtobuf] { + let target = |endpoint| { + OtlpTarget::new(&OtlpExportConfig { + endpoint, + service_name: ServiceName::Fixed("test-service"), + service_version: None, + resource_attributes: Vec::new(), + }) + .unwrap() + .with_protocol(protocol) + .with_tls(tls.clone()) + }; + assert!(matches!( + target("http://collector:4317").tracer_provider(), + Err(SetupError::TlsWithoutHttps { .. }) + )); + assert!(matches!( + target("http://collector:4317").log_exporter(), + Err(SetupError::TlsWithoutHttps { .. }) + )); + assert!(target("https://collector:4317").tracer_provider().is_ok()); + } + } + + #[test] + fn debug_output_names_headers_without_their_values() { + let target = OtlpTarget::new(&OtlpExportConfig { + endpoint: "http://collector:4317", + service_name: ServiceName::Fixed("test-service"), + service_version: None, + resource_attributes: Vec::new(), + }) + .unwrap() + .with_headers(parse_otlp_headers("authorization=Bearer%20secret-token")); + + let debug = format!("{target:?}"); + assert!(debug.contains("authorization"), "{debug}"); + assert!(!debug.contains("secret-token"), "{debug}"); + } + + #[tokio::test(flavor = "multi_thread")] + async fn trace_provider_sends_headers_as_grpc_metadata() { + crate::test_environment::Environment::new() + .remove("OTEL_EXPORTER_OTLP_HEADERS") + .remove("OTEL_EXPORTER_OTLP_TRACES_HEADERS") + .remove("OTEL_EXPORTER_OTLP_LOGS_HEADERS") + .run_async(async { + use opentelemetry::trace::Tracer as _; + + let collector = openshell_otel_test_support::OtlpTestServer::start().await; + let provider = OtlpTarget::new(&OtlpExportConfig { + endpoint: collector.endpoint(), + service_name: ServiceName::Fixed("test-service"), + service_version: None, + resource_attributes: Vec::new(), + }) + .unwrap() + .with_headers(parse_otlp_headers("authorization=Bearer%20token")) + .tracer_provider() + .unwrap(); + provider.tracer("test").in_span("grpc-span", |_| {}); + let flush = tokio::task::spawn_blocking(move || provider.force_flush()); + + collector.wait_for_export().await; + flush.await.unwrap().expect("flush succeeds"); + assert_eq!(collector.received().trace_authorizations, ["Bearer token"]); + }) + .await; + } + + #[tokio::test] + async fn malformed_endpoint_disables_export_with_a_reportable_error() { + let (provider, error) = provider_for(Some(OtlpExportConfig { + endpoint: "definitely not a url", + service_name: ServiceName::Fixed("test-service"), + service_version: None, + resource_attributes: Vec::new(), + })); + + assert!(provider.is_none()); + assert!(matches!(error, Some(SetupError::InvalidEndpoint { .. }))); + } +} diff --git a/crates/openshell-sandbox-backend/Cargo.toml b/crates/openshell-sandbox-backend/Cargo.toml index 8e8ff7cb05..f95c7bd6a2 100644 --- a/crates/openshell-sandbox-backend/Cargo.toml +++ b/crates/openshell-sandbox-backend/Cargo.toml @@ -43,6 +43,7 @@ libc = "0.2" rustix = { workspace = true, features = ["fs", "process"] } [dev-dependencies] +tokio = { workspace = true, features = ["test-util"] } jsonwebtoken = { workspace = true } [lints] diff --git a/crates/openshell-sandbox-backend/README.md b/crates/openshell-sandbox-backend/README.md index 8deb207f5d..b64aac2df3 100644 --- a/crates/openshell-sandbox-backend/README.md +++ b/crates/openshell-sandbox-backend/README.md @@ -199,8 +199,9 @@ selects which instance's generation-pinned TLS identity the supervisor then demands. Unknown, removed, and replaced handles are closed before any handshake. Handshakes are bounded globally and per route, and an authenticated connection only fills its own instance's spare queue. A routed -instance waits up to five minutes for its first connection, since it is -registered before its workload Pod is scheduled, and the usual connect window +instance waits up to its configured initial attachment budget (five minutes +by default) for its first connection, since it is registered before its +workload Pod is scheduled, and the usual connect window afterwards. The launch material and status of one hosted instance (`SandboxInstanceRegistration`, `SandboxInstanceStatusReport`) are defined by `openshell-supervisor-control`, which depends on this crate for the runtime diff --git a/crates/openshell-sandbox-backend/src/runtime.rs b/crates/openshell-sandbox-backend/src/runtime.rs index 9ea72d396c..878634ac8a 100644 --- a/crates/openshell-sandbox-backend/src/runtime.rs +++ b/crates/openshell-sandbox-backend/src/runtime.rs @@ -52,10 +52,6 @@ const ATTACH_REQUEST_TIMEOUT: Duration = Duration::from_mins(5); /// callers retry whole calls above this; past boot, exhausting this window /// means the remote boundary (or its launcher) is gone rather than still starting. const CONNECT_RETRY_TIMEOUT: Duration = Duration::from_secs(30); -/// Bound on a routed sandbox instance's first connection. The instance is -/// registered before its workload Pod is scheduled and its image pulled; the -/// driver reaps a bootstrap that is still incomplete after the same interval. -const ROUTED_FIRST_CONNECT_TIMEOUT: Duration = Duration::from_mins(5); /// Bounds the liveness check that decides whether a stream failure lost the /// whole connection. const CONNECTION_PROBE_TIMEOUT: Duration = Duration::from_secs(5); @@ -159,6 +155,7 @@ impl BoundaryTransport { async fn connect( &self, descriptor: &SandboxRuntimeDescriptor, + initial_connect_timeout: Duration, ) -> Result { let connector = match &self.0 { TransportSource::Dial => None, @@ -176,7 +173,7 @@ impl BoundaryTransport { let window = if registration.has_connected() { CONNECT_RETRY_TIMEOUT } else { - ROUTED_FIRST_CONNECT_TIMEOUT + initial_connect_timeout }; return tokio::time::timeout(window, registration.accept(&descriptor.tls)) .await @@ -327,6 +324,7 @@ pub struct OpenShellRuntimeBackend { ca_file_paths: Arc>>, provider_credentials: openshell_core::provider_credentials::ProviderCredentialState, sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, + attach_request_timeout: Duration, delegated: Option, resume: Option, takeover: Option, @@ -355,9 +353,11 @@ impl OpenShellRuntimeBackend { descriptor: SandboxRuntimeDescriptor, bearer: openshell_core::jwt::SessionBearerTokenSlot, transport: BoundaryTransport, + attach_request_timeout: Duration, ) -> Result<(Option, bool), BackendError> { let mut client = BoundaryClient::new(descriptor, bearer); client.transport = transport; + client.attach_request_timeout = attach_request_timeout; client.discover_policy().await } @@ -371,6 +371,7 @@ impl OpenShellRuntimeBackend { ca_file_paths, provider_credentials, sandbox_bearer, + attach_request_timeout: ATTACH_REQUEST_TIMEOUT, delegated: None, resume: None, takeover: None, @@ -411,6 +412,7 @@ impl OpenShellRuntimeBackend { descriptor: SandboxRuntimeDescriptor, driver_descriptor: Vec, bearer: openshell_core::jwt::SessionBearerTokenSlot, + attach_request_timeout: Duration, ) -> Result<(Option, bool), BackendError> { Self::discover_registered_policy_with_resume( backend_name, @@ -418,6 +420,7 @@ impl OpenShellRuntimeBackend { driver_descriptor, bearer, None, + attach_request_timeout, ) .await } @@ -429,15 +432,24 @@ impl OpenShellRuntimeBackend { driver_descriptor: Vec, bearer: openshell_core::jwt::SessionBearerTokenSlot, resume: Option, + attach_request_timeout: Duration, ) -> Result<(Option, bool), BackendError> { crate::delegated::validate_backend_name(&backend_name)?; validate_runtime_coordinates(&descriptor)?; let mut client = BoundaryClient::with_backend(descriptor, bearer, backend_name, driver_descriptor); client.resume = resume; + client.attach_request_timeout = attach_request_timeout; client.discover_policy().await } + /// Set the initial attachment and image-policy discovery budget. + #[must_use] + pub fn with_attach_request_timeout(mut self, timeout: Duration) -> Self { + self.attach_request_timeout = timeout; + self + } + /// Use the transport selected for the descriptor this backend will attach. #[must_use] pub fn with_transport(mut self, transport: BoundaryTransport) -> Self { @@ -505,6 +517,7 @@ impl OpenShellRuntimeBackend { descriptor.payload.clone(), ); client.transport = self.transport.clone(); + client.attach_request_timeout = self.attach_request_timeout; client.resume.clone_from(&self.resume); if let Some(takeover) = &self.takeover { client.supervisor_instance_id = takeover.supervisor_instance_id; @@ -1584,6 +1597,7 @@ struct BoundaryClient { backend_supports_suspend: AtomicBool, supervisor_instance_id: crate::boundary_protocol::SupervisorInstanceId, sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, + attach_request_timeout: Duration, grpc_channel: tokio::sync::Mutex>, mediation: tokio::sync::Mutex>>, mediation_open: tokio::sync::Mutex<()>, @@ -1787,6 +1801,7 @@ impl BoundaryClient { backend_supports_suspend: AtomicBool::new(false), supervisor_instance_id: crate::boundary_protocol::SupervisorInstanceId::new(), sandbox_bearer, + attach_request_timeout: ATTACH_REQUEST_TIMEOUT, grpc_channel: tokio::sync::Mutex::new(None), mediation: tokio::sync::Mutex::new(None), mediation_open: tokio::sync::Mutex::new(()), @@ -1871,7 +1886,7 @@ impl BoundaryClient { let remember_attach = matches!(request, Request::Attach { .. }); let remember_confirm = matches!(request, Request::Confirm); let timeout = if remember_attach || matches!(request, Request::DiscoverPolicy) { - ATTACH_REQUEST_TIMEOUT + self.attach_request_timeout } else { REQUEST_TIMEOUT }; @@ -2457,6 +2472,7 @@ impl BoundaryClient { ) -> Result<(tonic::transport::Channel, Arc), BackendError> { let runtime_descriptor = self.runtime_descriptor.clone(); let source = self.transport.clone(); + let initial_connect_timeout = self.attach_request_timeout; let transport = Arc::new(TransportAbort::default()); let connector_transport = transport.clone(); let authority = match &runtime_descriptor.transport { @@ -2490,7 +2506,7 @@ impl BoundaryClient { return Err(AbortableTransport::aborted_error()); } source - .connect(&runtime_descriptor) + .connect(&runtime_descriptor, initial_connect_timeout) .await .map(|inner| TokioIo::new(AbortableTransport { inner, abort })) .map_err(|error| std::io::Error::other(error.to_string())) @@ -2699,13 +2715,14 @@ async fn connect_boundary_using_connector( authority, addresses, } => { - let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addresses) - .await - .map_err(|error| { - BackendError::Unavailable(format!( - "connect to boundary TLS endpoint {authority}: {error}" - )) - })?; + let stream = + openshell_core::net::connect_tcp_nodelay_best_effort(addresses.as_slice()) + .await + .map_err(|error| { + BackendError::Unavailable(format!( + "connect to boundary TLS endpoint {authority}: {error}" + )) + })?; enable_boundary_tcp_keepalive(&stream); Box::new(stream) } @@ -3044,6 +3061,7 @@ mod tests { use crate::isolation_protocol::exchange_frame; use rustls::pki_types::PrivateKeyDer; + use std::future::{Future as _, poll_fn}; use std::path::PathBuf; use std::pin::Pin; use std::task::{Context, Poll}; @@ -3493,6 +3511,7 @@ mod tests { tls_runtime_descriptor(address, client_tls), test_bearer(&"a".repeat(32)), BoundaryTransport::default(), + ATTACH_REQUEST_TIMEOUT, ) .await; assert!( @@ -3536,6 +3555,7 @@ mod tests { tls_runtime_descriptor(address, client_tls), test_bearer(&"a".repeat(32)), BoundaryTransport::default(), + ATTACH_REQUEST_TIMEOUT, ) .await; assert!( @@ -3559,7 +3579,9 @@ mod tests { async fn boundary_tcp_connections_enable_keepalive() { let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let address = listener.local_addr().unwrap(); - let connected = tokio::spawn(async move { tokio::net::TcpStream::connect(address).await }); + let connected = tokio::spawn(async move { + openshell_core::net::connect_tcp_nodelay_best_effort(address).await + }); let (_server, _) = listener.accept().await.unwrap(); let client = connected.await.unwrap().unwrap(); @@ -4160,6 +4182,125 @@ mod tests { server_config: Arc, } + #[tokio::test(start_paused = true)] + async fn configured_initial_timeout_bounds_attach_and_discovery() { + for attach in [false, true] { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let descriptor = tls_runtime_descriptor( + listener.local_addr().unwrap(), + test_certificate().client_tls, + ); + let mut client = BoundaryClient::new(descriptor, test_bearer(&"a".repeat(32))); + client.transport = + BoundaryTransport::connector(Arc::new(ReverseTestConnector(listener))); + client.attach_request_timeout = Duration::from_secs(2); + let request = if attach { + Request::Attach { + supervisor_instance_id: client.supervisor_instance_id, + policy: Box::new(SandboxPolicyWire::from(sandbox().policy)), + supervisor_takeover: None, + supervisor_recovery_timeout_secs: None, + } + } else { + Request::DiscoverPolicy + }; + let mut waiting = Box::pin(client.call_idempotent(request)); + poll_fn(|context| { + assert!(waiting.as_mut().poll(context).is_pending()); + Poll::Ready(()) + }) + .await; + tokio::time::advance(Duration::from_secs(2)).await; + let result = poll_fn(|context| { + let Poll::Ready(result) = waiting.as_mut().poll(context) else { + panic!("initial request must finish at its configured two-second deadline"); + }; + Poll::Ready(result) + }) + .await; + assert!(matches!(result, Err(BackendError::Unavailable(message)) + if message.contains("idempotent control request timed out"))); + } + } + + #[tokio::test(start_paused = true)] + async fn configured_routed_bootstrap_wait_preserves_reconnect_window() { + let certificate = test_certificate(); + let mut descriptor = + tls_runtime_descriptor("127.0.0.1:1".parse().unwrap(), certificate.client_tls); + let router = + crate::runtime_routed::RuntimeRouter::bind("127.0.0.1:0".parse().unwrap()).unwrap(); + let route = crate::runtime_routed::new_route_handle(); + descriptor.transport = SandboxTransport::RuntimeInitiatedRouted { + authority: "sandbox.test".to_string(), + route: route.clone(), + }; + let registration = router.register(&route, &descriptor.tls).unwrap(); + let transport = BoundaryTransport::routed(&descriptor, registration.clone()).unwrap(); + let budget = Duration::from_mins(15); + let mut waiting = Box::pin(transport.connect(&descriptor, budget)); + poll_fn(|context| { + assert!(waiting.as_mut().poll(context).is_pending()); + Poll::Ready(()) + }) + .await; + tokio::time::advance(Duration::from_mins(6)).await; + poll_fn(|context| { + assert!( + waiting.as_mut().poll(context).is_pending(), + "configured bootstrap survives the old five-minute ceiling" + ); + Poll::Ready(()) + }) + .await; + // Real socket/TLS progress uses a normal clock; virtual time is only + // needed to cross the long bootstrap and recovery deadlines. + tokio::time::resume(); + let address = router.local_addr(); + let runtime = tokio::spawn(async move { + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) + .await + .unwrap(); + stream + .write_all(&crate::runtime_routed::encode_route_preface(&route).unwrap()) + .await + .unwrap(); + tokio_rustls::TlsAcceptor::from(certificate.server_config) + .accept(stream) + .await + .unwrap() + }); + let stream = tokio::time::timeout(Duration::from_secs(5), &mut waiting) + .await + .unwrap() + .unwrap(); + drop(waiting); + assert!(registration.has_connected()); + drop(stream); + drop(runtime.await.unwrap()); + + tokio::time::pause(); + let mut reconnect = Box::pin(transport.connect(&descriptor, budget)); + poll_fn(|context| { + assert!(reconnect.as_mut().poll(context).is_pending()); + Poll::Ready(()) + }) + .await; + tokio::time::advance(Duration::from_secs(29)).await; + poll_fn(|context| { + assert!(reconnect.as_mut().poll(context).is_pending()); + Poll::Ready(()) + }) + .await; + tokio::time::advance(Duration::from_secs(1)).await; + // Resuming real I/O can leave the paused clock between Tokio's 1 ms timer ticks. + let result = tokio::time::timeout(Duration::from_millis(1), &mut reconnect) + .await + .expect("routed reconnect must finish within one timer tick after thirty seconds"); + assert!(matches!(result, Err(BackendError::Unavailable(message)) + if message.contains("timed out waiting for the sandbox runtime to connect"))); + } + fn test_certificate() -> TestCertificate { test_certificate_with_protocol_versions(&[&rustls::version::TLS13]) } @@ -4585,7 +4726,9 @@ mod tests { let address = listener.local_addr().unwrap(); let connector = Arc::new(ReverseTestConnector(listener)); let server = tokio::spawn(async move { - let stream = tokio::net::TcpStream::connect(address).await.unwrap(); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) + .await + .unwrap(); let stream = tokio_rustls::TlsAcceptor::from(certificate.server_config) .accept(stream) .await @@ -4619,7 +4762,9 @@ mod tests { let address = listener.local_addr().unwrap(); let connector = ReverseTestConnector(listener); let server = tokio::spawn(async move { - let stream = tokio::net::TcpStream::connect(address).await.unwrap(); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) + .await + .unwrap(); let _ = tokio_rustls::TlsAcceptor::from(server_certificate.server_config) .accept(stream) .await; @@ -4645,7 +4790,7 @@ mod tests { let server_name = rustls::pki_types::ServerName::try_from(certificate.client_tls.server_name.clone()) .expect("server name"); - let stream = tokio::net::TcpStream::connect(address) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) .await .expect("connect to TLS test server"); diff --git a/crates/openshell-sandbox-backend/src/runtime/test_support.rs b/crates/openshell-sandbox-backend/src/runtime/test_support.rs index 79c36e2959..0db379981f 100644 --- a/crates/openshell-sandbox-backend/src/runtime/test_support.rs +++ b/crates/openshell-sandbox-backend/src/runtime/test_support.rs @@ -680,7 +680,7 @@ impl super::BoundaryTransportConnector for Connector { _: &w::SandboxRuntimeDescriptor, ) -> Result { Ok(Box::new( - openshell_core::net::connect_tcp_nodelay_best_effort(&[self.address]) + openshell_core::net::connect_tcp_nodelay_best_effort(self.address) .await .map_err(|e| BackendError::Unavailable(e.to_string()))?, )) diff --git a/crates/openshell-sandbox-backend/src/runtime/tests/delegated.rs b/crates/openshell-sandbox-backend/src/runtime/tests/delegated.rs index 6275eff8bd..53360ea8d5 100644 --- a/crates/openshell-sandbox-backend/src/runtime/tests/delegated.rs +++ b/crates/openshell-sandbox-backend/src/runtime/tests/delegated.rs @@ -59,7 +59,8 @@ async fn external_backend( EXTERNAL_NAME.into(), descriptor.clone(), OPAQUE_PAYLOAD.to_vec(), - bearer.clone() + bearer.clone(), + ATTACH_REQUEST_TIMEOUT ) .await .unwrap(), @@ -168,7 +169,8 @@ async fn delegated_backend_rejects_reserved_name_and_changed_payload_before_io() crate::BACKEND_NAME.into(), descriptor, OPAQUE_PAYLOAD.to_vec(), - test_bearer(&"a".repeat(32)) + test_bearer(&"a".repeat(32)), + ATTACH_REQUEST_TIMEOUT ) .await, Err(BackendError::Descriptor(_)) diff --git a/crates/openshell-sandbox-backend/src/runtime_routed.rs b/crates/openshell-sandbox-backend/src/runtime_routed.rs index 41d75f2a79..6a9822ae04 100644 --- a/crates/openshell-sandbox-backend/src/runtime_routed.rs +++ b/crates/openshell-sandbox-backend/src/runtime_routed.rs @@ -590,7 +590,7 @@ mod tests { route: &str, identity: &SandboxTlsMaterial, ) -> std::io::Result> { - let mut stream = tokio::net::TcpStream::connect(router).await?; + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(router).await?; stream .write_all(&encode_route_preface(route).unwrap()) .await?; @@ -599,7 +599,9 @@ mod tests { /// A redirect-aware runtime: negotiate `route` with the router. async fn negotiate(router: SocketAddr, route: &str) -> (tokio::net::TcpStream, RouteReply) { - let mut stream = tokio::net::TcpStream::connect(router).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(router) + .await + .unwrap(); let reply = negotiate_route(&mut stream, route).await.unwrap(); (stream, reply) } @@ -756,7 +758,9 @@ mod tests { ); // A version 1 runtime cannot follow a redirect and gets no reply. - let mut stream = tokio::net::TcpStream::connect(address).await.unwrap(); + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) + .await + .unwrap(); stream .write_all(&encode_route_preface(&elsewhere).unwrap()) .await diff --git a/crates/openshell-sandbox/src/bin/seccomp-perf.rs b/crates/openshell-sandbox/src/bin/seccomp-perf.rs index 4114d9a7a9..09bda3449f 100644 --- a/crates/openshell-sandbox/src/bin/seccomp-perf.rs +++ b/crates/openshell-sandbox/src/bin/seccomp-perf.rs @@ -3,6 +3,10 @@ //! Microbenchmark entry point for the production seccomp network broker. +// Crate-scoped: Cargo rejects mixing `[lints] workspace = true` with a +// crate-specific lint table (see #5188). +#![warn(clippy::undocumented_unsafe_blocks)] + #[cfg(target_os = "linux")] use std::net::SocketAddr; diff --git a/crates/openshell-sandbox/src/boundary_io.rs b/crates/openshell-sandbox/src/boundary_io.rs index 7d0e65f7b0..39ebc7a219 100644 --- a/crates/openshell-sandbox/src/boundary_io.rs +++ b/crates/openshell-sandbox/src/boundary_io.rs @@ -457,7 +457,7 @@ impl BoundaryLoopbackConnector for LocalLoopbackConnector { runtime.ensure_active()?; } let addr = std::net::SocketAddr::new(target.host(), target.port()); - let stream = openshell_core::net::connect_tcp_nodelay_best_effort(&[addr]) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) .await .map_err(|e| BackendError::Process(format!("port-forward connect to {addr}: {e}")))?; if let Some(runtime) = &self.runtime { diff --git a/crates/openshell-sandbox/src/boundary_server.rs b/crates/openshell-sandbox/src/boundary_server.rs index a01f3b797e..07df331e07 100644 --- a/crates/openshell-sandbox/src/boundary_server.rs +++ b/crates/openshell-sandbox/src/boundary_server.rs @@ -4884,6 +4884,8 @@ mod linux { .map_err(|_| { io::Error::new(io::ErrorKind::InvalidInput, "sockaddr_vm exceeds socklen_t") })?; + // SAFETY: socket() creates a fresh descriptor and takes only + // scalar arguments; the result is checked below before any use. let raw_fd = unsafe { libc::socket( libc::AF_VSOCK, @@ -4894,6 +4896,8 @@ mod linux { if raw_fd < 0 { return Err(io::Error::last_os_error()); } + // SAFETY: raw_fd was checked non-negative above and is uniquely + // owned; this is its single wrap into an OwnedFd. let fd = unsafe { OwnedFd::from_raw_fd(raw_fd) }; let address = libc::sockaddr_vm { svm_family: family, @@ -4902,6 +4906,8 @@ mod linux { svm_cid: libc::VMADDR_CID_ANY, svm_zero: [0; 4], }; + // SAFETY: bind operates on the live OwnedFd with a fully + // initialized sockaddr_vm of the declared length. let result = unsafe { libc::bind( fd.as_raw_fd(), @@ -4912,6 +4918,7 @@ mod linux { if result < 0 { return Err(io::Error::last_os_error()); } + // SAFETY: listen operates on the live, bound OwnedFd. if unsafe { libc::listen(fd.as_raw_fd(), 16) } < 0 { return Err(io::Error::last_os_error()); } @@ -4935,6 +4942,9 @@ mod linux { listener, server_config, } => { + // SAFETY: accept4 takes the live listener descriptor with + // SOCK_CLOEXEC and a valid null address buffer; the + // returned descriptor is checked below before any use. let raw_fd = unsafe { libc::accept4( listener.as_raw_fd(), @@ -4946,8 +4956,11 @@ mod linux { if raw_fd < 0 { Err(io::Error::last_os_error()) } else { + // SAFETY: raw_fd was checked non-negative above and + // is uniquely owned; this is its single wrap. + let stream = unsafe { File::from_raw_fd(raw_fd) }; Ok(ControlStream::PendingTls { - stream: PlainControlStream::Vsock(unsafe { File::from_raw_fd(raw_fd) }), + stream: PlainControlStream::Vsock(stream), server_config: server_config.clone(), }) } @@ -5142,8 +5155,26 @@ mod linux { ) -> io::Result { match self { Self::Vsock(file) => { - let stream = - unsafe { std::os::unix::net::UnixStream::from_raw_fd(file.into_raw_fd()) }; + let raw_fd = file.into_raw_fd(); + // SAFETY: `into_raw_fd` handed over a valid, open, + // solely-owned descriptor exactly once, so this is its + // single re-wrap. The descriptor is an AF_VSOCK stream + // reinterpreted as a Unix stream; the stream only ever + // sees family-agnostic operations (fcntl, read/write, + // epoll registration via `from_std`, and shutdown), which + // behave the same on AF_VSOCK streams. `peer_addr`, + // `local_addr`, and `peer_cred` must never be called on + // it: they would fail at runtime on a vsock descriptor. + // The surface is fenced by this private enum and by the + // `Box` erasure in `into_tokio`. The + // dial side performs the same reinterpretation in the + // `RuntimeInitiatedVsock` arm of `dial_supervisor`, but + // files its vsock descriptors under the `Unix` variant + // (not `Vsock`) and uses only `take_error()` (a + // SOL_SOCKET getsockopt) and `writable()`, both + // family-agnostic; the allowed-ops list above covers + // this accept-side stream only. + let stream = unsafe { std::os::unix::net::UnixStream::from_raw_fd(raw_fd) }; stream.set_nonblocking(true)?; Ok(Box::new(tokio::net::UnixStream::from_std(stream)?)) } @@ -5333,6 +5364,10 @@ mod linux { } #[cfg(test)] + #[allow( + clippy::disallowed_methods, + reason = "tests dial from the confined workload, where socket options are not under test" + )] mod tests { use super::*; use openshell_core::jwt::{ diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index 7b1f4400c1..3c280cc977 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -3,6 +3,11 @@ //! Capability-free in-workload sandbox boundary. +// Crate-scoped: Cargo rejects mixing `[lints] workspace = true` with a +// crate-specific lint table (see #5188). +#![warn(clippy::undocumented_unsafe_blocks)] +#![cfg_attr(test, allow(clippy::undocumented_unsafe_blocks))] + #[cfg(target_os = "linux")] mod accept_errors; pub mod binary_pins; diff --git a/crates/openshell-sandbox/src/linux/seccomp_notify.rs b/crates/openshell-sandbox/src/linux/seccomp_notify.rs index bc98813b00..665cadcefe 100644 --- a/crates/openshell-sandbox/src/linux/seccomp_notify.rs +++ b/crates/openshell-sandbox/src/linux/seccomp_notify.rs @@ -831,8 +831,6 @@ fn verify_notification_sizes() -> io::Result<()> { } fn ioctl_ptr(fd: RawFd, request: libc::c_ulong, argument: *mut libc::c_void) -> io::Result { - // SAFETY: every caller supplies the UAPI structure encoded into `request`, - // alive and writable for the ioctl duration. // `libc::ioctl` models the request as `c_ulong` for glibc and `c_int` // for musl. Linux UAPI request values fit both representations. #[cfg(target_env = "musl")] @@ -841,6 +839,8 @@ fn ioctl_ptr(fd: RawFd, request: libc::c_ulong, argument: *mut libc::c_void) -> })?; #[cfg(target_env = "musl")] let request = libc::c_int::from_ne_bytes(request.to_ne_bytes()); + // SAFETY: every caller supplies the UAPI structure encoded into `request`, + // alive and writable for the ioctl duration. let result = unsafe { libc::ioctl(fd, request, argument) }; if result < 0 { Err(io::Error::last_os_error()) diff --git a/crates/openshell-sandbox/src/linux/socket_confinement.rs b/crates/openshell-sandbox/src/linux/socket_confinement.rs index f1872bc44c..e5d182b69a 100644 --- a/crates/openshell-sandbox/src/linux/socket_confinement.rs +++ b/crates/openshell-sandbox/src/linux/socket_confinement.rs @@ -106,6 +106,7 @@ fn probe_accept_inherits_binding() -> io::Result<()> { }; confine_to_loopback(&listener) .map_err(|error| probe_error("confine probe listener", &error))?; + #[allow(clippy::disallowed_methods, reason = "probe stream carries no data")] let _client = std::net::TcpStream::connect(listener.local_addr()?)?; let (accepted, _) = listener.accept()?; if bound_device(&accepted)?.as_deref() != Some(LOOPBACK_DEVICE) { diff --git a/crates/openshell-sandbox/src/main.rs b/crates/openshell-sandbox/src/main.rs index 5830d79dfe..400478a889 100644 --- a/crates/openshell-sandbox/src/main.rs +++ b/crates/openshell-sandbox/src/main.rs @@ -3,6 +3,11 @@ //! `OpenShell` capability-free in-workload sandbox boundary. +// Crate-scoped: Cargo rejects mixing `[lints] workspace = true` with a +// crate-specific lint table (see #5188). +#![warn(clippy::undocumented_unsafe_blocks)] +#![cfg_attr(test, allow(clippy::undocumented_unsafe_blocks))] + #[cfg(target_os = "linux")] use std::mem::size_of; use std::path::Path; @@ -685,6 +690,10 @@ fn probe_socket_virtualization() -> Result<()> { { return Err(std::io::Error::last_os_error()).into_diagnostic(); } + // SAFETY: the retained source is the exact injected OFD and + // the payload buffer stays live for its declared length; + // MSG_NOSIGNAL keeps a refused send from raising SIGPIPE in + // the dispatcher thread. let sent = unsafe { libc::send( retained.as_raw_fd(), @@ -796,6 +805,15 @@ fn set_child_core_limit() -> std::io::Result<()> { .map_err(std::io::Error::from) } +/// Reset every catchable signal disposition in the calling process to its +/// default. +/// +/// # Safety +/// +/// Destroys all installed handlers for signals 1 through 64 (skipping the +/// uncatchable SIGKILL and SIGSTOP) in the calling process. Must only run +/// in the forked pre-exec child, never in the launcher parent, which still +/// needs its own dispositions. #[cfg(target_os = "linux")] #[allow(unsafe_code)] unsafe fn reset_child_signal_dispositions() -> std::io::Result<()> { @@ -1120,6 +1138,10 @@ fn probe_tcp_dns_socket_round_trip() -> Result<()> { const DNS_QUERY: &[u8] = b"\x56\x78\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x05probe\x09openshell\x04test\x00\x00\x01\x00\x01"; + #[allow( + clippy::disallowed_methods, + reason = "the probe checks that setsockopt succeeds through the mediated path" + )] let mut stream = std::net::TcpStream::connect("127.0.0.53:53").into_diagnostic()?; stream.set_nodelay(true).into_diagnostic()?; stream @@ -1217,43 +1239,44 @@ fn probe_child_self_protection(sandbox_tgid: libc::pid_t, socket: libc::c_int) - ) }; require_probe_errno(read, libc::EPERM, "process_vm_readv sandbox")?; - // SAFETY: signal zero would only probe process existence if the filter did - // not reject the trusted sandbox target. require_probe_errno( + // SAFETY: signal zero would only probe process existence if the + // filter did not reject the trusted sandbox target. isize::try_from(unsafe { libc::kill(sandbox_tgid, 0) }).expect("kill result fits isize"), libc::EPERM, "kill sandbox", )?; // SAFETY: scalar syscall arguments request a read-only resource query; // the child filter rejects non-self targets. + let prlimit_result = isize::try_from(unsafe { + libc::syscall(libc::SYS_prlimit64, sandbox_tgid, libc::RLIMIT_CORE, 0, 0) + }) + .expect("prlimit result fits isize"); + require_probe_errno(prlimit_result, libc::EPERM, "prlimit sandbox")?; + // SAFETY: signal zero on the process group would only probe group + // existence if the filter did not reject the trusted sandbox target. + let group_kill_result = unsafe { libc::kill(-sandbox_tgid, 0) }; require_probe_errno( - isize::try_from(unsafe { - libc::syscall(libc::SYS_prlimit64, sandbox_tgid, libc::RLIMIT_CORE, 0, 0) - }) - .expect("prlimit result fits isize"), - libc::EPERM, - "prlimit sandbox", - )?; - require_probe_errno( - isize::try_from(unsafe { libc::kill(-sandbox_tgid, 0) }).expect("kill result fits isize"), + isize::try_from(group_kill_result).expect("kill result fits isize"), libc::EPERM, "process-group signal", )?; + // SAFETY: F_SETOWN would only record the sandbox target as the owner of + // the child's own socket if the filter did not reject it first. + let fcntl_result = unsafe { libc::fcntl(socket, libc::F_SETOWN, sandbox_tgid) }; require_probe_errno( - isize::try_from(unsafe { libc::fcntl(socket, libc::F_SETOWN, sandbox_tgid) }) - .expect("fcntl result fits isize"), + isize::try_from(fcntl_result).expect("fcntl result fits isize"), libc::EPERM, "fcntl F_SETOWN", )?; let mut owner = sandbox_tgid; - require_probe_errno( - isize::try_from(unsafe { - libc::syscall(libc::SYS_ioctl, socket, 0x8901_u32, &raw mut owner) - }) - .expect("ioctl result fits isize"), - libc::EPERM, - "ioctl FIOSETOWN", - )?; + // SAFETY: FIOSETOWN would only record the sandbox target as the owner of + // the child's own socket if the filter did not reject it first. + let ioctl_result = isize::try_from(unsafe { + libc::syscall(libc::SYS_ioctl, socket, 0x8901_u32, &raw mut owner) + }) + .expect("ioctl result fits isize"); + require_probe_errno(ioctl_result, libc::EPERM, "ioctl FIOSETOWN")?; Ok(()) } @@ -2039,4 +2062,149 @@ mod tests { let final_path = dest_dir.join("openshell-sandbox"); assert!(final_path.exists(), "binary should land inside dest dir"); } + + /// Environment variable marking the re-executed probe child process. + const PROBE_CHILD_ENV: &str = "OPENSHELL_SANDBOX_PROBE_CHILD"; + + /// Re-executes `test_path` (the full libtest path, module prefix + /// included) in a fresh process so it can change its rlimits and load + /// seccomp filters without contaminating the test harness. The child + /// asserts on its own probe results; this parent only requires a + /// successful exit. A fresh executable process, unlike a raw `fork` child + /// of the multithreaded test harness, may allocate freely after the + /// filter is installed. + fn run_isolated_probe_child(test_path: &str) { + let output = std::process::Command::new(std::env::current_exe().expect("test executable")) + .args(["--exact", test_path, "--nocapture"]) + .env(PROBE_CHILD_ENV, "1") + .output() + .expect("start isolated probe child process"); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!( + output.status.success(), + "isolated probe child failed:\nstdout: {stdout}\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + // `--exact` matches full test paths; a wrong name would silently + // select zero tests and exit successfully. + assert!( + stdout.contains("1 passed"), + "isolated probe child matched no test:\nstdout: {stdout}" + ); + } + + /// Prepares a probe child: zeroes the core-dump limit the probe requires + /// and allows the unprivileged seccomp install. + #[allow(unsafe_code)] + fn prepare_probe_child() -> std::io::Result<()> { + nix::sys::resource::setrlimit(nix::sys::resource::Resource::RLIMIT_CORE, 0, 0) + .map_err(std::io::Error::from)?; + // SAFETY: PR_SET_NO_NEW_PRIVS is a one-way scalar transition required + // before loading an unprivileged seccomp filter. + if unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) } < 0 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) + } + + /// The production child filter rejects every self-protection probe with + /// EPERM. Running `probe_child_self_protection` against that exact filter + /// keeps each hoisted probe result wired to its own errno check. The test + /// re-executes itself in a fresh process so the rlimit and filter changes + /// never touch the test harness process. + #[test] + fn probe_child_self_protection_accepts_the_production_child_filter() { + if std::env::var_os(PROBE_CHILD_ENV).is_none() { + run_isolated_probe_child( + "tests::probe_child_self_protection_accepts_the_production_child_filter", + ); + return; + } + prepare_probe_child().expect("prepare probe child"); + let tgid = std::process::id(); + let sandbox_tgid = libc::pid_t::try_from(tgid).expect("test pid fits pid_t"); + let mut program = openshell_sandbox::linux::child_seccomp::prepare(tgid) + .expect("prepare production child filter"); + program.install().expect("install production child filter"); + probe_child_self_protection(sandbox_tgid, -1) + .expect("probes pass under the production child filter"); + } + + /// Denies every probe except the FIOSETOWN ioctl with EPERM; the kernel + /// then rejects the ioctl's invalid descriptor with EBADF. The + /// distinct errno only satisfies the check that belongs to the ioctl probe + /// when every check runs directly after its own syscall. + fn probe_adjacency_filter() -> seccompiler::BpfProgram { + use seccompiler::{ + SeccompAction, SeccompCmpArgLen, SeccompCmpOp, SeccompCondition, SeccompFilter, + SeccompRule, + }; + use std::collections::BTreeMap; + + let mut rules: BTreeMap> = BTreeMap::new(); + rules.insert(libc::SYS_process_vm_readv, Vec::new()); + rules.insert(libc::SYS_kill, Vec::new()); + rules.insert( + libc::SYS_prlimit64, + vec![ + SeccompRule::new(vec![ + SeccompCondition::new(0, SeccompCmpArgLen::Dword, SeccompCmpOp::Ne, 0) + .expect("valid prlimit64 argument condition"), + ]) + .expect("valid prlimit64 rule"), + ], + ); + rules.insert( + libc::SYS_fcntl, + vec![ + SeccompRule::new(vec![ + SeccompCondition::new( + 1, + SeccompCmpArgLen::Dword, + SeccompCmpOp::Eq, + u64::from(libc::F_SETOWN.cast_unsigned()), + ) + .expect("valid fcntl command condition"), + ]) + .expect("valid fcntl rule"), + ], + ); + let architecture = std::env::consts::ARCH + .try_into() + .expect("supported target architecture"); + SeccompFilter::new( + rules, + SeccompAction::Allow, + SeccompAction::Errno(libc::EPERM.cast_unsigned()), + architecture, + ) + .expect("build probe filter") + .try_into() + .expect("compile probe filter") + } + + #[test] + fn probe_child_self_protection_checks_each_syscall_errno_independently() { + if std::env::var_os(PROBE_CHILD_ENV).is_none() { + run_isolated_probe_child( + "tests::probe_child_self_protection_checks_each_syscall_errno_independently", + ); + return; + } + prepare_probe_child().expect("prepare probe child"); + let filter = probe_adjacency_filter(); + seccompiler::apply_filter(&filter).expect("install adjacency filter"); + let sandbox_tgid = libc::pid_t::try_from(std::process::id()).expect("test pid fits pid_t"); + // The invalid descriptor is enough: the fcntl probe is rejected by + // the filter before the kernel validates it, and the unfiltered + // ioctl probe reports EBADF from the kernel. + let error = probe_child_self_protection(sandbox_tgid, -1) + .expect_err("adjacency filter must let the ioctl probe through"); + assert!( + error + .to_string() + .contains("ioctl FIOSETOWN was not rejected with errno"), + "unexpected probe failure: {error}" + ); + } } diff --git a/crates/openshell-sandbox/src/network_broker.rs b/crates/openshell-sandbox/src/network_broker.rs index 4d84c8734c..323f1ac715 100644 --- a/crates/openshell-sandbox/src/network_broker.rs +++ b/crates/openshell-sandbox/src/network_broker.rs @@ -1691,10 +1691,12 @@ fn connect_exact(fd: RawFd, address: SocketAddr) -> io::Result<()> { return Err(io::Error::last_os_error()); } let changed_flags = original_flags & libc::O_NONBLOCK == 0; - if changed_flags - && unsafe { libc::fcntl(fd, libc::F_SETFL, original_flags | libc::O_NONBLOCK) } < 0 - { - return Err(io::Error::last_os_error()); + if changed_flags { + // SAFETY: F_SETFL operates on the live retained socket descriptor + // with the flag word read by the documented F_GETFL above. + if unsafe { libc::fcntl(fd, libc::F_SETFL, original_flags | libc::O_NONBLOCK) } < 0 { + return Err(io::Error::last_os_error()); + } } let result = with_sockaddr(address, |pointer, length| { // SAFETY: pointer/length describe a live native sockaddr and `fd` is @@ -1712,8 +1714,8 @@ fn connect_exact(fd: RawFd, address: SocketAddr) -> io::Result<()> { events: libc::POLLOUT, revents: 0, }; - // SAFETY: poll points to one live pollfd. let timeout = i32::try_from(RELAY_CONNECT_TIMEOUT.as_millis()).map_err(io::Error::other)?; + // SAFETY: poll points to one live pollfd. if unsafe { libc::poll(&raw mut poll, 1, timeout) } <= 0 { return Err(io::Error::from_raw_os_error(libc::ETIMEDOUT)); } @@ -1738,9 +1740,16 @@ fn connect_exact(fd: RawFd, address: SocketAddr) -> io::Result<()> { Err(io::Error::from_raw_os_error(socket_error)) } }); - let restore = if changed_flags && unsafe { libc::fcntl(fd, libc::F_SETFL, original_flags) } < 0 - { - Err(io::Error::last_os_error()) + let restore = if changed_flags { + // SAFETY: F_SETFL restores the original flag word read by the + // documented F_GETFL; the restore runs even when the connect + // attempt failed. + let restore_result = unsafe { libc::fcntl(fd, libc::F_SETFL, original_flags) }; + if restore_result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } } else { Ok(()) }; @@ -1882,6 +1891,10 @@ fn error_to_errno(error: &io::Error) -> i32 { } #[cfg(test)] +#[allow( + clippy::disallowed_methods, + reason = "tests dial from the confined workload, where socket options are not under test" +)] mod tests { use super::*; use crate::linux::socket_confinement; @@ -2483,6 +2496,168 @@ mod tests { ); } + /// Runs `child` in a forked process and returns the bytes it writes to a + /// pipe. Assertions belong in the parent: a panic in the child would + /// unwind into a copy of the test harness. + #[allow(unsafe_code)] + fn fork_capture(child: impl FnOnce() -> Vec) -> Vec { + let (read_fd, write_fd) = nix::unistd::pipe().expect("create probe pipe"); + // SAFETY: fork isolates the rlimit and seccomp changes the callers + // make. The child writes one result and exits without running + // inherited destructors or the test harness. + match unsafe { nix::unistd::fork() }.expect("fork should succeed") { + nix::unistd::ForkResult::Child => { + drop(read_fd); + let result = child(); + let _ = rustix::io::write(&write_fd, &result); + drop(write_fd); + // SAFETY: terminate the fork child without unwinding through + // the parent's test harness state. + unsafe { libc::_exit(0) }; + } + nix::unistd::ForkResult::Parent { child } => { + drop(write_fd); + let mut result = Vec::new(); + let mut buffer = [0_u8; 512]; + loop { + match rustix::io::read(&read_fd, &mut buffer) { + Ok(0) => break, + Ok(read) => result.extend_from_slice(&buffer[..read]), + Err(error) if error == rustix::io::Errno::INTR => {} + Err(error) => panic!("read probe result: {error}"), + } + } + drop(read_fd); + match nix::sys::wait::waitpid(child, None).expect("waitpid should succeed") { + nix::sys::wait::WaitStatus::Exited(_, 0) => {} + status => panic!("probe child exited unexpectedly: {status:?}"), + } + result + } + } + } + + #[test] + fn connect_exact_returns_a_failed_nonblocking_transition() { + // O_PATH descriptors support F_GETFL but reject F_SETFL with EBADF. + // That reaches the set-failure return without invalidating a + // descriptor that is also being connected. + let path = std::ffi::CString::new("/").expect("static path is NUL-free"); + // SAFETY: path is a live NUL-terminated directory path; a successful + // open returns one newly owned descriptor. + let descriptor = unsafe { libc::open(path.as_ptr(), libc::O_PATH | libc::O_CLOEXEC) }; + assert!( + descriptor >= 0, + "open O_PATH: {}", + io::Error::last_os_error() + ); + // SAFETY: the successful open transferred ownership of the descriptor. + let descriptor = unsafe { OwnedFd::from_raw_fd(descriptor) }; + + let error = connect_exact(descriptor.as_raw_fd(), "127.0.0.1:9".parse().unwrap()) + .expect_err("an O_PATH descriptor rejects F_SETFL"); + assert_eq!(error.raw_os_error(), Some(libc::EBADF)); + } + + /// Denies only the `F_SETFL` restore that `connect_exact` performs after a + /// bounded connect: the initial transition carries `O_NONBLOCK`, the restore + /// does not. The connect itself still succeeds, so the restore error is + /// the one the caller observes. + fn flag_restore_failure_filter() -> seccompiler::BpfProgram { + use seccompiler::{ + SeccompAction, SeccompCmpArgLen, SeccompCmpOp, SeccompCondition, SeccompFilter, + SeccompRule, + }; + use std::collections::BTreeMap; + + let mut rules: BTreeMap> = BTreeMap::new(); + rules.insert( + libc::SYS_fcntl, + vec![ + SeccompRule::new(vec![ + SeccompCondition::new( + 1, + SeccompCmpArgLen::Dword, + SeccompCmpOp::Eq, + u64::from(libc::F_SETFL.cast_unsigned()), + ) + .expect("valid fcntl command condition"), + SeccompCondition::new( + 2, + SeccompCmpArgLen::Dword, + SeccompCmpOp::MaskedEq(u64::from(libc::O_NONBLOCK.cast_unsigned())), + 0, + ) + .expect("valid fcntl flag condition"), + ]) + .expect("valid fcntl rule"), + ], + ); + let architecture = std::env::consts::ARCH + .try_into() + .expect("supported target architecture"); + SeccompFilter::new( + rules, + SeccompAction::Allow, + SeccompAction::Errno(libc::EPERM.cast_unsigned()), + architecture, + ) + .expect("build restore filter") + .try_into() + .expect("compile restore filter") + } + + #[test] + fn connect_exact_returns_the_flag_restore_failure() { + // Nothing can invalidate the retained descriptor between the two fcntl + // calls, so a filter that denies only the restore transition is the + // way to reach the restore-failure arm. + let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); + let address = listener.local_addr().expect("listener address"); + let filter = flag_restore_failure_filter(); + let result = fork_capture(move || { + // SAFETY: PR_SET_NO_NEW_PRIVS is a one-way scalar transition + // required before loading an unprivileged seccomp filter. + if unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) } < 0 { + return i32::MIN.to_ne_bytes().to_vec(); + } + if seccompiler::apply_filter(&filter).is_err() { + return i32::MIN.to_ne_bytes().to_vec(); + } + // SAFETY: socket returns one newly owned descriptor on success. + let descriptor = unsafe { + libc::socket( + libc::AF_INET, + libc::SOCK_STREAM | libc::SOCK_CLOEXEC, + libc::IPPROTO_TCP, + ) + }; + if descriptor < 0 { + return i32::MIN.to_ne_bytes().to_vec(); + } + // SAFETY: the successful socket call returned one owned descriptor. + let descriptor = unsafe { OwnedFd::from_raw_fd(descriptor) }; + match connect_exact(descriptor.as_raw_fd(), address) { + Ok(()) => 0_i32.to_ne_bytes().to_vec(), + Err(error) => error + .raw_os_error() + .unwrap_or(i32::MAX) + .to_ne_bytes() + .to_vec(), + } + }); + let errno = i32::from_ne_bytes( + result + .try_into() + .expect("probe child writes one i32 status"), + ); + assert_eq!( + errno, + libc::EPERM, + "restore failure must be returned to the caller" + ); + } + #[test] fn dns_admissions_reclaim_closed_sockets_at_the_bound() { let admissions = Mutex::new(HashMap::new()); diff --git a/crates/openshell-sandbox/src/process.rs b/crates/openshell-sandbox/src/process.rs index a8995bc7da..4967d20086 100644 --- a/crates/openshell-sandbox/src/process.rs +++ b/crates/openshell-sandbox/src/process.rs @@ -31,6 +31,9 @@ use tracing::debug; #[cfg(unix)] #[allow(unsafe_code, clippy::useless_conversion)] fn set_controlling_tty(fd: libc::c_int) -> std::io::Result<()> { + // SAFETY: fd is the slave PTY of a pre-exec child that already ran + // setsid; argument zero requests ownership without stealing the + // terminal from another session. if unsafe { libc::ioctl(fd, libc::TIOCSCTTY.into(), 0) } < 0 { return Err(std::io::Error::last_os_error()); } @@ -626,6 +629,9 @@ impl ProcessHandle { #[cfg(target_os = "linux")] let mut prepared_sandbox = prepared_sandbox; #[allow(unsafe_code)] + // SAFETY: pre_exec only registers the hook; the closure runs + // post-fork and pre-exec in the child process, never in the + // launcher parent. unsafe { cmd.pre_exec(move || { if let Some(slave_fd) = terminal_slave_fd { diff --git a/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs index d180920325..2dbdb53428 100644 --- a/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs @@ -126,6 +126,9 @@ fn build_supervisor_prelude_rules() -> BTreeMap> { fn set_no_new_privs() -> Result<()> { // libc/syscall FFI requires unsafe #[allow(unsafe_code)] + // SAFETY: prctl receives only scalar arguments; PR_SET_NO_NEW_PRIVS is + // required before an unprivileged seccomp filter load and is + // irreversible. let rc = unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; if rc != 0 { return Err(miette::miette!( diff --git a/crates/openshell-sdk/src/client.rs b/crates/openshell-sdk/src/client.rs index 1f03916ef2..8241ac49b5 100644 --- a/crates/openshell-sdk/src/client.rs +++ b/crates/openshell-sdk/src/client.rs @@ -89,6 +89,16 @@ impl OpenShellClient { self.channel.clone() } + /// Clone of the auth interceptor backing [`OpenShellClient::raw_grpc`]. + /// + /// tonic allows a single interceptor per channel, so callers that need + /// to stack another concern (e.g. trace-context forwarding) on top of + /// authentication can compose around this clone — it shares the same + /// live bearer slot, so OIDC refresh still reaches the composed stack. + pub fn auth_interceptor(&self) -> EdgeAuthInterceptor { + self.interceptor.clone() + } + /// Authenticated gRPC client for the main `OpenShell` service. /// /// Use this when the curated surface below doesn't expose the RPC or @@ -1770,6 +1780,31 @@ mod tests { ); } + #[tokio::test] + async fn auth_interceptor_clone_sees_refreshed_token() { + use tonic::service::Interceptor as _; + + // The clone handed out by `auth_interceptor()` must share the live + // bearer slot: callers compose other concerns (e.g. the CLI's + // trace-forwarding gateway interceptor) around it, and a forced OIDC + // refresh must reach those already-built stacks. + let calls = Arc::new(AtomicUsize::new(0)); + let client = oidc_client_with_refresher(Arc::clone(&calls)); + let mut interceptor = client.auth_interceptor(); + + assert!(client.force_refresh().await.unwrap()); + + let request = interceptor.call(tonic::Request::new(())).unwrap(); + assert_eq!( + request + .metadata() + .get("authorization") + .and_then(|value| value.to_str().ok()), + Some("Bearer token-1"), + "refreshed token must reach the cloned interceptor" + ); + } + #[test] fn store_bearer_rejects_malformed_token_and_keeps_previous() { let slot: BearerSlot = Arc::new(RwLock::new(None)); diff --git a/crates/openshell-sdk/src/transport.rs b/crates/openshell-sdk/src/transport.rs index 616f5099bb..eb2dd87b06 100644 --- a/crates/openshell-sdk/src/transport.rs +++ b/crates/openshell-sdk/src/transport.rs @@ -6,7 +6,6 @@ use crate::config::{AuthConfig, ClientConfig}; use crate::edge_tunnel; use crate::error::{Result, SdkError}; -use openshell_core::net::set_tcp_nodelay_best_effort; use rustls::{ client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier}, pki_types::{CertificateDer, ServerName, UnixTime}, @@ -235,8 +234,7 @@ impl tower::Service for InsecureTlsConnector { let host = uri.host().unwrap_or("localhost").to_string(); let port = uri.port_u16().unwrap_or(443); let addr = format!("{host}:{port}"); - let tcp = tokio::net::TcpStream::connect(addr).await?; - set_tcp_nodelay_best_effort(&tcp); + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(addr).await?; let server_name = ServerName::try_from(host)?; let tls_stream = tls_connector.connect(server_name, tcp).await?; Ok(hyper_util::rt::TokioIo::new(tls_stream)) diff --git a/crates/openshell-server/src/auth/workspace_authz.rs b/crates/openshell-server/src/auth/workspace_authz.rs index bc5cac8778..035d8aa354 100644 --- a/crates/openshell-server/src/auth/workspace_authz.rs +++ b/crates/openshell-server/src/auth/workspace_authz.rs @@ -156,7 +156,7 @@ pub async fn authorize_workspace( &user.identity.subject, ) .await - .map_err(|e| Status::internal(format!("membership lookup failed: {e}")))?; + .map_err(|e| crate::grpc::persistence_error_to_status(e, "membership lookup"))?; let Some(member) = member else { let workspace_arg = shell_quote_for_hint(&workspace); diff --git a/crates/openshell-server/src/cli.rs b/crates/openshell-server/src/cli.rs index 55ad3091ea..8458bc5dd6 100644 --- a/crates/openshell-server/src/cli.rs +++ b/crates/openshell-server/src/cli.rs @@ -679,6 +679,19 @@ fn prepare_server_config_with_drivers( config = config.with_ssh_session_ttl_secs(ttl); } + if let Some(gateway_file) = file.as_ref().map(|f| &f.openshell.gateway) { + apply_connection_cap( + "max_ssh_connections_per_sandbox", + gateway_file.max_ssh_connections_per_sandbox, + &mut config.max_ssh_connections_per_sandbox, + )?; + apply_connection_cap( + "max_service_connections_per_sandbox", + gateway_file.max_service_connections_per_sandbox, + &mut config.max_service_connections_per_sandbox, + )?; + } + if let Some(mode) = file .as_ref() .and_then(|f| f.openshell.gateway.policy_validation_failure_mode) @@ -705,6 +718,27 @@ fn prepare_server_config_with_drivers( config.image_preparation_timeout_seconds = seconds; } + if let Some(raw) = std::env::var_os("OPENSHELL_PROVISIONING_REPAIR_WINDOW_SECONDS") { + let raw = raw.to_str().ok_or_else(|| { + miette::miette!( + "OPENSHELL_PROVISIONING_REPAIR_WINDOW_SECONDS must be an integer number of seconds" + ) + })?; + let window = raw.parse::().map_err(|_| { + miette::miette!( + "OPENSHELL_PROVISIONING_REPAIR_WINDOW_SECONDS must be an integer number of seconds" + ) + })?; + validate_provisioning_repair_window_seconds(window)?; + config.provisioning_repair_window_seconds = window; + } else if let Some(window) = file + .as_ref() + .and_then(|f| f.openshell.gateway.provisioning_repair_window_seconds) + { + validate_provisioning_repair_window_seconds(window)?; + config.provisioning_repair_window_seconds = window; + } + if let Some(issuer) = args.oidc_issuer.clone() { config = config.with_oidc(openshell_core::OidcConfig { issuer, @@ -770,6 +804,13 @@ fn prepare_server_config_with_drivers( }) } +const UNAUTHENTICATED_USERS_WARNING: &str = "unauthenticated user access is enabled; only use this for trusted local development or a fully trusted fronting proxy"; + +fn unauthenticated_users_allowed(file: Option<&ConfigFile>) -> bool { + file.and_then(|file| file.openshell.gateway.auth.as_ref()) + .is_some_and(|auth| auth.allow_unauthenticated_users) +} + /// Reject an implicit development gateway before starting any listeners or services. fn validate_user_auth(config: &openshell_core::Config) -> Result<()> { if !config.auth.allow_unauthenticated_users @@ -919,9 +960,7 @@ async fn run_from_args( ); } if prepared.config.auth.allow_unauthenticated_users { - warn!( - "Unauthenticated user access enabled — only use this for trusted local development or a fully trusted fronting proxy" - ); + warn!("{UNAUTHENTICATED_USERS_WARNING}"); } info!(bind = %prepared.config.bind_address, "Starting OpenShell server"); @@ -1161,7 +1200,12 @@ fn run_effective_config_preflight( })(); match (validation, path.as_ref()) { - (Ok(driver), _) => Ok(driver), + (Ok(driver), _) => { + if unauthenticated_users_allowed(file.as_ref()) { + eprintln!("warning: {UNAUTHENTICATED_USERS_WARNING}"); + } + Ok(driver) + } (Err(_), Some(path)) => Err(miette::miette!( "{}", config_file::ConfigPreflightError::invalid_current(path) @@ -1624,6 +1668,15 @@ fn merge_file_into_args(args: &mut RunArgs, file: &GatewayFileSection, matches: } } +fn validate_provisioning_repair_window_seconds(window: i64) -> Result<()> { + if !(1..=86_400).contains(&window) { + return Err(miette::miette!( + "provisioning_repair_window_seconds must be between 1 and 86400" + )); + } + Ok(()) +} + fn validate_grpc_rate_limit_args(requests: Option, window_seconds: Option) -> Result<()> { let disabled = matches!(requests, Some(0)) || matches!(window_seconds, Some(0)); if disabled { @@ -1643,6 +1696,17 @@ fn validate_grpc_rate_limit_args(requests: Option, window_seconds: Option, cap: &mut u32) -> Result<()> { + if let Some(value) = value { + if value == 0 { + return Err(miette::miette!("{name} must be at least 1")); + } + *cap = value; + } + Ok(()) +} + fn normalize_compute_driver_socket_args(args: &mut RunArgs) -> Result<()> { let Some(socket) = args.compute_driver_socket.as_ref() else { return Ok(()); @@ -1725,6 +1789,20 @@ mod tests { } } + #[test] + fn preflight_detects_unauthenticated_users_opt_in() { + let parse = + |auth: &str| config_file_from_toml(&format!("[openshell]\nversion = 2\n{auth}")); + assert!(!super::unauthenticated_users_allowed(None)); + assert!(!super::unauthenticated_users_allowed(Some(&parse("")))); + assert!(!super::unauthenticated_users_allowed(Some(&parse( + "[openshell.gateway.auth]\nallow_unauthenticated_users = false\n" + )))); + assert!(super::unauthenticated_users_allowed(Some(&parse( + "[openshell.gateway.auth]\nallow_unauthenticated_users = true\n" + )))); + } + #[test] fn startup_rejects_missing_user_auth_before_starting_services() { for auth in [ @@ -1757,11 +1835,7 @@ mod tests { .build() .unwrap(); let error = runtime - .block_on(super::run_from_args( - args, - matches, - test_registry("test", true), - )) + .block_on(super::run_from_args(args, matches, test_registry("test"))) .unwrap_err() .to_string(); for expected in [ @@ -1917,12 +1991,9 @@ mod tests { } } - fn test_registry(name: &str, singleplayer: bool) -> crate::ComputeDriverRegistry { - let mut registration = + fn test_registry(name: &str) -> crate::ComputeDriverRegistry { + let registration = crate::ComputeDriverRegistration::new(name, 100, None, TestFactory).unwrap(); - if singleplayer { - registration = registration.with_local_singleplayer(); - } let mut registry = crate::ComputeDriverRegistry::new(); registry.install(registration).unwrap(); registry @@ -1935,8 +2006,7 @@ mod tests { Some(detect_registered_local), TestFactory, ) - .unwrap() - .with_local_singleplayer(); + .unwrap(); let mut registry = crate::ComputeDriverRegistry::new(); registry.install(registration).unwrap(); registry @@ -2196,7 +2266,7 @@ mod tests { "sqlite::memory:", "--disable-tls", ]); - let registry = test_registry("podman", true); + let registry = test_registry("podman"); let prepared = super::prepare_server_config_with_drivers(&mut args, &matches, ®istry) @@ -2506,7 +2576,7 @@ mod tests { "#, ) .unwrap(); - let registry = test_registry("local", true); + let registry = test_registry("local"); let preflight = |argv: &[&str]| { let (run, matches) = parse_with_args(argv); super::run_config_preflight_with_drivers( @@ -2561,7 +2631,7 @@ mod tests { .set("OPENSHELL_DRIVERS", "podman,docker") .run(|| { let (run, matches) = parse_with_args(&["openshell-gateway"]); - let registry = test_registry("podman", true); + let registry = test_registry("podman"); let error = super::run_config_preflight_with_drivers( super::ConfigPreflightArgs::default(), @@ -2648,7 +2718,7 @@ mod tests { .remove("OPENSHELL_COMPUTE_DRIVER") .remove("OPENSHELL_DRIVERS") .run(|| { - preflight(&test_registry("local", false)) + preflight(&test_registry("local")) .expect_err("a pool table without a shared supervision driver must fail"); let mut registry = crate::ComputeDriverRegistry::new(); @@ -2845,7 +2915,7 @@ mod tests { "--enable-mtls-auth", "true", ]); - let registry = test_registry("shared", false); + let registry = test_registry("shared"); super::run_config_preflight_with_drivers( super::ConfigPreflightArgs::default(), @@ -3466,7 +3536,7 @@ require_client_auth = true "--metrics-tls-key", "/metrics/tls.key", ]); - let registry = test_registry("podman", true); + let registry = test_registry("podman"); let Err(error) = super::prepare_server_config_with_drivers(&mut args, &matches, ®istry) @@ -3664,7 +3734,7 @@ require_client_auth = true .remove("OPENSHELL_GATEWAY_CONFIG") .remove("OPENSHELL_DRIVERS") .run(|| { - let registry = test_registry("shared", false); + let registry = test_registry("shared"); for (oidc_issuer, expected) in [(None, false), (Some("https://idp.example.com"), false)] @@ -3790,8 +3860,7 @@ require_client_auth = true Some(detect_registered_local), TestFactory, ) - .unwrap() - .with_local_singleplayer(), + .unwrap(), ) .unwrap(); let (preflight_args, preflight_matches) = parse_with_args(&argv); @@ -3831,8 +3900,7 @@ require_client_auth = true Some(detect_registered_local), TestFactory, ) - .unwrap() - .with_local_singleplayer(), + .unwrap(), ) .unwrap(); @@ -4088,7 +4156,7 @@ enabled = false .remove("OPENSHELL_DRIVERS") .remove("OPENSHELL_ENABLE_OPERATOR_AUTH") .run(|| { - let registry = test_registry("shared", false); + let registry = test_registry("shared"); for tls_args in [vec!["--disable-tls"], vec![]] { let mut argv = vec![ "openshell-gateway", @@ -4945,6 +5013,176 @@ mem_mib = "not-a-number" }); } + #[test] + fn server_config_prepares_provisioning_repair_window_from_toml() { + assert_eq!( + openshell_core::Config::new(None).provisioning_repair_window_seconds, + 300 + ); + for seconds in [1, 86_400] { + super::validate_provisioning_repair_window_seconds(seconds).unwrap(); + } + let state = tempfile::tempdir().unwrap(); + let tls = tempfile::tempdir().unwrap(); + Environment::new() + .set("XDG_STATE_HOME", state.path().to_str().unwrap()) + .set("OPENSHELL_LOCAL_TLS_DIR", tls.path().to_str().unwrap()) + .run(|| { + let state = + std::path::PathBuf::from(std::env::var_os("XDG_STATE_HOME").unwrap()); + let config_path = state.join("gateway.toml"); + std::fs::write( + &config_path, + "[openshell]\nversion = 2\n[openshell.gateway]\nprovisioning_repair_window_seconds = 900\n", + ) + .unwrap(); + let (mut args, matches) = parse_with_args(&[ + "openshell-gateway", + "--config", + config_path.to_str().unwrap(), + "--db-url", + "sqlite::memory:", + "--compute-driver", + "podman", + "--disable-tls", + ]); + let prepared = super::prepare_server_config(&mut args, &matches) + .expect("server config is prepared"); + assert_eq!(prepared.config.provisioning_repair_window_seconds, 900); + }); + } + + #[test] + fn server_config_rejects_unbounded_provisioning_repair_window() { + let state = tempfile::tempdir().unwrap(); + let tls = tempfile::tempdir().unwrap(); + Environment::new() + .set("XDG_STATE_HOME", state.path().to_str().unwrap()) + .set("OPENSHELL_LOCAL_TLS_DIR", tls.path().to_str().unwrap()) + .run(|| { + let state = + std::path::PathBuf::from(std::env::var_os("XDG_STATE_HOME").unwrap()); + let config_path = state.join("gateway.toml"); + for window in [-1, 0, 86_401] { + std::fs::write( + &config_path, + format!( + "[openshell]\nversion = 2\n[openshell.gateway]\nprovisioning_repair_window_seconds = {window}\n" + ), + ) + .unwrap(); + let (mut args, matches) = parse_with_args(&[ + "openshell-gateway", + "--config", + config_path.to_str().unwrap(), + "--db-url", + "sqlite::memory:", + "--compute-driver", + "podman", + "--disable-tls", + ]); + let Err(error) = super::prepare_server_config(&mut args, &matches) else { + panic!("unbounded repair window must be rejected"); + }; + assert!( + error + .to_string() + .contains("provisioning_repair_window_seconds must be between 1 and 86400") + ); + } + }); + } + + #[test] + fn server_config_env_overrides_provisioning_repair_window() { + let state = tempfile::tempdir().unwrap(); + let tls = tempfile::tempdir().unwrap(); + Environment::new() + .set("XDG_STATE_HOME", state.path().to_str().unwrap()) + .set("OPENSHELL_LOCAL_TLS_DIR", tls.path().to_str().unwrap()) + .set("OPENSHELL_PROVISIONING_REPAIR_WINDOW_SECONDS", "600") + .run(|| { + let state = + std::path::PathBuf::from(std::env::var_os("XDG_STATE_HOME").unwrap()); + let config_path = state.join("gateway.toml"); + std::fs::write( + &config_path, + "[openshell]\nversion = 2\n[openshell.gateway]\nprovisioning_repair_window_seconds = 900\n", + ) + .unwrap(); + let (mut args, matches) = parse_with_args(&[ + "openshell-gateway", + "--config", + config_path.to_str().unwrap(), + "--db-url", + "sqlite::memory:", + "--compute-driver", + "podman", + "--disable-tls", + ]); + let prepared = super::prepare_server_config(&mut args, &matches) + .expect("server config is prepared"); + assert_eq!(prepared.config.provisioning_repair_window_seconds, 600); + }); + } + + #[test] + fn server_config_sets_and_validates_per_sandbox_connection_caps() { + let state = tempfile::tempdir().unwrap(); + let tls = tempfile::tempdir().unwrap(); + Environment::new() + .set("XDG_STATE_HOME", state.path().to_str().unwrap()) + .set("OPENSHELL_LOCAL_TLS_DIR", tls.path().to_str().unwrap()) + .run(|| { + let state = std::path::PathBuf::from(std::env::var_os("XDG_STATE_HOME").unwrap()); + let config_path = state.join("gateway.toml"); + let prepare = |gateway: &str| { + std::fs::write( + &config_path, + format!("[openshell]\nversion = 2\n[openshell.gateway]\n{gateway}\n"), + ) + .unwrap(); + let (mut args, matches) = parse_with_args(&[ + "openshell-gateway", + "--config", + config_path.to_str().unwrap(), + "--db-url", + "sqlite::memory:", + "--compute-driver", + "docker", + "--disable-tls", + ]); + super::prepare_server_config(&mut args, &matches) + }; + + let defaults = prepare("").expect("defaults"); + assert_eq!(defaults.config.max_ssh_connections_per_sandbox, 20); + assert_eq!(defaults.config.max_service_connections_per_sandbox, 20); + + let set = prepare( + "max_ssh_connections_per_sandbox = 5\nmax_service_connections_per_sandbox = 50", + ) + .expect("caps are applied"); + assert_eq!(set.config.max_ssh_connections_per_sandbox, 5); + assert_eq!(set.config.max_service_connections_per_sandbox, 50); + + for key in [ + "max_ssh_connections_per_sandbox", + "max_service_connections_per_sandbox", + ] { + let Err(error) = prepare(&format!("{key} = 0")) else { + panic!("a zero cap must be rejected"); + }; + assert!( + error + .to_string() + .contains(&format!("{key} must be at least 1")), + "{error}" + ); + } + }); + } + #[test] fn explicit_launch_signing_config_ignores_partial_local_bundle() { let directory = tempfile::tempdir().unwrap(); diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index a6768040e6..4ba1311d54 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -937,6 +937,7 @@ pub struct ComputeRuntime { driver_process: Option>, default_image: String, image_preparation_timeout_seconds: u32, + provisioning_repair_window_ms: i64, sandbox_updates_enabled: bool, update_recovery_sweep: Arc, update_recovery_backoff: Arc>>, @@ -1059,6 +1060,7 @@ impl ComputeRuntime { driver_process, default_image, image_preparation_timeout_seconds: 1800, + provisioning_repair_window_ms: 300_000, sandbox_updates_enabled: false, update_recovery_sweep: Arc::new(tokio::sync::Semaphore::new(1)), update_recovery_backoff: Arc::default(), @@ -1342,6 +1344,20 @@ impl ComputeRuntime { Ok(self) } + /// Validate the repair window once at startup. Persisted attempts keep + /// their original deadline even if the operator changes this value on + /// restart. + pub(crate) fn with_provisioning_repair_window_seconds( + mut self, + window: i64, + ) -> Result { + if !(1..=86_400).contains(&window) { + return Err("provisioning_repair_window_seconds must be between 1 and 86400".into()); + } + self.provisioning_repair_window_ms = window * 1000; + Ok(self) + } + #[must_use] pub fn driver_info_snapshots(&self) -> &[ComputeDriverInfoSnapshot] { std::slice::from_ref(&self.driver_info) @@ -2522,6 +2538,7 @@ impl ComputeRuntime { reason, message, self.image_preparation_timeout_seconds, + self.provisioning_repair_window_ms, ); if let Some(metadata) = sandbox.metadata.as_mut() { next_identity_for_update.write(&mut metadata.annotations); @@ -2954,6 +2971,7 @@ impl ComputeRuntime { "ResumeRolledBack", "Resumed workload stopped after its runtime binding could not be committed", self.image_preparation_timeout_seconds, + self.provisioning_repair_window_ms, ); } if self.restore_lifecycle_snapshot(&settled, &rollback).await { @@ -3031,6 +3049,7 @@ impl ComputeRuntime { "Suspended", "Sandbox remains suspended after failed resume", self.image_preparation_timeout_seconds, + self.provisioning_repair_window_ms, ); let _ = self .restore_lifecycle_snapshot(transition, &suspended) @@ -3171,6 +3190,7 @@ impl ComputeRuntime { &reason, &message, self.image_preparation_timeout_seconds, + self.provisioning_repair_window_ms, ); }, ) @@ -5615,7 +5635,9 @@ impl ComputeRuntime { let ready_container_exit = existing_phase == SandboxPhase::Ready && (driver_snapshot_confirms_stopped(&incoming) || driver_snapshot_reports_runtime_restart(&incoming)); - if existing_phase != SandboxPhase::Starting && !ready_container_exit { + let stopped_bootstrap = + existing_phase == SandboxPhase::Stopped && driver_snapshot_confirms_starting(&incoming); + if existing_phase != SandboxPhase::Starting && !ready_container_exit && !stopped_bootstrap { return self.apply_sandbox_update_locked(incoming, existing).await; } @@ -5625,6 +5647,8 @@ impl ComputeRuntime { // stop the new generation before the replacement supervisor connects. // The replacement may already be Ready when the old exit arrives, so // terminal container snapshots need the same check after readiness. + // A bootstrap event queued before Stop reached Stopped must also be + // revalidated so it cannot undo the completed stop. // Release this sandbox's local mutation lock, wait for that lifecycle // operation, and then reread both the driver and store before applying // an authoritative observation. Taking the per-sandbox gate only for @@ -5654,7 +5678,10 @@ impl ComputeRuntime { match observed { Ok(Some(live)) if live.id == incoming.id && live.status.is_some() => incoming = live, Ok(Some(_) | None) | Err(_) - if matches!(current_phase, SandboxPhase::Starting | SandboxPhase::Ready) => + if matches!( + current_phase, + SandboxPhase::Starting | SandboxPhase::Ready | SandboxPhase::Stopped + ) => { warn!( sandbox_id = %incoming.id, @@ -5958,7 +5985,9 @@ impl ComputeRuntime { return Err("restarting sandbox requires a fresh supervisor instance".into()); } let provisioning = admission - .map(|admission| admitted_provisioning(¤t, admission)) + .map(|admission| { + admitted_provisioning(¤t, admission, self.provisioning_repair_window_ms) + }) .transpose()? .flatten(); let expected_resource_version = sandbox_resource_version(¤t); @@ -6573,6 +6602,12 @@ impl ComputeRuntime { sandbox_id: &str, ) -> Result { let _guard = self.lock_sandbox_for_lifecycle(delete_guard).await; + // A missing sandbox row means the sandbox is gone. Several readers make + // that inference with their own error policy -- this teardown, the + // push-log handler's store recheck (`ensure_log_stream_sandbox_exists` + // in grpc/policy.rs), and the `WatchSandbox` producer in + // grpc/sandbox.rs -- so a change to deletion semantics (soft delete, + // migration windows, record recreation) must update every one of them. let record = self .store .get(Sandbox::object_type(), sandbox_id) @@ -6591,9 +6626,10 @@ impl ComputeRuntime { } fn cleanup_sandbox_state(&self, sandbox_id: &str) { - // `tracing_log_bus.remove` also clears the platform event bus and resets - // the shared cursor allocator last (see its docs). - self.tracing_log_bus.remove(sandbox_id); + // `tracing_log_bus.close` also clears the platform event bus and leaves + // a terminal tombstone in the shared cursor allocator, so late + // publishers cannot re-create the buffers this frees (see its docs). + self.tracing_log_bus.close(sandbox_id); self.sandbox_watch_bus.remove(sandbox_id); } @@ -7861,6 +7897,7 @@ fn record_session_admission( fn admitted_provisioning( sandbox: &Sandbox, admission: &openshell_core::proto::SandboxConfigurationAdmission, + repair_window_ms: i64, ) -> Result, String> { let Some(mut record) = sandbox .status @@ -7873,9 +7910,9 @@ fn admitted_provisioning( if !provisioning_deadline::allows_admission(&record, now_ms) { return Err("provisioning deadline expired".to_string()); } - provisioning_deadline::record_admission_start(&mut record, now_ms)?; + provisioning_deadline::record_admission_start(&mut record, now_ms, repair_window_ms)?; if admission.state == i32::from(openshell_core::proto::ConfigurationAdmissionState::Rejected) { - provisioning_deadline::record_rejection(&mut record, now_ms)?; + provisioning_deadline::record_rejection(&mut record, now_ms, repair_window_ms)?; } Ok(Some(record)) } @@ -8229,6 +8266,7 @@ fn apply_lifecycle_phase( reason: &str, message: &str, image_preparation_timeout_seconds: u32, + provisioning_repair_window_ms: i64, ) { let preserve_exit_code = phase == SandboxPhase::Stopping && (sandbox.phase() == SandboxPhase::Completed as i32 @@ -8260,6 +8298,7 @@ fn apply_lifecycle_phase( status.provisioning = Some(provisioning_deadline::new_preparation_record( openshell_core::time::now_ms(), image_preparation_timeout_seconds, + provisioning_repair_window_ms, )); status.configuration_admission = Some(openshell_core::proto::SandboxConfigurationAdmission { @@ -8726,6 +8765,7 @@ pub fn new_test_runtime_with_driver( driver_process: None, default_image: "openshell/sandbox:test".to_string(), image_preparation_timeout_seconds: 1800, + provisioning_repair_window_ms: 300_000, sandbox_updates_enabled: false, update_recovery_sweep: Arc::new(tokio::sync::Semaphore::new(1)), update_recovery_backoff: Arc::default(), @@ -10024,6 +10064,7 @@ mod tests { driver_process: None, default_image: "openshell/sandbox:test".to_string(), image_preparation_timeout_seconds: 1800, + provisioning_repair_window_ms: 300_000, sandbox_updates_enabled: false, update_recovery_sweep: Arc::new(Semaphore::new(1)), update_recovery_backoff: Arc::default(), @@ -13216,6 +13257,7 @@ mod tests { provisioning_deadline::record_admission_start( status.provisioning.as_mut().unwrap(), openshell_core::time::now_ms(), + runtime.provisioning_repair_window_ms, ) .unwrap(); status.configuration_admission.as_mut().unwrap().state = @@ -14499,9 +14541,109 @@ mod tests { assert_eq!(current.phase(), SandboxPhase::Stopped as i32); } + #[tokio::test] + async fn stale_bootstrap_snapshot_waits_for_stop_and_revalidates() { + let sandbox = sandbox_record("sb-stop-race", "sandbox-stop-race", SandboxPhase::Ready); + let mut stopped = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + stopped.status = Some(make_driver_status(make_driver_condition( + "ContainerExited", + "container stopped by request", + ))); + let mut bootstrapping = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + let mut status = make_driver_status(make_driver_condition( + "DependenciesNotReady", + "old bootstrap snapshot queued before stop", + )); + status.conditions.push(DriverCondition { + r#type: "Bootstrapping".to_string(), + status: "True".to_string(), + reason: "GenerationStarting".to_string(), + ..Default::default() + }); + bootstrapping.status = Some(status); + + for (observation, outcome) in [ + ( + "stopped", + ControlledGetOutcome::Sandbox(Box::new(stopped.clone())), + ), + ("missing", ControlledGetOutcome::Missing), + ( + "unavailable", + ControlledGetOutcome::Error("driver unavailable"), + ), + ( + "wrong sandbox", + ControlledGetOutcome::Sandbox(Box::new(DriverSandbox { + id: "other-sandbox".to_string(), + ..stopped.clone() + })), + ), + ( + "missing status", + ControlledGetOutcome::Sandbox(Box::new(DriverSandbox { + status: None, + ..stopped.clone() + })), + ), + ] { + let driver = ControlledDriver::new(); + driver.block_stop(); + driver.set_get_outcome(outcome); + let runtime = test_runtime(driver.clone()).await; + runtime.store.put_message(&sandbox).await.unwrap(); + + let stop_runtime = runtime.clone(); + let sandbox_name = sandbox.object_name().to_string(); + let stop = + tokio::spawn( + async move { stop_runtime.stop_sandbox("default", &sandbox_name).await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.stop_started.notified()) + .await + .expect("stop did not reach the driver"); + runtime.apply_sandbox_update(stopped.clone()).await.unwrap(); + + let gate = runtime.lifecycle_gates.gate_for(sandbox.object_id()); + let update_runtime = runtime.clone(); + let stale_bootstrap = bootstrapping.clone(); + let update = + tokio::spawn( + async move { update_runtime.apply_sandbox_update(stale_bootstrap).await }, + ); + tokio::time::timeout(Duration::from_secs(1), async { + while Arc::strong_count(&gate) < 3 { + assert!( + !update.is_finished(), + "{observation}: queued bootstrap must wait for the active stop" + ); + tokio::task::yield_now().await; + } + }) + .await + .expect("bootstrap update did not wait on the lifecycle gate"); + + driver.release_stop(); + stop.await.unwrap().unwrap(); + update.await.unwrap().unwrap(); + let current = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!( + current.phase(), + SandboxPhase::Stopped as i32, + "{observation}: stale bootstrap must not revive the stopped sandbox" + ); + } + } + #[tokio::test] async fn active_bootstrap_snapshot_recovers_stale_stopped_view() { - let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let driver = ControlledDriver::new(); + let runtime = test_runtime(driver.clone()).await; let sandbox = sandbox_record("sb-restart", "sandbox-restart", SandboxPhase::Stopped); runtime.store.put_message(&sandbox).await.unwrap(); let mut bootstrapping = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); @@ -14518,6 +14660,9 @@ mod tests { }); bootstrapping.status = Some(status); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new( + bootstrapping.clone(), + ))); runtime.apply_sandbox_update(bootstrapping).await.unwrap(); let current = runtime @@ -15519,6 +15664,105 @@ mod tests { )); } + #[tokio::test] + async fn delete_tombstones_sandbox_log_and_event_buses() { + // #5402: cleanup frees the sandbox's log and platform-event buffers. + // Resetting the cursor space (`remove`) lets a late publisher re-create + // both, and nothing ever frees them again. The delete path must leave a + // terminal tombstone in the shared allocator instead. + let runtime = test_runtime(ControlledDriver::new()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let log_line = |message: &str| openshell_core::proto::SandboxLogLine { + sandbox_id: "sb-1".to_string(), + event_time: openshell_core::time::timestamp_from_millis(1_000).ok(), + level: "INFO".to_string(), + target: "delete-tombstone".to_string(), + message: message.to_string(), + source: "sandbox".to_string(), + fields: HashMap::new(), + }; + let platform_event = |message: &str| openshell_core::proto::SandboxStreamEvent { + payload: Some(openshell_core::proto::sandbox_stream_event::Payload::Event( + PlatformEvent { + source: "test".to_string(), + reason: "Delete".to_string(), + message: message.to_string(), + ..Default::default() + }, + )), + cursor: String::new(), + }; + + // Both buses hold buffers before the delete, so cleanup has memory to + // reclaim and the assertions below observe a real transition. + runtime + .tracing_log_bus + .publish_external(log_line("before delete")); + runtime + .tracing_log_bus + .platform_event_bus + .publish("sb-1", platform_event("before delete")); + assert_eq!(runtime.tracing_log_bus.tail("sb-1", 10).len(), 1); + assert_eq!( + runtime + .tracing_log_bus + .platform_event_bus + .tail("sb-1", 10) + .len(), + 1 + ); + + // The real delete: request, driver acknowledgement, then the Deleted + // watch event that owns gateway-side cleanup. + runtime + .delete_sandbox("default", "sandbox-a") + .await + .expect("the delete request is accepted"); + runtime + .apply_deleted("sb-1") + .await + .expect("the deleted watch event completes cleanup"); + + assert!( + runtime.tracing_log_bus.is_closed("sb-1"), + "delete cleanup must tombstone the sandbox's cursor space, not reset it" + ); + assert!(runtime.tracing_log_bus.tail("sb-1", 10).is_empty()); + assert!( + runtime + .tracing_log_bus + .platform_event_bus + .tail("sb-1", 10) + .is_empty() + ); + + // Late publishers: an open PushSandboxLogs stream, a driver event + // queued behind the Deleted event, and the gateway tracing layer. + runtime + .tracing_log_bus + .publish_external(log_line("after delete")); + runtime + .tracing_log_bus + .platform_event_bus + .publish("sb-1", platform_event("after delete")); + + assert!( + runtime.tracing_log_bus.tail("sb-1", 10).is_empty(), + "a late log publish must not re-create the buffers cleanup freed" + ); + assert!( + runtime + .tracing_log_bus + .platform_event_bus + .tail("sb-1", 10) + .is_empty(), + "a late platform event must not re-create the buffers cleanup freed" + ); + assert!(runtime.tracing_log_bus.is_closed("sb-1")); + } + #[tokio::test] async fn apply_deleted_releases_driver_resources_for_out_of_band_removal() { // Regression test for #2352: a container removed without going @@ -19258,7 +19502,10 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" async fn seed_provisioning_attempt(runtime: &ComputeRuntime) -> Sandbox { let mut sandbox = sandbox_record("sb-ttl", "sandbox-ttl", SandboxPhase::Provisioning); let status = sandbox.status.as_mut().unwrap(); - status.provisioning = Some(provisioning_deadline::new_record(0)); + status.provisioning = Some(provisioning_deadline::new_record( + 0, + runtime.provisioning_repair_window_ms, + )); status.configuration_activated = Some(false); status.configuration_admission = Some(openshell_core::proto::SandboxConfigurationAdmission { @@ -19345,9 +19592,12 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" "result-owner", SandboxPhase::Provisioning, ); - sandbox.status.as_mut().unwrap().provisioning = Some( - provisioning_deadline::new_preparation_record(openshell_core::time::now_ms(), 1800), - ); + sandbox.status.as_mut().unwrap().provisioning = + Some(provisioning_deadline::new_preparation_record( + openshell_core::time::now_ms(), + 1800, + 300_000, + )); let creating = runtime.clone(); let create = tokio::spawn(async move { creating.create_sandbox(sandbox, None, false).await }); @@ -19439,8 +19689,9 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" other.lifecycle_gates = Arc::new(LifecycleGateRegistry::default()); let now = openshell_core::time::now_ms(); let mut sandbox = sandbox_record("sb-stop-lease", "stop-lease", SandboxPhase::Provisioning); - sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_preparation_record(now, 1)); + sandbox.status.as_mut().unwrap().provisioning = Some( + provisioning_deadline::new_preparation_record(now, 1, 300_000), + ); let creating = runtime.clone(); let create = tokio::spawn(async move { creating.create_sandbox(sandbox, None, false).await }); @@ -19537,6 +19788,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" status.provisioning = Some(provisioning_deadline::new_preparation_record( openshell_core::time::now_ms(), 1800, + 300_000, )); let creating = runtime.clone(); let create = @@ -19643,6 +19895,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" status.main_process_instance_id = "instance-1".into(); status.provisioning = Some(provisioning_deadline::new_record( openshell_core::time::now_ms(), + 300_000, )); let old_generation = sandbox_runtime_generation(&sandbox).unwrap(); runtime.store.put_message(&sandbox).await.unwrap(); @@ -19757,8 +20010,11 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" let driver = ControlledDriver::new(); let runtime = test_runtime(driver.clone()).await; let mut previous = sandbox_record("sb-start-owner", "start-owner", SandboxPhase::Stopped); - let mut record = - provisioning_deadline::new_preparation_record(openshell_core::time::now_ms(), 1800); + let mut record = provisioning_deadline::new_preparation_record( + openshell_core::time::now_ms(), + 1800, + 300_000, + ); record.driver_operation_id = "old-operation".into(); previous.status.as_mut().unwrap().provisioning = Some(record); runtime.store.put_message(&previous).await.unwrap(); @@ -19829,7 +20085,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" SandboxPhase::Provisioning, ); sandbox.status.as_mut().unwrap().provisioning = Some( - provisioning_deadline::new_record(openshell_core::time::now_ms()), + provisioning_deadline::new_record(openshell_core::time::now_ms(), 300_000), ); runtime.store.put_message(&sandbox).await.unwrap(); let owned = runtime @@ -19908,6 +20164,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" status.main_process_instance_id = "instance-1".into(); status.provisioning = Some(provisioning_deadline::new_record( openshell_core::time::now_ms(), + 300_000, )); runtime.store.put_message(&sandbox).await.unwrap(); runtime @@ -19968,8 +20225,9 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" let pool = sqlx::SqlitePool::connect(&database_url).await.unwrap(); let now = openshell_core::time::now_ms(); let mut sandbox = sandbox_record("sb-monitor", "monitor", SandboxPhase::Provisioning); - sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_preparation_record(now, 60)); + sandbox.status.as_mut().unwrap().provisioning = Some( + provisioning_deadline::new_preparation_record(now, 60, 300_000), + ); let creating = runtime.clone(); let create = tokio::spawn(async move { creating.create_sandbox(sandbox, None, false).await }); @@ -20021,8 +20279,9 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" let mut runtime = test_runtime(driver.clone()).await; let now = openshell_core::time::now_ms(); let mut sandbox = sandbox_record("sb-owned", "owned", SandboxPhase::Provisioning); - sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_preparation_record(now, 60)); + sandbox.status.as_mut().unwrap().provisioning = Some( + provisioning_deadline::new_preparation_record(now, 60, 300_000), + ); let root = tempfile::tempdir().unwrap(); let upload = stage_create_test_upload(&mut runtime, &mut sandbox, root.path()); let creating = runtime.clone(); @@ -20101,9 +20360,12 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" driver.set_delete_outcome(ControlledDeleteOutcome::Ok(false)); let mut runtime = test_runtime(driver).await; let mut sandbox = sandbox_record("sb-rejected", "rejected", SandboxPhase::Provisioning); - sandbox.status.as_mut().unwrap().provisioning = Some( - provisioning_deadline::new_preparation_record(openshell_core::time::now_ms(), 60), - ); + sandbox.status.as_mut().unwrap().provisioning = + Some(provisioning_deadline::new_preparation_record( + openshell_core::time::now_ms(), + 60, + 300_000, + )); let root = tempfile::tempdir().unwrap(); let upload = stage_create_test_upload(&mut runtime, &mut sandbox, root.path()); let error = runtime @@ -20181,9 +20443,12 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" runtime.store = Arc::new(Store::connect(&database_url).await.unwrap()); let pool = sqlx::SqlitePool::connect(&database_url).await.unwrap(); let mut sandbox = sandbox_record("sb-settlement", "settlement", SandboxPhase::Provisioning); - sandbox.status.as_mut().unwrap().provisioning = Some( - provisioning_deadline::new_preparation_record(openshell_core::time::now_ms(), 60), - ); + sandbox.status.as_mut().unwrap().provisioning = + Some(provisioning_deadline::new_preparation_record( + openshell_core::time::now_ms(), + 60, + 300_000, + )); let creating = runtime.clone(); let create = tokio::spawn(async move { creating.create_sandbox(sandbox, None, false).await }); @@ -20241,7 +20506,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" driver.block_create(); let mut runtime = test_runtime(driver.clone()).await; let now = openshell_core::time::now_ms(); - let preparation = provisioning_deadline::new_preparation_record(now, 1); + let preparation = provisioning_deadline::new_preparation_record(now, 1, 300_000); let mut sandbox = sandbox_record("sb-create-ttl", "create-ttl", SandboxPhase::Provisioning); sandbox.status.as_mut().unwrap().provisioning = Some(preparation.clone()); let upload_root = tempfile::tempdir().unwrap(); @@ -20373,8 +20638,9 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" "compensation-ttl", SandboxPhase::Provisioning, ); - sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_preparation_record(now, 1)); + sandbox.status.as_mut().unwrap().provisioning = Some( + provisioning_deadline::new_preparation_record(now, 1, 300_000), + ); runtime.store.put_message(&sandbox).await.unwrap(); let stale = runtime .store @@ -20423,8 +20689,9 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" "create-retry", SandboxPhase::Provisioning, ); - sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_preparation_record(now, 1)); + sandbox.status.as_mut().unwrap().provisioning = Some( + provisioning_deadline::new_preparation_record(now, 1, 300_000), + ); let original_attempt = sandbox_provisioning_attempt_id(&sandbox) .unwrap() .to_owned(); @@ -20578,9 +20845,12 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" async fn failed_create_cleanup_preserves_retry_after_cas_conflict() { let runtime = test_runtime(ControlledDriver::new()).await; let mut sandbox = sandbox_record("sb-retry-cas", "retry-cas", SandboxPhase::Provisioning); - sandbox.status.as_mut().unwrap().provisioning = Some( - provisioning_deadline::new_preparation_record(openshell_core::time::now_ms(), 1), - ); + sandbox.status.as_mut().unwrap().provisioning = + Some(provisioning_deadline::new_preparation_record( + openshell_core::time::now_ms(), + 1, + 300_000, + )); runtime.store.put_message(&sandbox).await.unwrap(); let stale = runtime .store @@ -20600,6 +20870,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" "Starting", "Sandbox start requested", runtime.image_preparation_timeout_seconds, + runtime.provisioning_repair_window_ms, ); }, ) @@ -20631,9 +20902,12 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" "binding-retry", SandboxPhase::Provisioning, ); - sandbox.status.as_mut().unwrap().provisioning = Some( - provisioning_deadline::new_preparation_record(openshell_core::time::now_ms(), 1), - ); + sandbox.status.as_mut().unwrap().provisioning = + Some(provisioning_deadline::new_preparation_record( + openshell_core::time::now_ms(), + 1, + 300_000, + )); runtime.store.put_message(&sandbox).await.unwrap(); let stale = runtime .store @@ -20653,6 +20927,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" "Starting", "Sandbox start requested", runtime.image_preparation_timeout_seconds, + runtime.provisioning_repair_window_ms, ); current.set_phase(SandboxPhase::Ready.into()); set_compute_runtime_binding(current, "retry-runtime"); @@ -20703,7 +20978,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" let mut runtime = test_runtime(driver.clone()).await; runtime.driver_info.supports_sandbox_authentication = true; let now = openshell_core::time::now_ms(); - let preparation = provisioning_deadline::new_preparation_record(now, 1); + let preparation = provisioning_deadline::new_preparation_record(now, 1, 300_000); let mut sandbox = sandbox_record("sb-late-create", "late-create", SandboxPhase::Provisioning); sandbox.status.as_mut().unwrap().provisioning = Some(preparation.clone()); @@ -20776,7 +21051,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" driver.block_start(); let runtime = test_runtime(driver.clone()).await; let now = openshell_core::time::now_ms(); - let preparation = provisioning_deadline::new_preparation_record(now, 1800); + let preparation = provisioning_deadline::new_preparation_record(now, 1800, 300_000); let mut sandbox = sandbox_record("sb-recovery-ttl", "recovery-ttl", SandboxPhase::Starting); sandbox.status.as_mut().unwrap().provisioning = Some(preparation.clone()); runtime.store.put_message(&sandbox).await.unwrap(); @@ -20859,8 +21134,9 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" let driver = ControlledDriver::new(); let runtime = test_runtime(driver.clone()).await; let mut sandbox = sandbox_record("sb-prepare", "prepare", SandboxPhase::Provisioning); - sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_preparation_record(0, 1800)); + sandbox.status.as_mut().unwrap().provisioning = Some( + provisioning_deadline::new_preparation_record(0, 1800, 300_000), + ); runtime.store.put_message(&sandbox).await.unwrap(); let sandbox = runtime .store @@ -20930,29 +21206,30 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" #[tokio::test] async fn provisioning_timeout_persists_error_before_cleanup_and_preserves_record() { let driver = ControlledDriver::new(); - let runtime = test_runtime(driver.clone()).await; + let runtime = test_runtime(driver.clone()) + .await + .with_provisioning_repair_window_seconds(900) + .unwrap(); let sandbox = seed_provisioning_attempt(&runtime).await; let gate = runtime.lifecycle_gates.lock_for("sb-ttl").await; let sandbox_guard = runtime.lock_sandbox_for_lifecycle(&gate).await; assert!( runtime - .claim_provisioning_timeout(&sandbox, 299_999) + .claim_provisioning_timeout(&sandbox, 899_999) .await .unwrap() .is_none() ); let expired = runtime - .claim_provisioning_timeout(&sandbox, 300_000) + .claim_provisioning_timeout(&sandbox, 900_000) .await .unwrap() .unwrap(); drop(sandbox_guard); assert_eq!(expired.phase(), i32::from(SandboxPhase::Error)); - assert!( - ready_condition(&expired) - .unwrap() - .message - .contains("Provider requires an inspected endpoint") + assert_eq!( + ready_condition(&expired).unwrap().message, + "Provisioning repair window expired: Provider requires an inspected endpoint" ); assert_eq!(driver.stop_calls(), 0); runtime @@ -21445,10 +21722,15 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" let now = openshell_core::time::now_ms(); let mut sandbox = sandbox_record("sb-reset", "reset", SandboxPhase::Provisioning); sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_record(now - 100_000)); - provisioning_deadline::refresh_configuration(&runtime.store, &mut sandbox, now - 100_000) - .await - .unwrap(); + Some(provisioning_deadline::new_record(now - 100_000, 300_000)); + provisioning_deadline::refresh_configuration( + &runtime.store, + &mut sandbox, + now - 100_000, + 300_000, + ) + .await + .unwrap(); runtime.store.put_message(&sandbox).await.unwrap(); let policy = openshell_policy::restrictive_default_policy(); let hash = openshell_core::policy_identity::deterministic_policy_hash(&policy); @@ -21501,7 +21783,7 @@ endpoint = { kind = "unix", socket_path = "/run/agent-substrate/isolation.sock" let now = openshell_core::time::now_ms(); let mut sandbox = sandbox_record("sb-busy-ttl", "busy-ttl", SandboxPhase::Starting); sandbox.status.as_mut().unwrap().provisioning = - Some(provisioning_deadline::new_record(now)); + Some(provisioning_deadline::new_record(now, 300_000)); runtime.store.put_message(&sandbox).await.unwrap(); let gate = runtime.lifecycle_gates.lock_for("sb-busy-ttl").await; runtime diff --git a/crates/openshell-server/src/compute/provisioning_deadline.rs b/crates/openshell-server/src/compute/provisioning_deadline.rs index cb351e64d0..5f74a7b65f 100644 --- a/crates/openshell-server/src/compute/provisioning_deadline.rs +++ b/crates/openshell-server/src/compute/provisioning_deadline.rs @@ -12,8 +12,6 @@ use openshell_core::proto::SandboxProvisioning; use openshell_core::time::{timestamp_from_millis, timestamp_to_millis}; use serde::{Deserialize, Serialize}; -const REPAIR_WINDOW_MS: i64 = 300_000; - /// How long one scan pass waits, in total, for sandboxes' local keys. A sandbox /// whose keys are still busy is skipped until the next pass. Keys are taken in /// ascending order, so the scan can hold X(sandbox) while it waits for @@ -136,7 +134,12 @@ impl ProvisioningDeadline { }; } - pub fn new(attempt_id: String, change: ConfigurationChange, now_ms: i64) -> Self { + pub fn new( + attempt_id: String, + change: ConfigurationChange, + now_ms: i64, + repair_window_ms: i64, + ) -> Self { Self { attempt_id, change, @@ -144,7 +147,7 @@ impl ProvisioningDeadline { preparation_deadline_at_ms: None, admission_start_at_ms: None, state: DeadlineState::Armed { - deadline_at_ms: now_ms.saturating_add(REPAIR_WINDOW_MS), + deadline_at_ms: now_ms.saturating_add(repair_window_ms), }, } } @@ -156,7 +159,7 @@ impl ProvisioningDeadline { /// Only an authenticated report for the current supervisor may call this. /// Persist it with the report: duplicate delivery or restart must not grant /// another admission window, and a late registration cannot revive compute. - fn start_admission(&mut self, attempt_id: &str, now_ms: i64) -> bool { + fn start_admission(&mut self, attempt_id: &str, now_ms: i64, repair_window_ms: i64) -> bool { if attempt_id != self.attempt_id || !self.is_preparing() || now_ms < self.change.committed_at_ms @@ -168,7 +171,7 @@ impl ProvisioningDeadline { } self.admission_start_at_ms = Some(now_ms); self.state = DeadlineState::Armed { - deadline_at_ms: now_ms.saturating_add(REPAIR_WINDOW_MS), + deadline_at_ms: now_ms.saturating_add(repair_window_ms), }; true } @@ -182,7 +185,12 @@ impl ProvisioningDeadline { /// Apply a newer, committed change to an active attempt. Delayed observation /// uses commit time, not poll time. A change after expiry cannot revive it. - pub fn configuration_changed(&mut self, attempt_id: &str, change: ConfigurationChange) -> bool { + pub fn configuration_changed( + &mut self, + attempt_id: &str, + change: ConfigurationChange, + repair_window_ms: i64, + ) -> bool { let Some(deadline_at_ms) = self.deadline_at_ms() else { return false; }; @@ -196,7 +204,7 @@ impl ProvisioningDeadline { if !self.is_preparing() { self.state = DeadlineState::Armed { deadline_at_ms: deadline_at_ms - .max(change.committed_at_ms.saturating_add(REPAIR_WINDOW_MS)), + .max(change.committed_at_ms.saturating_add(repair_window_ms)), }; } self.change = change; @@ -206,7 +214,13 @@ impl ProvisioningDeadline { /// Only the first accepted rejection for the current change grants a repair /// window. Retrying, reconnecting, or changing diagnostic text does not. - pub fn rejected(&mut self, attempt_id: &str, change_id: &str, now_ms: i64) -> bool { + pub fn rejected( + &mut self, + attempt_id: &str, + change_id: &str, + now_ms: i64, + repair_window_ms: i64, + ) -> bool { let Some(deadline_at_ms) = self.deadline_at_ms() else { return false; }; @@ -220,7 +234,7 @@ impl ProvisioningDeadline { } self.first_rejection_at_ms = Some(now_ms); self.state = DeadlineState::Armed { - deadline_at_ms: deadline_at_ms.max(now_ms.saturating_add(REPAIR_WINDOW_MS)), + deadline_at_ms: deadline_at_ms.max(now_ms.saturating_add(repair_window_ms)), }; true } @@ -281,7 +295,7 @@ pub(super) fn driver_operation_pending(sandbox: &openshell_core::proto::Sandbox) /// Adopt an existing untimed attempt without granting it a new preparation phase. /// New create/start operations use `new_preparation_record` instead. -pub fn new_record(now_ms: i64) -> SandboxProvisioning { +pub fn new_record(now_ms: i64, repair_window_ms: i64) -> SandboxProvisioning { let mut record = SandboxProvisioning::default(); ProvisioningDeadline::new( uuid::Uuid::new_v4().to_string(), @@ -290,6 +304,7 @@ pub fn new_record(now_ms: i64) -> SandboxProvisioning { committed_at_ms: now_ms, }, now_ms, + repair_window_ms, ) .write_record(&mut record); record @@ -298,8 +313,12 @@ pub fn new_record(now_ms: i64) -> SandboxProvisioning { /// Create an independent attempt with a fixed preparation budget. The gateway /// validates the configured seconds before constructing the runtime. Reconnects /// and driver progress must never call this function. -pub fn new_preparation_record(now_ms: i64, timeout_seconds: u32) -> SandboxProvisioning { - let mut record = new_record(now_ms); +pub fn new_preparation_record( + now_ms: i64, + timeout_seconds: u32, + repair_window_ms: i64, +) -> SandboxProvisioning { + let mut record = new_record(now_ms, repair_window_ms); let ceiling = now_ms.saturating_add(i64::from(timeout_seconds) * 1_000); record.preparation_deadline = timestamp_from_millis(ceiling).ok(); record.deadline.clone_from(&record.preparation_deadline); @@ -309,18 +328,31 @@ pub fn new_preparation_record(now_ms: i64, timeout_seconds: u32) -> SandboxProvi /// The caller has authenticated the supervisor and checked its instance fence. /// A Pending registration may start timing before configuration validation. /// Persist this transition in the same CAS as the supervisor report. -pub fn record_admission_start(record: &mut SandboxProvisioning, now_ms: i64) -> Result<(), String> { +pub fn record_admission_start( + record: &mut SandboxProvisioning, + now_ms: i64, + repair_window_ms: i64, +) -> Result<(), String> { let mut deadline = ProvisioningDeadline::from_record(record)?; - deadline.start_admission(&record.attempt_id, now_ms); + deadline.start_admission(&record.attempt_id, now_ms, repair_window_ms); deadline.write_record(record); Ok(()) } /// Apply the first accepted rejection under the same CAS as admission evidence. /// The report's generation and supervisor instance must already be validated. -pub fn record_rejection(record: &mut SandboxProvisioning, now_ms: i64) -> Result<(), String> { +pub fn record_rejection( + record: &mut SandboxProvisioning, + now_ms: i64, + repair_window_ms: i64, +) -> Result<(), String> { let mut deadline = ProvisioningDeadline::from_record(record)?; - deadline.rejected(&record.attempt_id, &record.configuration_change_id, now_ms); + deadline.rejected( + &record.attempt_id, + &record.configuration_change_id, + now_ms, + repair_window_ms, + ); deadline.write_record(record); Ok(()) } @@ -401,6 +433,7 @@ pub async fn refresh_configuration( store: &crate::persistence::Store, sandbox: &mut openshell_core::proto::Sandbox, now_ms: i64, + repair_window_ms: i64, ) -> Result<(), String> { use openshell_core::proto::SandboxPhase; if !matches!( @@ -412,7 +445,7 @@ pub async fn refresh_configuration( let status = sandbox.status.get_or_insert_with(Default::default); let record = status .provisioning - .get_or_insert_with(|| new_record(now_ms)); + .get_or_insert_with(|| new_record(now_ms, repair_window_ms)); if record.deadline.is_none() { return Ok(()); } @@ -427,7 +460,7 @@ pub async fn refresh_configuration( // Taking the first source snapshot does not grant a second initial window. deadline.change.id = change.id; } else { - deadline.configuration_changed(&record.attempt_id, change); + deadline.configuration_changed(&record.attempt_id, change, repair_window_ms); } deadline.write_record(record); Ok(()) @@ -518,7 +551,13 @@ impl super::ComputeRuntime { continue; }; let previous = current.clone(); - refresh_configuration(&self.store, &mut current, now_ms).await?; + refresh_configuration( + &self.store, + &mut current, + now_ms, + self.provisioning_repair_window_ms, + ) + .await?; if current != previous { current = self .store @@ -613,10 +652,10 @@ impl super::ComputeRuntime { let message = if preparation_expired { "Image preparation or initial supervisor startup exceeded its absolute deadline".to_string() } else if diagnostic.is_empty() { - "Provisioning repair window expired after 300 seconds".to_string() + "Provisioning repair window expired".to_string() } else { format!( - "Provisioning repair window expired after 300 seconds: {diagnostic}" + "Provisioning repair window expired: {diagnostic}" ) }; status @@ -814,6 +853,41 @@ impl super::ComputeRuntime { mod tests { use super::*; + #[test] + fn configured_repair_window_preserves_persisted_deadlines() { + use prost::Message; + let mut record = new_preparation_record(0, 1800, 900_000); + record_admission_start(&mut record, 600_000, 900_000).unwrap(); + assert_eq!( + ProvisioningDeadline::from_record(&record) + .unwrap() + .deadline_at_ms(), + Some(1_500_000) + ); + record_rejection(&mut record, 610_000, 900_000).unwrap(); + let bytes = record.encode_to_vec(); + let mut restored = SandboxProvisioning::decode(bytes.as_slice()).unwrap(); + record_admission_start(&mut restored, 700_000, 1_200_000).unwrap(); + record_rejection(&mut restored, 700_000, 1_200_000).unwrap(); + let mut timer = ProvisioningDeadline::from_record(&restored).unwrap(); + assert_eq!(timer.deadline_at_ms(), Some(1_510_000)); + assert!(!timer.configuration_changed( + "previous-attempt", + change("late", 800_000), + 1_200_000 + )); + assert!(timer.expire( + &restored.attempt_id, + &restored.configuration_change_id, + 1_510_000 + )); + assert!(!timer.configuration_changed( + &restored.attempt_id, + change("after-expiry", 1_500_000), + 1_200_000 + )); + } + #[test] fn existing_wire_record_does_not_gain_preparation_time() { use prost::Message; @@ -824,7 +898,7 @@ mod tests { assert!(!record.driver_operation_pending); assert!(record.driver_operation_id.is_empty()); let before = record.clone(); - record_admission_start(&mut record, 299_999).unwrap(); + record_admission_start(&mut record, 299_999, 300_000).unwrap(); assert_eq!(record, before); assert!(record.preparation_deadline.is_none()); assert!(!allows_admission(&record, 300_000)); @@ -832,18 +906,18 @@ mod tests { #[test] fn preparation_over_five_minutes_gets_a_full_admission_repair_window() { - let record = new_preparation_record(0, 1800); + let record = new_preparation_record(0, 1800, 300_000); let mut timer = ProvisioningDeadline::from_record(&record).unwrap(); let attempt = record.attempt_id; let change_id = record.configuration_change_id; assert!(!timer.expire(&attempt, &change_id, 600_000)); assert!(!timer.ready(&attempt, &change_id, 600_000)); - assert!(timer.start_admission(&attempt, 600_000)); + assert!(timer.start_admission(&attempt, 600_000, 300_000)); assert_eq!(timer.deadline_at_ms(), Some(900_000)); - assert!(timer.rejected(&attempt, &change_id, 601_000)); + assert!(timer.rejected(&attempt, &change_id, 601_000, 300_000)); assert_eq!(timer.deadline_at_ms(), Some(901_000)); - assert!(!timer.start_admission(&attempt, 800_000)); - assert!(timer.configuration_changed(&attempt, change("updated", 800_000))); + assert!(!timer.start_admission(&attempt, 800_000, 300_000)); + assert!(timer.configuration_changed(&attempt, change("updated", 800_000), 300_000)); assert_eq!(timer.deadline_at_ms(), Some(1_100_000)); assert_eq!(timer.admission_start_at_ms, Some(600_000)); assert_eq!(timer.preparation_deadline_at_ms, Some(1_800_000)); @@ -852,15 +926,15 @@ mod tests { #[test] fn preparation_config_changes_and_restart_preserve_the_absolute_ceiling() { use prost::Message; - let mut record = new_preparation_record(0, 1800); + let mut record = new_preparation_record(0, 1800, 300_000); record.driver_operation_pending = true; record.driver_operation_id = "operation-1".into(); let mut timer = ProvisioningDeadline::from_record(&record).unwrap(); let attempt = record.attempt_id.clone(); - assert!(timer.configuration_changed(&attempt, change("first", 600_000))); - assert!(timer.configuration_changed(&attempt, change("last", 1_799_000))); - assert!(!timer.configuration_changed(&attempt, change("last", 1_799_500))); - assert!(!timer.rejected(&attempt, "last", 1_799_500)); + assert!(timer.configuration_changed(&attempt, change("first", 600_000), 300_000)); + assert!(timer.configuration_changed(&attempt, change("last", 1_799_000), 300_000)); + assert!(!timer.configuration_changed(&attempt, change("last", 1_799_500), 300_000)); + assert!(!timer.rejected(&attempt, "last", 1_799_500, 300_000)); timer.write_record(&mut record); let bytes = record.encode_to_vec(); let restored = SandboxProvisioning::decode(bytes.as_slice()).unwrap(); @@ -868,30 +942,30 @@ mod tests { assert_eq!(restored.driver_operation_id, "operation-1"); let mut timer = ProvisioningDeadline::from_record(&restored).unwrap(); assert_eq!(timer.deadline_at_ms(), Some(1_800_000)); - assert!(!timer.start_admission("previous-attempt", 1_799_999)); - assert!(!timer.start_admission(&attempt, 1_800_000)); + assert!(!timer.start_admission("previous-attempt", 1_799_999, 300_000)); + assert!(!timer.start_admission(&attempt, 1_800_000, 300_000)); assert!(timer.expire(&attempt, "last", 1_800_000)); - assert!(!timer.start_admission(&attempt, 1_799_999)); + assert!(!timer.start_admission(&attempt, 1_799_999, 300_000)); assert!(!timer.ready(&attempt, "last", 1_799_999)); - assert!(!timer.configuration_changed(&attempt, change("late", 1_799_999))); + assert!(!timer.configuration_changed(&attempt, change("late", 1_799_999), 300_000)); } #[test] fn admission_registration_roundtrip_does_not_restart_repair() { use prost::Message; - let mut record = new_preparation_record(0, 1800); - record_admission_start(&mut record, 600_000).unwrap(); + let mut record = new_preparation_record(0, 1800, 300_000); + record_admission_start(&mut record, 600_000, 300_000).unwrap(); let before = record.clone(); let bytes = record.encode_to_vec(); let mut restored = SandboxProvisioning::decode(bytes.as_slice()).unwrap(); - record_admission_start(&mut restored, 899_999).unwrap(); + record_admission_start(&mut restored, 899_999, 300_000).unwrap(); assert_eq!(restored, before); assert!(!allows_admission(&restored, 900_000)); } #[test] fn malformed_preparation_timing_cannot_grant_admission() { - let mut record = new_preparation_record(0, 1800); + let mut record = new_preparation_record(0, 1800, 300_000); record.deadline = timestamp_from_millis(1_800_001).ok(); assert!(ProvisioningDeadline::from_record(&record).is_err()); record.deadline = None; @@ -904,8 +978,9 @@ mod tests { #[test] fn protobuf_roundtrip_retains_deadline_and_cleanup_progress() { use prost::Message; - let mut deadline = ProvisioningDeadline::new("attempt".into(), change("change", 0), 0); - assert!(deadline.rejected("attempt", "change", 1_000)); + let mut deadline = + ProvisioningDeadline::new("attempt".into(), change("change", 0), 0, 300_000); + assert!(deadline.rejected("attempt", "change", 1_000, 300_000)); let mut record = SandboxProvisioning::default(); deadline.write_record(&mut record); let bytes = record.encode_to_vec(); @@ -929,7 +1004,7 @@ mod tests { #[test] fn malformed_persisted_timing_cannot_grant_another_window() { assert!(ProvisioningDeadline::from_record(&SandboxProvisioning::default()).is_err()); - let mut record = new_record(0); + let mut record = new_record(0, 300_000); record.timeout_time = timestamp_from_millis(300_000).ok(); assert!(ProvisioningDeadline::from_record(&record).is_err()); record.timeout_time = None; @@ -945,7 +1020,7 @@ mod tests { } fn initial() -> ProvisioningDeadline { - ProvisioningDeadline::new("attempt-1".into(), change("change-1", 0), 0) + ProvisioningDeadline::new("attempt-1".into(), change("change-1", 0), 0, 300_000) } #[test] @@ -959,9 +1034,9 @@ mod tests { #[test] fn repeated_failures_do_not_extend_the_repair_window() { let mut timer = initial(); - assert!(timer.rejected("attempt-1", "change-1", 5_000)); + assert!(timer.rejected("attempt-1", "change-1", 5_000, 300_000)); for now in (7_000..305_000).step_by(2_000) { - assert!(!timer.rejected("attempt-1", "change-1", now)); + assert!(!timer.rejected("attempt-1", "change-1", now, 300_000)); } assert_eq!(timer.deadline_at_ms(), Some(305_000)); assert!(timer.expire("attempt-1", "change-1", 305_000)); @@ -970,12 +1045,12 @@ mod tests { #[test] fn change_and_first_failed_load_each_reset_once() { let mut timer = initial(); - assert!(timer.configuration_changed("attempt-1", change("change-2", 100_000))); + assert!(timer.configuration_changed("attempt-1", change("change-2", 100_000), 300_000)); assert_eq!(timer.deadline_at_ms(), Some(400_000)); - assert!(timer.rejected("attempt-1", "change-2", 105_000)); + assert!(timer.rejected("attempt-1", "change-2", 105_000, 300_000)); assert_eq!(timer.deadline_at_ms(), Some(405_000)); - assert!(!timer.configuration_changed("attempt-1", change("change-2", 110_000))); - assert!(!timer.rejected("attempt-1", "change-2", 110_000)); + assert!(!timer.configuration_changed("attempt-1", change("change-2", 110_000), 300_000)); + assert!(!timer.rejected("attempt-1", "change-2", 110_000, 300_000)); assert_eq!(timer.deadline_at_ms(), Some(405_000)); } @@ -983,17 +1058,17 @@ mod tests { fn delayed_observation_uses_commit_time() { let mut timer = initial(); // Caller can observe this later, but it committed before the deadline. - assert!(timer.configuration_changed("attempt-1", change("change-2", 100_000))); + assert!(timer.configuration_changed("attempt-1", change("change-2", 100_000), 300_000)); assert_eq!(timer.deadline_at_ms(), Some(400_000)); - assert!(!timer.configuration_changed("attempt-1", change("older", 50_000))); + assert!(!timer.configuration_changed("attempt-1", change("older", 50_000), 300_000)); } #[test] fn stale_attempts_and_changes_cannot_extend_or_expire() { let mut timer = initial(); - assert!(!timer.rejected("old-attempt", "change-1", 1_000)); - assert!(!timer.rejected("attempt-1", "old-change", 1_000)); - assert!(!timer.configuration_changed("old-attempt", change("change-2", 1_000))); + assert!(!timer.rejected("old-attempt", "change-1", 1_000, 300_000)); + assert!(!timer.rejected("attempt-1", "old-change", 1_000, 300_000)); + assert!(!timer.configuration_changed("old-attempt", change("change-2", 1_000), 300_000)); assert!(!timer.expire("old-attempt", "change-1", 400_000)); assert!(!timer.expire("attempt-1", "old-change", 400_000)); assert_eq!(timer.deadline_at_ms(), Some(300_000)); @@ -1002,45 +1077,50 @@ mod tests { #[test] fn late_change_failure_or_ready_cannot_escape_expiry() { let mut timer = initial(); - assert!(!timer.configuration_changed("attempt-1", change("change-2", 300_000))); - assert!(!timer.rejected("attempt-1", "change-1", 300_000)); + assert!(!timer.configuration_changed("attempt-1", change("change-2", 300_000), 300_000)); + assert!(!timer.rejected("attempt-1", "change-1", 300_000, 300_000)); assert!(!timer.ready("attempt-1", "change-1", 300_000)); assert!(timer.expire("attempt-1", "change-1", 300_000)); assert!(!timer.ready("attempt-1", "change-1", 300_001)); - assert!(!timer.configuration_changed("attempt-1", change("change-2", 100_000))); + assert!(!timer.configuration_changed("attempt-1", change("change-2", 100_000), 300_000)); } #[test] fn ready_disarms_and_cannot_rearm_for_a_failed_live_update() { let mut timer = initial(); assert!(timer.ready("attempt-1", "change-1", 50_000)); - assert!(!timer.rejected("attempt-1", "change-1", 60_000)); - assert!(!timer.configuration_changed("attempt-1", change("change-2", 60_000))); + assert!(!timer.rejected("attempt-1", "change-1", 60_000, 300_000)); + assert!(!timer.configuration_changed("attempt-1", change("change-2", 60_000), 300_000)); assert!(!timer.expire("attempt-1", "change-1", 400_000)); } #[test] fn restart_roundtrip_preserves_first_failure_and_deadline() { let mut timer = initial(); - assert!(timer.rejected("attempt-1", "change-1", 5_000)); + assert!(timer.rejected("attempt-1", "change-1", 5_000, 300_000)); let encoded = serde_json::to_vec(&timer).unwrap(); let mut restored: ProvisioningDeadline = serde_json::from_slice(&encoded).unwrap(); assert_eq!(restored, timer); - assert!(!restored.rejected("attempt-1", "change-1", 100_000)); + assert!(!restored.rejected("attempt-1", "change-1", 100_000, 300_000)); assert_eq!(restored.deadline_at_ms(), Some(305_000)); } #[test] fn explicit_retry_is_a_new_attempt_and_old_cleanup_cannot_expire_it() { - let mut timer = - ProvisioningDeadline::new("attempt-2".into(), change("change-2", 350_000), 400_000); + let mut timer = ProvisioningDeadline::new( + "attempt-2".into(), + change("change-2", 350_000), + 400_000, + 300_000, + ); assert_eq!(timer.deadline_at_ms(), Some(700_000)); assert!(!timer.expire("attempt-1", "change-2", 800_000)); } #[test] fn legacy_initialization_grants_one_window_from_adoption() { - let timer = ProvisioningDeadline::new("adopted".into(), change("legacy", 0), 1_000_000); + let timer = + ProvisioningDeadline::new("adopted".into(), change("legacy", 0), 1_000_000, 300_000); assert_eq!(timer.deadline_at_ms(), Some(1_300_000)); } } diff --git a/crates/openshell-server/src/compute/tests/error_recovery.rs b/crates/openshell-server/src/compute/tests/error_recovery.rs index 8d269d1804..89a58632ea 100644 --- a/crates/openshell-server/src/compute/tests/error_recovery.rs +++ b/crates/openshell-server/src/compute/tests/error_recovery.rs @@ -371,7 +371,7 @@ async fn pending_driver_operation_blocks_error_recovery_without_mutation() { let driver = ControlledDriver::new(); let runtime = test_runtime(driver.clone()).await; let mut sandbox = error_sandbox_record("pending-id", "pending", "StartFailed"); - let mut operation = provisioning_deadline::new_record(openshell_core::time::now_ms()); + let mut operation = provisioning_deadline::new_record(openshell_core::time::now_ms(), 300_000); // Even a completion timestamp cannot override an unsettled original driver // operation. This also fences malformed/stale timeout cleanup evidence. operation.timeout_time = openshell_core::time::timestamp_from_millis(300_000).ok(); @@ -401,7 +401,7 @@ async fn timeout_stop_preserves_cleanup_and_direct_start_recovery() { driver.track_compute.store(true, Ordering::SeqCst); let runtime = test_runtime(driver.clone()).await; let mut sandbox = error_sandbox_record("timeout-id", "timeout", "ProvisioningTimedOut"); - let mut operation = provisioning_deadline::new_record(0); + let mut operation = provisioning_deadline::new_record(0, 300_000); operation.timeout_time = openshell_core::time::timestamp_from_millis(300_000).ok(); operation.cleanup_retry_time = openshell_core::time::timestamp_from_millis(310_000).ok(); if cleanup_complete { @@ -474,7 +474,7 @@ async fn reclaimed_timeout_can_still_start_directly_without_stop() { "direct-timeout", "ImagePreparationTimedOut", ); - let mut operation = provisioning_deadline::new_record(0); + let mut operation = provisioning_deadline::new_record(0, 300_000); operation.timeout_time = openshell_core::time::timestamp_from_millis(300_000).ok(); operation.cleanup_completed_time = openshell_core::time::timestamp_from_millis(320_000).ok(); sandbox.status.as_mut().unwrap().provisioning = Some(operation); diff --git a/crates/openshell-server/src/compute/tests/suspend_resume.rs b/crates/openshell-server/src/compute/tests/suspend_resume.rs index 0e59e4ec90..395a663493 100644 --- a/crates/openshell-server/src/compute/tests/suspend_resume.rs +++ b/crates/openshell-server/src/compute/tests/suspend_resume.rs @@ -685,9 +685,12 @@ async fn stale_resume_compensation_cannot_clear_a_newer_operation_intent() { let driver = ControlledDriver::new(); let runtime = suspend_capable_runtime(driver.clone()).await; let mut sandbox = sandbox_record("sb-resume-owner", "resume-owner", SandboxPhase::Starting); - sandbox.status.as_mut().unwrap().provisioning = Some( - provisioning_deadline::new_preparation_record(openshell_core::time::now_ms(), 1800), - ); + sandbox.status.as_mut().unwrap().provisioning = + Some(provisioning_deadline::new_preparation_record( + openshell_core::time::now_ms(), + 1800, + 300_000, + )); let generation = sandbox_runtime_generation(&sandbox).unwrap(); write_start_intent( &mut sandbox.metadata.as_mut().unwrap().annotations, diff --git a/crates/openshell-server/src/config_file.rs b/crates/openshell-server/src/config_file.rs index 865f6b1ed1..a2df105c55 100644 --- a/crates/openshell-server/src/config_file.rs +++ b/crates/openshell-server/src/config_file.rs @@ -146,9 +146,19 @@ pub struct GatewayFileSection { // ── Sandbox / SSH ──────────────────────────────────────────────────── #[serde(default)] pub ssh_session_ttl_secs: Option, + /// Per-sandbox cap on concurrent SSH relays; omit to keep the default of 20. + #[serde(default)] + pub max_ssh_connections_per_sandbox: Option, + /// Per-sandbox cap on concurrent service forward relays; omit to keep the + /// default of 20. + #[serde(default)] + pub max_service_connections_per_sandbox: Option, /// Absolute preparation budget for new attempts; existing deadlines persist. #[serde(default)] pub image_preparation_timeout_seconds: Option, + /// Admission repair window in seconds, between 1 and 86400. + #[serde(default)] + pub provisioning_repair_window_seconds: Option, /// Explicit activation only after the compatible deployment rollout ends. #[serde(default)] pub enable_sandbox_updates: Option, diff --git a/crates/openshell-server/src/grpc/mod.rs b/crates/openshell-server/src/grpc/mod.rs index 2efcb9873e..69208348de 100644 --- a/crates/openshell-server/src/grpc/mod.rs +++ b/crates/openshell-server/src/grpc/mod.rs @@ -84,7 +84,8 @@ use crate::ServerState; /// CAS conflicts (optimistic concurrency failures) are mapped to `ABORTED` /// to signal that the client should retry with fresh data. Mutation lock /// timeouts are mapped to `UNAVAILABLE` with a retry delay: the mutation lock -/// was not acquired, so this request's guarded writes did not run. Other +/// was not acquired, so this request's guarded writes did not run. So is a +/// database without a free connection, which ran no statement. Other /// persistence errors are mapped to `INTERNAL`. pub fn persistence_error_to_status( err: crate::persistence::PersistenceError, @@ -108,6 +109,11 @@ pub fn persistence_error_to_status( format!("{operation} timed out waiting for a concurrent mutation; retry the request"), std::time::Duration::from_secs(1), ), + PersistenceError::Unavailable(_) => openshell_core::rpc_error::unavailable( + "DATABASE_UNAVAILABLE", + format!("{operation} failed: the database is unavailable; retry the request"), + std::time::Duration::from_secs(1), + ), other => Status::internal(format!("{operation} failed: {other}")), } } @@ -1367,6 +1373,40 @@ mod tests { ); } + #[test] + fn persistence_error_to_status_maps_database_unavailable_to_unavailable() { + let status = persistence_error_to_status( + crate::persistence::PersistenceError::Unavailable( + "pool timed out while waiting for an open connection".into(), + ), + "workspace lookup", + ); + + assert_eq!(status.code(), tonic::Code::Unavailable); + assert_eq!( + status.message(), + "workspace lookup failed: the database is unavailable; retry the request" + ); + let details = openshell_core::rpc_error::decode_details(&status).expect("error details"); + assert_eq!( + details.error_info().expect("error info").reason, + "DATABASE_UNAVAILABLE" + ); + assert_eq!( + details.retry_info().expect("retry info").retry_delay, + Some(std::time::Duration::from_secs(1)) + ); + assert!( + !openshell_core::replica_routing::retry_unrouted( + Some(&tonic::metadata::AsciiMetadataValue::from_static( + "openshell-0" + )), + &status, + ), + "a gateway that answered must not look like an unrouted edge failure" + ); + } + #[test] fn public_resource_capabilities_preserves_absence() { let absent: Option = None; diff --git a/crates/openshell-server/src/grpc/policy.rs b/crates/openshell-server/src/grpc/policy.rs index cc83789417..ee6f07f9b1 100644 --- a/crates/openshell-server/src/grpc/policy.rs +++ b/crates/openshell-server/src/grpc/policy.rs @@ -5115,6 +5115,7 @@ pub(super) async fn handle_report_sandbox_configuration( &state.store, &mut sandbox, current_time_ms(), + state.config.provisioning_repair_window_seconds * 1000, ) .await .map_err(Status::internal)?; @@ -5208,11 +5209,19 @@ pub(super) async fn handle_report_sandbox_configuration( .map_err(Status::internal)?; return Err(Status::failed_precondition("provisioning deadline expired")); } - crate::compute::provisioning_deadline::record_admission_start(record, now_ms) - .map_err(Status::internal)?; + crate::compute::provisioning_deadline::record_admission_start( + record, + now_ms, + state.config.provisioning_repair_window_seconds * 1000, + ) + .map_err(Status::internal)?; if reported == ConfigurationAdmissionState::Rejected { - crate::compute::provisioning_deadline::record_rejection(record, now_ms) - .map_err(Status::internal)?; + crate::compute::provisioning_deadline::record_rejection( + record, + now_ms, + state.config.provisioning_repair_window_seconds * 1000, + ) + .map_err(Status::internal)?; } } let updated = state @@ -5455,6 +5464,35 @@ pub(super) async fn handle_get_sandbox_logs( Ok(Response::new(GetSandboxLogsResponse { logs, buffer_total })) } +/// How often an open push stream re-reads its sandbox record to detect that +/// the sandbox was deleted. +/// +/// Batches flush every 500ms, so this costs roughly one primary-key read per +/// stream per second, and only when a batch arrives. +const LOG_STREAM_EXISTENCE_RECHECK_PERIOD: std::time::Duration = std::time::Duration::from_secs(1); + +/// Mutable state for one `PushSandboxLogs` stream. +#[derive(Debug, Default)] +struct LogStreamState { + /// Sandbox id the first batch validated as belonging to the caller. + validated_sandbox_id: Option, + /// When the sandbox record was last confirmed to exist. + last_existence_check: Option, +} + +impl LogStreamState { + /// True when the cached existence proof is stale enough to re-read. + fn should_recheck(&self, now: std::time::Instant) -> bool { + self.last_existence_check.is_none_or(|checked_at| { + now.duration_since(checked_at) >= LOG_STREAM_EXISTENCE_RECHECK_PERIOD + }) + } + + fn mark_checked(&mut self, now: std::time::Instant) { + self.last_existence_check = Some(now); + } +} + pub(super) async fn handle_push_sandbox_logs( state: &Arc, request: Request>, @@ -5467,7 +5505,7 @@ pub(super) async fn handle_push_sandbox_logs( .cloned() .ok_or_else(|| Status::unauthenticated("missing principal"))?; let mut stream = request.into_inner(); - let mut validated_sandbox_id = None; + let mut stream_state = LogStreamState::default(); while let Some(batch) = stream .message() @@ -5478,11 +5516,15 @@ pub(super) async fn handle_push_sandbox_logs( continue; } - ensure_log_stream_sandbox_scope( + // Auth first: a cross-sandbox push must keep the audited + // PermissionDenied rather than learn about existence. + ensure_log_stream_sandbox_scope(state, &principal, &batch.sandbox_id, &mut stream_state) + .await?; + ensure_log_stream_sandbox_exists( state, - &principal, + &mut stream_state, &batch.sandbox_id, - &mut validated_sandbox_id, + std::time::Instant::now(), ) .await?; @@ -5511,6 +5553,55 @@ pub(super) async fn handle_push_sandbox_logs( Ok(Response::new(PushSandboxLogsResponse {})) } +/// End an open push stream once the sandbox behind it is gone. +/// +/// A tombstone from [`crate::tracing_bus::TracingLogBus::close`] is checked on +/// every batch and costs no store read. It ends the stream on the replica that +/// ran delete cleanup. The store is re-read at most once per +/// [`LOG_STREAM_EXISTENCE_RECHECK_PERIOD`], which ends a stream whose record +/// disappears while the stream is still open, as a best effort on the other +/// replicas. Freeing another replica's buffers is a separate concern (#5401); +/// only the replica that removed the record arms a permanent tombstone. +/// +/// Only a missing record ends the stream. A transient store failure is not +/// proof of absence, so it keeps the stream open and the next cadence retries. +/// +/// A missing sandbox row means the sandbox is gone. Several readers make that +/// inference with their own error policy -- delete cleanup, this recheck, and +/// the `WatchSandbox` producer in grpc/sandbox.rs -- so a change to deletion +/// semantics (soft delete, migration windows, record recreation) must update +/// every one of them. +async fn ensure_log_stream_sandbox_exists( + state: &Arc, + stream_state: &mut LogStreamState, + sandbox_id: &str, + now: std::time::Instant, +) -> Result<(), Status> { + if state.tracing_log_bus.is_closed(sandbox_id) { + return Err(Status::not_found("sandbox not found")); + } + if !stream_state.should_recheck(now) { + return Ok(()); + } + + match state.store.get(Sandbox::object_type(), sandbox_id).await { + Ok(Some(_)) => { + stream_state.mark_checked(now); + Ok(()) + } + Ok(None) => Err(Status::not_found("sandbox not found")), + Err(error) => { + warn!( + sandbox_id, + %error, + "Sandbox existence recheck failed; keeping the log stream open" + ); + stream_state.mark_checked(now); + Ok(()) + } + } +} + fn validate_sandbox_log_timestamps(logs: &[SandboxLogLine]) -> Result<(), Status> { for (index, log) in logs.iter().enumerate() { if let Some(event_time) = log.event_time.as_ref() { @@ -5522,16 +5613,21 @@ fn validate_sandbox_log_timestamps(logs: &[SandboxLogLine]) -> Result<(), Status Ok(()) } +/// Validate one batch's sandbox scope and, on the first batch, its existence. +/// +/// On the first batch this reads the sandbox record and marks the +/// existence-recheck clock, so the already-paid read also seeds the cadence +/// instead of the caller issuing a second one on the next batch. async fn ensure_log_stream_sandbox_scope( state: &Arc, principal: &Principal, sandbox_id: &str, - validated_sandbox_id: &mut Option, + stream_state: &mut LogStreamState, ) -> Result<(), Status> { // Guard every batch, so switching to another sandbox mid-stream is // recorded like any other cross-sandbox attempt. crate::auth::guard::ensure_sandbox_scope(principal, sandbox_id)?; - if let Some(validated) = validated_sandbox_id.as_deref() { + if let Some(validated) = stream_state.validated_sandbox_id.as_deref() { if sandbox_id != validated { return Err(Status::permission_denied( "log stream sandbox_id changed after validation", @@ -5546,7 +5642,8 @@ async fn ensure_log_stream_sandbox_scope( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - *validated_sandbox_id = Some(sandbox_id.to_string()); + stream_state.validated_sandbox_id = Some(sandbox_id.to_string()); + stream_state.mark_checked(std::time::Instant::now()); Ok(()) } @@ -8468,6 +8565,322 @@ mod tests { server.abort(); } + #[test] + fn log_stream_state_rechecks_only_after_the_period() { + let mut stream_state = LogStreamState::default(); + let now = std::time::Instant::now(); + assert!( + stream_state.should_recheck(now), + "a stream that never read the record must check" + ); + + stream_state.mark_checked(now); + assert!(!stream_state.should_recheck(now)); + let just_inside = now + + LOG_STREAM_EXISTENCE_RECHECK_PERIOD + .checked_sub(std::time::Duration::from_millis(1)) + .expect("the recheck period must be at least a millisecond"); + assert!(!stream_state.should_recheck(just_inside)); + assert!(stream_state.should_recheck(now + LOG_STREAM_EXISTENCE_RECHECK_PERIOD)); + } + + #[tokio::test] + async fn log_stream_tombstone_ends_stream_without_a_store_read() { + // The record is present, so only the tombstone can end the stream: a + // path that consulted the store first would report it alive. + let state = test_server_state().await; + let sandbox_id = "sb-recheck-tombstone"; + state + .store + .put_message(&test_sandbox( + sandbox_id, + "recheck-tombstone", + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + let mut stream_state = LogStreamState::default(); + let now = std::time::Instant::now(); + stream_state.mark_checked(now); + state.tracing_log_bus.close(sandbox_id); + + let error = ensure_log_stream_sandbox_exists(&state, &mut stream_state, sandbox_id, now) + .await + .expect_err("a tombstoned sandbox must end the stream on the next batch"); + + assert_eq!(error.code(), Code::NotFound); + } + + #[tokio::test] + async fn log_stream_recheck_ends_stream_when_the_record_is_gone() { + let state = test_server_state().await; + let sandbox_id = "sb-recheck-gone"; + let mut stream_state = LogStreamState::default(); + let checked_at = std::time::Instant::now(); + stream_state.mark_checked(checked_at); + + let error = ensure_log_stream_sandbox_exists( + &state, + &mut stream_state, + sandbox_id, + checked_at + LOG_STREAM_EXISTENCE_RECHECK_PERIOD, + ) + .await + .expect_err("a missing record must end the stream"); + + assert_eq!(error.code(), Code::NotFound); + assert!( + !state.tracing_log_bus.is_closed(sandbox_id), + "one absent sample must not arm a permanent tombstone" + ); + } + + #[tokio::test] + async fn log_stream_recheck_keeps_stream_when_the_record_still_exists() { + let state = test_server_state().await; + let sandbox_id = "sb-recheck-alive"; + state + .store + .put_message(&test_sandbox( + sandbox_id, + "recheck-alive", + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + let mut stream_state = LogStreamState::default(); + let checked_at = std::time::Instant::now(); + stream_state.mark_checked(checked_at); + let now = checked_at + LOG_STREAM_EXISTENCE_RECHECK_PERIOD; + + ensure_log_stream_sandbox_exists(&state, &mut stream_state, sandbox_id, now) + .await + .expect("a present record keeps the stream open"); + + assert!(!state.tracing_log_bus.is_closed(sandbox_id)); + assert!( + !stream_state.should_recheck(now), + "the confirmed read restarts the cadence" + ); + } + + #[tokio::test] + async fn log_stream_recheck_survives_a_transient_store_failure() { + let state = test_server_state().await; + let sandbox_id = "sb-recheck-transient"; + state + .store + .put_message(&test_sandbox( + sandbox_id, + "recheck-transient", + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + let mut stream_state = LogStreamState::default(); + let checked_at = std::time::Instant::now(); + stream_state.mark_checked(checked_at); + let now = checked_at + LOG_STREAM_EXISTENCE_RECHECK_PERIOD; + + // A failed read is not proof of absence: every later read fails too. + state.store.close().await; + + ensure_log_stream_sandbox_exists(&state, &mut stream_state, sandbox_id, now) + .await + .expect("a store failure must not end the stream"); + assert!(!state.tracing_log_bus.is_closed(sandbox_id)); + + ensure_log_stream_sandbox_exists( + &state, + &mut stream_state, + sandbox_id, + now + LOG_STREAM_EXISTENCE_RECHECK_PERIOD, + ) + .await + .expect("the stream must survive to the next batch"); + } + + #[tokio::test] + async fn push_sandbox_logs_stream_ends_not_found_after_sandbox_delete() { + use openshell_core::proto::open_shell_client::OpenShellClient; + use openshell_core::proto::open_shell_server::OpenShellServer; + use tokio_stream::wrappers::{ReceiverStream, TcpListenerStream}; + + let state = test_server_state().await; + let sandbox_id = "sb-log-deleted"; + state + .store + .put_message(&test_sandbox( + sandbox_id, + "log-deleted", + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let principal = crate::auth::principal::test_support::sandbox(sandbox_id); + let service = OpenShellServer::with_interceptor( + crate::grpc::OpenShellService::new(state.clone()), + move |mut request: Request<()>| { + request.extensions_mut().insert(principal.clone()); + Ok(request) + }, + ); + let server = tokio::spawn( + tonic::transport::Server::builder() + .add_service(service) + .serve_with_incoming(TcpListenerStream::new(listener)), + ); + let mut client = OpenShellClient::connect(format!("http://{address}")) + .await + .unwrap(); + + let (tx, rx) = tokio::sync::mpsc::channel(4); + let call = + tokio::spawn(async move { client.push_sandbox_logs(ReceiverStream::new(rx)).await }); + + let batch = |message: &str, millis: i64| PushSandboxLogsRequest { + sandbox_id: sandbox_id.to_string(), + logs: vec![SandboxLogLine { + sandbox_id: sandbox_id.to_string(), + event_time: openshell_core::time::timestamp_from_millis(millis).ok(), + level: "INFO".to_string(), + target: "delete-test".to_string(), + message: message.to_string(), + source: "sandbox".to_string(), + fields: HashMap::new(), + }], + }; + + tx.send(batch("before delete", 1)).await.unwrap(); + // Wait for the first batch to be ingested, so the second batch is the + // one that meets the tombstone. + tokio::time::timeout(std::time::Duration::from_secs(5), async { + while state.tracing_log_bus.tail(sandbox_id, 10).is_empty() { + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + }) + .await + .expect("the first batch must be ingested"); + + // What `cleanup_sandbox_state` does on the replica that serves the + // delete. + state.tracing_log_bus.close(sandbox_id); + + tx.send(batch("after delete", 2)).await.unwrap(); + + let error = call + .await + .expect("client task must not panic") + .expect_err("a deleted sandbox must end its push stream"); + assert_eq!(error.code(), Code::NotFound); + server.abort(); + } + + #[tokio::test] + async fn push_sandbox_logs_probes_a_sandbox_record_once_per_validated_batch() { + use crate::otel_tracing::test_exporter; + use openshell_core::proto::open_shell_client::OpenShellClient; + use openshell_core::proto::open_shell_server::OpenShellServer; + use tokio_stream::wrappers::{ReceiverStream, TcpListenerStream}; + + let state = test_server_state().await; + let sandbox_id = "sb-log-seeded-recheck"; + state + .store + .put_message(&test_sandbox( + sandbox_id, + "log-seeded-recheck", + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let principal = crate::auth::principal::test_support::sandbox(sandbox_id); + let service = OpenShellServer::with_interceptor( + crate::grpc::OpenShellService::new(state.clone()), + move |mut request: Request<()>| { + request.extensions_mut().insert(principal.clone()); + Ok(request) + }, + ); + let server = tokio::spawn( + tonic::transport::Server::builder() + .add_service(service) + .serve_with_incoming(TcpListenerStream::new(listener)), + ); + let mut client = OpenShellClient::connect(format!("http://{address}")) + .await + .unwrap(); + let (tx, rx) = tokio::sync::mpsc::channel(4); + let call = + tokio::spawn(async move { client.push_sandbox_logs(ReceiverStream::new(rx)).await }); + + // Count the sandbox-record reads the handler makes. The first batch's + // scope check already fetched the record, so the recheck clock must be + // seeded from it: a second read in the same batch would double what + // every open stream pays at startup. + let traced = test_exporter::install_traced(); + + tx.send(PushSandboxLogsRequest { + sandbox_id: sandbox_id.to_string(), + logs: vec![SandboxLogLine { + sandbox_id: sandbox_id.to_string(), + event_time: openshell_core::time::timestamp_from_millis(1).ok(), + level: "INFO".to_string(), + target: "seeded-recheck".to_string(), + message: "first batch".to_string(), + source: "sandbox".to_string(), + fields: HashMap::new(), + }], + }) + .await + .unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(5), async { + while state.tracing_log_bus.tail(sandbox_id, 10).is_empty() { + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + }) + .await + .expect("the first batch must be ingested"); + + let record_reads: Vec<_> = traced + .wait_for_spans(|spans| { + spans.iter().any(|span| { + span.name == "store.get" + && test_exporter::attribute(span, "object.id").as_deref() + == Some(sandbox_id) + }) + }) + .await + .into_iter() + .filter(|span| { + span.name == "store.get" + && test_exporter::attribute(span, "object.id").as_deref() == Some(sandbox_id) + }) + .collect(); + assert_eq!( + record_reads.len(), + 1, + "a validated batch must read the sandbox record once, not twice: the \ + scope check proves existence, so only the recheck-cadence seed prevents \ + a duplicate read per open stream" + ); + + drop(tx); + call.await + .expect("client task must not panic") + .expect("closing the client stream ends the push cleanly"); + server.abort(); + } + #[test] fn sandbox_log_timestamp_validation_rejects_malformed_batches_atomically() { let logs = vec![ @@ -8676,7 +9089,7 @@ mod tests { Vec::new(), ); sandbox.set_phase(SandboxPhase::Provisioning.into()); - let preparation = new_preparation_record(current_time_ms() - 600_000, 1800); + let preparation = new_preparation_record(current_time_ms() - 600_000, 1800, 300_000); sandbox.status.as_mut().unwrap().provisioning = Some(preparation.clone()); state.store.put_message(&sandbox).await.unwrap(); let instance_id = uuid::Uuid::new_v4().to_string(); @@ -8765,7 +9178,8 @@ mod tests { Vec::new(), ); sandbox.set_phase(SandboxPhase::Provisioning.into()); - sandbox.status.as_mut().unwrap().provisioning = Some(new_preparation_record(0, 1800)); + sandbox.status.as_mut().unwrap().provisioning = + Some(new_preparation_record(0, 1800, 300_000)); state.store.put_message(&sandbox).await.unwrap(); let error = handle_report_sandbox_configuration( &state, @@ -11990,15 +12404,15 @@ mod tests { } let req = with_sandbox(Request::new(()), "sb-a"); let principal = req.extensions().get::().unwrap().clone(); - let mut validated = None; + let mut stream_state = LogStreamState::default(); let capture = openshell_ocsf::test_support::OcsfCapture::default(); let _guard = capture.install(); - ensure_log_stream_sandbox_scope(&state, &principal, "sb-a", &mut validated) + ensure_log_stream_sandbox_scope(&state, &principal, "sb-a", &mut stream_state) .await .expect("first frame should validate"); assert!(capture.events().is_empty(), "own sandbox is not a denial"); - let err = ensure_log_stream_sandbox_scope(&state, &principal, "sb-b", &mut validated) + let err = ensure_log_stream_sandbox_scope(&state, &principal, "sb-b", &mut stream_state) .await .expect_err("later frame must not switch sandbox ids"); @@ -12014,9 +12428,9 @@ mod tests { let state = test_server_state().await; let req = with_sandbox(Request::new(()), "sb-a"); let principal = req.extensions().get::().unwrap().clone(); - let mut validated = None; + let mut stream_state = LogStreamState::default(); - let err = ensure_log_stream_sandbox_scope(&state, &principal, "sb-a", &mut validated) + let err = ensure_log_stream_sandbox_scope(&state, &principal, "sb-a", &mut stream_state) .await .expect_err("missing sandbox must not validate"); diff --git a/crates/openshell-server/src/grpc/policy/endpoint_status.rs b/crates/openshell-server/src/grpc/policy/endpoint_status.rs index 88efafada1..6d35131cc5 100644 --- a/crates/openshell-server/src/grpc/policy/endpoint_status.rs +++ b/crates/openshell-server/src/grpc/policy/endpoint_status.rs @@ -738,9 +738,11 @@ async fn invalidate_sandbox_endpoint_status_once( // Lifecycle writers on other replicas take no distributed guard, and a // delete between the re-read and the write surfaces as a database // error. Retry with a fresh read after this guard drops. - Err(error @ (PersistenceError::Conflict { .. } | PersistenceError::Database(_))) => { - Ok(StartupAttempt::Retry(error)) - } + Err( + error @ (PersistenceError::Conflict { .. } + | PersistenceError::Database(_) + | PersistenceError::Unavailable(_)), + ) => Ok(StartupAttempt::Retry(error)), Err(error) => Err(super::super::persistence_error_to_status( error, "invalidate tool server endpoint status during gateway startup", diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index 86ff12b170..35e3122bd4 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -56,7 +56,7 @@ use std::pin::Pin; use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; use std::task::{Context, Poll}; -use tokio::net::{TcpListener, TcpStream}; +use tokio::net::TcpListener; use tokio::sync::{broadcast, mpsc, oneshot}; use tokio_stream::wrappers::ReceiverStream; use tonic::{Request, Response, Status}; @@ -741,12 +741,14 @@ async fn handle_create_sandbox_inner( crate::compute::provisioning_deadline::new_preparation_record( now_ms, state.config.image_preparation_timeout_seconds, + state.config.provisioning_repair_window_seconds * 1000, ), ); crate::compute::provisioning_deadline::refresh_configuration( &state.store, &mut sandbox, now_ms, + state.config.provisioning_repair_window_seconds * 1000, ) .await .map_err(Status::internal)?; @@ -2484,7 +2486,7 @@ pub(super) async fn handle_watch_sandbox( // Events drained above the publication watermark. They are held // back rather than emitted so the client's highest delivered cursor // is never above an event still queued on the other source. - let mut deferred: Vec = Vec::new(); + let mut deferred: Vec> = Vec::new(); loop { // Events withheld by the previous round are already in hand, so @@ -2673,7 +2675,11 @@ pub(super) async fn handle_watch_sandbox( continue; } } - if tx.send(Ok(cursored.event)).await.is_err() { + if tx + .send(Ok(Arc::unwrap_or_clone(cursored).event)) + .await + .is_err() + { return; } } @@ -2911,9 +2917,48 @@ pub(super) async fn handle_forward_tcp( Ok(Response::new(stream)) } +/// The per-sandbox relay budget a forward stream draws from. SSH relays and +/// service forwards are counted apart so open forwards cannot starve SSH. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum RelayBudget { + Ssh, + Service, +} + +impl RelayBudget { + fn of(init: &TcpForwardInit) -> Self { + match init.target { + Some(tcp_forward_init::Target::Tcp(_)) => Self::Service, + _ => Self::Ssh, + } + } + + fn counts(self, state: &ServerState) -> &std::sync::Mutex> { + match self { + Self::Ssh => &state.ssh_connections_by_sandbox, + Self::Service => &state.service_connections_by_sandbox, + } + } + + fn limit(self, config: &openshell_core::Config) -> u32 { + match self { + Self::Ssh => config.max_ssh_connections_per_sandbox, + Self::Service => config.max_service_connections_per_sandbox, + } + } + + const fn exhausted_message(self) -> &'static str { + match self { + Self::Ssh => "sandbox SSH connection limit reached", + Self::Service => "sandbox service forward connection limit reached", + } + } +} + struct ForwardConnectionGuard { state: Arc, token: Option, + budget: RelayBudget, sandbox_id: String, } @@ -2922,7 +2967,7 @@ impl Drop for ForwardConnectionGuard { if let Some(token) = self.token.as_deref() { decrement_ssh_connection_count(&self.state.ssh_connections_by_token, token); } - decrement_ssh_connection_count(&self.state.ssh_connections_by_sandbox, &self.sandbox_id); + decrement_ssh_connection_count(self.budget.counts(&self.state), &self.sandbox_id); } } @@ -2931,7 +2976,8 @@ impl Drop for ForwardConnectionGuard { /// `target.tcp` streams are authorized by the caller's gateway principal, which /// `handle_forward_tcp` has already checked against the sandbox's workspace, so /// they carry no session token and touch no store: the only per-connection -/// state is the in-memory per-sandbox connection count. `target.ssh` streams +/// state is the in-memory per-sandbox connection count, kept in a budget apart +/// from SSH relays (see [`RelayBudget`]). `target.ssh` streams /// keep requiring the `CreateSshSession` token because the process that opens /// them is an ssh `ProxyCommand` that holds nothing else. A token supplied with /// a TCP target (older clients) is still validated and counted per token. @@ -2942,9 +2988,10 @@ async fn acquire_forward_connection_guard( sandbox: &Sandbox, ) -> Result { let sandbox_id = sandbox.object_id().to_string(); + let budget = RelayBudget::of(init); let token = init.authorization_token.trim(); if token.is_empty() { - if !matches!(init.target, Some(tcp_forward_init::Target::Tcp(_))) { + if budget == RelayBudget::Ssh { return Err(crate::auth::audit::reject_credential( Some(principal), AuthRejection::new( @@ -2954,10 +3001,11 @@ async fn acquire_forward_connection_guard( ), )); } - acquire_sandbox_connection_slot(&state.ssh_connections_by_sandbox, &sandbox_id)?; + acquire_sandbox_connection_slot(state, budget, &sandbox_id)?; return Ok(ForwardConnectionGuard { state: state.clone(), token: None, + budget, sandbox_id, }); } @@ -2966,16 +3014,12 @@ async fn acquire_forward_connection_guard( .await .map_err(|rejection| crate::auth::audit::reject_credential(Some(principal), rejection))?; crate::auth::audit::emit_ssh_session_logon(&state.logons, principal, token); - acquire_ssh_connection_slots( - &state.ssh_connections_by_token, - &state.ssh_connections_by_sandbox, - token, - &sandbox_id, - )?; + acquire_ssh_connection_slots(state, budget, token, &sandbox_id)?; Ok(ForwardConnectionGuard { state: state.clone(), token: Some(token.to_string()), + budget, sandbox_id, }) } @@ -3039,16 +3083,15 @@ async fn validate_ssh_forward_token( } const MAX_CONNECTIONS_PER_TOKEN: u32 = 3; -const MAX_CONNECTIONS_PER_SANDBOX: u32 = 20; fn acquire_ssh_connection_slots( - token_counts: &std::sync::Mutex>, - sandbox_counts: &std::sync::Mutex>, + state: &ServerState, + budget: RelayBudget, token: &str, sandbox_id: &str, ) -> Result<(), Status> { { - let mut counts = token_counts.lock().unwrap(); + let mut counts = state.ssh_connections_by_token.lock().unwrap(); let count = counts.entry(token.to_string()).or_insert(0); if *count >= MAX_CONNECTIONS_PER_TOKEN { return Err(Status::resource_exhausted( @@ -3058,26 +3101,24 @@ fn acquire_ssh_connection_slots( *count += 1; } - if let Err(status) = acquire_sandbox_connection_slot(sandbox_counts, sandbox_id) { - decrement_ssh_connection_count(token_counts, token); + if let Err(status) = acquire_sandbox_connection_slot(state, budget, sandbox_id) { + decrement_ssh_connection_count(&state.ssh_connections_by_token, token); return Err(status); } Ok(()) } -/// Reserve one of the per-sandbox relay slots shared by SSH sessions and -/// service forwards. +/// Reserve one of the sandbox's relay slots in the given budget. fn acquire_sandbox_connection_slot( - sandbox_counts: &std::sync::Mutex>, + state: &ServerState, + budget: RelayBudget, sandbox_id: &str, ) -> Result<(), Status> { - let mut counts = sandbox_counts.lock().unwrap(); + let mut counts = budget.counts(state).lock().unwrap(); let count = counts.entry(sandbox_id.to_string()).or_insert(0); - if *count >= MAX_CONNECTIONS_PER_SANDBOX { - return Err(Status::resource_exhausted( - "sandbox SSH connection limit reached", - )); + if *count >= budget.limit(&state.config) { + return Err(Status::resource_exhausted(budget.exhausted_message())); } *count += 1; Ok(()) @@ -3889,12 +3930,10 @@ async fn run_interactive_exec_with_russh( ))); } - let stream = TcpStream::connect(("127.0.0.1", local_proxy_port)) - .await - .map_err(|e| Status::internal(format!("failed to connect to ssh proxy: {e}")))?; - // russh client end of the loopback exec bridge — disable Nagle so keystroke - // and PTY tinygrams don't stall on delayed ACKs. - set_tcp_nodelay_best_effort(&stream); + let stream = + openshell_core::net::connect_tcp_nodelay_best_effort(("127.0.0.1", local_proxy_port)) + .await + .map_err(|e| Status::internal(format!("failed to connect to ssh proxy: {e}")))?; let config = Arc::new(exec_ssh_client_config()); let remote_sshid = Arc::new(std::sync::Mutex::new(None)); @@ -4149,12 +4188,10 @@ async fn run_exec_with_russh( ))); } - let stream = TcpStream::connect(("127.0.0.1", local_proxy_port)) - .await - .map_err(|e| Status::internal(format!("failed to connect to ssh proxy: {e}")))?; - // russh client end of the loopback exec bridge — disable Nagle so keystroke - // and PTY tinygrams don't stall on delayed ACKs. - set_tcp_nodelay_best_effort(&stream); + let stream = + openshell_core::net::connect_tcp_nodelay_best_effort(("127.0.0.1", local_proxy_port)) + .await + .map_err(|e| Status::internal(format!("failed to connect to ssh proxy: {e}")))?; let config = Arc::new(exec_ssh_client_config()); let remote_sshid = Arc::new(std::sync::Mutex::new(None)); @@ -4681,9 +4718,9 @@ mod tests { }) } - fn sandbox_connection_count(state: &ServerState, sandbox_id: &str) -> u32 { - state - .ssh_connections_by_sandbox + fn sandbox_connection_count(state: &ServerState, budget: RelayBudget, sandbox_id: &str) -> u32 { + budget + .counts(state) .lock() .unwrap() .get(sandbox_id) @@ -4710,10 +4747,13 @@ mod tests { "no per-token accounting without a token" ); assert!(state.ssh_connections_by_token.lock().unwrap().is_empty()); - assert_eq!(sandbox_connection_count(&state, sandbox.object_id()), 1); + assert_eq!( + sandbox_connection_count(&state, RelayBudget::Service, sandbox.object_id()), + 1 + ); drop(guard); assert_eq!( - sandbox_connection_count(&state, sandbox.object_id()), + sandbox_connection_count(&state, RelayBudget::Service, sandbox.object_id()), 0, "dropping the guard releases the sandbox slot" ); @@ -4808,7 +4848,10 @@ mod tests { err.message(), "authorization_token is required for SSH targets" ); - assert_eq!(sandbox_connection_count(&state, sandbox.object_id()), 0); + assert_eq!( + sandbox_connection_count(&state, RelayBudget::Ssh, sandbox.object_id()), + 0 + ); assert_ssh_token_failure(&events, "missing_credential", ""); } @@ -4827,7 +4870,10 @@ mod tests { assert_eq!(err.code(), tonic::Code::Unauthenticated); assert_eq!(err.message(), "SSH session token not found"); - assert_eq!(sandbox_connection_count(&state, sandbox.object_id()), 0); + assert_eq!( + sandbox_connection_count(&state, RelayBudget::Service, sandbox.object_id()), + 0 + ); assert_ssh_token_failure(&events, "invalid_credential", "canary-ssh-token-unknown"); } @@ -4941,13 +4987,14 @@ mod tests { } #[tokio::test] - async fn tcp_forwards_without_token_share_the_per_sandbox_connection_cap() { + async fn tcp_forwards_are_capped_per_sandbox() { let state = test_server_state().await; let sandbox = test_sandbox("fwd", Vec::new()); let init = forward_init(loopback_tcp_target(), ""); + let cap = openshell_core::config::DEFAULT_MAX_CONNECTIONS_PER_SANDBOX; let mut guards = Vec::new(); - for _ in 0..MAX_CONNECTIONS_PER_SANDBOX { + for _ in 0..cap { guards.push( acquire_forward_connection_guard(&state, &forward_caller(), &init, &sandbox) .await @@ -4959,10 +5006,13 @@ mod tests { .err() .expect("cap reached"); assert_eq!(err.code(), tonic::Code::ResourceExhausted); - assert_eq!(err.message(), "sandbox SSH connection limit reached"); assert_eq!( - sandbox_connection_count(&state, sandbox.object_id()), - MAX_CONNECTIONS_PER_SANDBOX + err.message(), + "sandbox service forward connection limit reached" + ); + assert_eq!( + sandbox_connection_count(&state, RelayBudget::Service, sandbox.object_id()), + cap ); guards.pop(); @@ -4971,7 +5021,65 @@ mod tests { .expect("slot freed by a closed connection"); drop(guard); drop(guards); - assert_eq!(sandbox_connection_count(&state, sandbox.object_id()), 0); + assert_eq!( + sandbox_connection_count(&state, RelayBudget::Service, sandbox.object_id()), + 0 + ); + } + + #[tokio::test] + async fn ssh_and_service_budgets_are_independent() { + let mut state = test_server_state().await; + { + let config = &mut Arc::get_mut(&mut state).expect("sole state owner").config; + config.max_ssh_connections_per_sandbox = 1; + config.max_service_connections_per_sandbox = 2; + } + let sandbox = test_sandbox("fwd", Vec::new()); + store_ssh_session(&state, "canary-ssh-token-a", "sandbox-fwd", false, 0).await; + store_ssh_session(&state, "canary-ssh-token-b", "sandbox-fwd", false, 0).await; + let tcp = forward_init(loopback_tcp_target(), ""); + let ssh = |token: &str| { + forward_init( + tcp_forward_init::Target::Ssh(SshRelayTarget::default()), + token, + ) + }; + let acquire = |init: TcpForwardInit| { + let state = state.clone(); + let sandbox = sandbox.clone(); + async move { + acquire_forward_connection_guard(&state, &forward_caller(), &init, &sandbox).await + } + }; + + let _forwards = [ + acquire(tcp.clone()).await.expect("first forward"), + acquire(tcp.clone()).await.expect("second forward"), + ]; + let err = acquire(tcp).await.err().expect("forward budget spent"); + assert_eq!( + err.message(), + "sandbox service forward connection limit reached" + ); + + let _ssh = acquire(ssh("canary-ssh-token-a")) + .await + .expect("open forwards leave the SSH budget alone"); + let err = acquire(ssh("canary-ssh-token-b")) + .await + .err() + .expect("SSH budget spent"); + assert_eq!(err.code(), tonic::Code::ResourceExhausted); + assert_eq!(err.message(), "sandbox SSH connection limit reached"); + assert_eq!( + sandbox_connection_count(&state, RelayBudget::Ssh, sandbox.object_id()), + 1 + ); + assert_eq!( + sandbox_connection_count(&state, RelayBudget::Service, sandbox.object_id()), + 2 + ); } #[test] diff --git a/crates/openshell-server/src/grpc/workspace.rs b/crates/openshell-server/src/grpc/workspace.rs index 56156f7d3f..9f3be64703 100644 --- a/crates/openshell-server/src/grpc/workspace.rs +++ b/crates/openshell-server/src/grpc/workspace.rs @@ -130,7 +130,7 @@ pub async fn resolve_workspace( let ws: Option = store .get_message_by_name("", &name) .await - .map_err(|e| Status::internal(format!("workspace lookup failed: {e}")))?; + .map_err(|e| super::persistence_error_to_status(e, "workspace lookup"))?; match ws { Some(ws) => { diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index 02c1ba16b3..8ecfbb0b99 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -355,6 +355,9 @@ pub struct ServerState { /// Active SSH tunnel connection counts per sandbox id. pub ssh_connections_by_sandbox: Mutex>, + /// Active service forward connection counts per sandbox id. + pub service_connections_by_sandbox: Mutex>, + /// Registry of active supervisor sessions and pending relay channels. /// /// Stored as `Arc` so compiled compute drivers can be constructed before @@ -656,6 +659,7 @@ impl ServerState { telemetry: telemetry::TelemetryState::new(), ssh_connections_by_token: Mutex::new(HashMap::new()), ssh_connections_by_sandbox: Mutex::new(HashMap::new()), + service_connections_by_sandbox: Mutex::new(HashMap::new()), supervisor_sessions, gateway_shutting_down: AtomicBool::new(false), config_delivery, @@ -827,14 +831,21 @@ pub(crate) async fn run_server( .map_err(|error| Error::config(format!("middleware registration failed: {error}")))?, ); - let store = Arc::new( - Store::connect_with_pool_sizes( - database_url, - config.database_max_connections, - config.database_lock_max_connections, - ) - .await?, - ); + let shutdown = shutdown_signal(); + tokio::pin!(shutdown); + let mut health_listener = HealthListener::new(config.health_bind_address); + let Some(store) = connect_store( + &config, + database_url, + &mut health_listener, + shutdown.as_mut(), + ) + .await? + else { + info!("Shutdown signal received while waiting for the database; stopping gateway"); + return Ok(()); + }; + let store = Arc::new(store); let credentials = credentials::CredentialRuntime::from_config_file_with_store( &config, config_file.as_ref(), @@ -1153,26 +1164,7 @@ pub(crate) async fn run_server( // Create the multiplexed service let service = MultiplexService::new(state.clone()); - // Bind the unauthenticated health endpoint on a separate port when configured. - if let Some(health_bind_address) = config.health_bind_address { - let health_listener = TcpListener::bind(health_bind_address).await.map_err(|e| { - Error::transport(format!( - "failed to bind health port {health_bind_address}: {e}" - )) - })?; - info!(address = %health_bind_address, "Health server listening"); - // `health_router` returns immediately; the listener serves - // `Initializing → 503` until the background monitor publishes the - // first real probe outcome, so the endpoint is always responsive. - let router = health_router(store.clone()); - tokio::spawn(async move { - if let Err(e) = axum::serve(health_listener, router.into_make_service()).await { - error!("Health server error: {e}"); - } - }); - } else { - info!("Health server disabled"); - } + health_listener.serve(store.clone()).await?; // Bind the Prometheus metrics endpoint on a dedicated port when configured. if let Some((metrics_bind_address, prometheus_handle)) = metrics_endpoint { @@ -1311,7 +1303,7 @@ pub(crate) async fn run_server( provider_profile_sources::spawn_refresh(state.clone()); provider_refresh::spawn_refresh_worker(state.clone(), Duration::from_mins(1)); - shutdown_signal().await; + shutdown.await; info!("Shutdown signal received; stopping gateway"); state.gateway_shutting_down.store(true, Ordering::Release); state.supervisor_sessions.close_admission(); @@ -1607,6 +1599,112 @@ fn spawn_gateway_connection( } } +/// First delay between attempts to reach an unreachable database at startup. +const DATABASE_CONNECT_RETRY_INITIAL_DELAY: Duration = Duration::from_secs(1); + +/// Longest delay between attempts to reach an unreachable database at startup. +const DATABASE_CONNECT_RETRY_MAX_DELAY: Duration = Duration::from_secs(2); + +/// Connect to the database, retrying while it is unreachable. +/// +/// The first failed attempt binds the health endpoint and starts listening +/// for `shutdown`; a gateway running as PID 1 ignores SIGTERM until then. +/// Returns `None` if `shutdown` completes first. +async fn connect_store( + config: &Config, + database_url: &str, + health_listener: &mut HealthListener, + mut shutdown: std::pin::Pin<&mut impl Future>, +) -> Result> { + let mut delay = DATABASE_CONNECT_RETRY_INITIAL_DELAY; + let mut waiting = false; + loop { + let attempt = Store::try_connect_with_pool_sizes( + database_url, + config.database_max_connections, + config.database_lock_max_connections, + ); + let result = if waiting { + tokio::select! { + () = &mut shutdown => return Ok(None), + result = attempt => result?, + } + } else { + attempt.await? + }; + let error = match result { + Ok(store) => return Ok(Some(store)), + Err(error @ persistence::PersistenceError::Unavailable(_)) => error, + Err(error) => return Err(Error::execution(error.to_string())), + }; + if !waiting { + health_listener.bind().await?; + waiting = true; + } + warn!( + error = %error, + retry_in_ms = u64::try_from(delay.as_millis()).unwrap_or(u64::MAX), + "database unreachable; retrying" + ); + tokio::select! { + () = &mut shutdown => return Ok(None), + () = tokio::time::sleep(delay) => {} + } + delay = (delay * 2).min(DATABASE_CONNECT_RETRY_MAX_DELAY); + } +} + +/// The unauthenticated health endpoint, served on its own port when +/// configured. +struct HealthListener { + address: Option, + state: Option>, +} + +impl HealthListener { + const fn new(address: Option) -> Self { + Self { + address, + state: None, + } + } + + /// Start serving probes, with `/readyz` reporting `Initializing` until + /// [`Self::serve`] publishes the database's health. + async fn bind(&mut self) -> Result<()> { + let Some(address) = self.address else { + return Ok(()); + }; + if self.state.is_some() { + return Ok(()); + } + let listener = TcpListener::bind(address) + .await + .map_err(|e| Error::transport(format!("failed to bind health port {address}: {e}")))?; + info!(address = %address, "Health server listening"); + let (sender, receiver) = watch::channel(readiness::HealthState::Initializing); + let router = http::health_router_from_receiver(receiver); + tokio::spawn(async move { + if let Err(e) = axum::serve(listener, router.into_make_service()).await { + error!("Health server error: {e}"); + } + }); + self.state = Some(sender); + Ok(()) + } + + /// Bind if not yet serving and publish `store`'s health from now on. + async fn serve(mut self, store: Arc) -> Result<()> { + self.bind().await?; + if let Some(sender) = self.state { + readiness::DatabaseHealthMonitor::spawn_into(store, sender); + } else { + info!("Health server disabled"); + } + Ok(()) + } +} + async fn shutdown_signal() { #[cfg(unix)] { @@ -1779,19 +1877,6 @@ impl ComputeDriverRegistration { }) } - /// Compatibility no-op retained for source compatibility with schema-v1 - /// factory registrations. Schema v2 never inherits gateway keys into a - /// driver table; move every driver setting under - /// `[openshell.drivers.]`. - #[deprecated( - since = "0.0.0", - note = "schema v2 does not inherit gateway keys into driver configuration" - )] - #[must_use] - pub fn with_inherited_config_keys(self, _keys: &'static [&'static str]) -> Self { - self - } - /// Assign a bounded telemetry category chosen by the binary composition /// boundary. Runtime driver names are never used as telemetry values. #[must_use] @@ -1800,13 +1885,6 @@ impl ComputeDriverRegistration { self } - /// Compatibility no-op retained for existing factory registrations. - /// Gateway mTLS user authentication is independent of compute drivers. - #[must_use] - pub fn with_local_singleplayer(self) -> Self { - self - } - /// Attach process-wide tracing for this in-process compiled driver. #[must_use] pub fn with_in_process_tracing( @@ -2284,6 +2362,10 @@ async fn build_compute_runtime( .and_then(|runtime| { runtime.with_image_preparation_timeout(config.image_preparation_timeout_seconds) }) + .and_then(|runtime| { + runtime + .with_provisioning_repair_window_seconds(config.provisioning_repair_window_seconds) + }) .map_err(Error::config)?; Ok(runtime.with_telemetry_compute_driver(telemetry_compute_driver)) } @@ -2539,7 +2621,7 @@ mod tests { use std::time::{Duration, Instant}; use tempfile::{TempDir, tempdir}; use tokio::io::{AsyncReadExt, AsyncWriteExt}; - use tokio::net::{TcpListener, TcpStream}; + use tokio::net::TcpListener; use tokio::sync::watch; use tokio_rustls::TlsConnector; @@ -3089,7 +3171,7 @@ mod tests { let connect_addr: SocketAddr = format!("127.0.0.1:{}", addr.port()) .parse() .expect("failed to build loopback connect addr"); - let mut stream = TcpStream::connect(connect_addr) + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(connect_addr) .await .expect("failed to connect to test listener"); stream @@ -3298,7 +3380,7 @@ mod tests { )); let connector = test_tls_connector(&tls_dir.path().join("ca.pem"), None); - let stream = TcpStream::connect(address) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) .await .expect("failed to connect to metrics listener"); let mut tls_stream = connector @@ -3324,7 +3406,7 @@ mod tests { "expected a successful HTTPS metrics response" ); - let mut plaintext = TcpStream::connect(address) + let mut plaintext = openshell_core::net::connect_tcp_nodelay_best_effort(address) .await .expect("failed to connect for plaintext request"); plaintext @@ -3403,7 +3485,7 @@ mod tests { &directory.path().join("ca.pem"), Some((&metrics_client_cert, &metrics_client_key)), ); - let stream = TcpStream::connect(address) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) .await .expect("failed to connect with metrics client certificate"); let mut metrics_client = metrics_connector @@ -3433,7 +3515,7 @@ mod tests { &directory.path().join("ca.pem"), Some((&gateway_client_cert, &gateway_client_key)), ); - let stream = TcpStream::connect(address) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) .await .expect("failed to connect with gateway client certificate"); let rejected_response = match gateway_connector @@ -3467,7 +3549,7 @@ mod tests { ); let anonymous_connector = test_tls_connector(&directory.path().join("ca.pem"), None); - let stream = TcpStream::connect(address) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) .await .expect("failed to connect without a client certificate"); let anonymous_response = match anonymous_connector @@ -3978,6 +4060,158 @@ mod tests { "persisted sandbox start must not run before the gateway listener is bound" ); } + + /// A loopback address where nothing listens until a test binds it. + async fn reserved_loopback_address() -> SocketAddr { + TcpListener::bind("127.0.0.1:0") + .await + .expect("reserve a loopback port") + .local_addr() + .expect("loopback address") + } + + /// Status code of `GET path`, or `None` when nothing answers. + async fn http_status(address: SocketAddr, path: &str) -> Option { + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(address) + .await + .ok()?; + stream + .write_all( + format!("GET {path} HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n") + .as_bytes(), + ) + .await + .ok()?; + let mut response = String::new(); + stream.read_to_string(&mut response).await.ok()?; + response.split_whitespace().nth(1)?.parse().ok() + } + + async fn wait_for_http_status(address: SocketAddr, path: &str, expected: u16) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(20); + while http_status(address, path).await != Some(expected) { + assert!( + tokio::time::Instant::now() < deadline, + "{path} never returned {expected}" + ); + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + + #[tokio::test] + async fn connect_store_serves_probes_while_the_database_is_unreachable() { + let url = crate::persistence::PostgresStore::refusing_url_for_tests().await; + let health_address = reserved_loopback_address().await; + let mut health = super::HealthListener::new(Some(health_address)); + let (stop, stopped) = tokio::sync::oneshot::channel::<()>(); + let shutdown = async { + let _ = stopped.await; + }; + tokio::pin!(shutdown); + let config = Config::new(None); + + let probe = async { + wait_for_http_status(health_address, "/healthz", 200).await; + assert_eq!(http_status(health_address, "/readyz").await, Some(503)); + stop.send(()).expect("connect_store is still waiting"); + }; + let (result, ()) = tokio::join!( + super::connect_store(&config, &url, &mut health, shutdown.as_mut()), + probe + ); + + assert!( + result.expect("shutdown is not an error").is_none(), + "shutdown ends the wait without a store" + ); + } + + #[tokio::test] + async fn connect_store_returns_a_configuration_error_without_retrying() { + let health_address = reserved_loopback_address().await; + let mut health = super::HealthListener::new(Some(health_address)); + let shutdown = std::future::pending::<()>(); + tokio::pin!(shutdown); + + let result = tokio::time::timeout( + Duration::from_secs(5), + super::connect_store( + &Config::new(None), + "postgres://openshell@127.0.0.1:1/openshell?sslmode=invalid", + &mut health, + shutdown.as_mut(), + ), + ) + .await + .expect("a configuration error is not retried"); + + assert!(result.is_err()); + assert_eq!(http_status(health_address, "/healthz").await, None); + } + + #[tokio::test] + #[ignore = "requires a disposable PostgreSQL database in OPENSHELL_TEST_POSTGRES_URL; run mise run test:rust:postgres"] + async fn postgres_connect_store_retries_until_the_database_answers() { + let schema = crate::persistence::test_postgres::TestSchema::create("startup").await; + let upstream = url::Url::parse(schema.url()).expect("parse the PostgreSQL URL"); + let upstream_addresses: Vec = tokio::net::lookup_host(( + upstream.host_str().expect("TCP PostgreSQL host"), + upstream.port().unwrap_or(5432), + )) + .await + .expect("resolve PostgreSQL") + .collect(); + let late_address = reserved_loopback_address().await; + let mut url = upstream.clone(); + url.set_host(Some("127.0.0.1")).expect("set host"); + url.set_port(Some(late_address.port())).expect("set port"); + let health_address = reserved_loopback_address().await; + let mut health = super::HealthListener::new(Some(health_address)); + let shutdown = std::future::pending::<()>(); + tokio::pin!(shutdown); + let config = Config::new(None); + + // PostgreSQL becomes reachable only after the gateway reports that it + // is waiting. + let late_database = async { + wait_for_http_status(health_address, "/healthz", 200).await; + assert_eq!(http_status(health_address, "/readyz").await, Some(503)); + let listener = TcpListener::bind(late_address) + .await + .expect("bind the late PostgreSQL address"); + loop { + let (mut client, _) = listener.accept().await.expect("accept"); + openshell_core::net::set_tcp_nodelay_best_effort(&client); + let mut upstream = openshell_core::net::connect_tcp_nodelay_best_effort( + upstream_addresses.as_slice(), + ) + .await + .expect("connect to PostgreSQL"); + tokio::spawn(async move { + let _ = tokio::io::copy_bidirectional(&mut client, &mut upstream).await; + }); + } + }; + let store = tokio::select! { + result = tokio::time::timeout( + Duration::from_mins(1), + super::connect_store(&config, url.as_str(), &mut health, shutdown.as_mut()), + ) => result + .expect("connect_store finishes once PostgreSQL answers") + .expect("connect") + .expect("no shutdown"), + () = late_database => unreachable!("the forwarder runs until the store connects"), + }; + store.ping().await.expect("ping the connected store"); + + health + .serve(Arc::new(store)) + .await + .expect("keep serving the bound health listener"); + wait_for_http_status(health_address, "/readyz", 200).await; + + schema.drop_schema().await; + } } #[cfg(test)] diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 438b7d4bc6..6e33dc0051 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -1795,7 +1795,9 @@ mod tests { path: &str, headers: &[(&str, &str)], ) -> Response { - let stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let (mut sender, conn) = hyper::client::conn::http1::Builder::new() .handshake(TokioIo::new(stream)) .await diff --git a/crates/openshell-server/src/persistence/mod.rs b/crates/openshell-server/src/persistence/mod.rs index 30280b96bb..d61451604d 100644 --- a/crates/openshell-server/src/persistence/mod.rs +++ b/crates/openshell-server/src/persistence/mod.rs @@ -69,6 +69,11 @@ pub enum PersistenceError { /// request's guarded writes did not run; the operation is safe to retry. #[error("mutation lock timeout: {0}")] LockTimeout(String), + /// The database is not serving. At request time, no connection was + /// available before the pool's acquire timeout, so the failed statement + /// was never sent. + #[error("database unavailable: {0}")] + Unavailable(String), } impl PersistenceError { @@ -475,6 +480,22 @@ impl Store { max_connections: Option, lock_max_connections: Option, ) -> CoreResult { + Self::try_connect_with_pool_sizes(url, max_connections, lock_max_connections) + .await? + .map_err(|e| CoreError::execution(e.to_string())) + } + + /// One attempt of [`Self::connect_with_pool_sizes`]. + /// + /// Configuration errors fail the outer result. Connection and migration + /// errors keep their [`PersistenceError`] kind in the inner result, so a + /// caller can retry [`PersistenceError::Unavailable`] until the database + /// is reachable. + pub(crate) async fn try_connect_with_pool_sizes( + url: &str, + max_connections: Option, + lock_max_connections: Option, + ) -> CoreResult> { if url.starts_with("postgres://") || url.starts_with("postgresql://") { if let Some(configured) = max_connections && configured < postgres::MIN_MAX_CONNECTIONS @@ -486,28 +507,25 @@ impl Store { postgres::MIN_MAX_CONNECTIONS ))); } - let store = PostgresStore::connect(url, max_connections, lock_max_connections) - .await - .map_err(|e| CoreError::execution(e.to_string()))?; - store - .migrate() - .await - .map_err(|e| CoreError::execution(e.to_string()))?; - Ok(Self::Postgres(store)) + Ok(async { + let store = + PostgresStore::connect(url, max_connections, lock_max_connections).await?; + store.migrate().await?; + Ok(Self::Postgres(store)) + } + .await) } else if url.starts_with("sqlite:") { if lock_max_connections.is_some() { tracing::warn!( "ignoring the configured mutation lock pool ceiling: SQLite has no lock pool" ); } - let store = SqliteStore::connect(url, max_connections) - .await - .map_err(|e| CoreError::execution(e.to_string()))?; - store - .migrate() - .await - .map_err(|e| CoreError::execution(e.to_string()))?; - Ok(Self::Sqlite(store)) + Ok(async { + let store = SqliteStore::connect(url, max_connections).await?; + store.migrate().await?; + Ok(Self::Sqlite(store)) + } + .await) } else { Err(CoreError::config(format!( "unsupported database URL scheme: {url}" @@ -1446,6 +1464,9 @@ pub fn current_time_ms() -> i64 { } fn map_db_error(error: &sqlx::Error) -> PersistenceError { + if matches!(error, sqlx::Error::PoolTimedOut) { + return PersistenceError::Unavailable(error.to_string()); + } if let sqlx::Error::Database(db) = error && db.is_unique_violation() { diff --git a/crates/openshell-server/src/persistence/mutation_lock_pg_tests.rs b/crates/openshell-server/src/persistence/mutation_lock_pg_tests.rs index 3bf37d1a33..0d05b4d05c 100644 --- a/crates/openshell-server/src/persistence/mutation_lock_pg_tests.rs +++ b/crates/openshell-server/src/persistence/mutation_lock_pg_tests.rs @@ -17,7 +17,7 @@ use super::mutation_lock::{ GLOBAL_MUTATION_LOCK_KEY, LOCK_CONNECTION_MIN_BUDGET, MUTATION_LOCK_POOL_MAX_CONNECTIONS, MUTATION_LOCK_TIMEOUT, MutationLockKey, MutationLockSet, }; -use super::postgres::LOCK_CONNECTION_RELEASE_TIMEOUT; +use super::postgres::{DATA_POOL_ACQUIRE_TIMEOUT, LOCK_CONNECTION_RELEASE_TIMEOUT}; use super::test_postgres::TestSchema; use super::{ DistributedMutationGuard, PersistenceError, PersistenceResult, PostgresStore, Store, @@ -220,7 +220,7 @@ impl StallingProxy { let stalled = receiver.clone(); connections.spawn(async move { let upstream = openshell_core::net::connect_tcp_nodelay_best_effort( - &addresses, + addresses.as_slice(), ).await?; let (mut client_read, mut client_write) = client.into_split(); let (mut upstream_read, mut upstream_write) = upstream.into_split(); @@ -1261,6 +1261,77 @@ async fn postgres_mutation_lock_connection_failure_is_not_a_lock_timeout() { fixture.finish(vec![saturated, unanswered]).await; } +/// A data-pool caller whose `PostgreSQL` stops answering fails as retryable +/// within the acquire timeout, and the next caller after it answers again +/// gets a fresh connection. +#[tokio::test] +#[ignore = "requires a disposable PostgreSQL database in OPENSHELL_TEST_POSTGRES_URL; run mise run test:rust:postgres"] +async fn postgres_data_pool_fails_unavailable_while_postgres_does_not_answer() { + let schema = TestSchema::create("unavailable").await; + let proxy = StallingProxy::start(schema.url()).await; + let store = Store::Postgres( + PostgresStore::connect_with_lock_pool_size(&proxy.url, None, 1) + .await + .expect("connect a store through the proxy"), + ); + store.ping().await.expect("ping before the network stalls"); + + proxy.stalled.send_replace(true); + let started = Instant::now(); + let result = store.ping().await; + let waited = started.elapsed(); + assert!( + matches!(result, Err(PersistenceError::Unavailable(_))), + "{result:?}" + ); + assert!( + waited < DATA_POOL_ACQUIRE_TIMEOUT + Duration::from_secs(2), + "ping waited {waited:?}" + ); + + proxy.stalled.send_replace(false); + tokio::time::timeout(Duration::from_secs(5), store.ping()) + .await + .expect("ping after the network recovers is bounded") + .expect("ping after the network recovers"); + + drop(store); + schema.drop_schema().await; +} + +/// Migrating against a read-only server, as a standby before promotion, +/// fails as retryable rather than as a fatal migration error. +#[tokio::test] +#[ignore = "requires a disposable PostgreSQL database in OPENSHELL_TEST_POSTGRES_URL; run mise run test:rust:postgres"] +async fn postgres_migrate_on_a_read_only_server_fails_unavailable() { + let schema = TestSchema::create("read_only").await; + let mut url = url::Url::parse(schema.url()).expect("parse the PostgreSQL URL"); + let pairs: Vec<(String, String)> = url + .query_pairs() + .map(|(key, value)| { + let value = if key == "options" { + format!("{value} -cdefault_transaction_read_only=on") + } else { + value.into_owned() + }; + (key.into_owned(), value) + }) + .collect(); + url.query_pairs_mut().clear().extend_pairs(pairs); + let store = PostgresStore::connect(url.as_str(), None, None) + .await + .expect("connect to the read-only server"); + + let result = store.migrate().await; + assert!( + matches!(result, Err(PersistenceError::Unavailable(_))), + "{result:?}" + ); + + drop(store); + schema.drop_schema().await; +} + #[tokio::test] #[ignore = "requires a disposable PostgreSQL database in OPENSHELL_TEST_POSTGRES_URL; run mise run test:rust:postgres"] async fn postgres_mutation_lock_unrelated_sandboxes_outpace_global_serialization() { diff --git a/crates/openshell-server/src/persistence/postgres.rs b/crates/openshell-server/src/persistence/postgres.rs index ca070bedd6..45511726fd 100644 --- a/crates/openshell-server/src/persistence/postgres.rs +++ b/crates/openshell-server/src/persistence/postgres.rs @@ -275,6 +275,46 @@ pub const DEFAULT_MAX_CONNECTIONS: u32 = 10; /// wait on itself until the acquire times out. pub const MIN_MAX_CONNECTIONS: u32 = 2; +/// How long a data-pool caller waits for a connection, including opening one. +pub(super) const DATA_POOL_ACQUIRE_TIMEOUT: Duration = Duration::from_secs(5); + +/// Whether a connect or migration failure means `PostgreSQL` is not serving +/// yet, so startup retries it. A bad host, credentials, or certificate stays +/// fatal. +fn is_unavailable_at_startup(error: &sqlx::Error) -> bool { + use std::io::ErrorKind; + match error { + sqlx::Error::PoolTimedOut => true, + sqlx::Error::Io(error) => matches!( + error.kind(), + ErrorKind::ConnectionRefused + | ErrorKind::ConnectionReset + | ErrorKind::ConnectionAborted + | ErrorKind::NotConnected + | ErrorKind::BrokenPipe + | ErrorKind::TimedOut + | ErrorKind::UnexpectedEof + | ErrorKind::HostUnreachable + | ErrorKind::NetworkUnreachable + ), + // read_only_sql_transaction, too_many_connections, admin_shutdown, + // crash_shutdown, cannot_connect_now. + sqlx::Error::Database(error) => matches!( + error.code().as_deref(), + Some("25006" | "53300" | "57P01" | "57P02" | "57P03") + ), + _ => false, + } +} + +fn map_startup_error(error: &sqlx::Error) -> PersistenceError { + if is_unavailable_at_startup(error) { + PersistenceError::Unavailable(error.to_string()) + } else { + map_db_error(error) + } +} + impl PostgresStore { /// Connect and size the data and lock pools, falling back to /// [`DEFAULT_MAX_CONNECTIONS`] and `MUTATION_LOCK_POOL_MAX_CONNECTIONS`. @@ -304,9 +344,10 @@ impl PostgresStore { ); let pool = PgPoolOptions::new() .max_connections(max_connections) + .acquire_timeout(DATA_POOL_ACQUIRE_TIMEOUT) .connect(url) .await - .map_err(|e| map_db_error(&e))?; + .map_err(|e| map_startup_error(&e))?; let lock_pool = PgPoolOptions::new() .max_connections(lock_pool_size) .min_connections(0) @@ -351,22 +392,30 @@ impl PostgresStore { POSTGRES_MIGRATOR .run(&self.pool) .await - .map_err(|e| map_migrate_error(&e))?; + .map_err(|e| match &e { + sqlx::migrate::MigrateError::Execute(error) + | sqlx::migrate::MigrateError::ExecuteMigration(error, _) + if is_unavailable_at_startup(error) => + { + PersistenceError::Unavailable(e.to_string()) + } + _ => map_migrate_error(&e), + })?; self.migrate_legacy_time_payloads().await } async fn migrate_legacy_time_payloads(&self) -> PersistenceResult<()> { - let mut transaction = self.pool.begin().await.map_err(|e| map_db_error(&e))?; + let mut transaction = self.pool.begin().await.map_err(|e| map_startup_error(&e))?; // Serialize this application-level data migration across gateway replicas. sqlx::query("SELECT pg_advisory_xact_lock(3052)") .execute(&mut *transaction) .await - .map_err(|e| map_db_error(&e))?; + .map_err(|e| map_startup_error(&e))?; let rows = sqlx::query("SELECT id, object_type, payload FROM objects ORDER BY id FOR UPDATE") .fetch_all(&mut *transaction) .await - .map_err(|e| map_db_error(&e))?; + .map_err(|e| map_startup_error(&e))?; for row in rows { let id: String = row.try_get("id").map_err(|e| map_db_error(&e))?; @@ -384,11 +433,14 @@ impl PostgresStore { .bind(id) .execute(&mut *transaction) .await - .map_err(|e| map_db_error(&e))?; + .map_err(|e| map_startup_error(&e))?; } } - transaction.commit().await.map_err(|e| map_db_error(&e)) + transaction + .commit() + .await + .map_err(|e| map_startup_error(&e)) } /// Verify the database is reachable by acquiring a pooled connection @@ -2283,6 +2335,55 @@ impl PostgresStore { mod tests { use super::*; + #[tokio::test] + async fn connect_to_a_refusing_server_fails_unavailable_within_the_acquire_timeout() { + let url = PostgresStore::refusing_url_for_tests().await; + let started = std::time::Instant::now(); + + let error = PostgresStore::connect(&url, None, None) + .await + .expect_err("nothing listens on the port"); + + assert!( + matches!(error, PersistenceError::Unavailable(_)), + "{error:?}" + ); + assert!( + started.elapsed() < DATA_POOL_ACQUIRE_TIMEOUT + Duration::from_secs(3), + "connect waited {:?}", + started.elapsed() + ); + } + + #[test] + fn startup_retries_io_errors_only_while_postgres_does_not_answer() { + use std::io::ErrorKind; + let io = |kind: ErrorKind| map_startup_error(&sqlx::Error::Io(kind.into())); + for kind in [ + ErrorKind::ConnectionRefused, + ErrorKind::ConnectionReset, + ErrorKind::TimedOut, + ErrorKind::HostUnreachable, + ErrorKind::UnexpectedEof, + ] { + assert!( + matches!(io(kind), PersistenceError::Unavailable(_)), + "{kind:?}" + ); + } + // An unresolvable host, a missing certificate file, a failed TLS handshake. + for kind in [ + ErrorKind::Other, + ErrorKind::NotFound, + ErrorKind::InvalidData, + ] { + assert!( + matches!(io(kind), PersistenceError::Database(_)), + "{kind:?}" + ); + } + } + #[tokio::test] async fn lock_connection_timeout_is_contention_only_if_the_pool_filled_during_the_wait() { let store = PostgresStore::connect_lazy_for_tests( diff --git a/crates/openshell-server/src/persistence/sandbox_image.rs b/crates/openshell-server/src/persistence/sandbox_image.rs index 386fe78136..70877c4d52 100644 --- a/crates/openshell-server/src/persistence/sandbox_image.rs +++ b/crates/openshell-server/src/persistence/sandbox_image.rs @@ -854,7 +854,8 @@ mod tests { #[test] fn image_completion_preserves_concurrent_attachment_clock_edits() { let mut original = sandbox(); - let mut previous = crate::compute::provisioning_deadline::new_preparation_record(0, 1800); + let mut previous = + crate::compute::provisioning_deadline::new_preparation_record(0, 1800, 300_000); previous.driver_operation_id = "prior-operation".into(); previous.attachment_change_id = "before-attachment".into(); original.status.as_mut().unwrap().provisioning = Some(previous); diff --git a/crates/openshell-server/src/readiness.rs b/crates/openshell-server/src/readiness.rs index 7ca21605a8..fed1ffd102 100644 --- a/crates/openshell-server/src/readiness.rs +++ b/crates/openshell-server/src/readiness.rs @@ -129,6 +129,17 @@ impl DatabaseHealthMonitor { Self { receiver: rx } } + /// Spawn a monitor with the production defaults that publishes to a + /// channel whose receivers already serve probes. + pub fn spawn_into(store: Arc, sender: watch::Sender) { + tokio::spawn(monitor_loop( + store, + sender, + DEFAULT_CHECK_INTERVAL, + DEFAULT_CHECK_TIMEOUT, + )); + } + /// Subscribe to state updates. /// /// Returned receivers always observe the latest value with no lock diff --git a/crates/openshell-server/src/tls.rs b/crates/openshell-server/src/tls.rs index b8850044ae..f2de2e18e8 100644 --- a/crates/openshell-server/src/tls.rs +++ b/crates/openshell-server/src/tls.rs @@ -677,7 +677,7 @@ mod tests { let addr = listen_addr; client_handles.push(tokio::spawn(async move { for _ in 0..cycles_per_client { - let tcp = TcpStream::connect(addr) + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(addr) .await .expect("client connect failed"); let _tls = connector @@ -747,7 +747,9 @@ mod tests { .await .expect("TLS accept failed"); }); - let tcp_1 = TcpStream::connect(addr1).await.expect("connect failed"); + let tcp_1 = openshell_core::net::connect_tcp_nodelay_best_effort(addr1) + .await + .expect("connect failed"); let tls_1 = connector .connect(server_name.clone(), tcp_1) .await @@ -773,7 +775,9 @@ mod tests { .await .expect("TLS accept failed"); }); - let tcp_2 = TcpStream::connect(addr2).await.expect("connect failed"); + let tcp_2 = openshell_core::net::connect_tcp_nodelay_best_effort(addr2) + .await + .expect("connect failed"); let tls_2 = connector .connect(server_name.clone(), tcp_2) .await @@ -872,7 +876,7 @@ mod tests { acc.acceptor().accept(stream).await }); - let tcp = TcpStream::connect(listen_addr) + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(listen_addr) .await .expect("connect failed"); let handshake_result = tokio::time::timeout( @@ -1004,7 +1008,7 @@ mod tests { let connector = tokio_rustls::TlsConnector::from(new_ca_client_config); let server_name = rustls::pki_types::ServerName::try_from("localhost").expect("invalid server name"); - let tcp = TcpStream::connect(listen_addr) + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(listen_addr) .await .expect("connect failed"); connector @@ -1079,7 +1083,7 @@ mod tests { // server rejects the client cert (TLS 1.3 post-handshake auth), // so the authoritative check is the server-side accept result above. let connector = tokio_rustls::TlsConnector::from(old_ca_client_config); - let tcp = TcpStream::connect(listen_addr) + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(listen_addr) .await .expect("connect failed"); let _ = connector.connect(server_name, tcp).await; @@ -1296,7 +1300,7 @@ mod tests { }); let connector = tokio_rustls::TlsConnector::from(client_config.clone()); - let tcp = TcpStream::connect(listen_addr) + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(listen_addr) .await .expect("connect failed"); let server_name = "external.example.com" @@ -1332,7 +1336,7 @@ mod tests { }); let connector = tokio_rustls::TlsConnector::from(client_config); - let tcp = TcpStream::connect(listen_addr) + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(listen_addr) .await .expect("connect failed"); let server_name = "localhost".try_into().expect("invalid server name"); diff --git a/crates/openshell-server/src/tracing_bus.rs b/crates/openshell-server/src/tracing_bus.rs index 92d30f5571..cb3ecd40dc 100644 --- a/crates/openshell-server/src/tracing_bus.rs +++ b/crates/openshell-server/src/tracing_bus.rs @@ -41,10 +41,17 @@ pub(crate) struct CursoredEvent { pub(crate) event: SandboxStreamEvent, } +/// Slot capacity of a per-sandbox broadcast channel. +const BROADCAST_CAPACITY: usize = 1024; + #[derive(Debug, Clone)] struct PerSandbox { - sender: broadcast::Sender, - tail: VecDeque, + /// Created on first subscribe. Every slot is allocated with the channel, + /// so a sandbox nobody watches must not pay for one. + sender: Option>>, + /// Shares each event's allocation with the channel; an inline + /// `CursoredEvent` is sized for the full `Sandbox` payload variant. + tail: VecDeque>, /// Highest seq this bus has evicted from `tail`. 0 = nothing trimmed. /// /// Under the shared cursor space each bus's tail is non-contiguous in the @@ -55,13 +62,42 @@ struct PerSandbox { impl PerSandbox { fn new() -> Self { - let (tx, _rx) = broadcast::channel(1024); Self { - sender: tx, + sender: None, tail: VecDeque::new(), last_trimmed_seq: 0, } } + + fn subscribe(&mut self) -> broadcast::Receiver> { + self.sender + .get_or_insert_with(|| broadcast::channel(BROADCAST_CAPACITY).0) + .subscribe() + } + + fn push(&mut self, cursored: CursoredEvent, tail_cap: usize) { + let cursored = Arc::new(cursored); + if let Some(sender) = &self.sender { + let _ = sender.send(Arc::clone(&cursored)); + } + self.tail.push_back(cursored); + while self.tail.len() > tail_cap { + if let Some(trimmed) = self.tail.pop_front() { + self.last_trimmed_seq = trimmed.seq; + } + } + } + + fn tail(&self, max: usize) -> Vec> { + let skip = self.tail.len().saturating_sub(max); + self.tail.iter().skip(skip).cloned().collect() + } +} + +/// Deep-clone shared events. Call only after releasing the bus lock, so a +/// large replay does not stall publishers. +fn unshare(events: Vec>) -> Vec { + events.into_iter().map(Arc::unwrap_or_clone).collect() } /// The requested resume cursor is older than the oldest buffered event; @@ -84,6 +120,22 @@ pub struct ResumeGap { struct CursorSpace { epoch: Uuid, next: u64, + /// Terminal tombstone written by [`TracingLogBus::close`]. A closed space + /// never allocates another cursor, so late publishers cannot re-create the + /// buffers deletion just freed. + closed: bool, +} + +impl CursorSpace { + /// A bare tombstone for an id that never published, so a subscribe-only + /// sandbox can be closed too. No epoch is ever handed out for it. + fn tombstone() -> Self { + Self { + epoch: Uuid::nil(), + next: 1, + closed: true, + } + } } /// Identity and extent of a sandbox's current cursor space. @@ -108,8 +160,9 @@ struct SeqAllocator { impl SeqAllocator { /// Lock the cursor space. /// - /// Publication and teardown each hold this guard across their bus-map - /// mutation, which is what keeps cursors monotonic. Allocating and then + /// Publication, subscription, and teardown each hold this guard across + /// their bus-map mutation, which is what keeps cursors monotonic and makes + /// the closed check atomic with the entry it guards. Allocating and then /// releasing would let a teardown reset the counter in between, so the /// in-flight event lands in a freshly recreated entry carrying a cursor from /// the old space while the next publish restarts at 1. @@ -128,38 +181,70 @@ impl SeqAllocator { /// The epoch is minted on the vacant path only — once per cursor space, not /// once per publish — so generating it under the allocator lock costs /// nothing on the hot path. - fn next_locked(spaces: &mut HashMap, sandbox_id: &str) -> (Uuid, u64) { + /// + /// Returns `None` for a closed space: the timeline is over and the caller + /// must drop the event rather than re-create the buffers [`TracingLogBus::close`] + /// freed. + fn next_locked( + spaces: &mut HashMap, + sandbox_id: &str, + ) -> Option<(Uuid, u64)> { let space = spaces .entry(sandbox_id.to_string()) .or_insert_with(|| CursorSpace { epoch: Uuid::new_v4(), next: 1, + closed: false, }); + if space.closed { + return None; + } let seq = space.next; space.next += 1; - (space.epoch, seq) + Some((space.epoch, seq)) } /// Identity and extent of this sandbox's space, or `None` when no event has - /// been published into it. + /// been published into it or the space is closed. /// /// A resume cursor is validated against this: a different epoch means the /// cursor belongs to a space that no longer exists, and `None` means there /// is no space to resume into at all. An empty `tail_after` result is not /// on its own proof that a cursor is still valid. fn space(&self, sandbox_id: &str) -> Option { - self.lock().get(sandbox_id).map(|space| CursorSpaceInfo { - epoch: space.epoch, - highest_seq: space.next.saturating_sub(1), - }) + self.lock() + .get(sandbox_id) + .filter(|space| !space.closed) + .map(|space| CursorSpaceInfo { + epoch: space.epoch, + highest_seq: space.next.saturating_sub(1), + }) } + + /// True when `sandbox_id` carries a tombstone in the locked space map. + /// + /// Shared by `subscribe` (which must test it atomically with the bus-map + /// insert) and `is_closed`, so the predicate has one definition. + fn is_closed_locked(spaces: &HashMap, sandbox_id: &str) -> bool { + spaces.get(sandbox_id).is_some_and(|space| space.closed) + } +} + +/// A receiver that is already closed, for a tombstoned id. +/// +/// Nothing is ever sent for a closed id, so the sender is dropped immediately +/// and the receiver reports `Closed` on first use. Capacity is irrelevant. +fn closed_receiver() -> broadcast::Receiver> { + let (sender, receiver) = broadcast::channel(1); + drop(sender); + receiver } fn tail_after_impl( - tail: &VecDeque, + tail: &VecDeque>, last_trimmed_seq: u64, after_seq: u64, -) -> Result, ResumeGap> { +) -> Result>, ResumeGap> { // Gap iff this bus dropped an event the client still needs, i.e. the // highest seq we evicted is newer than the client's position. Judged only // on this bus's own evictions — the other bus owns the seqs missing here. @@ -173,7 +258,7 @@ fn tail_after_impl( // Skippable events (seq <= after_seq) are the oldest, at the front, so a // take-while would stop before reaching the wanted ones. Filter the whole // tail instead; order is preserved and caught-up yields an empty vec. - let res: Vec = tail + let res: Vec> = tail .iter() .filter(|cursored| cursored.seq > after_seq) .cloned() @@ -210,18 +295,77 @@ impl TracingLogBus { } } - fn sender_for(&self, sandbox_id: &str) -> broadcast::Sender { + pub(crate) fn subscribe(&self, sandbox_id: &str) -> broadcast::Receiver> { + // Hold the allocator across the bus-map insert so the closed check is + // atomic with a concurrent [`Self::close`], mirroring `publish`. A + // closed id yields a receiver that is already closed instead of a new + // entry: re-creating the entry would undo the memory `close` reclaimed. + let spaces = self.seq.lock(); + if SeqAllocator::is_closed_locked(&spaces, sandbox_id) { + return closed_receiver(); + } let mut inner = self.inner.lock().expect("tracing bus lock poisoned"); inner .per_id .entry(sandbox_id.to_string()) .or_insert_with(PerSandbox::new) - .sender - .clone() + .subscribe() + } + + /// Terminally close a sandbox's log and platform event streams. + /// + /// This is the deletion path: it frees both buses' buffers and leaves a + /// tombstone in the shared cursor space, so a late publisher — an open + /// `PushSandboxLogs` stream, a driver event queued after `Deleted`, the + /// gateway tracing layer including OCSF audit events — drops its event + /// instead of re-creating the ~1.7 MiB of channel and tail that nothing + /// would ever free again. Idempotent. + /// + /// Lifecycle: a sandbox's space stays open until [`Self::remove`] retires + /// it or this call closes it. `remove` is the reset path (re-publishing + /// mints a new epoch, as the resume tests pin). `closed` is terminal: + /// `publish`, `subscribe`, and the platform bus all refuse the id, the + /// cursor-space lookup reports no space, and `remove` leaves the tombstone + /// in place. + /// + /// Tombstones are unbounded on purpose. Each costs roughly a hundred + /// bytes, against the ~1.7 MiB of buffers it replaces; sandbox ids are + /// fresh UUIDs that are never reused, and all bus state is cleared on + /// gateway restart. + /// + /// The whole sequence runs under the cursor-space lock, so it is atomic + /// against publication and subscription on either bus. The platform bus + /// removal only takes its own map lock, never the allocator, so the + /// documented allocator -> bus map order holds. + pub fn close(&self, sandbox_id: &str) { + let mut spaces = self.seq.lock(); + self.drop_bus_entries_locked(sandbox_id); + spaces + .entry(sandbox_id.to_string()) + .and_modify(|space| space.closed = true) + .or_insert_with(CursorSpace::tombstone); + } + + /// Drop both buses' per-sandbox entries. + /// + /// The caller holds the cursor-space lock. The platform bus removal takes + /// only its own map lock and never reaches for the allocator, so this + /// preserves the documented allocator -> bus map order. + fn drop_bus_entries_locked(&self, sandbox_id: &str) { + { + let mut inner = self.inner.lock().expect("tracing bus lock poisoned"); + inner.per_id.remove(sandbox_id); + } + self.platform_event_bus.remove(sandbox_id); } - pub(crate) fn subscribe(&self, sandbox_id: &str) -> broadcast::Receiver { - self.sender_for(sandbox_id).subscribe() + /// True once [`Self::close`] has run for this sandbox. + /// + /// The push-stream handler uses this as a cheap, lock-only existence + /// check: a tombstoned id can never gain a buffer again, so it needs no + /// store read. + pub fn is_closed(&self, sandbox_id: &str) -> bool { + SeqAllocator::is_closed_locked(&self.seq.lock(), sandbox_id) } /// Remove all bus entries for the given sandbox id, including the platform @@ -242,27 +386,29 @@ impl TracingLogBus { /// replacement space. Both broadcast senders are dropped here too, so a /// stream that was live across the reset ends rather than silently /// continuing into a different space. + /// + /// [`Self::close`] is the terminal counterpart: a removed space can be + /// republished into, a closed one cannot. On a closed id this is a no-op on + /// the allocator, because dropping the tombstone there would let a late + /// publisher re-create the buffers deletion just freed. pub fn remove(&self, sandbox_id: &str) { let mut spaces = self.seq.lock(); - { - let mut inner = self.inner.lock().expect("tracing bus lock poisoned"); - inner.per_id.remove(sandbox_id); + self.drop_bus_entries_locked(sandbox_id); + if spaces.get(sandbox_id).is_none_or(|space| !space.closed) { + spaces.remove(sandbox_id); } - // Takes only the platform bus map lock; never reaches for `spaces`. - self.platform_event_bus.remove(sandbox_id); - spaces.remove(sandbox_id); } pub(crate) fn tail(&self, sandbox_id: &str, max: usize) -> Vec { - let inner = self.inner.lock().expect("tracing bus lock poisoned"); - inner + let events = self + .inner + .lock() + .expect("tracing bus lock poisoned") .per_id .get(sandbox_id) - .map(|d| d.tail.iter().rev().take(max).cloned().collect::>()) - .unwrap_or_default() - .into_iter() - .rev() - .collect::>() + .map(|per| per.tail(max)) + .unwrap_or_default(); + unshare(events) } /// Identity and extent of this sandbox's current cursor space. @@ -279,11 +425,17 @@ impl TracingLogBus { sandbox_id: &str, after_seq: u64, ) -> Result, ResumeGap> { - let inner = self.inner.lock().expect("tracing bus lock poisoned"); - inner.per_id.get(sandbox_id).map_or_else( - || Ok(Vec::new()), - |per| tail_after_impl(&per.tail, per.last_trimmed_seq, after_seq), - ) + let events = self + .inner + .lock() + .expect("tracing bus lock poisoned") + .per_id + .get(sandbox_id) + .map_or_else( + || Ok(Vec::new()), + |per| tail_after_impl(&per.tail, per.last_trimmed_seq, after_seq), + )?; + Ok(unshare(events)) } /// Publish a log line from an external source (e.g., sandbox push). @@ -309,25 +461,21 @@ impl TracingLogBus { fn publish(&self, sandbox_id: &str, mut event: SandboxStreamEvent, tail_cap: usize) { // Hold the cursor space across the tail insert so a teardown cannot // reset the counter between allocation and insertion. Lock order is - // allocator -> bus map, matching `remove`. + // allocator -> bus map, matching `remove` and `close`. let mut spaces = self.seq.lock(); - let (epoch, seq) = SeqAllocator::next_locked(&mut spaces, sandbox_id); + // A closed space is over: drop the event instead of allocating a + // cursor and re-creating the buffers `close` freed. + let Some((epoch, seq)) = SeqAllocator::next_locked(&mut spaces, sandbox_id) else { + return; + }; event.cursor = WatchCursor::new(epoch, seq).encode(); let mut inner = self.inner.lock().expect("tracing bus lock poisoned"); - let per = inner + inner .per_id .entry(sandbox_id.to_string()) - .or_insert_with(PerSandbox::new); - - let cursored = CursoredEvent { seq, event }; - let _ = per.sender.send(cursored.clone()); - per.tail.push_back(cursored); - while per.tail.len() > tail_cap { - if let Some(trimmed) = per.tail.pop_front() { - per.last_trimmed_seq = trimmed.seq; - } - } + .or_insert_with(PerSandbox::new) + .push(CursoredEvent { seq, event }, tail_cap); } } @@ -420,6 +568,20 @@ fn display_level(target: &str, level: &str) -> String { mod tests { use super::*; + /// Assert that neither bus holds a per-sandbox entry. + #[track_caller] + fn assert_bus_maps_empty(bus: &TracingLogBus) { + assert!(bus.inner.lock().unwrap().per_id.is_empty()); + assert!( + bus.platform_event_bus + .inner + .lock() + .unwrap() + .per_id + .is_empty() + ); + } + #[test] fn cross_sandbox_denials_remain_gateway_wide() { use crate::auth::{audit, principal::test_support}; @@ -496,7 +658,7 @@ mod tests { .try_recv() .expect("sandbox audit event must reach live stream"); let Some(openshell_core::proto::sandbox_stream_event::Payload::Log(log)) = - live.event.payload + &live.event.payload else { panic!("expected log payload"); }; @@ -546,13 +708,13 @@ mod tests { } /// Build a contiguous tail with seqs `lo..=hi`. - fn tail_of(lo: u64, hi: u64) -> VecDeque { - (lo..=hi).map(cursored).collect() + fn tail_of(lo: u64, hi: u64) -> VecDeque> { + (lo..=hi).map(|seq| Arc::new(cursored(seq))).collect() } /// Extract seqs from a run of buffered events, in order. - fn cursors(events: &[CursoredEvent]) -> Vec { - events.iter().map(|c| c.seq).collect() + fn cursors>(events: &[E]) -> Vec { + events.iter().map(|c| c.borrow().seq).collect() } #[test] @@ -621,7 +783,10 @@ mod tests { // Simulate the shared cursor space: this bus only owns seqs 2 and 4 // (the other bus owns 1 and 3), and never trimmed. Resuming from 0 must // not report a gap just because seq 1 is absent here. - let tail: VecDeque = [cursored(2), cursored(4)].into_iter().collect(); + let tail: VecDeque> = [cursored(2), cursored(4)] + .into_iter() + .map(Arc::new) + .collect(); let events = tail_after_impl(&tail, 0, 0).expect("no gap"); assert_eq!(cursors(&events), vec![2, 4]); } @@ -679,6 +844,34 @@ mod tests { assert_eq!(events, vec![2]); } + #[test] + fn broadcast_channel_is_created_only_on_subscribe() { + let bus = TracingLogBus::new(); + let sandbox_id = "sb-lazy"; + let has_sender = |bus: &TracingLogBus| { + bus.inner.lock().unwrap().per_id[sandbox_id] + .sender + .is_some() + && bus.platform_event_bus.inner.lock().unwrap().per_id[sandbox_id] + .sender + .is_some() + }; + + bus.publish_external(make_log_event(sandbox_id, "unwatched")); + bus.platform_event_bus.publish(sandbox_id, stream_event(0)); + assert!(!has_sender(&bus), "publishing must not allocate a channel"); + assert_eq!(bus.tail(sandbox_id, 10).len(), 1); + + let mut log_rx = bus.subscribe(sandbox_id); + let mut platform_rx = bus.platform_event_bus.subscribe(sandbox_id); + assert!(has_sender(&bus)); + + bus.publish_external(make_log_event(sandbox_id, "watched")); + bus.platform_event_bus.publish(sandbox_id, stream_event(0)); + assert!(log_rx.try_recv().is_ok()); + assert!(platform_rx.try_recv().is_ok()); + } + #[test] fn tracing_log_bus_remove_cleans_up_all_maps() { let bus = TracingLogBus::new(); @@ -705,11 +898,12 @@ mod tests { assert_eq!(bus.cursor_space(sandbox_id), None); - // Subscribing must not mint a space. `sender_for` touches only the bus - // map, never the allocator -- and the watch handler subscribes before - // it validates the resume cursor. If a subscription could manufacture - // an epoch, a client resuming against a torn-down sandbox would create - // the very space its stale cursor is then checked against. + // Subscribing must not mint a space. `subscribe` takes the allocator + // lock to check the tombstone, but only inserts into the bus map -- and + // the watch handler subscribes before it validates the resume cursor. + // If a subscription could manufacture an epoch, a client resuming + // against a torn-down sandbox would create the very space its stale + // cursor is then checked against. let _log_rx = bus.subscribe(sandbox_id); let _platform_rx = bus.platform_event_bus.subscribe(sandbox_id); assert_eq!(bus.cursor_space(sandbox_id), None); @@ -867,6 +1061,222 @@ mod tests { bus.remove("nonexistent"); } + #[test] + fn publish_after_close_drops_without_recreating_buffers() { + // The issue's leak: late publishers used to re-create the buffers a + // deleted sandbox's cleanup had just freed, and nothing ever freed them + // again. + let bus = TracingLogBus::new(); + let sandbox_id = "sb-closed-publish"; + bus.publish_external(make_log_event(sandbox_id, "before")); + bus.platform_event_bus.publish(sandbox_id, stream_event(0)); + bus.close(sandbox_id); + + bus.publish_external(make_log_event(sandbox_id, "after")); + bus.platform_event_bus.publish(sandbox_id, stream_event(0)); + + assert!( + bus.tail(sandbox_id, 10).is_empty(), + "a closed sandbox must drop log lines" + ); + assert!( + bus.platform_event_bus.tail(sandbox_id, 10).is_empty(), + "a closed sandbox must drop platform events" + ); + assert_bus_maps_empty(&bus); + assert_eq!(bus.cursor_space(sandbox_id), None); + assert!(bus.is_closed(sandbox_id)); + } + + #[test] + fn tracing_log_bus_close_closes_active_receivers() { + let bus = TracingLogBus::new(); + let sandbox_id = "sb-close-receivers"; + let mut log_rx = bus.subscribe(sandbox_id); + let mut platform_rx = bus.platform_event_bus.subscribe(sandbox_id); + + bus.publish_external(make_log_event(sandbox_id, "hello")); + assert!(log_rx.try_recv().is_ok()); + + bus.close(sandbox_id); + + match log_rx.try_recv() { + Err(broadcast::error::TryRecvError::Closed) => {} // expected + other => panic!("expected Closed, got {other:?}"), + } + match platform_rx.try_recv() { + Err(broadcast::error::TryRecvError::Closed) => {} // expected + other => panic!("expected Closed, got {other:?}"), + } + // A close that tombstoned but forgot the map removal would still pass + // the receiver assertions above; the map lengths are what prove the + // buffers were actually freed. + assert_bus_maps_empty(&bus); + } + + #[test] + fn subscribe_after_close_yields_closed_receiver_and_creates_no_entry() { + let bus = TracingLogBus::new(); + let sandbox_id = "sb-sub-after-close"; + bus.close(sandbox_id); + + let mut log_rx = bus.subscribe(sandbox_id); + assert!(matches!( + log_rx.try_recv(), + Err(broadcast::error::TryRecvError::Closed) + )); + let mut platform_rx = bus.platform_event_bus.subscribe(sandbox_id); + assert!(matches!( + platform_rx.try_recv(), + Err(broadcast::error::TryRecvError::Closed) + )); + + assert_bus_maps_empty(&bus); + } + + #[test] + fn close_is_idempotent_and_remove_keeps_the_tombstone() { + let bus = TracingLogBus::new(); + let sandbox_id = "sb-close-idempotent"; + + bus.publish_external(make_log_event(sandbox_id, "line")); + bus.close(sandbox_id); + bus.close(sandbox_id); + assert!(bus.is_closed(sandbox_id)); + assert_eq!(bus.seq.lock().len(), 1); + + // `remove` is the reset path, but a closed id is terminal: the + // tombstone must survive so a late publisher still cannot re-create the + // buffers. + bus.remove(sandbox_id); + assert!(bus.is_closed(sandbox_id)); + bus.publish_external(make_log_event(sandbox_id, "late")); + assert!(bus.tail(sandbox_id, 10).is_empty()); + assert_bus_maps_empty(&bus); + } + + #[test] + fn close_without_prior_publish_inserts_a_tombstone() { + // A subscribe-only sandbox has no cursor space to close, but deleting + // it must still refuse later publishes. + let bus = TracingLogBus::new(); + let sandbox_id = "sb-subscribe-only"; + let _rx = bus.subscribe(sandbox_id); + bus.close(sandbox_id); + + assert!(bus.is_closed(sandbox_id)); + bus.publish_external(make_log_event(sandbox_id, "late")); + assert!(bus.tail(sandbox_id, 10).is_empty()); + assert_bus_maps_empty(&bus); + } + + #[test] + fn cursor_space_reports_none_for_a_closed_sandbox() { + let bus = TracingLogBus::new(); + let sandbox_id = "sb-closed-space"; + bus.publish_external(make_log_event(sandbox_id, "line")); + assert!(bus.cursor_space(sandbox_id).is_some()); + + bus.close(sandbox_id); + + assert_eq!( + bus.cursor_space(sandbox_id), + None, + "a tombstone must not look like a live space" + ); + } + + #[test] + fn tracing_layer_after_close_allocates_nothing() { + // The gateway tracing layer -- including the OCSF audit event emitted + // while serving the delete -- publishes for any event carrying a + // `sandbox_id`. Pin that route against the tombstone too. + use tracing_subscriber::prelude::*; + + let bus = TracingLogBus::new(); + let sandbox_id = "sb-layer-closed"; + bus.publish_external(make_log_event(sandbox_id, "before")); + bus.close(sandbox_id); + + let subscriber = tracing_subscriber::registry().with(bus.layer()); + tracing::subscriber::with_default(subscriber, || { + tracing::info!(sandbox_id = "sb-layer-closed", "late gateway event"); + }); + + assert!(bus.tail(sandbox_id, 10).is_empty()); + // The layer must not re-create a closed entry. + assert_bus_maps_empty(&bus); + let spaces = bus.seq.lock(); + assert_eq!(spaces.len(), 1, "only the tombstone may remain"); + assert!(spaces[sandbox_id].closed); + } + + #[test] + fn concurrent_publish_and_close_never_recreates_buffers() { + // A publish that allocated before `close` must not land in a re-created + // entry. The tombstone check and the bus-map insert share the allocator + // lock, so once `close` returns for an id nothing can refill its tail. + for round in 0..64 { + let bus = TracingLogBus::new(); + let sandbox_id = format!("sb-close-race-{round}"); + let barrier = Arc::new(std::sync::Barrier::new(5)); + + let publishers: Vec<_> = (0..4) + .map(|_| { + let bus = bus.clone(); + let barrier = Arc::clone(&barrier); + let sandbox_id = sandbox_id.clone(); + std::thread::spawn(move || { + barrier.wait(); + for _ in 0..64 { + bus.publish_external(make_log_event(&sandbox_id, "x")); + bus.platform_event_bus.publish(&sandbox_id, stream_event(0)); + } + }) + }) + .collect(); + + barrier.wait(); + bus.close(&sandbox_id); + + for publisher in publishers { + publisher.join().expect("publisher thread panicked"); + } + + assert!(bus.is_closed(&sandbox_id)); + assert!( + bus.tail(&sandbox_id, usize::MAX).is_empty(), + "a closed sandbox must never accumulate a tail again" + ); + assert!( + bus.platform_event_bus + .tail(&sandbox_id, usize::MAX) + .is_empty() + ); + assert_bus_maps_empty(&bus); + assert_eq!(bus.seq.lock().len(), 1, "only the tombstone remains"); + } + } + + #[test] + fn repeated_create_delete_cycles_keep_bus_maps_bounded() { + // The RSS proxy: buffers are freed every cycle and only the small + // tombstone accumulates. + let bus = TracingLogBus::new(); + for index in 0..1_000 { + let sandbox_id = format!("sb-cycle-{index}"); + bus.publish_external(make_log_event(&sandbox_id, "line")); + bus.platform_event_bus.publish(&sandbox_id, stream_event(0)); + bus.close(&sandbox_id); + assert_bus_maps_empty(&bus); + } + assert_eq!( + bus.seq.lock().len(), + 1_000, + "one tombstone per deleted sandbox" + ); + } + #[test] fn display_level_maps_ocsf_target_to_ocsf() { assert_eq!(display_level(OCSF_TARGET, "INFO"), "OCSF"); @@ -1019,18 +1429,20 @@ impl PlatformEventBus { } } - fn sender_for(&self, sandbox_id: &str) -> broadcast::Sender { + pub(crate) fn subscribe(&self, sandbox_id: &str) -> broadcast::Receiver> { + // Same guard as `TracingLogBus::subscribe`: take the allocator lock so + // the closed check is atomic with `TracingLogBus::close`, and return an + // already-closed receiver for a tombstoned id. + let spaces = self.seq.lock(); + if SeqAllocator::is_closed_locked(&spaces, sandbox_id) { + return closed_receiver(); + } let mut inner = self.inner.lock().expect("platform event bus lock poisoned"); inner .per_id .entry(sandbox_id.to_string()) .or_insert_with(PerSandbox::new) - .sender - .clone() - } - - pub(crate) fn subscribe(&self, sandbox_id: &str) -> broadcast::Receiver { - self.sender_for(sandbox_id).subscribe() + .subscribe() } pub(crate) fn publish(&self, sandbox_id: &str, mut event: SandboxStreamEvent) { @@ -1038,36 +1450,30 @@ impl PlatformEventBus { // lock order as `TracingLogBus::publish`), so teardown cannot reset the // counter underneath an in-flight publish. let mut spaces = self.seq.lock(); - let (epoch, seq) = SeqAllocator::next_locked(&mut spaces, sandbox_id); + let Some((epoch, seq)) = SeqAllocator::next_locked(&mut spaces, sandbox_id) else { + return; + }; event.cursor = WatchCursor::new(epoch, seq).encode(); let mut inner = self.inner.lock().expect("platform event bus lock poisoned"); - let per = inner + inner .per_id .entry(sandbox_id.to_string()) - .or_insert_with(PerSandbox::new); - - let cursored = CursoredEvent { seq, event }; - let _ = per.sender.send(cursored.clone()); - per.tail.push_back(cursored); - while per.tail.len() > Self::DEFAULT_TAIL { - if let Some(trimmed) = per.tail.pop_front() { - per.last_trimmed_seq = trimmed.seq; - } - } + .or_insert_with(PerSandbox::new) + .push(CursoredEvent { seq, event }, Self::DEFAULT_TAIL); } /// Return buffered platform events for replay to late subscribers. pub(crate) fn tail(&self, sandbox_id: &str, max: usize) -> Vec { - let inner = self.inner.lock().expect("platform event bus lock poisoned"); - inner + let events = self + .inner + .lock() + .expect("platform event bus lock poisoned") .per_id .get(sandbox_id) - .map(|d| d.tail.iter().rev().take(max).cloned().collect::>()) - .unwrap_or_default() - .into_iter() - .rev() - .collect() + .map(|per| per.tail(max)) + .unwrap_or_default(); + unshare(events) } pub(crate) fn tail_after( @@ -1075,11 +1481,17 @@ impl PlatformEventBus { sandbox_id: &str, after_seq: u64, ) -> Result, ResumeGap> { - let inner = self.inner.lock().expect("platform event bus lock poisoned"); - inner.per_id.get(sandbox_id).map_or_else( - || Ok(Vec::new()), - |per| tail_after_impl(&per.tail, per.last_trimmed_seq, after_seq), - ) + let events = self + .inner + .lock() + .expect("platform event bus lock poisoned") + .per_id + .get(sandbox_id) + .map_or_else( + || Ok(Vec::new()), + |per| tail_after_impl(&per.tail, per.last_trimmed_seq, after_seq), + )?; + Ok(unshare(events)) } /// Remove the bus entry for the given sandbox id. diff --git a/crates/openshell-server/tests/auth_endpoint_integration.rs b/crates/openshell-server/tests/auth_endpoint_integration.rs index b4d1e6436e..55bacbd925 100644 --- a/crates/openshell-server/tests/auth_endpoint_integration.rs +++ b/crates/openshell-server/tests/auth_endpoint_integration.rs @@ -182,7 +182,9 @@ async fn start_auth_server(bind_addr: SocketAddr) -> (SocketAddr, tokio::task::J /// Make an HTTP/1.1 GET request and return (status, body). async fn http_get(addr: SocketAddr, uri: &str, headers: &[(&str, &str)]) -> (StatusCode, String) { - let stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let (mut sender, conn) = hyper::client::conn::http1::Builder::new() .handshake(TokioIo::new(stream)) .await @@ -440,7 +442,9 @@ async fn plaintext_server_accepts_grpc_and_http() { ); // HTTP over plaintext - let stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let (mut sender, conn) = hyper::client::conn::http1::Builder::new() .handshake(TokioIo::new(stream)) .await diff --git a/crates/openshell-server/tests/health_endpoint_integration.rs b/crates/openshell-server/tests/health_endpoint_integration.rs index 45cd501357..b450701afe 100644 --- a/crates/openshell-server/tests/health_endpoint_integration.rs +++ b/crates/openshell-server/tests/health_endpoint_integration.rs @@ -30,7 +30,7 @@ async fn start_health_server( } async fn http_get_json(addr: std::net::SocketAddr, path: &str) -> (StatusCode, Value) { - let stream = tokio::net::TcpStream::connect(addr) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) .await .expect("connect test HTTP client"); let (mut sender, conn) = hyper::client::conn::http1::Builder::new() diff --git a/crates/openshell-server/tests/multiplex_integration.rs b/crates/openshell-server/tests/multiplex_integration.rs index 2474ccebf4..70ad7126f5 100644 --- a/crates/openshell-server/tests/multiplex_integration.rs +++ b/crates/openshell-server/tests/multiplex_integration.rs @@ -48,7 +48,9 @@ async fn serves_grpc_and_http_on_same_port() { let response = client.health(HealthRequest {}).await.unwrap(); assert_eq!(response.get_ref().status, ServiceStatus::Healthy as i32); - let stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let (mut sender, conn) = hyper::client::conn::http1::Builder::new() .handshake(TokioIo::new(stream)) .await diff --git a/crates/openshell-server/tests/ws_tunnel_integration.rs b/crates/openshell-server/tests/ws_tunnel_integration.rs index 5e0d58ec9b..6ff320edcf 100644 --- a/crates/openshell-server/tests/ws_tunnel_integration.rs +++ b/crates/openshell-server/tests/ws_tunnel_integration.rs @@ -409,7 +409,9 @@ async fn ws_tunnel_grpc_health_through_websocket() { // Also verify HTTP /healthz works directly (not through tunnel — WS is // for gRPC; HTTP healthz goes through the multiplexed service directly). - let stream = TcpStream::connect(stack.grpc_addr).await.unwrap(); + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(stack.grpc_addr) + .await + .unwrap(); let (mut sender, conn) = hyper::client::conn::http1::Builder::new() .handshake(TokioIo::new(stream)) .await diff --git a/crates/openshell-supervisor-middleware/src/headers.rs b/crates/openshell-supervisor-middleware/src/headers.rs index 056ad32cf7..49d994d790 100644 --- a/crates/openshell-supervisor-middleware/src/headers.rs +++ b/crates/openshell-supervisor-middleware/src/headers.rs @@ -286,25 +286,31 @@ fn is_safe_value(value: &str) -> bool { } fn is_request_protected(name: &str) -> bool { - matches!( - name, - "authorization" - | "proxy-authorization" - | "proxy-authenticate" - | "cookie" - | "host" - | "content-length" - | "transfer-encoding" - | "connection" - | "proxy-connection" - | "keep-alive" - | "te" - | "trailer" - | "upgrade" - ) || name.starts_with("x-amz-") + is_platform_correlation_header(name) + || matches!( + name, + "authorization" + | "proxy-authorization" + | "proxy-authenticate" + | "cookie" + | "host" + | "content-length" + | "transfer-encoding" + | "connection" + | "proxy-connection" + | "keep-alive" + | "te" + | "trailer" + | "upgrade" + ) + || name.starts_with("x-amz-") || name.starts_with("x-openshell-credential") } +fn is_platform_correlation_header(name: &str) -> bool { + name == "x-openshell-request-id" +} + /// Return whether a response header can carry authentication material and must /// never be exposed to middleware. #[must_use] @@ -322,7 +328,8 @@ pub fn is_response_credential_header(name: &str) -> bool { } fn is_response_protected(name: &str) -> bool { - is_response_credential_header(name) + is_platform_correlation_header(name) + || is_response_credential_header(name) || matches!( name, "connection" @@ -408,6 +415,39 @@ mod tests { ); } + #[test] + fn platform_request_id_is_immutable_for_all_http_phases() { + for authority in [ + HeaderAuthority::Request, + HeaderAuthority::Response, + HeaderAuthority::ResponseTrailers, + ] { + let name = "X-OpenShell-Request-ID"; + let write_error = apply( + authority, + &[], + &[], + &[write(name, "forged", ExistingHeaderAction::Overwrite)], + ) + .expect_err("platform request ID write"); + assert_eq!( + write_error, + HeaderMutationError::Protected { + name: name.to_string() + } + ); + + let remove_error = apply(authority, &[], &[], &[remove(name)]) + .expect_err("platform request ID removal"); + assert_eq!( + remove_error, + HeaderMutationError::Protected { + name: name.to_string() + } + ); + } + } + #[test] fn unsafe_header_value_is_rejected() { let error = apply( diff --git a/crates/openshell-supervisor-network/Cargo.toml b/crates/openshell-supervisor-network/Cargo.toml index 7043f30d24..e51f282cd5 100644 --- a/crates/openshell-supervisor-network/Cargo.toml +++ b/crates/openshell-supervisor-network/Cargo.toml @@ -40,7 +40,7 @@ ipnet = "2" miette = { workspace = true } prost-types = { workspace = true } rcgen = { workspace = true } -regorus = { version = "0.9", default-features = false, features = ["std", "arc", "glob", "yaml"] } +regorus = { version = "0.9", default-features = false, features = ["std", "arc", "glob"] } reqwest = { workspace = true } rustls = { workspace = true } rustls-native-certs = { workspace = true } diff --git a/crates/openshell-supervisor-network/README.md b/crates/openshell-supervisor-network/README.md index 38a7fb6add..c4163b885d 100644 --- a/crates/openshell-supervisor-network/README.md +++ b/crates/openshell-supervisor-network/README.md @@ -51,3 +51,23 @@ domain event and a detection finding without credential values or placeholders. Explicit body and WebSocket rewrite flags authorize those locations for ordinary static credentials. SigV4 signing retains the endpoint-scoped resolver's existing trusted access. + +## HTTP enforcement correlation + +Each inspected HTTP request has one UUID shared by middleware, OCSF +`metadata.correlation_uid`, and the protected `x-openshell-request-id` response +header. Application correlation must use another header. The receipt is only a +join key: it does not establish identity, authorization or complete delivery. +L4 and passthrough traffic do not provide this header guarantee. + +The audit state commits one primary HTTP Activity record when the final response +head is written, with an early-termination fallback. It retains the original +request and process context while consolidating middleware and credential +outcomes. Routine outcomes do not add records; existing security findings and +post-header failure records remain distinct. A process crash or indefinitely +hung request cannot be finalized by the fallback. + +Request IDs do not depend on OpenTelemetry or collector availability. This layer +adds no request spans or workload-controlled supervisor trace context and keeps +the existing bounded deny-span path. Request-stage tracing is a separate layer +that requires operator-controlled supervisor sampling. diff --git a/crates/openshell-supervisor-network/src/host.rs b/crates/openshell-supervisor-network/src/host.rs index d6fbe2e54f..411af0299f 100644 --- a/crates/openshell-supervisor-network/src/host.rs +++ b/crates/openshell-supervisor-network/src/host.rs @@ -245,7 +245,6 @@ mod tests { MiddlewareEndpointSelector, NetworkMiddlewareConfig, SandboxPolicy as ProtoSandboxPolicy, }; use tokio::io::{AsyncReadExt, AsyncWriteExt}; - use tokio::net::TcpStream; use super::*; @@ -345,7 +344,9 @@ mod tests { .contains("BEGIN CERTIFICATE") ); - let mut client = TcpStream::connect(addr).await.unwrap(); + let mut client = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); client .write_all( b"GET http://policy.local/v1/policy/current HTTP/1.1\r\n\ diff --git a/crates/openshell-supervisor-network/src/l7/correlation.rs b/crates/openshell-supervisor-network/src/l7/correlation.rs new file mode 100644 index 0000000000..a65f175cc2 --- /dev/null +++ b/crates/openshell-supervisor-network/src/l7/correlation.rs @@ -0,0 +1,458 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Per-request receipts and consolidated HTTP audit records. + +use std::sync::Mutex; +use std::{ + pin::Pin, + task::{Context, Poll}, +}; + +use openshell_ocsf::{ + ActionId, DispositionId, Endpoint, HttpActivityBuilder, HttpActivityEvent, HttpRequest, + HttpResponse, OcsfEvent, SeverityId, StatusId, +}; +use tokio::io::{AsyncRead, AsyncWrite, ReadBuf}; + +use crate::l7::provider::L7Request; +use crate::l7::relay::L7EvalContext; + +#[derive(Debug)] +pub struct RequestCorrelation { + request_id: String, + inspected: bool, + // Keep the audit payload off the async state machine. + state: Box>, +} + +#[derive(Debug, Default)] +struct RequestState { + primary: Option, + seeded: bool, + status: Option, + upstream_status: Option, + outcome: Option<&'static str>, + blocked: bool, + failure_reason: Option<&'static str>, + late_failure_emitted: bool, +} + +impl RequestCorrelation { + #[must_use] + pub(crate) fn new(request: &L7Request, ctx: &L7EvalContext) -> Self { + Self::build(request, ctx, uuid::Uuid::new_v4().to_string()) + } + + pub(crate) fn uninspected() -> Self { + Self { + request_id: uuid::Uuid::new_v4().to_string(), + inspected: false, + state: Box::default(), + } + } + + #[cfg(test)] + #[must_use] + pub(crate) fn with_request_id( + request: &L7Request, + ctx: &L7EvalContext, + request_id: String, + ) -> Self { + Self::build(request, ctx, request_id) + } + + fn build(request: &L7Request, ctx: &L7EvalContext, request_id: String) -> Self { + let target = request.target.split(['?', '#']).next().unwrap_or("/"); + let target = if target.starts_with('/') { target } else { "/" }; + let path = openshell_core::secrets::redact_target_for_policy(target) + .unwrap_or_else(|_| "/".to_string()); + let fallback = HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(HttpRequest::new(&request.action, ctx.ocsf_url(&path))) + .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) + .message("HTTP request ended before a response was selected") + .build(); + Self { + request_id, + inspected: true, + state: Box::new(Mutex::new(RequestState { + primary: Some(fallback), + ..RequestState::default() + })), + } + } + + #[must_use] + pub(crate) fn request_id(&self) -> &str { + &self.request_id + } + + pub(crate) fn observe_response<'a, C>(&'a self, client: &'a mut C) -> AuditedClient<'a, C> { + AuditedClient { + inner: client, + correlation: self, + head: ResponseHead::default(), + } + } + + pub(crate) fn record_enforcement_action(&self, action: &str) { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .blocked |= action == "block"; + } + + pub(crate) fn record_final_http_outcome(&self, outcome: &'static str) { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .outcome = Some(outcome); + } + + pub(crate) fn record_response_status(&self, status: u16) { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .status = Some(status); + } + + pub(crate) fn record_upstream_status(&self, status: u16) { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .upstream_status = Some(status); + } + + pub(crate) fn record_failure_reason(&self, reason: &'static str) { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .failure_reason = Some(reason); + } + + pub(crate) fn emit(&self, event: OcsfEvent) { + if !self.inspected { + crate::telemetry::emit_decision([event], None); + return; + } + if let OcsfEvent::HttpActivity(mut incoming) = event { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if !state.seeded && state.primary.is_some() { + state.primary = Some(OcsfEvent::HttpActivity(incoming)); + state.seeded = true; + return; + } + if let Some(OcsfEvent::HttpActivity(primary)) = &mut state.primary { + merge_http_outcome(primary, incoming); + return; + } + let fail_open = incoming.action == Some(ActionId::Other) + && (incoming.disposition == Some(DispositionId::Allowed) + || incoming + .base + .unmapped + .as_ref() + .and_then(|fields| fields.get("response_middleware_outcome")) + .and_then(serde_json::Value::as_str) + == Some("failopen")); + if !is_failure(&incoming) || fail_open || state.late_failure_emitted { + return; + } + state.late_failure_emitted = true; + if let Some(code) = state.status { + incoming.http_response = Some(HttpResponse { code }); + } + drop(state); + self.emit_correlated(OcsfEvent::HttpActivity(incoming)); + } else { + if let OcsfEvent::NetworkActivity(network) = &event { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if !state.seeded + && (network.action == Some(ActionId::Denied) + || network.base.status == Some(StatusId::Failure)) + && let Some(OcsfEvent::HttpActivity(primary)) = &mut state.primary + { + primary.action = network.action; + primary.disposition = network.disposition; + primary.base.status = Some(StatusId::Failure); + primary + .base + .status_detail + .clone_from(&network.base.status_detail); + primary.base.severity = network.base.severity; + primary.base.message = + Some("HTTP request rejected before policy evaluation".into()); + primary.src_endpoint.clone_from(&network.src_endpoint); + primary.actor.clone_from(&network.actor); + } + } + self.emit_correlated(event); + } + } + + pub(crate) fn response_selected(&self, status: u16) { + self.record_response_status(status); + self.emit_primary(false); + } + + fn emit_primary(&self, incomplete: bool) { + if !self.inspected { + return; + } + let (mut event, status, upstream_status, outcome, blocked, failure_reason) = { + let mut state = self + .state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(primary) = state.primary.take() else { + return; + }; + ( + primary, + state.status, + state.upstream_status, + state.outcome, + state.blocked, + state.failure_reason, + ) + }; + if let OcsfEvent::HttpActivity(http) = &mut event { + let has_failure_reason = is_failure(http) && http.base.status_detail.is_some(); + if let Some(code) = status { + http.http_response = Some(HttpResponse { code }); + } + if let Some(upstream) = upstream_status.filter(|upstream| Some(*upstream) != status) + && let serde_json::Value::Object(unmapped) = http + .base + .unmapped + .get_or_insert_with(|| serde_json::json!({})) + { + unmapped.insert("upstream_response_code".into(), serde_json::json!(upstream)); + } + if blocked || outcome == Some("denied") { + if http.action != Some(ActionId::Denied) { + http.base.message = Some("HTTP request blocked by supervisor".into()); + } + http.action = Some(ActionId::Denied); + http.disposition = Some(DispositionId::Blocked); + http.base.status = Some(StatusId::Failure); + raise_severity(http, SeverityId::Medium); + if !has_failure_reason { + http.base.status_detail = Some( + failure_reason + .or(outcome) + .unwrap_or("supervisor_blocked") + .into(), + ); + } + } else if matches!(outcome, Some("platform_error" | "upstream_error")) { + http.base.status = Some(StatusId::Failure); + raise_severity(http, SeverityId::Low); + if !has_failure_reason { + http.base.status_detail = failure_reason.or(outcome).map(str::to_string); + } + } + if incomplete && outcome.is_none() { + http.base.status = Some(StatusId::Failure); + if !has_failure_reason { + http.base.status_detail = Some("request_incomplete".into()); + } + raise_severity(http, SeverityId::Low); + } + } + self.emit_correlated(event); + } + + fn emit_correlated(&self, mut event: OcsfEvent) { + event.set_correlation_uid(self.request_id.clone()); + crate::telemetry::emit_decision([event], None); + } +} + +fn is_failure(event: &HttpActivityEvent) -> bool { + event.action == Some(ActionId::Denied) || event.base.status == Some(StatusId::Failure) +} + +fn raise_severity(event: &mut HttpActivityEvent, severity: SeverityId) { + if event.base.severity.as_u8() < severity.as_u8() { + event.base.severity = severity; + } +} + +fn merge_http_outcome(primary: &mut HttpActivityEvent, incoming: HttpActivityEvent) { + let failed = is_failure(&incoming); + if primary.actor.is_none() { + primary.actor = incoming.actor; + } + if primary.src_endpoint.is_none() { + primary.src_endpoint = incoming.src_endpoint; + } + if primary.dst_endpoint.is_none() { + primary.dst_endpoint = incoming.dst_endpoint; + } + if primary.firewall_rule.is_none() { + primary.firewall_rule = incoming.firewall_rule; + } + if primary + .http_request + .as_ref() + .and_then(|request| request.url.as_ref()) + .is_none() + && incoming + .http_request + .as_ref() + .and_then(|request| request.url.as_ref()) + .is_some() + { + primary.http_request = incoming.http_request; + } + if !failed { + if incoming + .base + .unmapped + .as_ref() + .and_then(|fields| fields.get("transformed_body_policy")) + .and_then(serde_json::Value::as_str) + == Some("audit") + { + if let serde_json::Value::Object(fields) = primary + .base + .unmapped + .get_or_insert_with(|| serde_json::json!({})) + { + fields.insert("transformed_body_policy".into(), "audit".into()); + } + if !is_failure(primary) && incoming.base.status_detail.is_some() { + primary + .base + .status_detail + .clone_from(&incoming.base.status_detail); + } + } + if primary.action.is_none() { + primary.action = incoming.action; + primary.disposition = incoming.disposition; + primary.base.status = incoming.base.status; + primary.base.message = incoming.base.message; + primary.base.status_detail = incoming.base.status_detail; + } + return; + } + if incoming.action == Some(ActionId::Denied) { + primary.action = incoming.action; + primary.disposition = incoming.disposition; + } + primary.base.status = Some(StatusId::Failure); + raise_severity(primary, incoming.base.severity); + if incoming.base.message.is_some() { + primary.base.message = incoming.base.message; + } + if incoming.base.status_detail.is_some() { + primary.base.status_detail = incoming.base.status_detail; + } + if incoming.http_response.is_some() { + primary.http_response = incoming.http_response; + } + if let Some(serde_json::Value::Object(fields)) = incoming.base.unmapped + && let serde_json::Value::Object(existing) = primary + .base + .unmapped + .get_or_insert_with(|| serde_json::json!({})) + { + existing.extend(fields); + } +} + +/// Observe accepted response bytes without buffering or changing delivery. +pub struct AuditedClient<'a, C> { + inner: &'a mut C, + correlation: &'a RequestCorrelation, + head: ResponseHead, +} + +#[derive(Default)] +struct ResponseHead { + prefix: [u8; 12], + prefix_len: usize, + delimiter: u32, + complete: bool, +} + +impl ResponseHead { + fn accept(&mut self, bytes: &[u8], correlation: &RequestCorrelation) { + for &byte in bytes { + if self.complete { + break; + } + if self.prefix_len < self.prefix.len() { + self.prefix[self.prefix_len] = byte; + self.prefix_len += 1; + } + self.delimiter = (self.delimiter << 8) | u32::from(byte); + if self.delimiter != u32::from_be_bytes(*b"\r\n\r\n") { + continue; + } + let status = std::str::from_utf8(&self.prefix[9..12]) + .ok() + .and_then(|code| code.parse::().ok()); + if status.is_some_and(|code| (100..200).contains(&code) && code != 101) { + *self = Self::default(); + } else { + self.complete = true; + if let Some(status) = status { + correlation.response_selected(status); + } + } + } + } +} + +impl AsyncRead for AuditedClient<'_, C> { + fn poll_read( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + buf: &mut ReadBuf<'_>, + ) -> Poll> { + Pin::new(&mut *self.get_mut().inner).poll_read(cx, buf) + } +} + +impl AsyncWrite for AuditedClient<'_, C> { + fn poll_write( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + bytes: &[u8], + ) -> Poll> { + let this = self.get_mut(); + let result = Pin::new(&mut *this.inner).poll_write(cx, bytes); + if let Poll::Ready(Ok(accepted)) = result + && this.correlation.inspected + { + this.head.accept(&bytes[..accepted], this.correlation); + } + result + } + + fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut *self.get_mut().inner).poll_flush(cx) + } + + fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut *self.get_mut().inner).poll_shutdown(cx) + } +} + +impl Drop for RequestCorrelation { + fn drop(&mut self) { + self.emit_primary(true); + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/openshell-supervisor-network/src/l7/correlation/tests.rs b/crates/openshell-supervisor-network/src/l7/correlation/tests.rs new file mode 100644 index 0000000000..06471cd80e --- /dev/null +++ b/crates/openshell-supervisor-network/src/l7/correlation/tests.rs @@ -0,0 +1,393 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use super::*; +use openshell_ocsf::{ActivityId, NetworkActivityBuilder, Process}; +use tokio::io::AsyncWriteExt as _; + +fn request() -> L7Request { + L7Request { + action: "GET".into(), + target: "/data".into(), + query_params: std::collections::HashMap::new(), + raw_header: b"GET /data HTTP/1.1\r\nHost: example.test\r\n\r\n".to_vec(), + body_length: crate::l7::provider::BodyLength::None, + } +} + +fn context() -> L7EvalContext { + L7EvalContext { + host: "example.test".into(), + port: 443, + policy_name: "protected-api".into(), + ..Default::default() + } +} + +fn allowed() -> OcsfEvent { + HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(HttpRequest::new("GET", context().ocsf_url("/data"))) + .actor_process(Process::new("curl", 42).with_cmd_line("curl /data")) + .firewall_rule("protected-api", "l7") + .action(ActionId::Allowed) + .disposition(DispositionId::Allowed) + .status(StatusId::Success) + .message("HTTP request admitted by policy") + .status_detail("matched_rule") + .build() +} + +fn denied() -> OcsfEvent { + HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(HttpRequest { + http_method: "GET".parse().unwrap(), + url: None, + }) + .http_response(HttpResponse { code: 403 }) + .firewall_rule("protected-api", "credential-binding") + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .status(StatusId::Failure) + .severity(SeverityId::High) + .message("Credential use denied") + .status_detail("credential_endpoint_mismatch") + .build() +} + +fn http_events(capture: &openshell_ocsf::test_support::OcsfCapture) -> Vec { + capture + .json_events() + .into_iter() + .filter(|event| event["class_uid"] == 4002) + .collect() +} + +#[test] +fn request_ids_are_unique_canonical_uuids() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let first = RequestCorrelation::new(&request(), &context()); + let second = RequestCorrelation::new(&request(), &context()); + assert_ne!(first.request_id(), second.request_id()); + assert!(uuid::Uuid::parse_str(first.request_id()).is_ok()); + assert!(uuid::Uuid::parse_str(second.request_id()).is_ok()); +} + +#[tokio::test] +async fn primary_record_waits_for_final_response_headers_and_is_emitted_once_without_otel() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + let id = correlation.request_id().to_string(); + correlation.emit(allowed()); + let mut output = Vec::new(); + { + let mut client = correlation.observe_response(&mut output); + client + .write_all(b"HTTP/1.1 100 Continue\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 4\r\n") + .await + .unwrap(); + assert!(capture.events().is_empty()); + client.write_all(b"\r\nbody").await.unwrap(); + } + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["http_response"]["code"], 200); + assert_eq!(events[0]["http_request"]["url"]["path"], "/data"); + assert_eq!(events[0]["action"], "Allowed"); + assert_eq!(events[0]["status"], "Success"); + assert_eq!(events[0]["actor"]["process"]["pid"], 42); + assert_eq!(events[0]["metadata"]["correlation_uid"], id); + assert!(events[0].get("trace").is_none()); + assert!(output.ends_with(b"\r\n\r\nbody")); +} + +#[test] +fn later_credential_denial_preserves_policy_request_context() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + let id = correlation.request_id().to_string(); + correlation.emit(allowed()); + correlation.emit(denied()); + // A routine later middleware result cannot undo the denial. + correlation.emit(allowed()); + correlation.response_selected(403); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + let event = &events[0]; + assert_eq!(event["action"], "Denied"); + assert_eq!(event["disposition"], "Blocked"); + assert_eq!(event["status"], "Failure"); + assert_eq!(event["severity"], "High"); + assert_eq!(event["status_detail"], "credential_endpoint_mismatch"); + assert_eq!(event["http_response"]["code"], 403); + assert_eq!(event["http_request"]["url"]["path"], "/data"); + assert_eq!(event["firewall_rule"]["type"], "l7"); + assert_eq!(event["actor"]["process"]["pid"], 42); + assert_eq!(event["metadata"]["correlation_uid"], id); +} + +#[test] +fn local_refusals_and_upstream_failures_are_distinct() { + for (outcome, blocked, code, expected_action, expected_disposition) in [ + ("platform_error", true, 503, "Denied", "Blocked"), + ("upstream_error", false, 502, "Allowed", "Allowed"), + ("platform_error", false, 502, "Allowed", "Allowed"), + ] { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + correlation.emit(allowed()); + correlation.record_enforcement_action(if blocked { "block" } else { "allow" }); + correlation.record_final_http_outcome(outcome); + correlation.response_selected(code); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["action"], expected_action); + assert_eq!(events[0]["disposition"], expected_disposition); + assert_eq!(events[0]["status"], "Failure"); + assert_eq!(events[0]["http_response"]["code"], code); + } +} + +#[test] +fn remote_api_error_does_not_become_a_policy_denial() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + correlation.emit(allowed()); + correlation.record_final_http_outcome("allowed"); + correlation.response_selected(500); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["action"], "Allowed"); + assert_eq!(events[0]["status"], "Success"); + assert_eq!(events[0]["http_response"]["code"], 500); +} + +#[test] +fn pre_policy_rejection_retains_reason_without_claiming_policy_admission() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + let id = correlation.request_id().to_string(); + correlation.emit( + NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(ActivityId::Other) + .dst_endpoint(Endpoint::from_domain("example.test", 443)) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .status(StatusId::Failure) + .severity(SeverityId::Medium) + .status_detail("invalid_jsonrpc_request") + .actor_process(Process::new("curl", 42)) + .build(), + ); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.response_selected(400); + drop(correlation); + assert_eq!(capture.events().len(), 2); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["http_request"]["url"]["path"], "/data"); + assert_eq!(events[0]["status_detail"], "invalid_jsonrpc_request"); + assert_eq!( + events[0]["message"], + "HTTP request rejected before policy evaluation" + ); + assert_eq!(events[0]["action"], "Denied"); + assert_eq!(events[0]["actor"]["process"]["pid"], 42); + assert!(events[0].get("firewall_rule").is_none()); + assert!( + capture + .json_events() + .iter() + .all(|event| event["metadata"]["correlation_uid"] == id) + ); +} + +#[test] +fn late_failure_is_preserved_once_but_routine_late_records_are_not_duplicated() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + let id = correlation.request_id().to_string(); + correlation.emit(allowed()); + correlation.response_selected(200); + correlation.emit(allowed()); + correlation.emit(denied()); + correlation.emit(denied()); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 2); + assert_eq!(events[0]["action"], "Allowed"); + assert_eq!(events[0]["http_response"]["code"], 200); + assert_eq!(events[1]["action"], "Denied"); + assert_eq!(events[1]["status_detail"], "credential_endpoint_mismatch"); + assert_eq!(events[1]["http_response"]["code"], 200); + assert!( + events + .iter() + .all(|event| event["metadata"]["correlation_uid"] == id) + ); +} + +#[test] +fn unfinished_request_retains_safe_path_without_inventing_a_rule() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let mut request = request(); + request.target = "/data?secret=SENTINEL-QUERY".into(); + let id; + { + let correlation = RequestCorrelation::new(&request, &context()); + id = correlation.request_id().to_string(); + let mut head = ResponseHead::default(); + head.accept(b"HTTP/1.1 200 OK\r\nContent-Length: 4\r\n", &correlation); + assert!(capture.events().is_empty()); + } + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["metadata"]["correlation_uid"], id); + assert_eq!(events[0]["status_detail"], "request_incomplete"); + assert_eq!(events[0]["http_request"]["url"]["path"], "/data"); + assert!(events[0].get("http_response").is_none()); + assert!(events[0].get("firewall_rule").is_none()); + assert!(!events[0].to_string().contains("SENTINEL-QUERY")); +} + +#[test] +fn workload_traceparent_does_not_supply_audit_trace_identity() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let mut request = request(); + request.raw_header = b"GET /data HTTP/1.1\r\nHost: example.test\r\ntraceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01\r\n\r\n".to_vec(); + let correlation = RequestCorrelation::new(&request, &context()); + correlation.emit(allowed()); + correlation.response_selected(200); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert!(events[0].get("trace").is_none()); + assert!( + !events[0] + .to_string() + .contains("4bf92f3577b34da6a3ce929d0e0e4736") + ); +} + +#[tokio::test] +async fn uninspected_response_and_events_preserve_legacy_behavior() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::uninspected(); + assert!(uuid::Uuid::parse_str(correlation.request_id()).is_ok()); + let mut output = Vec::new(); + { + let mut client = correlation.observe_response(&mut output); + client + .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n") + .await + .unwrap(); + } + assert!(capture.events().is_empty()); + let expected = allowed(); + correlation.emit(expected.clone()); + drop(correlation); + assert_eq!(capture.events(), vec![expected]); + assert_eq!(output, b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"); +} + +#[test] +fn policy_classification_fills_an_existing_forward_request_context() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + let seed = HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(HttpRequest::new("GET", context().ocsf_url("/forward"))) + .actor_process(Process::new("python3", 77)) + .message("HTTP request awaiting inspection") + .build(); + correlation.emit(seed); + correlation.emit(allowed()); + correlation.response_selected(200); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["http_request"]["url"]["path"], "/forward"); + assert_eq!(events[0]["actor"]["process"]["pid"], 77); + assert_eq!(events[0]["action"], "Allowed"); + assert_eq!(events[0]["status"], "Success"); + assert_eq!(events[0]["firewall_rule"]["type"], "l7"); + assert_eq!(events[0]["message"], "HTTP request admitted by policy"); +} + +#[test] +fn failure_reason_replaces_only_a_prior_success_reason() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + correlation.emit(allowed()); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("platform_error"); + correlation.record_failure_reason("middleware_admission"); + correlation.response_selected(503); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["status_detail"], "middleware_admission"); + assert_eq!(events[0]["action"], "Denied"); + assert_eq!(events[0]["status"], "Failure"); +} + +#[test] +fn unfinished_admitted_request_does_not_keep_a_success_reason() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + let id = correlation.request_id().to_string(); + correlation.emit(allowed()); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 1); + assert_eq!(events[0]["action"], "Allowed"); + assert_eq!(events[0]["status"], "Failure"); + assert_eq!(events[0]["status_detail"], "request_incomplete"); + assert_eq!(events[0]["metadata"]["correlation_uid"], id); + assert_eq!(events[0]["http_request"]["url"]["path"], "/data"); + assert_eq!(events[0]["actor"]["process"]["pid"], 42); + assert!(events[0].get("http_response").is_none()); +} + +#[test] +fn late_fail_open_does_not_hide_a_later_blocking_failure() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = RequestCorrelation::new(&request(), &context()); + let id = correlation.request_id().to_string(); + correlation.emit(allowed()); + correlation.response_selected(200); + correlation.emit( + HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(HttpRequest::new("GET", context().ocsf_url("/data"))) + .action(ActionId::Other) + .disposition(DispositionId::Allowed) + .status(StatusId::Failure) + .status_detail("middleware_fail_open") + .build(), + ); + correlation.emit(denied()); + drop(correlation); + let events = http_events(&capture); + assert_eq!(events.len(), 2); + assert_eq!(events[1]["action"], "Denied"); + assert_eq!(events[1]["status_detail"], "credential_endpoint_mismatch"); + assert_eq!(events[1]["metadata"]["correlation_uid"], id); + assert_eq!(events[1]["http_response"]["code"], 200); +} diff --git a/crates/openshell-supervisor-network/src/l7/middleware.rs b/crates/openshell-supervisor-network/src/l7/middleware.rs index 73c317be6f..07ee742fc4 100644 --- a/crates/openshell-supervisor-network/src/l7/middleware.rs +++ b/crates/openshell-supervisor-network/src/l7/middleware.rs @@ -52,18 +52,19 @@ impl HttpMiddlewareExchange { } } - pub async fn apply_request( + pub(crate) async fn apply_request_correlated( &self, request: crate::l7::provider::L7Request, client: &mut C, ctx: &L7EvalContext, scheme: &str, transformed_body_policy: openshell_supervisor_middleware::TransformedBodyPolicy<'_>, + correlation: Option<&super::correlation::RequestCorrelation>, ) -> Result where C: AsyncRead + AsyncWrite + Unpin + Send, { - apply_middleware_chain_for_scheme_with_request_id( + apply_middleware_chain_for_scheme_with_request_id_correlated( request, client, ctx, @@ -73,6 +74,7 @@ impl HttpMiddlewareExchange { &self.generation_guard, transformed_body_policy, &self.request_id, + correlation, ) .await } @@ -493,10 +495,6 @@ fn middleware_admission_exhausted_event(ctx: &L7EvalContext) -> openshell_ocsf:: .build() } -fn emit_middleware_admission_exhausted(ctx: &L7EvalContext) { - egress_emit!(middleware_admission_exhausted_event(ctx)); -} - /// Largest body-buffering limit across the entries that actually resolved to a /// registered binding. Buffering for the most capable stage lets every stage /// that can handle the body run; stages whose own limit is smaller are failed @@ -516,6 +514,7 @@ pub(super) fn middleware_chain_body_limit( .max() } +#[cfg(test)] #[allow(clippy::too_many_arguments)] pub async fn apply_middleware_chain_with_request_id( req: crate::l7::provider::L7Request, @@ -541,6 +540,7 @@ pub async fn apply_middleware_chain_with_request_id, request_id: &str, +) -> Result { + apply_middleware_chain_for_scheme_with_request_id_correlated( + req, + client, + ctx, + scheme, + chain, + runner, + generation_guard, + transformed_body_policy, + request_id, + None, + ) + .await +} + +#[allow(clippy::too_many_arguments)] +pub async fn apply_middleware_chain_with_request_id_correlated< + C: AsyncRead + AsyncWrite + Unpin + Send, +>( + req: crate::l7::provider::L7Request, + client: &mut C, + ctx: &L7EvalContext, + chain: Vec, + runner: &openshell_supervisor_middleware::ChainRunner, + generation_guard: &PolicyGenerationGuard, + transformed_body_policy: openshell_supervisor_middleware::TransformedBodyPolicy<'_>, + request_id: &str, + correlation: Option<&super::correlation::RequestCorrelation>, +) -> Result { + apply_middleware_chain_for_scheme_with_request_id_correlated( + req, + client, + ctx, + ctx.request_scheme(), + chain, + runner, + generation_guard, + transformed_body_policy, + request_id, + correlation, + ) + .await +} + +#[allow(clippy::too_many_arguments)] +pub async fn apply_middleware_chain_for_scheme_with_request_id_correlated< + C: AsyncRead + AsyncWrite + Unpin + Send, +>( + req: crate::l7::provider::L7Request, + client: &mut C, + ctx: &L7EvalContext, + scheme: &str, + chain: Vec, + runner: &openshell_supervisor_middleware::ChainRunner, + generation_guard: &PolicyGenerationGuard, + transformed_body_policy: openshell_supervisor_middleware::TransformedBodyPolicy<'_>, + request_id: &str, + correlation: Option<&super::correlation::RequestCorrelation>, ) -> Result { if chain.is_empty() { return Ok(MiddlewareApplyResult::Allowed(req)); @@ -568,7 +627,7 @@ pub async fn apply_middleware_chain_for_scheme_with_request_id< admission, ) => Some(admission), openshell_supervisor_middleware::MiddlewareWorkAdmissionOutcome::QueueExhausted => { - emit_middleware_admission_exhausted(ctx); + emit_request_events(correlation, [middleware_admission_exhausted_event(ctx)]); return Ok(MiddlewareApplyResult::AdmissionExhausted); } } @@ -597,7 +656,7 @@ pub async fn apply_middleware_chain_for_scheme_with_request_id< admission, ) .await?; - emit_middleware_events(ctx, &req, &outcome); + emit_request_events(correlation, middleware_events(ctx, &req, &outcome)); return Ok(if outcome.allowed { MiddlewareApplyResult::Allowed(req) } else { @@ -620,7 +679,13 @@ pub async fn apply_middleware_chain_for_scheme_with_request_id< { crate::l7::rest::BufferResult::Buffered(buffered) => buffered, crate::l7::rest::BufferResult::OverCapacity { recoverable } => { - return Ok(resolve_unbuffered_body(ctx, req, &chain, recoverable)); + return Ok(resolve_unbuffered_body_correlated( + ctx, + req, + &chain, + recoverable, + correlation, + )); } }; let headers = safe_middleware_headers(&buffered.headers)?; @@ -647,7 +712,7 @@ pub async fn apply_middleware_chain_for_scheme_with_request_id< Some(admission), ) .await?; - emit_middleware_events(ctx, &req, &outcome); + emit_request_events(correlation, middleware_events(ctx, &req, &outcome)); if !outcome.allowed { return Ok(MiddlewareApplyResult::Denied { denial: outcome.denial, @@ -668,8 +733,11 @@ pub async fn send_middleware_rejection_response, redacted_target: &str, + request_id: Option<&str>, ) -> Result<()> { - let context = Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)); + let context = crate::l7::rest::DenyResponseContext::from_l7_context(ctx); + let context = request_id.map_or(context, |request_id| context.with_request_id(request_id)); + let context = Some(context); if let Some(denial) = denial { crate::l7::rest::send_middleware_deny_response( req, @@ -699,13 +767,16 @@ pub async fn send_middleware_admission_exhausted_response< client: &mut C, ctx: &L7EvalContext, redacted_target: &str, + request_id: Option<&str>, ) -> Result<()> { + let context = crate::l7::rest::DenyResponseContext::from_l7_context(ctx); + let context = request_id.map_or(context, |request_id| context.with_request_id(request_id)); crate::l7::rest::send_middleware_unavailable_response( req, &ctx.policy_name, client, Some(redacted_target), - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some(context), ) .await } @@ -783,24 +854,39 @@ pub(super) fn raw_query_from_request_headers(headers: &[u8]) -> Result { /// would individually fail with `request_body_over_capacity`; the aggregate is /// a deny unless every attached middleware is `fail_open`, and passing the /// body through is only safe when no bytes were consumed. +#[cfg(test)] pub(super) fn resolve_unbuffered_body( ctx: &L7EvalContext, req: crate::l7::provider::L7Request, chain: &[openshell_supervisor_middleware::DescribedChainEntry], recoverable: bool, +) -> MiddlewareApplyResult { + resolve_unbuffered_body_correlated(ctx, req, chain, recoverable, None) +} + +fn resolve_unbuffered_body_correlated( + ctx: &L7EvalContext, + req: crate::l7::provider::L7Request, + chain: &[openshell_supervisor_middleware::DescribedChainEntry], + recoverable: bool, + correlation: Option<&super::correlation::RequestCorrelation>, ) -> MiddlewareApplyResult { let all_fail_open = chain .iter() .all(|entry| entry.on_error() == openshell_supervisor_middleware::OnError::FailOpen); if recoverable && all_fail_open { - emit_middleware_body_unavailable(ctx, false); + emit_middleware_body_unavailable(ctx, false, correlation); return MiddlewareApplyResult::Allowed(req); } - emit_middleware_body_unavailable(ctx, true); + emit_middleware_body_unavailable(ctx, true, correlation); MiddlewareApplyResult::Denied { denial: None } } -fn emit_middleware_body_unavailable(ctx: &L7EvalContext, denied: bool) { +fn emit_middleware_body_unavailable( + ctx: &L7EvalContext, + denied: bool, + correlation: Option<&super::correlation::RequestCorrelation>, +) { let mut event = DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx()) .severity(if denied { SeverityId::High @@ -824,7 +910,7 @@ fn emit_middleware_body_unavailable(ctx: &L7EvalContext, denied: bool) { if denied { event = event.action(ActionId::Denied); } - egress_emit!(event.build()); + emit_request_events(correlation, [event.build()]); } /// Parse the raw header block into middleware-visible headers, preserving @@ -1009,6 +1095,7 @@ pub(super) fn middleware_events( /// Emit the OCSF events describing a middleware chain outcome through the /// tracing pipeline. +#[cfg(test)] fn emit_middleware_events( ctx: &L7EvalContext, req: &crate::l7::provider::L7Request, @@ -1017,6 +1104,19 @@ fn emit_middleware_events( crate::telemetry::emit_decision(middleware_events(ctx, req, outcome), None); } +pub fn emit_request_events( + correlation: Option<&super::correlation::RequestCorrelation>, + events: impl IntoIterator, +) { + if let Some(correlation) = correlation { + for event in events { + correlation.emit(event); + } + } else { + crate::telemetry::emit_decision(events, None); + } +} + #[cfg(test)] mod trace_tests { use super::*; @@ -1122,7 +1222,7 @@ mod trace_tests { let (events, received) = exported("warn", || { let ctx = L7EvalContext::default(); emit_middleware_uninspectable(&ctx, "unknown protocol", denied); - emit_middleware_body_unavailable(&ctx, denied); + emit_middleware_body_unavailable(&ctx, denied, None); }) .await; assert_eq!(events.len(), 2); @@ -1397,9 +1497,16 @@ mod tests { }; let (mut client, mut server) = tokio::io::duplex(4096); - send_middleware_rejection_response(&req, &mut server, &ctx, Some(&denial), "/v1/messages") - .await - .expect("send denial"); + send_middleware_rejection_response( + &req, + &mut server, + &ctx, + Some(&denial), + "/v1/messages", + Some("00000000-0000-4000-8000-000000000010"), + ) + .await + .expect("send denial"); drop(server); let mut response = Vec::new(); @@ -1408,9 +1515,13 @@ mod tests { .await .expect("read denial"); let response = String::from_utf8(response).expect("UTF-8 response"); + assert!( + response.contains("x-openshell-request-id: 00000000-0000-4000-8000-000000000010\r\n") + ); let (_, body) = response.split_once("\r\n\r\n").expect("HTTP response"); let body: serde_json::Value = serde_json::from_str(body).expect("JSON response"); assert_eq!(body["error"], "middleware_denied"); + assert_eq!(body["request_id"], "00000000-0000-4000-8000-000000000010"); assert_eq!(body["middleware"], "prototype-content-guard"); assert_eq!(body["reason_code"], "content_match"); assert!(body.get("rule_missing").is_none()); @@ -1490,9 +1601,16 @@ mod tests { }; let (mut client, mut server) = tokio::io::duplex(4096); - send_middleware_rejection_response(&req, &mut server, &ctx, None, "/v1/messages") - .await - .expect("send failure"); + send_middleware_rejection_response( + &req, + &mut server, + &ctx, + None, + "/v1/messages", + Some("00000000-0000-4000-8000-000000000011"), + ) + .await + .expect("send failure"); drop(server); let mut response = Vec::new(); @@ -1501,9 +1619,13 @@ mod tests { .await .expect("read failure"); let response = String::from_utf8(response).expect("UTF-8 response"); + assert!( + response.contains("x-openshell-request-id: 00000000-0000-4000-8000-000000000011\r\n") + ); let (_, body) = response.split_once("\r\n\r\n").expect("HTTP response"); let body: serde_json::Value = serde_json::from_str(body).expect("JSON response"); assert_eq!(body["error"], "middleware_failed"); + assert_eq!(body["request_id"], "00000000-0000-4000-8000-000000000011"); assert_eq!( body["detail"], "Request could not be processed by configured middleware" diff --git a/crates/openshell-supervisor-network/src/l7/mod.rs b/crates/openshell-supervisor-network/src/l7/mod.rs index c24777826e..93ab909ef3 100644 --- a/crates/openshell-supervisor-network/src/l7/mod.rs +++ b/crates/openshell-supervisor-network/src/l7/mod.rs @@ -8,6 +8,7 @@ //! doing a raw `copy_bidirectional`. Each request within the tunnel is parsed, //! evaluated against OPA policy, and either forwarded or denied. +pub(crate) mod correlation; pub mod graphql; pub(crate) mod http; pub mod jsonrpc; diff --git a/crates/openshell-supervisor-network/src/l7/relay.rs b/crates/openshell-supervisor-network/src/l7/relay.rs index 61c342bc17..70edcb1d6a 100644 --- a/crates/openshell-supervisor-network/src/l7/relay.rs +++ b/crates/openshell-supervisor-network/src/l7/relay.rs @@ -7,14 +7,16 @@ //! Parses each request within the tunnel, evaluates it against OPA policy, //! and either forwards or denies the request. +use crate::l7::correlation::RequestCorrelation; use crate::l7::middleware::{ - MiddlewareApplyResult, UninspectableTrafficGate, apply_middleware_chain_with_request_id, - emit_middleware_uninspectable, middleware_network_input, uninspectable_traffic_gate, + MiddlewareApplyResult, UninspectableTrafficGate, + apply_middleware_chain_with_request_id_correlated, emit_middleware_uninspectable, + middleware_network_input, uninspectable_traffic_gate, }; #[cfg(test)] use crate::l7::middleware::{ - middleware_chain_body_limit, middleware_events, middleware_request_input, - raw_query_from_request_headers, resolve_unbuffered_body, + apply_middleware_chain_with_request_id, middleware_chain_body_limit, middleware_events, + middleware_request_input, raw_query_from_request_headers, resolve_unbuffered_body, }; use crate::l7::provider::{L7Provider, RelayOutcome}; use crate::l7::rest::WebSocketExtensionMode; @@ -36,6 +38,8 @@ use std::sync::Arc; use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt}; use tracing::{debug, warn}; +#[cfg(test)] +mod correlation_dispatch_tests; #[cfg(test)] mod token_grant_ownership_tests; @@ -162,6 +166,7 @@ fn credential_generation_guard( /// /// A refused grant returns a [`TokenGrantInjectionError`](crate::l7::token_grant_injection::TokenGrantInjectionError) /// inside the report, so callers can tell it apart from other failures. +#[allow(clippy::too_many_arguments)] pub(crate) async fn prepare_inspected_request( req: crate::l7::provider::L7Request, ctx: &L7EvalContext, @@ -170,6 +175,7 @@ pub(crate) async fn prepare_inspected_request( request_info: &L7RequestInfo, request_id: &str, observation_context: Option<&openshell_core::endpoint_status::EndpointObservationContext>, + correlation: Option<&RequestCorrelation>, ) -> Result<(crate::l7::provider::L7Request, L7EvalContext)> { let mut scoped = ctx.clone(); let snapshot = crate::l7::token_grant_injection::pin_live_credentials(&mut scoped); @@ -212,7 +218,7 @@ pub(crate) async fn prepare_inspected_request( L7Protocol::Rest | L7Protocol::Websocket | L7Protocol::Sql => {} } let owners = admitted_token_grant_owners(engine, &scoped, ¤t_info)?; - crate::l7::token_grant_injection::inject_for_admitted_owners( + crate::l7::token_grant_injection::inject_for_admitted_owners_correlated( req, &scoped, snapshot, @@ -221,6 +227,7 @@ pub(crate) async fn prepare_inspected_request( observation_context.filter(|context| { context.matches_config(&config.policy_hash, Some(snapshot.revision)) }), + correlation, ) .await? } else { @@ -272,35 +279,48 @@ async fn reject_request_authority_mismatch( client: &mut W, ctx: &L7EvalContext, method: &str, + correlation: Option<&RequestCorrelation>, ) -> Result<()> where W: AsyncWrite + Unpin, { - let body = r#"{"error":"request_authority_mismatch","message":"HTTP request authority does not match the authorized tunnel endpoint"}"#; - let response = format!( - "HTTP/1.1 403 Forbidden\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", - body.len() - ); - client - .write_all(response.as_bytes()) - .await - .into_diagnostic()?; - client.flush().await.into_diagnostic()?; - - crate::telemetry::emit_decision( - [ - build_request_authority_mismatch_event(ctx, method), - build_request_authority_mismatch_finding(ctx), - ], - None, - ); + let activity = build_request_authority_mismatch_event(ctx, method); + let finding = build_request_authority_mismatch_finding(ctx); + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + correlation.emit(activity); + correlation.emit(finding); + crate::l7::rest::send_json_response_with_request_id( + &ctx.policy_name, + serde_json::json!({ + "error": "request_authority_mismatch", + "message": "HTTP request authority does not match the authorized tunnel endpoint", + }), + client, + "403 Forbidden", + correlation.request_id(), + ) + .await?; + } else { + crate::telemetry::emit_decision([activity, finding], None); + crate::l7::rest::send_json_response( + &ctx.policy_name, + serde_json::json!({ + "error": "request_authority_mismatch", + "message": "HTTP request authority does not match the authorized tunnel endpoint", + }), + client, + "403 Forbidden", + ) + .await?; + } Ok(()) } -/// Return the selected revision's inspection for policy evaluation, or emit a -/// rejection and return `None`. Non-MCP adapters retain their original inspection. -/// Record local rejections before client response delivery can fail. -pub(crate) async fn enforce_mcp_protocol_version( +#[allow(clippy::too_many_arguments)] +pub(crate) async fn enforce_mcp_protocol_version_correlated( config: &L7EndpointConfig, request: &crate::l7::provider::L7Request, mut info: crate::l7::jsonrpc::JsonRpcRequestInfo, @@ -308,6 +328,7 @@ pub(crate) async fn enforce_mcp_protocol_version( ctx: &L7EvalContext, redacted_target: &str, observer: Option<&EndpointObserver>, + correlation: Option<&RequestCorrelation>, ) -> Result> where W: AsyncWrite + Unpin, @@ -343,7 +364,7 @@ where crate::l7::mcp::selected_revision_context(request, version) ); let summary = l7_protocol_log_summary(None, Some(&info)); - egress_emit!(build_l7_request_event( + let event = build_l7_request_event( ctx, &request.action, redacted_target, @@ -351,7 +372,15 @@ where "l7-mcp", &reason, summary.as_deref(), - )); + ); + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_response_status(400); + correlation.record_final_http_outcome("denied"); + correlation.emit(event); + } else { + egress_emit!(event); + } emit_activity(ctx, true, "l7_parse_rejection"); let body = serde_json::json!({ "error": "invalid_mcp_request", @@ -362,13 +391,24 @@ where "method": request.action, "path": redacted_target, }); - crate::l7::rest::send_json_response( - &ctx.policy_name, - body, - client, - "400 Bad Request", - ) - .await?; + if let Some(correlation) = correlation { + crate::l7::rest::send_json_response_with_request_id( + &ctx.policy_name, + body, + client, + "400 Bad Request", + correlation.request_id(), + ) + .await?; + } else { + crate::l7::rest::send_json_response( + &ctx.policy_name, + body, + client, + "400 Bad Request", + ) + .await?; + } return Ok(None); } @@ -381,6 +421,7 @@ where ctx, redacted_target, observer, + correlation, ) .await?; return Ok(None); @@ -397,6 +438,7 @@ where ctx, redacted_target, observer, + correlation, ) .await?; Ok(None) @@ -405,6 +447,7 @@ where } /// Emit the same transport or policy rejection at initial and final inspection. +#[allow(clippy::too_many_arguments)] async fn reject_mcp_protocol_version( error: crate::l7::mcp::McpProtocolVersionError, request: &crate::l7::provider::L7Request, @@ -413,6 +456,7 @@ async fn reject_mcp_protocol_version( ctx: &L7EvalContext, redacted_target: &str, observer: Option<&EndpointObserver>, + correlation: Option<&RequestCorrelation>, ) -> Result<()> where W: AsyncWrite + Unpin, @@ -424,7 +468,7 @@ where } let reason = error.rejection_detail(request); let summary = l7_protocol_log_summary(None, Some(info)); - egress_emit!(build_l7_request_event( + let event = build_l7_request_event( ctx, &request.action, redacted_target, @@ -432,7 +476,21 @@ where "l7-mcp", &reason, summary.as_deref(), - )); + ); + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_response_status(match error { + crate::l7::mcp::McpProtocolVersionError::InvalidHeader + | crate::l7::mcp::McpProtocolVersionError::UnsupportedHeaderValue + | crate::l7::mcp::McpProtocolVersionError::InvalidRequestMetadata => 400, + crate::l7::mcp::McpProtocolVersionError::MethodNotAllowed => 405, + crate::l7::mcp::McpProtocolVersionError::NotAllowed(_) => 403, + }); + correlation.record_final_http_outcome("denied"); + correlation.emit(event); + } else { + egress_emit!(event); + } let deny_group = match error { crate::l7::mcp::McpProtocolVersionError::NotAllowed(_) => "l7_policy", crate::l7::mcp::McpProtocolVersionError::InvalidHeader @@ -452,19 +510,32 @@ where }); let allowed_methods = (error == crate::l7::mcp::McpProtocolVersionError::MethodNotAllowed).then_some("POST"); - crate::l7::rest::send_json_response_with_allow( - &ctx.policy_name, - body, - client, - error.http_status(), - allowed_methods, - ) - .await?; + if let Some(correlation) = correlation { + crate::l7::rest::send_json_response_with_allow_and_request_id( + &ctx.policy_name, + body, + client, + error.http_status(), + allowed_methods, + correlation.request_id(), + ) + .await?; + } else { + crate::l7::rest::send_json_response_with_allow( + &ctx.policy_name, + body, + client, + error.http_status(), + allowed_methods, + ) + .await?; + } Ok(()) } /// Reinspect the buffered outgoing MCP request after request transformations. /// The forwarding adapter must call this before any upstream request write. +#[cfg(test)] pub(crate) async fn enforce_final_mcp_protocol_version( config: &L7EndpointConfig, request: &crate::l7::provider::L7Request, @@ -473,6 +544,31 @@ pub(crate) async fn enforce_final_mcp_protocol_version( redacted_target: &str, observer: Option<&EndpointObserver>, ) -> Result +where + W: AsyncWrite + Unpin, +{ + enforce_final_mcp_protocol_version_correlated( + config, + request, + client, + ctx, + redacted_target, + observer, + None, + ) + .await +} + +#[allow(clippy::too_many_arguments)] +pub(crate) async fn enforce_final_mcp_protocol_version_correlated( + config: &L7EndpointConfig, + request: &crate::l7::provider::L7Request, + client: &mut W, + ctx: &L7EvalContext, + redacted_target: &str, + observer: Option<&EndpointObserver>, + correlation: Option<&RequestCorrelation>, +) -> Result where W: AsyncWrite + Unpin, { @@ -483,7 +579,7 @@ where request, crate::l7::jsonrpc::JsonRpcInspectionOptions::for_config(config), )?; - Ok(enforce_mcp_protocol_version( + Ok(enforce_mcp_protocol_version_correlated( config, request, info, @@ -491,6 +587,7 @@ where ctx, redacted_target, observer, + correlation, ) .await? .is_some()) @@ -593,6 +690,7 @@ pub(crate) async fn reject_credential_resolution( ctx: &L7EvalContext, method: &str, error: &secrets::UnresolvedPlaceholderError, + correlation: Option<&RequestCorrelation>, ) -> Result<()> where W: AsyncWrite + Unpin, @@ -605,22 +703,21 @@ where "500 Internal Server Error" }; let body = if placement_mismatch { - r#"{"error":"credential_placement_mismatch","message":"Credential is not authorized for this request location"}"# + serde_json::json!({ + "error": "credential_placement_mismatch", + "message": "Credential is not authorized for this request location", + }) } else if endpoint_mismatch { - r#"{"error":"credential_endpoint_mismatch","message":"Credential is not authorized for this request endpoint"}"# + serde_json::json!({ + "error": "credential_endpoint_mismatch", + "message": "Credential is not authorized for this request endpoint", + }) } else { - r#"{"error":"credential_unavailable","message":"Credential placeholder could not be resolved"}"# + serde_json::json!({ + "error": "credential_unavailable", + "message": "Credential placeholder could not be resolved", + }) }; - let response = format!( - "HTTP/1.1 {status}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", - body.len() - ); - client - .write_all(response.as_bytes()) - .await - .into_diagnostic()?; - client.flush().await.into_diagnostic()?; - let mut events = if placement_mismatch { vec![ HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) @@ -652,28 +749,62 @@ where if endpoint_mismatch { events.push(build_credential_endpoint_mismatch_finding(ctx)); } - crate::telemetry::emit_decision(events, None); + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_response_status(if endpoint_mismatch || placement_mismatch { + 403 + } else { + 500 + }); + correlation.record_final_http_outcome(if endpoint_mismatch || placement_mismatch { + "denied" + } else { + "platform_error" + }); + for event in events { + correlation.emit(event); + } + crate::l7::rest::send_json_response_with_request_id( + &ctx.policy_name, + body, + client, + status, + correlation.request_id(), + ) + .await?; + } else { + crate::telemetry::emit_decision(events, None); + crate::l7::rest::send_json_response(&ctx.policy_name, body, client, status).await?; + } Ok(()) } pub(crate) async fn reject_body_credential( client: &mut C, error: secrets::body::BodyCredentialError, + ctx: &L7EvalContext, + correlation: Option<&RequestCorrelation>, ) -> Result<()> { let body = serde_json::json!({"error": { "code": "credential_placeholder_in_request_body", "reason": error.reason(), "message": "A credential placeholder in the request body cannot be forwarded. Remove the reference from conversation history or restore provider access; body credential rewriting is disabled." - }}).to_string(); - let response = format!( - "HTTP/1.1 403 Forbidden\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", - body.len() - ); - client - .write_all(response.as_bytes()) + }}); + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + crate::l7::rest::send_json_response_with_request_id( + &ctx.policy_name, + body, + client, + "403 Forbidden", + correlation.request_id(), + ) .await - .into_diagnostic()?; - client.flush().await.into_diagnostic() + } else { + crate::l7::rest::send_json_response(&ctx.policy_name, body, client, "403 Forbidden").await + } } struct InspectedForwarding<'a> { @@ -681,6 +812,7 @@ struct InspectedForwarding<'a> { engine: &'a TunnelPolicyEngine, request_info: &'a L7RequestInfo, request_id: &'a str, + correlation: Option<&'a RequestCorrelation>, response_chain: &'a [openshell_supervisor_middleware::ChainEntry], websocket_middleware: bool, observation_context: Option<&'a openshell_core::endpoint_status::EndpointObservationContext>, @@ -705,6 +837,7 @@ where engine, request_info, request_id, + correlation, response_chain, websocket_middleware, observation_context, @@ -717,11 +850,17 @@ where request_info, request_id, observation_context, + correlation, ) .await { Ok(prepared) => prepared, Err(error) => { + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_response_status(502); + correlation.record_final_http_outcome("platform_error"); + } if let Some(refusal) = error.downcast_ref::() { @@ -740,7 +879,8 @@ where write_bad_gateway_response(client, Some(request_id)).await?; } else { warn!(error = %error, "Token grant failed before forwarding"); - write_bad_gateway_response(client, None).await?; + write_bad_gateway_response(client, correlation.map(RequestCorrelation::request_id)) + .await?; } return Ok(None); } @@ -790,11 +930,15 @@ where engine.middleware_runner(), Some(engine.generation_guard()), )), - observer.as_ref(), + RelayHttpInstrumentation { + observer: observer.as_ref(), + correlation, + }, ) .await } +#[cfg(test)] async fn relay_http_request_with_credential_rejection( request: &crate::l7::provider::L7Request, client: &mut C, @@ -814,11 +958,17 @@ where options, ctx, response_middleware, - None, + RelayHttpInstrumentation::default(), ) .await } +#[derive(Clone, Copy, Default)] +struct RelayHttpInstrumentation<'a> { + observer: Option<&'a EndpointObserver>, + correlation: Option<&'a RequestCorrelation>, +} + async fn relay_http_request_with_credential_rejection_observed( request: &crate::l7::provider::L7Request, client: &mut C, @@ -826,20 +976,25 @@ async fn relay_http_request_with_credential_rejection_observed( options: crate::l7::rest::RelayRequestOptions<'_>, ctx: &L7EvalContext, response_middleware: Option>, - observer: Option<&EndpointObserver>, + instrumentation: RelayHttpInstrumentation<'_>, ) -> Result> where C: AsyncRead + AsyncWrite + Unpin, U: AsyncRead + AsyncWrite + Unpin, { + let RelayHttpInstrumentation { + observer, + correlation, + } = instrumentation; match Box::pin( - crate::l7::rest::relay_http_request_with_response_middleware_guarded_observed( + crate::l7::rest::relay_http_request_with_response_middleware_guarded_observed_correlated( request, client, upstream, options, response_middleware, observer, + correlation, ), ) .await @@ -854,14 +1009,15 @@ where observer.observe(EndpointResult::PolicyDenied); } let _ = upstream.shutdown().await; - reject_body_credential(client, *error).await?; + reject_body_credential(client, *error, ctx, correlation).await?; Ok(None) } else if let Some(error) = report.downcast_ref::() { if let Some(observer) = observer { observer.observe_credential_failure(error.is_endpoint_mismatch()); } - reject_credential_resolution(client, ctx, &request.action, error).await?; + reject_credential_resolution(client, ctx, &request.action, error, correlation) + .await?; Ok(None) } else if report .downcast_ref::() @@ -870,6 +1026,15 @@ where if let Some(observer) = observer { observer.observe_credential_failure(false); } + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("platform_error"); + let mut event = build_credential_resolution_event(ctx, &request.action, false); + if let openshell_ocsf::OcsfEvent::HttpActivity(http) = &mut event { + http.http_response = None; + } + correlation.emit(event); + } Err(report) } else { Err(report) @@ -1027,6 +1192,15 @@ fn parse_rejection_detail(error: &str, mode: ParseRejectionMode) -> String { } fn emit_parse_rejection(ctx: &L7EvalContext, detail: &str, engine_type: &str) { + emit_parse_rejection_correlated(ctx, detail, engine_type, None); +} + +fn emit_parse_rejection_correlated( + ctx: &L7EvalContext, + detail: &str, + engine_type: &str, + correlation: Option<&RequestCorrelation>, +) { let policy_name = if ctx.policy_name.is_empty() { "-" } else { @@ -1046,7 +1220,11 @@ fn emit_parse_rejection(ctx: &L7EvalContext, detail: &str, engine_type: &str) { )) .status_detail(detail) .build(); - egress_emit!(event); + if let Some(correlation) = correlation { + correlation.emit(event); + } else { + egress_emit!(event); + } emit_activity(ctx, true, "l7_parse_rejection"); } @@ -1075,6 +1253,7 @@ async fn deny_unsupported_upgrade_if_requested( ctx: &L7EvalContext, observer: Option<&EndpointObserver>, client: &mut C, + correlation: Option<&RequestCorrelation>, ) -> Result where C: AsyncRead + AsyncWrite + Unpin + Send, @@ -1088,17 +1267,34 @@ where if let Some(observer) = observer { observer.observe(EndpointResult::PolicyDenied); } - emit_parse_rejection(ctx, detail, engine_type_for_protocol(config.protocol)); - crate::l7::rest::send_json_response( - &ctx.policy_name, - serde_json::json!({ - "error": "unsupported_l7_protocol", - "detail": detail, - }), - client, - "403 Forbidden", - ) - .await?; + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + } + emit_parse_rejection_correlated( + ctx, + detail, + engine_type_for_protocol(config.protocol), + correlation, + ); + let body = serde_json::json!({ + "error": "unsupported_l7_protocol", + "detail": detail, + }); + if let Some(correlation) = correlation { + crate::l7::rest::send_json_response_with_request_id( + &ctx.policy_name, + body, + client, + "403 Forbidden", + correlation.request_id(), + ) + .await?; + } else { + crate::l7::rest::send_json_response(&ctx.policy_name, body, client, "403 Forbidden") + .await?; + } Ok(true) } @@ -1235,21 +1431,37 @@ where return Ok(()); } }; + let correlation = RequestCorrelation::new(&req, ctx); + let mut audited_client = correlation.observe_response(client); + let client = &mut audited_client; if !request_authority_matches_endpoint(&req, ctx) { - reject_request_authority_mismatch(client, ctx, &req.action).await?; + reject_request_authority_mismatch(client, ctx, &req.action, Some(&correlation)).await?; return Ok(()); } let route_target = match secrets::redact_target_for_policy(&req.target) { Ok(target) => target, Err(error) => { - reject_credential_resolution(client, ctx, &req.action, &error).await?; + reject_credential_resolution(client, ctx, &req.action, &error, Some(&correlation)) + .await?; return Ok(()); } }; let Some(config) = select_l7_config_for_path(configs, &route_target) else { let reason = "no L7 endpoint path matched request"; - emit_l7_request_log(ctx, &req.action, &route_target, "deny", "l7", reason, None); + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + emit_l7_request_log_correlated( + ctx, + &req.action, + &route_target, + "deny", + "l7", + reason, + None, + Some(&correlation), + ); crate::l7::rest::RestProvider::default() .deny_with_redacted_target( &req, @@ -1257,7 +1469,10 @@ where reason, client, None, - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some( + crate::l7::rest::DenyResponseContext::from_l7_context(ctx) + .with_request_id(correlation.request_id()), + ), ) .await?; return Ok(()); @@ -1285,7 +1500,21 @@ where if let Some(observer) = observer.as_ref() { observer.observe(EndpointResult::PolicyDenied); } - emit_parse_rejection(ctx, detail, engine_type_for_protocol(config.protocol)); + emit_parse_rejection_correlated( + ctx, + detail, + engine_type_for_protocol(config.protocol), + Some(&correlation), + ); + { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + } + let deny_context = crate::l7::rest::DenyResponseContext::from_l7_context(ctx); + let deny_context = Some(&correlation).map_or(deny_context, |correlation| { + deny_context.with_request_id(correlation.request_id()) + }); crate::l7::rest::RestProvider::default() .deny_with_redacted_target( &req, @@ -1293,13 +1522,20 @@ where detail, client, None, - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some(deny_context), ) .await?; return Ok(()); } - if deny_unsupported_upgrade_if_requested(&req, config, ctx, observer.as_ref(), client) - .await? + if deny_unsupported_upgrade_if_requested( + &req, + config, + ctx, + observer.as_ref(), + client, + Some(&correlation), + ) + .await? { return Ok(()); } @@ -1324,7 +1560,16 @@ where } else { let detail = parse_rejection_detail(&e.to_string(), ParseRejectionMode::L7Endpoint); - emit_parse_rejection(ctx, &detail, "l7-graphql"); + { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + } + emit_parse_rejection_correlated( + ctx, + &detail, + "l7-graphql", + Some(&correlation), + ); } return Ok(()); } @@ -1360,7 +1605,16 @@ where &e.to_string(), ParseRejectionMode::L7Endpoint, ); - emit_parse_rejection(ctx, &detail, "l7-jsonrpc"); + { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + } + emit_parse_rejection_correlated( + ctx, + &detail, + "l7-jsonrpc", + Some(&correlation), + ); } return Ok(()); } @@ -1380,13 +1634,14 @@ where if let Some(observer) = observer.as_ref() { observer.observe_credential_failure(error.is_endpoint_mismatch()); } - reject_credential_resolution(client, ctx, &req.action, &error).await?; + reject_credential_resolution(client, ctx, &req.action, &error, Some(&correlation)) + .await?; return Ok(()); } }; if let Some(info) = jsonrpc_info.take() { - let Some(inspected) = enforce_mcp_protocol_version( + let Some(inspected) = enforce_mcp_protocol_version_correlated( config, &req, info, @@ -1394,6 +1649,7 @@ where ctx, &redacted_target, observer.as_ref(), + Some(&correlation), ) .await? else { @@ -1410,14 +1666,34 @@ where }; let websocket_request = crate::l7::rest::request_is_websocket_upgrade(&req.raw_header); if config.protocol == L7Protocol::Websocket && !websocket_request { + let reason = "websocket endpoint requires a valid WebSocket upgrade request"; + { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + } + emit_l7_request_log_correlated( + ctx, + &request_info.action, + &redacted_target, + "deny", + "l7-websocket", + reason, + None, + Some(&correlation), + ); + let deny_context = crate::l7::rest::DenyResponseContext::from_l7_context(ctx); + let deny_context = Some(&correlation).map_or(deny_context, |correlation| { + deny_context.with_request_id(correlation.request_id()) + }); crate::l7::rest::RestProvider::default() .deny_with_redacted_target( &req, &ctx.policy_name, - "websocket endpoint requires a valid WebSocket upgrade request", + reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some(deny_context), ) .await?; return Ok(()); @@ -1441,10 +1717,19 @@ where (false, EnforcementMode::Audit) => "audit", (false, EnforcementMode::Enforce) => "deny", }; + { + correlation.record_enforcement_action(if allowed { + "allow" + } else if config.enforcement == EnforcementMode::Audit && !force_deny { + "audit" + } else { + "block" + }); + } let engine_type = engine_type_for_protocol(config.protocol); let protocol_summary = l7_protocol_log_summary(graphql_info.as_ref(), jsonrpc_info.as_ref()); - emit_l7_request_log( + emit_l7_request_log_correlated( ctx, &request_info.action, &redacted_target, @@ -1452,19 +1737,21 @@ where engine_type, &reason, protocol_summary.as_deref(), + Some(&correlation), ); if allowed || (config.enforcement == EnforcementMode::Audit && !force_deny) { let chain = engine.query_middleware_chain(&middleware_network_input(ctx))?; let response_chain = chain.clone(); - let request_id = uuid::Uuid::new_v4().to_string(); + let request_id = correlation.request_id(); let websocket_chain = websocket_request.then(|| chain.clone()); // Route selection resolved `config` per request, so re-check the // body against that protocol's policy after every transforming // stage (a no-op for REST and websocket, whose policy inputs the // chain cannot mutate). - let validate = transformed_body_validator(config, &engine, ctx, &request_info); - let middleware_result = apply_middleware_chain_with_request_id( + let validate = + transformed_body_validator(config, &engine, ctx, &request_info, Some(&correlation)); + let middleware_result = apply_middleware_chain_with_request_id_correlated( req, client, ctx, @@ -1472,7 +1759,8 @@ where engine.middleware_runner(), engine.generation_guard(), openshell_supervisor_middleware::TransformedBodyPolicy::Reevaluate(&validate), - &request_id, + request_id, + Some(&correlation), ) .await; let req = match middleware_result? { @@ -1481,6 +1769,15 @@ where if let Some(observer) = observer.as_ref() { observer.observe(EndpointResult::PolicyDenied); } + { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if denial.is_some() { + "denied" + } else { + "platform_error" + }); + } let denied_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -1494,11 +1791,17 @@ where ctx, denial.as_ref(), &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); } MiddlewareApplyResult::AdmissionExhausted => { + { + correlation.record_enforcement_action("block"); + correlation.record_response_status(503); + correlation.record_final_http_outcome("platform_error"); + } let unavailable_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -1511,6 +1814,7 @@ where client, ctx, &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); @@ -1529,18 +1833,33 @@ where Ok(preflight) => preflight, Err(error) => { warn!(error = %error, "WebSocket middleware preflight failed"); - write_bad_gateway_response(client, None).await?; + { + correlation.record_enforcement_action("block"); + correlation.record_response_status(502); + correlation.record_final_http_outcome("platform_error"); + } + write_bad_gateway_response(client, Some(correlation.request_id())).await?; return Ok(()); } }; crate::l7::middleware::emit_websocket_preflight_events(ctx, &preflight); if preflight.terminal_reason.is_some() { + { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if preflight.denial.is_some() { + "denied" + } else { + "platform_error" + }); + } crate::l7::middleware::send_middleware_rejection_response( &req, client, ctx, preflight.denial.as_ref(), &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); @@ -1560,7 +1879,8 @@ where config, engine: &engine, request_info: &request_info, - request_id: &request_id, + request_id, + correlation: Some(&correlation), response_chain: &response_chain, websocket_middleware: middleware_session.is_some(), observation_context: observation_context.as_ref(), @@ -1652,6 +1972,14 @@ where if let Some(observer) = observer.as_ref() { observer.observe(EndpointResult::PolicyDenied); } + { + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + } + let deny_context = crate::l7::rest::DenyResponseContext::from_l7_context(ctx); + let deny_context = Some(&correlation).map_or(deny_context, |correlation| { + deny_context.with_request_id(correlation.request_id()) + }); crate::l7::rest::RestProvider::default() .deny_with_redacted_target( &req, @@ -1659,7 +1987,7 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some(deny_context), ) .await?; return Ok(()); @@ -1677,7 +2005,8 @@ fn select_l7_config_for_path<'a>( .max_by_key(|config| config.path_specificity()) } -fn emit_l7_request_log( +#[allow(clippy::too_many_arguments)] +fn emit_l7_request_log_correlated( ctx: &L7EvalContext, action: &str, redacted_target: &str, @@ -1685,6 +2014,7 @@ fn emit_l7_request_log( engine_type: &str, reason: &str, protocol_summary: Option<&str>, + correlation: Option<&RequestCorrelation>, ) { let event = build_l7_request_event( ctx, @@ -1695,7 +2025,11 @@ fn emit_l7_request_log( reason, protocol_summary, ); - egress_emit!(event); + if let Some(correlation) = correlation { + correlation.emit(event); + } else { + egress_emit!(event); + } emit_activity(ctx, decision_str == "deny", "l7_policy"); } @@ -2099,11 +2433,23 @@ where return Ok(()); // Close connection on parse error } }; + let correlation = RequestCorrelation::new(&req, ctx); + let mut audited_client = correlation.observe_response(client); + let client = &mut audited_client; if !request_authority_matches_endpoint(&req, ctx) { - reject_request_authority_mismatch(client, ctx, &req.action).await?; + reject_request_authority_mismatch(client, ctx, &req.action, Some(&correlation)).await?; return Ok(()); } - if deny_unsupported_upgrade_if_requested(&req, config, ctx, None, client).await? { + if deny_unsupported_upgrade_if_requested( + &req, + config, + ctx, + None, + client, + Some(&correlation), + ) + .await? + { return Ok(()); } @@ -2116,7 +2462,8 @@ where let redacted_target = match secrets::redact_target_for_policy(&req.target) { Ok(target) => target, Err(error) => { - reject_credential_resolution(client, ctx, &req.action, &error).await?; + reject_credential_resolution(client, ctx, &req.action, &error, Some(&correlation)) + .await?; return Ok(()); } }; @@ -2130,14 +2477,31 @@ where }; let websocket_request = crate::l7::rest::request_is_websocket_upgrade(&req.raw_header); if config.protocol == L7Protocol::Websocket && !websocket_request { + let reason = "websocket endpoint requires a valid WebSocket upgrade request"; + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); + emit_l7_request_log_correlated( + ctx, + &request_info.action, + &redacted_target, + "deny", + "l7-websocket", + reason, + None, + Some(&correlation), + ); provider .deny_with_redacted_target( &req, &ctx.policy_name, - "websocket endpoint requires a valid WebSocket upgrade request", + reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some( + crate::l7::rest::DenyResponseContext::from_l7_context(ctx) + .with_request_id(correlation.request_id()), + ), ) .await?; return Ok(()); @@ -2169,6 +2533,13 @@ where (false, EnforcementMode::Audit, _) => "audit", (false, EnforcementMode::Enforce, _) => "deny", }; + correlation.record_enforcement_action(if allowed { + "allow" + } else if config.enforcement == EnforcementMode::Audit { + "audit" + } else { + "block" + }); // Log every L7 decision as an OCSF HTTP Activity event. // Uses redacted_target (path only, no query params) to avoid logging secrets. @@ -2202,18 +2573,18 @@ where request_info.action, ctx.host, ctx.port, redacted_target, reason, )) .build(); - egress_emit!(event); + correlation.emit(event); } if allowed || config.enforcement == EnforcementMode::Audit { let chain = engine.query_middleware_chain(&middleware_network_input(ctx))?; let response_chain = chain.clone(); - let request_id = uuid::Uuid::new_v4().to_string(); + let request_id = correlation.request_id(); let websocket_chain = websocket_request.then(|| chain.clone()); // REST and websocket-upgrade policy evaluates only the method, // path, and query, which a middleware result cannot mutate, so no // per-stage body re-check is needed. - let middleware_result = apply_middleware_chain_with_request_id( + let middleware_result = apply_middleware_chain_with_request_id_correlated( req, client, ctx, @@ -2221,12 +2592,20 @@ where engine.middleware_runner(), engine.generation_guard(), openshell_supervisor_middleware::TransformedBodyPolicy::NotPolicyRelevant, - &request_id, + request_id, + Some(&correlation), ) .await; let req = match middleware_result? { MiddlewareApplyResult::Allowed(request) => request, MiddlewareApplyResult::Denied { denial, .. } => { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if denial.is_some() { + "denied" + } else { + "platform_error" + }); let denied_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -2240,11 +2619,15 @@ where ctx, denial.as_ref(), &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); } MiddlewareApplyResult::AdmissionExhausted => { + correlation.record_enforcement_action("block"); + correlation.record_response_status(503); + correlation.record_final_http_outcome("platform_error"); let unavailable_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -2257,6 +2640,7 @@ where client, ctx, &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); @@ -2275,18 +2659,29 @@ where Ok(preflight) => preflight, Err(error) => { warn!(error = %error, "WebSocket middleware preflight failed"); - write_bad_gateway_response(client, None).await?; + correlation.record_enforcement_action("block"); + correlation.record_response_status(502); + correlation.record_final_http_outcome("platform_error"); + write_bad_gateway_response(client, Some(correlation.request_id())).await?; return Ok(()); } }; crate::l7::middleware::emit_websocket_preflight_events(ctx, &preflight); if preflight.terminal_reason.is_some() { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if preflight.denial.is_some() { + "denied" + } else { + "platform_error" + }); crate::l7::middleware::send_middleware_rejection_response( &req, client, ctx, preflight.denial.as_ref(), &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); @@ -2306,7 +2701,8 @@ where config, engine, request_info: &request_info, - request_id: &request_id, + request_id, + correlation: Some(&correlation), response_chain: &response_chain, websocket_middleware: middleware_session.is_some(), observation_context: observation_context.as_ref(), @@ -2386,6 +2782,8 @@ where } } else { // Enforce mode: deny with 403 and close connection (use redacted target) + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); provider .deny_with_redacted_target( &req, @@ -2393,7 +2791,10 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some( + crate::l7::rest::DenyResponseContext::from_l7_context(ctx) + .with_request_id(correlation.request_id()), + ), ) .await?; return Ok(()); @@ -2528,6 +2929,9 @@ where let req = parsed.request; let jsonrpc_info = parsed.info; + let correlation = RequestCorrelation::new(&req, ctx); + let mut audited_client = correlation.observe_response(client); + let client = &mut audited_client; let observer = EndpointObserver::begin_captured( ctx.endpoint_observation_tx.as_ref(), config, @@ -2539,11 +2943,18 @@ where if let Some(observer) = observer.as_ref() { observer.observe(EndpointResult::PolicyDenied); } - reject_request_authority_mismatch(client, ctx, &req.action).await?; + reject_request_authority_mismatch(client, ctx, &req.action, Some(&correlation)).await?; return Ok(()); } - if deny_unsupported_upgrade_if_requested(&req, config, ctx, observer.as_ref(), client) - .await? + if deny_unsupported_upgrade_if_requested( + &req, + config, + ctx, + observer.as_ref(), + client, + Some(&correlation), + ) + .await? { return Ok(()); } @@ -2557,12 +2968,13 @@ where if let Some(observer) = observer.as_ref() { observer.observe_credential_failure(error.is_endpoint_mismatch()); } - reject_credential_resolution(client, ctx, &req.action, &error).await?; + reject_credential_resolution(client, ctx, &req.action, &error, Some(&correlation)) + .await?; return Ok(()); } }; - let Some(jsonrpc_info) = enforce_mcp_protocol_version( + let Some(jsonrpc_info) = enforce_mcp_protocol_version_correlated( config, &req, jsonrpc_info, @@ -2570,6 +2982,7 @@ where ctx, &redacted_target, observer.as_ref(), + Some(&correlation), ) .await? else { @@ -2604,6 +3017,13 @@ where (false, EnforcementMode::Audit) => "audit", (false, EnforcementMode::Enforce) => "deny", }; + correlation.record_enforcement_action(if allowed { + "allow" + } else if config.enforcement == EnforcementMode::Audit && !force_deny { + "audit" + } else { + "block" + }); { let (action_id, disposition_id, severity) = match decision_str { @@ -2636,19 +3056,20 @@ where &reason, )) .build(); - egress_emit!(event); + correlation.emit(event); } if allowed || (config.enforcement == EnforcementMode::Audit && !force_deny) { let chain = engine.query_middleware_chain(&middleware_network_input(ctx))?; let response_chain = chain.clone(); - let request_id = uuid::Uuid::new_v4().to_string(); + let request_id = correlation.request_id(); // Policy admitted the original body above; re-check the body // against the same body-aware policy after every transforming // stage so a middleware cannot smuggle a denied operation to the // upstream or the next stage. - let validate = transformed_body_validator(config, engine, ctx, &request_info); - let req = match apply_middleware_chain_with_request_id( + let validate = + transformed_body_validator(config, engine, ctx, &request_info, Some(&correlation)); + let req = match apply_middleware_chain_with_request_id_correlated( req, client, ctx, @@ -2656,7 +3077,8 @@ where engine.middleware_runner(), engine.generation_guard(), openshell_supervisor_middleware::TransformedBodyPolicy::Reevaluate(&validate), - &request_id, + request_id, + Some(&correlation), ) .await? { @@ -2665,6 +3087,13 @@ where if let Some(observer) = observer.as_ref() { observer.observe(EndpointResult::PolicyDenied); } + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if denial.is_some() { + "denied" + } else { + "platform_error" + }); let denied_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -2678,11 +3107,15 @@ where ctx, denial.as_ref(), &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); } MiddlewareApplyResult::AdmissionExhausted => { + correlation.record_enforcement_action("block"); + correlation.record_response_status(503); + correlation.record_final_http_outcome("platform_error"); let unavailable_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -2695,6 +3128,7 @@ where client, ctx, &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); @@ -2710,7 +3144,8 @@ where config, engine, request_info: &request_info, - request_id: &request_id, + request_id, + correlation: Some(&correlation), response_chain: &response_chain, websocket_middleware: false, observation_context: observation_context.as_ref(), @@ -2738,6 +3173,8 @@ where if let Some(observer) = observer.as_ref() { observer.observe(EndpointResult::PolicyDenied); } + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); crate::l7::rest::RestProvider::default() .deny_with_redacted_target( &req, @@ -2745,7 +3182,10 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some( + crate::l7::rest::DenyResponseContext::from_l7_context(ctx) + .with_request_id(correlation.request_id()), + ), ) .await?; return Ok(()); @@ -2802,12 +3242,24 @@ where return Ok(()); } }; + let correlation = RequestCorrelation::new(&req, ctx); + let mut audited_client = correlation.observe_response(client); + let client = &mut audited_client; if !request_authority_matches_endpoint(&req, ctx) { - reject_request_authority_mismatch(client, ctx, &req.action).await?; + reject_request_authority_mismatch(client, ctx, &req.action, Some(&correlation)).await?; return Ok(()); } - if deny_unsupported_upgrade_if_requested(&req, config, ctx, None, client).await? { + if deny_unsupported_upgrade_if_requested( + &req, + config, + ctx, + None, + client, + Some(&correlation), + ) + .await? + { return Ok(()); } @@ -2830,7 +3282,9 @@ where } else { let detail = parse_rejection_detail(&e.to_string(), ParseRejectionMode::L7Endpoint); - emit_parse_rejection(ctx, &detail, "l7-graphql"); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + emit_parse_rejection_correlated(ctx, &detail, "l7-graphql", Some(&correlation)); } return Ok(()); } @@ -2840,7 +3294,7 @@ where // without Host must still be denied if that body contains reserved // credential markers, so repeat the authority check on the full request. if !request_authority_matches_endpoint(&req, ctx) { - reject_request_authority_mismatch(client, ctx, &req.action).await?; + reject_request_authority_mismatch(client, ctx, &req.action, Some(&correlation)).await?; return Ok(()); } @@ -2851,7 +3305,8 @@ where let redacted_target = match secrets::redact_target_for_policy(&req.target) { Ok(target) => target, Err(error) => { - reject_credential_resolution(client, ctx, &req.action, &error).await?; + reject_credential_resolution(client, ctx, &req.action, &error, Some(&correlation)) + .await?; return Ok(()); } }; @@ -2886,6 +3341,13 @@ where (false, EnforcementMode::Audit) => "audit", (false, EnforcementMode::Enforce) => "deny", }; + correlation.record_enforcement_action(if allowed { + "allow" + } else if config.enforcement == EnforcementMode::Audit && !force_deny { + "audit" + } else { + "block" + }); { let (action_id, disposition_id, severity) = match decision_str { @@ -2918,19 +3380,20 @@ where request_info.action, ctx.host, ctx.port, redacted_target, reason, )) .build(); - egress_emit!(event); + correlation.emit(event); } if allowed || (config.enforcement == EnforcementMode::Audit && !force_deny) { let chain = engine.query_middleware_chain(&middleware_network_input(ctx))?; let response_chain = chain.clone(); - let request_id = uuid::Uuid::new_v4().to_string(); + let request_id = correlation.request_id(); // Policy admitted the original body above; re-check the body // against the same body-aware policy after every transforming // stage so a middleware cannot smuggle a denied operation to the // upstream or the next stage. - let validate = transformed_body_validator(config, engine, ctx, &request_info); - let req = match apply_middleware_chain_with_request_id( + let validate = + transformed_body_validator(config, engine, ctx, &request_info, Some(&correlation)); + let req = match apply_middleware_chain_with_request_id_correlated( req, client, ctx, @@ -2938,12 +3401,20 @@ where engine.middleware_runner(), engine.generation_guard(), openshell_supervisor_middleware::TransformedBodyPolicy::Reevaluate(&validate), - &request_id, + request_id, + Some(&correlation), ) .await? { MiddlewareApplyResult::Allowed(request) => request, MiddlewareApplyResult::Denied { denial, .. } => { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if denial.is_some() { + "denied" + } else { + "platform_error" + }); let denied_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -2957,11 +3428,15 @@ where ctx, denial.as_ref(), &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); } MiddlewareApplyResult::AdmissionExhausted => { + correlation.record_enforcement_action("block"); + correlation.record_response_status(503); + correlation.record_final_http_outcome("platform_error"); let unavailable_request = crate::l7::provider::L7Request { action: request_info.action.clone(), target: redacted_target.clone(), @@ -2974,6 +3449,7 @@ where client, ctx, &redacted_target, + Some(correlation.request_id()), ) .await?; return Ok(()); @@ -2989,7 +3465,8 @@ where config, engine, request_info: &request_info, - request_id: &request_id, + request_id, + correlation: Some(&correlation), response_chain: &response_chain, websocket_middleware: false, observation_context: observation_context.as_ref(), @@ -3026,6 +3503,8 @@ where } } } else { + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); crate::l7::rest::RestProvider::default() .deny_with_redacted_target( &req, @@ -3033,7 +3512,10 @@ where &reason, client, Some(&redacted_target), - Some(crate::l7::rest::DenyResponseContext::from_l7_context(ctx)), + Some( + crate::l7::rest::DenyResponseContext::from_l7_context(ctx) + .with_request_id(correlation.request_id()), + ), ) .await?; return Ok(()); @@ -3279,6 +3761,7 @@ fn reevaluate_transformed_body( ctx: &L7EvalContext, request_info: &L7RequestInfo, body: &[u8], + correlation: Option<&RequestCorrelation>, ) -> Result> { let (engine_type, transformed_info) = match config.protocol { // REST and websocket-upgrade policy evaluates only the method, path, @@ -3326,7 +3809,14 @@ fn reevaluate_transformed_body( if let Some(reason) = l7_request_hard_deny_reason(config.protocol, &transformed_info) { let reason = format!("middleware transformation rejected: {reason}"); - emit_transformed_body_decision(ctx, request_info, engine_type, "deny", &reason); + emit_transformed_body_decision( + ctx, + request_info, + engine_type, + "deny", + &reason, + correlation, + ); return Ok(Some(reason)); } @@ -3336,10 +3826,17 @@ fn reevaluate_transformed_body( } let reason = format!("middleware transformation denied by policy: {reason}"); if config.enforcement == EnforcementMode::Audit { - emit_transformed_body_decision(ctx, request_info, engine_type, "audit", &reason); + emit_transformed_body_decision( + ctx, + request_info, + engine_type, + "audit", + &reason, + correlation, + ); return Ok(None); } - emit_transformed_body_decision(ctx, request_info, engine_type, "deny", &reason); + emit_transformed_body_decision(ctx, request_info, engine_type, "deny", &reason, correlation); Ok(Some(reason)) } @@ -3351,8 +3848,11 @@ pub(crate) fn transformed_body_validator<'a>( engine: &'a TunnelPolicyEngine, ctx: &'a L7EvalContext, request_info: &'a L7RequestInfo, + correlation: Option<&'a RequestCorrelation>, ) -> impl Fn(&[u8]) -> Result> + Send + Sync + 'a { - move |body: &[u8]| reevaluate_transformed_body(config, engine, ctx, request_info, body) + move |body: &[u8]| { + reevaluate_transformed_body(config, engine, ctx, request_info, body, correlation) + } } /// Log the post-transformation policy decision as an OCSF HTTP Activity @@ -3364,6 +3864,7 @@ fn emit_transformed_body_decision( engine_type: &str, decision_str: &str, reason: &str, + correlation: Option<&RequestCorrelation>, ) { let (action_id, disposition_id, severity) = match decision_str { "deny" => (ActionId::Denied, DispositionId::Blocked, SeverityId::Medium), @@ -3373,7 +3874,7 @@ fn emit_transformed_body_decision( SeverityId::Informational, ), }; - let event = HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + let mut event = HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(ActivityId::Other) .action(action_id) .disposition(disposition_id) @@ -3384,12 +3885,18 @@ fn emit_transformed_body_decision( )) .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) .firewall_rule(&ctx.policy_name, engine_type) + .status_detail(reason) .message(format!( "L7_REQUEST_TRANSFORMED {decision_str} {} {}:{}{} reason={}", request_info.action, ctx.host, ctx.port, request_info.target, reason )) .build(); - egress_emit!(event); + if decision_str == "audit" + && let openshell_ocsf::OcsfEvent::HttpActivity(http) = &mut event + { + http.base.unmapped = Some(serde_json::json!({"transformed_body_policy": "audit"})); + } + crate::l7::middleware::emit_request_events(correlation, [event]); } fn jsonrpc_policy_input(info: &crate::l7::jsonrpc::JsonRpcRequestInfo) -> serde_json::Value { @@ -3570,8 +4077,20 @@ where .as_ref() .filter(|_| ctx.token_grant_resolver.is_some()) .and_then(EndpointObservationSender::capture); + let middleware_selection = middleware_engine.map(|engine| { + engine.query_middleware_chain_with_generation(&middleware_network_input(ctx)) + }); + let inspected = matches!( + &middleware_selection, + Some(Ok((chain, _))) if !chain.is_empty() + ); // Read next request from client. - let req = match provider.parse_request(client).await { + let parsed = if inspected { + provider.parse_request(client).await + } else { + provider.parse_uninspected_request(client).await + }; + let req = match parsed { Ok(Some(req)) => req, Ok(None) => break, // Client closed connection. Err(e) => { @@ -3584,8 +4103,16 @@ where return Ok(()); } }; + let correlation = if inspected { + RequestCorrelation::new(&req, ctx) + } else { + RequestCorrelation::uninspected() + }; + let receipt = inspected.then_some(&correlation); + let mut audited_client = correlation.observe_response(client); + let client = &mut audited_client; if !request_authority_matches_endpoint(&req, ctx) { - reject_request_authority_mismatch(client, ctx, &req.action).await?; + reject_request_authority_mismatch(client, ctx, &req.action, receipt).await?; return Ok(()); } if close_if_stale(generation_guard, ctx) { @@ -3598,7 +4125,7 @@ where let redacted_target = match secrets::redact_target_for_policy(&req.target) { Ok(target) => target, Err(error) => { - reject_credential_resolution(client, ctx, &req.action, &error).await?; + reject_credential_resolution(client, ctx, &req.action, &error, receipt).await?; return Ok(()); } }; @@ -3622,14 +4149,13 @@ where req.action, ctx.host, ctx.port, redacted_target, )) .build(); - egress_emit!(event); + correlation.emit(event); } - let request_id = uuid::Uuid::new_v4().to_string(); + let request_id = correlation.request_id().to_string(); let mut response_selection = None; - let req = if let Some(engine) = middleware_engine { - let input = middleware_network_input(ctx); - let (chain, generation) = engine.query_middleware_chain_with_generation(&input)?; + let req = if let Some((engine, selection)) = middleware_engine.zip(middleware_selection) { + let (chain, generation) = selection?; if generation != generation_guard.captured_generation() { return Ok(()); } @@ -3643,17 +4169,25 @@ where // The passthrough path enforces no L7 policy, so there is no // body-aware decision to re-check after a transformation. let result = exchange - .apply_request( + .apply_request_correlated( req, client, ctx, ctx.request_scheme(), openshell_supervisor_middleware::TransformedBodyPolicy::NotPolicyRelevant, + receipt, ) .await?; let request = match result { MiddlewareApplyResult::Allowed(request) => request, MiddlewareApplyResult::Denied { denial, .. } => { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome(if denial.is_some() { + "denied" + } else { + "platform_error" + }); + correlation.record_response_status(403); let denied_request = crate::l7::provider::L7Request { action: "HTTP".into(), target: redacted_target.clone(), @@ -3667,11 +4201,15 @@ where ctx, denial.as_ref(), &redacted_target, + receipt.map(RequestCorrelation::request_id), ) .await?; return Ok(()); } MiddlewareApplyResult::AdmissionExhausted => { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("platform_error"); + correlation.record_response_status(503); let unavailable_request = crate::l7::provider::L7Request { action: "HTTP".into(), target: redacted_target.clone(), @@ -3684,6 +4222,7 @@ where client, ctx, &redacted_target, + receipt.map(RequestCorrelation::request_id), ) .await?; return Ok(()); @@ -3698,18 +4237,22 @@ where let injected = match crate::l7::token_grant_injection::pin_live_credentials(&mut grant_ctx) { Some(snapshot) => { - crate::l7::token_grant_injection::inject_for_selectors( + crate::l7::token_grant_injection::inject_for_selectors_correlated( req, &grant_ctx, &snapshot, &request_id, observation_context.as_ref(), + receipt, ) .await } None => Ok(req), }; let Ok(req_with_auth) = injected else { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("platform_error"); + correlation.record_response_status(502); // The refusal event names the credential and request ID. // The sandbox-level grant record does not require an MCP endpoint identity. write_bad_gateway_response(client, Some(&request_id)).await?; @@ -3726,7 +4269,7 @@ where // Forward request with credential rewriting and relay the response. // relay_http_request_with_resolver handles both directions: it sends // the request upstream and reads the response back to the client. - let Some(outcome) = relay_http_request_with_credential_rejection( + let Some(outcome) = relay_http_request_with_credential_rejection_observed( &req_with_auth, client, upstream, @@ -3738,6 +4281,10 @@ where }, ctx, response_middleware, + RelayHttpInstrumentation { + observer: None, + correlation: receipt, + }, ) .await? else { @@ -3808,12 +4355,12 @@ mod trace_tests { ..Default::default() }; let mut response = Vec::new(); - // The in-memory span processor exports with a futures executor. tokio::task::block_in_place(|| { tokio::runtime::Handle::current().block_on(reject_request_authority_mismatch( &mut response, &ctx, "POST", + None, )) }) .unwrap(); @@ -3894,6 +4441,7 @@ mod tests { #[tokio::test] async fn body_denial_returns_actionable_local_json() { + const REQUEST_ID: &str = "8b02c179-9e6b-4eb0-948b-d9f40582fc44"; let req = crate::l7::provider::L7Request { action: "POST".into(), target: "/v1/responses".into(), query_params: TestHashMap::new(), raw_header: b"POST /v1/responses HTTP/1.1\r\nHost: api.openai.com\r\nContent-Length: 25\r\n\r\nopenshell:resolve:env:KEY".to_vec(), @@ -3901,7 +4449,12 @@ mod tests { }; let (mut client, mut caller) = tokio::io::duplex(4096); let (mut upstream, mut server) = tokio::io::duplex(4096); - let outcome = relay_http_request_with_credential_rejection( + let ctx = L7EvalContext { + policy_name: "body-policy".into(), + ..Default::default() + }; + let correlation = RequestCorrelation::with_request_id(&req, &ctx, REQUEST_ID.into()); + let outcome = relay_http_request_with_credential_rejection_observed( &req, &mut client, &mut upstream, @@ -3909,8 +4462,12 @@ mod tests { deny_uninspected_credentials: true, ..Default::default() }, - &L7EvalContext::default(), + &ctx, None, + RelayHttpInstrumentation { + observer: None, + correlation: Some(&correlation), + }, ) .await .unwrap(); @@ -3919,6 +4476,10 @@ mod tests { let mut response = String::new(); caller.read_to_string(&mut response).await.unwrap(); assert!(response.starts_with("HTTP/1.1 403 Forbidden")); + assert!( + response.contains(&format!("x-openshell-request-id: {REQUEST_ID}\r\n")), + "{response}" + ); let body: serde_json::Value = serde_json::from_str(response.split_once("\r\n\r\n").unwrap().1).unwrap(); assert_eq!( @@ -3926,6 +4487,7 @@ mod tests { "credential_placeholder_in_request_body" ); assert_eq!(body["error"]["reason"], "classification_unavailable"); + assert_eq!(body["request_id"], REQUEST_ID); let mut sent = String::new(); server.read_to_string(&mut sent).await.unwrap(); assert!(!sent.contains("openshell:resolve:")); @@ -3999,7 +4561,10 @@ mod tests { }, &L7EvalContext::default(), None, - Some(&observer), + RelayHttpInstrumentation { + observer: Some(&observer), + correlation: None, + }, ) .await; if disconnect_client { @@ -4335,6 +4900,7 @@ mod tests { resolver: &SecretResolver, options: crate::l7::rest::RelayRequestOptions<'_>, ) { + const REQUEST_ID: &str = "1329fced-3bb8-4810-bc25-45a1282997ad"; let (mut client_peer, mut client) = tokio::io::duplex(8192); let (mut upstream, mut upstream_peer) = tokio::io::duplex(8192); let ctx = L7EvalContext { @@ -4345,8 +4911,10 @@ mod tests { secret_resolver: Some(Arc::new(resolver.clone())), ..Default::default() }; + let correlation = + RequestCorrelation::with_request_id(&request, &ctx, REQUEST_ID.to_string()); - let outcome = relay_http_request_with_credential_rejection( + let outcome = relay_http_request_with_credential_rejection_observed( &request, &mut client, &mut upstream, @@ -4356,6 +4924,10 @@ mod tests { }, &ctx, None, + RelayHttpInstrumentation { + observer: None, + correlation: Some(&correlation), + }, ) .await .expect("typed credential denial"); @@ -4370,6 +4942,13 @@ mod tests { response.contains("credential_endpoint_mismatch"), "{response}" ); + assert!( + response.contains(&format!("x-openshell-request-id: {REQUEST_ID}\r\n")), + "{response}" + ); + let body: serde_json::Value = + serde_json::from_str(response.split_once("\r\n\r\n").unwrap().1).unwrap(); + assert_eq!(body["request_id"], REQUEST_ID); let mut forwarded = Vec::new(); upstream_peer.read_to_end(&mut forwarded).await.unwrap(); assert!( @@ -5009,19 +5588,81 @@ mod tests { body_length: crate::l7::provider::BodyLength::ContentLength(0), }; - assert_credential_relay_rejected( - request, - resolver.as_ref(), + assert_credential_relay_rejected( + request, + resolver.as_ref(), + crate::l7::rest::RelayRequestOptions { + credential_signing: crate::l7::CredentialSigning::SigV4NoBody, + signing_service: "execute-api", + signing_region: "us-west-2", + host: "denied.example.test", + port: 443, + ..Default::default() + }, + ) + .await; + } + + #[tokio::test] + async fn correlated_credential_unavailable_preserves_transport_failure() { + const REQUEST_ID: &str = "250b9326-7a4f-4aa0-a6a9-7e0a1d96beaa"; + let request = crate::l7::provider::L7Request { + action: "GET".to_string(), + target: "/v1/resource".to_string(), + query_params: TestHashMap::new(), + raw_header: b"GET /v1/resource HTTP/1.1\r\nHost: api.example.test\r\n\r\n".to_vec(), + body_length: crate::l7::provider::BodyLength::None, + }; + let ctx = L7EvalContext { + host: "api.example.test".into(), + port: 443, + policy_name: "sigv4-policy".into(), + ..Default::default() + }; + let correlation = + RequestCorrelation::with_request_id(&request, &ctx, REQUEST_ID.to_string()); + let (mut client_peer, mut client) = tokio::io::duplex(8192); + let (mut upstream, mut upstream_peer) = tokio::io::duplex(8192); + + let error = relay_http_request_with_credential_rejection_observed( + &request, + &mut client, + &mut upstream, crate::l7::rest::RelayRequestOptions { credential_signing: crate::l7::CredentialSigning::SigV4NoBody, signing_service: "execute-api", signing_region: "us-west-2", - host: "denied.example.test", + host: "api.example.test", port: 443, ..Default::default() }, + &ctx, + None, + RelayHttpInstrumentation { + observer: None, + correlation: Some(&correlation), + }, ) - .await; + .await + .expect_err("credential availability failure remains a transport failure"); + assert!( + error + .downcast_ref::() + .is_some() + ); + drop(client); + drop(upstream); + + let mut response = String::new(); + client_peer.read_to_string(&mut response).await.unwrap(); + assert!( + response.is_empty(), + "no new synthetic HTTP response: {response}" + ); + + let mut forwarded = Vec::new(); + upstream_peer.read_to_end(&mut forwarded).await.unwrap(); + assert!(forwarded.is_empty()); } fn install_builtin_middleware(engine: &OpaEngine) { @@ -7564,20 +8205,24 @@ network_policies: enforcement: &str, replacement: &'static [u8], ) -> (String, Option) { + use tracing::instrument::WithSubscriber as _; let (config, tunnel_engine, ctx) = jsonrpc_transforming_relay_parts(enforcement, replacement); let (mut app, mut relay_client) = tokio::io::duplex(8192); let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); - let relay = tokio::spawn(async move { - relay_jsonrpc( - &config, - &tunnel_engine, - &mut relay_client, - &mut relay_upstream, - &ctx, - ) - .await - }); + let relay = tokio::spawn( + async move { + relay_jsonrpc( + &config, + &tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + } + .with_current_subscriber(), + ); let body = br#"{"jsonrpc":"2.0","id":1,"method":"reports.list"}"#; let request = format!( @@ -7639,6 +8284,56 @@ network_policies: assert!(upstream_seen.is_none(), "upstream must not see the request"); } + #[tokio::test] + async fn transformed_jsonrpc_policy_outcome_has_one_primary_and_shared_receipt() { + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + for enforcement in ["enforce", "audit"] { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let subscriber = tracing_subscriber::registry().with(capture.clone()); + let (response, upstream_seen) = run_jsonrpc_transform_case( + enforcement, + br#"{"jsonrpc":"2.0","id":1,"method":"admin.delete"}"#, + ) + .with_subscriber(subscriber) + .await; + let request_id = response + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("x-openshell-request-id") + .then(|| value.trim()) + }) + .expect("inspected response receipt"); + let events = capture.json_events(); + let primary: Vec<_> = events + .iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!(primary.len(), 1, "{enforcement}: {events:?}"); + assert_eq!(primary[0]["metadata"]["correlation_uid"], request_id); + assert_eq!(primary[0]["http_request"]["url"]["path"], "/rpc"); + assert_eq!(primary[0]["firewall_rule"]["type"], "l7-jsonrpc"); + if enforcement == "enforce" { + assert!(upstream_seen.is_none()); + assert_eq!(primary[0]["action"], "Denied"); + assert_eq!(primary[0]["http_response"]["code"], 403); + assert!( + primary[0]["status_detail"] + .as_str() + .unwrap() + .contains("middleware transformation denied") + ); + } else { + assert!(upstream_seen.unwrap().contains("admin.delete")); + assert_eq!(primary[0]["action"], "Allowed"); + assert_eq!(primary[0]["http_response"]["code"], 204); + assert_eq!(primary[0]["unmapped"]["transformed_body_policy"], "audit"); + } + } + } + #[tokio::test] async fn transformed_jsonrpc_body_policy_deny_forwards_under_audit() { // Under enforcement: audit a policy deny of the transformed body is @@ -8501,6 +9196,145 @@ network_policies: assert!(!serialized.contains(RAW_SECRET)); } + #[tokio::test] + async fn passthrough_receipts_require_selected_middleware_and_keep_one_primary() { + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + for selected in [false, true] { + let middleware = if selected { + "network_middlewares:\n scheme:\n middleware: test/scheme\n on_error: fail_closed\n endpoints:\n include: [api.example.test]\n" + } else { + "" + }; + let engine = OpaEngine::from_strings( + TEST_POLICY, + &format!("{middleware}network_policies: {{}}\n"), + ) + .unwrap(); + engine.set_middleware_runner_for_tests( + openshell_supervisor_middleware::ChainRunner::new(Arc::new(SchemeRecorder( + Arc::new(std::sync::Mutex::new(Vec::new())), + ))), + ); + let generation_guard = engine + .generation_guard(engine.current_generation()) + .unwrap(); + let state = ProviderCredentialState::from_bound_environment( + 1, + TestHashMap::from([("API_TOKEN".into(), "synthetic-live-credential".into())]), + TestHashMap::new(), + TestHashMap::new(), + TestHashMap::from([( + "API_TOKEN".into(), + StaticCredentialBinding { + endpoints: vec![StaticCredentialEndpointBinding { + host: "api.example.test".into(), + port: 8080, + path: "/v1/**".into(), + }], + credential_identity: "provider-a:API_TOKEN".into(), + workload_credential_handle: String::new(), + }, + )]), + Vec::new(), + ) + .unwrap(); + let placeholder = state.snapshot().child_env["API_TOKEN"].clone(); + let ctx = L7EvalContext { + host: "api.example.test".into(), + port: 8080, + request_default_port: Some(8080), + policy_name: "passthrough_api".into(), + binary_path: "/usr/bin/curl".into(), + provider_credentials: Some(state), + ..Default::default() + }; + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let subscriber = tracing_subscriber::registry().with(capture.clone()); + let (mut app, mut relay_client) = tokio::io::duplex(8192); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); + let relay = async move { + relay_passthrough_with_credentials( + &mut relay_client, + &mut relay_upstream, + &ctx, + &generation_guard, + Some(&engine), + ) + .await + } + .with_subscriber(subscriber); + let request = format!( + "GET /v1/messages HTTP/1.1\r\nHost: api.example.test:8080\r\nAuthorization: Bearer {placeholder}\r\nX-OpenShell-Request-ID: application-in\r\nConnection: close\r\n\r\n" + ); + let exchange = async { + tokio::join!( + relay, + async { + app.write_all(request.as_bytes()).await.unwrap(); + let mut response = String::new(); + app.read_to_string(&mut response).await.unwrap(); + response + }, + async { + let forwarded = read_http_headers(&mut upstream).await; + upstream.write_all(b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\nX-OpenShell-Request-ID: application-out\r\nConnection: close\r\n\r\n").await.unwrap(); + forwarded + }, + ) + }; + let (relay, response, forwarded) = + tokio::time::timeout(std::time::Duration::from_secs(5), Box::pin(exchange)) + .await + .expect("credential-only CONNECT exchange"); + relay.unwrap(); + let forwarded = String::from_utf8(forwarded).unwrap(); + assert!( + forwarded.contains("Authorization: Bearer synthetic-live-credential"), + "{forwarded}" + ); + let records = capture.json_events(); + let primary: Vec<_> = records + .iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!(primary.len(), 1, "selected={selected}: {records:?}"); + assert_eq!(primary[0]["action"], "Allowed"); + assert_eq!(primary[0]["http_request"]["url"]["path"], "/v1/messages"); + assert!( + !serde_json::to_string(&records) + .unwrap() + .contains("synthetic-live-credential") + ); + if selected { + assert!(!forwarded.contains("application-in")); + assert!(!response.contains("application-out")); + let receipt = response + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("x-openshell-request-id") + .then(|| value.trim()) + }) + .expect("middleware-selected receipt"); + assert!(uuid::Uuid::parse_str(receipt).is_ok()); + assert_eq!(primary[0]["metadata"]["correlation_uid"], receipt); + assert_eq!(primary[0]["http_response"]["code"], 204); + } else { + assert!( + forwarded.contains("X-OpenShell-Request-ID: application-in"), + "{forwarded}" + ); + assert!( + response.contains("X-OpenShell-Request-ID: application-out"), + "{response}" + ); + assert!(primary[0]["metadata"].get("correlation_uid").is_none()); + } + } + } + #[tokio::test] async fn passthrough_relay_runs_middleware_redaction() { // A no-protocol endpoint takes the credential-injection passthrough path; @@ -8786,6 +9620,124 @@ network_policies: .unwrap(); } + async fn assert_websocket_endpoint_rejects_plain_http(route_selected: bool) { + let data = r" +network_policies: + ws_api: + name: ws_api + endpoints: + - host: gateway.example.test + port: 443 + path: /ws + protocol: websocket + enforcement: enforce + rules: + - allow: + method: GET + path: /ws + binaries: + - { path: /usr/bin/node } +"; + let engine = OpaEngine::from_strings(TEST_POLICY, data).unwrap(); + let input = NetworkInput { + host: "gateway.example.test".into(), + port: 443, + binary_path: PathBuf::from("/usr/bin/node"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + let (endpoints, generation) = engine + .query_endpoint_configs_with_generation(&input) + .unwrap(); + let configs: Vec<_> = endpoints + .iter() + .map(|endpoint| crate::l7::parse_l7_config(endpoint).unwrap()) + .collect(); + let tunnel_engine = engine.clone_engine_for_tunnel(generation).unwrap(); + let ctx = L7EvalContext { + host: "gateway.example.test".into(), + port: 443, + request_default_port: Some(443), + policy_name: "ws_api".into(), + binary_path: "/usr/bin/node".into(), + ..Default::default() + }; + let (mut app, mut relay_client) = tokio::io::duplex(8192); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); + let relay = tokio::spawn(async move { + if route_selected { + relay_with_route_selection( + &configs, + tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + } else { + relay_with_inspection( + &configs[0], + tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + } + }); + + app.write_all( + b"GET /ws HTTP/1.1\r\nHost: gateway.example.test\r\nConnection: close\r\n\r\n", + ) + .await + .unwrap(); + let response = tokio::time::timeout( + std::time::Duration::from_secs(1), + read_http_headers(&mut app), + ) + .await + .expect("denial should reach client"); + let response = String::from_utf8_lossy(&response); + assert!(response.contains("403 Forbidden"), "{response}"); + let request_id = response + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("x-openshell-request-id") + .then(|| value.trim()) + }) + .expect("denial should carry a supervisor receipt"); + assert!(uuid::Uuid::parse_str(request_id).is_ok()); + + let mut upstream_bytes = [0u8; 16]; + assert!(matches!( + tokio::time::timeout( + std::time::Duration::from_millis(100), + upstream.read(&mut upstream_bytes) + ) + .await, + Err(_) | Ok(Ok(0)) + )); + + drop(app); + tokio::time::timeout(std::time::Duration::from_secs(1), relay) + .await + .expect("relay should finish") + .unwrap() + .unwrap(); + } + + #[tokio::test] + async fn websocket_endpoint_plain_http_denial_has_receipt() { + assert_websocket_endpoint_rejects_plain_http(false).await; + } + + #[tokio::test] + async fn route_selected_websocket_plain_http_denial_has_receipt() { + assert_websocket_endpoint_rejects_plain_http(true).await; + } + #[tokio::test] async fn passthrough_relay_injects_token_grant_authorization_header() { let (generation_guard, ctx, fixture) = @@ -9849,12 +10801,23 @@ network_policies: .await .unwrap(); - let mut response = [0u8; 1024]; - let n = tokio::time::timeout(std::time::Duration::from_secs(1), app.read(&mut response)) - .await - .expect("denial should reach client") - .unwrap(); - assert!(String::from_utf8_lossy(&response[..n]).contains("403 Forbidden")); + let response = tokio::time::timeout( + std::time::Duration::from_secs(1), + read_http_headers(&mut app), + ) + .await + .expect("denial should reach client"); + let response = String::from_utf8_lossy(&response); + assert!(response.contains("403 Forbidden")); + let request_id = response + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("x-openshell-request-id") + .then(|| value.trim()) + }) + .expect("denial should carry a supervisor receipt"); + assert!(uuid::Uuid::parse_str(request_id).is_ok()); let mut upstream_bytes = [0u8; 16]; assert!(matches!( @@ -11025,7 +11988,7 @@ network_policies: let _guard = capture.install(); let (_app, mut client) = tokio::io::duplex(8192); assert!( - deny_unsupported_upgrade_if_requested(&request, config, &ctx, None, &mut client) + deny_unsupported_upgrade_if_requested(&request, config, &ctx, None, &mut client, None) .await .unwrap() ); @@ -13430,6 +14393,7 @@ network_policies: #[tokio::test] async fn final_mcp_version_check_reclassifies_a_rewritten_initialize_body() { + const REQUEST_ID: &str = "36a15952-692c-44d8-a373-eeca9735fb6b"; let (config, _, ctx) = mcp_test_relay_context(); let final_body = br#"{"jsonrpc":"2.0","id":2,"method":"tools/list"}"#; let mut raw_header = format!( @@ -13446,14 +14410,17 @@ network_policies: body_length: crate::l7::provider::BodyLength::ContentLength(final_body.len() as u64), }; let (mut client, mut relay_client) = tokio::io::duplex(2048); + let correlation = + RequestCorrelation::with_request_id(&request, &ctx, REQUEST_ID.to_string()); - let allowed = enforce_final_mcp_protocol_version( + let allowed = enforce_final_mcp_protocol_version_correlated( &config, &request, &mut relay_client, &ctx, "/mcp", None, + Some(&correlation), ) .await .expect("final request inspection"); @@ -13461,6 +14428,7 @@ network_policies: !allowed, "rewritten non-initialize request must require a version" ); + drop(correlation); drop(relay_client); let mut response = Vec::new(); @@ -13471,6 +14439,13 @@ network_policies: response.contains("mcp_protocol_version_not_allowed"), "{response}" ); + assert!( + response.contains(&format!("x-openshell-request-id: {REQUEST_ID}\r\n")), + "{response}" + ); + let response_body: serde_json::Value = + serde_json::from_str(response.split_once("\r\n\r\n").unwrap().1).unwrap(); + assert_eq!(response_body["request_id"], REQUEST_ID); } #[tokio::test] diff --git a/crates/openshell-supervisor-network/src/l7/relay/correlation_dispatch_tests.rs b/crates/openshell-supervisor-network/src/l7/relay/correlation_dispatch_tests.rs new file mode 100644 index 0000000000..4289799446 --- /dev/null +++ b/crates/openshell-supervisor-network/src/l7/relay/correlation_dispatch_tests.rs @@ -0,0 +1,492 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Correlation contracts for the production single-config relay dispatch. + +use super::L7EvalContext; +use crate::l7::L7EndpointConfig; +use crate::opa::{NetworkInput, OpaEngine, TunnelPolicyEngine}; +use std::path::PathBuf; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt}; +use tracing::instrument::WithSubscriber as _; +use tracing_subscriber::layer::SubscriberExt as _; + +const POLICY: &str = include_str!("../../../data/sandbox-policy.rego"); + +struct Fixture { + config: L7EndpointConfig, + tunnel_engine: TunnelPolicyEngine, + middleware_engine: OpaEngine, + ctx: L7EvalContext, +} + +struct Exchange { + response: Vec, + forwarded: Vec, + ocsf_events: Vec, +} + +fn fixture(data: &str, host: &str, policy_name: &str) -> Fixture { + let middleware_engine = OpaEngine::from_strings(POLICY, data).expect("valid test policy"); + let input = NetworkInput { + host: host.into(), + port: 8000, + binary_path: PathBuf::from("/usr/bin/python3"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + let (endpoint_config, generation) = middleware_engine + .query_endpoint_config_with_generation(&input) + .expect("query endpoint config"); + let config = crate::l7::parse_l7_config( + &endpoint_config.expect("single configured endpoint should match"), + ) + .expect("parse endpoint config"); + let tunnel_engine = middleware_engine + .clone_engine_for_tunnel(generation) + .expect("pin test policy generation"); + let ctx = L7EvalContext { + host: host.into(), + port: 8000, + request_default_port: Some(8000), + policy_name: policy_name.into(), + binary_path: "/usr/bin/python3".into(), + ..Default::default() + }; + Fixture { + config, + tunnel_engine, + middleware_engine, + ctx, + } +} + +async fn read_http_message(reader: &mut R) -> Vec { + let mut message = Vec::new(); + let mut byte = [0_u8; 1]; + while !message.ends_with(b"\r\n\r\n") { + reader + .read_exact(&mut byte) + .await + .expect("complete HTTP headers"); + message.push(byte[0]); + } + let headers = std::str::from_utf8(&message).expect("ASCII test headers"); + let body_len = headers + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("content-length").then(|| { + value + .trim() + .parse::() + .expect("numeric content length") + }) + }) + .unwrap_or_default(); + let header_len = message.len(); + message.resize(header_len + body_len, 0); + reader + .read_exact(&mut message[header_len..]) + .await + .expect("complete HTTP body"); + message +} + +fn request_id(response: &[u8]) -> &str { + let headers_end = response + .windows(4) + .position(|window| window == b"\r\n\r\n") + .expect("complete response headers"); + std::str::from_utf8(&response[..headers_end]) + .expect("ASCII response headers") + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("x-openshell-request-id") + .then(|| value.trim()) + }) + .expect("response carries a request receipt") +} + +async fn run_allowed( + fixture: Fixture, + request: Vec, + upstream_response: &'static [u8], +) -> Exchange { + let ocsf = openshell_ocsf::test_support::OcsfCapture::default(); + let subscriber = tracing_subscriber::registry().with(ocsf.clone()); + let (mut app, mut relay_client) = tokio::io::duplex(32 * 1024); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(32 * 1024); + let Fixture { + config, + tunnel_engine, + middleware_engine, + ctx, + } = fixture; + + let exchange = async { + tracing::callsite::rebuild_interest_cache(); + tokio::join!( + crate::proxy::relay_inspected_http_stream_for_test( + &mut relay_client, + &mut relay_upstream, + vec![config], + tunnel_engine, + &middleware_engine, + &ctx, + ), + async { + app.write_all(&request).await.expect("send request"); + app.flush().await.expect("flush request"); + read_http_message(&mut app).await + }, + async { + let forwarded = read_http_message(&mut upstream).await; + upstream + .write_all(upstream_response) + .await + .expect("send upstream response"); + upstream.flush().await.expect("flush upstream response"); + forwarded + }, + ) + } + .with_subscriber(subscriber); + let (relay, response, forwarded) = + tokio::time::timeout(std::time::Duration::from_secs(5), Box::pin(exchange)) + .await + .expect("single-config relay exchange completes"); + relay.expect("relay succeeds"); + Exchange { + response, + forwarded, + ocsf_events: ocsf.json_events(), + } +} + +async fn run_denied(fixture: Fixture, request: Vec) -> Exchange { + let ocsf = openshell_ocsf::test_support::OcsfCapture::default(); + let subscriber = tracing_subscriber::registry().with(ocsf.clone()); + let (mut app, mut relay_client) = tokio::io::duplex(32 * 1024); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(32 * 1024); + let Fixture { + config, + tunnel_engine, + middleware_engine, + ctx, + } = fixture; + + let exchange = async { + tracing::callsite::rebuild_interest_cache(); + tokio::join!( + crate::proxy::relay_inspected_http_stream_for_test( + &mut relay_client, + &mut relay_upstream, + vec![config], + tunnel_engine, + &middleware_engine, + &ctx, + ), + async { + app.write_all(&request).await.expect("send request"); + app.flush().await.expect("flush request"); + read_http_message(&mut app).await + }, + async { + let mut forwarded = Vec::new(); + let _ = tokio::time::timeout( + std::time::Duration::from_millis(100), + upstream.read_to_end(&mut forwarded), + ) + .await; + forwarded + }, + ) + } + .with_subscriber(subscriber); + let (relay, response, forwarded) = + tokio::time::timeout(std::time::Duration::from_secs(5), Box::pin(exchange)) + .await + .expect("single-config denial completes"); + relay.expect("relay succeeds"); + Exchange { + response, + forwarded, + ocsf_events: ocsf.json_events(), + } +} + +fn assert_primary_record<'a>( + exchange: &'a Exchange, + engine_type: &str, + status: u16, + action: &str, + path: &str, +) -> &'a serde_json::Value { + let receipt = request_id(&exchange.response); + assert!( + uuid::Uuid::parse_str(receipt).is_ok(), + "canonical receipt: {receipt}" + ); + let primary: Vec<_> = exchange + .ocsf_events + .iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!( + primary.len(), + 1, + "one HTTP Activity: {:?}", + exchange.ocsf_events + ); + let event = primary[0]; + assert_eq!(event["metadata"]["correlation_uid"], receipt); + assert_eq!(event["firewall_rule"]["type"], engine_type); + assert_eq!(event["http_response"]["code"], status); + assert_eq!(event["action"], action); + assert_eq!(event["http_request"]["url"]["path"], path); + assert!( + event.get("trace").is_none(), + "receipts must work without tracing" + ); + event +} + +fn mcp_fixture() -> Fixture { + fixture( + r#" +network_policies: + mcp_api: + name: mcp_api + endpoints: + - host: mcp.example.test + port: 8000 + path: /mcp + protocol: mcp + enforcement: enforce + mcp: + versions: ["2025-11-25"] + rules: + - allow: + method: initialize + binaries: + - { path: /usr/bin/python3 } +"#, + "mcp.example.test", + "mcp_api", + ) +} + +fn request(host: &str, path: &str, extra_headers: &str, body: &str) -> Vec { + format!( + "POST {path} HTTP/1.1\r\nHost: {host}:8000\r\nContent-Type: application/json\r\n{extra_headers}Content-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len(), + ) + .into_bytes() +} + +#[tokio::test] +async fn inspected_keep_alive_requests_have_distinct_receipts_without_tracing() { + let Fixture { + config, + tunnel_engine, + middleware_engine, + ctx, + } = fixture( + r" +network_policies: + rest_api: + name: rest_api + endpoints: + - host: rest.example.test + port: 8000 + path: /item + protocol: rest + enforcement: enforce + rules: + - allow: { method: GET, path: /item } + binaries: + - { path: /usr/bin/python3 } +", + "rest.example.test", + "rest_api", + ); + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let subscriber = tracing_subscriber::registry().with(capture.clone()); + let (mut app, mut relay_client) = tokio::io::duplex(8192); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); + let exchange = async { + tokio::join!( + crate::proxy::relay_inspected_http_stream_for_test( + &mut relay_client, + &mut relay_upstream, + vec![config], + tunnel_engine, + &middleware_engine, + &ctx, + ), + async { + let mut responses = Vec::new(); + for (sampled, connection) in [("01", "keep-alive"), ("00", "close")] { + let request = format!( + "GET /item HTTP/1.1\r\nHost: rest.example.test:8000\r\ntraceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-{sampled}\r\nConnection: {connection}\r\n\r\n", + ); + app.write_all(request.as_bytes()).await.expect("send request"); + app.flush().await.expect("flush request"); + responses.push(read_http_message(&mut app).await); + } + responses + }, + async { + for connection in ["keep-alive", "close"] { + let request = read_http_message(&mut upstream).await; + assert!(request.starts_with(b"GET /item HTTP/1.1\r\n")); + let response = format!("HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: {connection}\r\n\r\n"); + upstream.write_all(response.as_bytes()).await.expect("upstream response"); + upstream.flush().await.expect("flush response"); + } + }, + ) + }.with_subscriber(subscriber); + let (relay, responses, ()) = + tokio::time::timeout(std::time::Duration::from_secs(5), Box::pin(exchange)) + .await + .expect("two inspected requests complete"); + relay.expect("inspected relay"); + let receipts: Vec<_> = responses + .iter() + .map(|response| request_id(response)) + .collect(); + assert_eq!(receipts.len(), 2); + assert_ne!(receipts[0], receipts[1]); + let records: Vec<_> = capture + .json_events() + .into_iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!( + records.len(), + 2, + "one primary per inspected request: {records:?}" + ); + for (record, receipt) in records.iter().zip(receipts) { + assert!(uuid::Uuid::parse_str(receipt).is_ok()); + assert_eq!(record["metadata"]["correlation_uid"], receipt); + assert_eq!(record["action"], "Allowed"); + assert_eq!(record["http_response"]["code"], 200); + assert_eq!(record["http_request"]["url"]["path"], "/item"); + assert!(record.get("trace").is_none()); + assert!( + !record + .to_string() + .contains("4bf92f3577b34da6a3ce929d0e0e4736") + ); + } +} + +#[tokio::test] +async fn production_single_mcp_allow_correlates_record_and_response_receipt() { + let body = r#"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}"#; + let exchange = Box::pin(run_allowed( + mcp_fixture(), + request("mcp.example.test", "/mcp", "", body), + b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 2\r\nConnection: close\r\n\r\n{}", + )) + .await; + + assert!(exchange.forwarded.starts_with(b"POST /mcp HTTP/1.1\r\n")); + assert!(exchange.response.starts_with(b"HTTP/1.1 200 OK\r\n")); + assert_primary_record(&exchange, "l7-mcp", 200, "Allowed", "/mcp"); +} + +#[tokio::test] +async fn production_single_mcp_invalid_version_correlates_denial_and_receipt() { + let body = r#"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}"#; + let exchange = Box::pin(run_denied( + mcp_fixture(), + request( + "mcp.example.test", + "/mcp", + "MCP-Protocol-Version: 1900-01-01\r\n", + body, + ), + )) + .await; + + assert!(exchange.forwarded.is_empty()); + assert!( + exchange + .response + .starts_with(b"HTTP/1.1 400 Bad Request\r\n") + ); + assert!( + String::from_utf8_lossy(&exchange.response).contains("unsupported_mcp_protocol_version") + ); + assert_primary_record(&exchange, "l7-mcp", 400, "Denied", "/mcp"); +} + +#[tokio::test] +async fn production_single_mcp_policy_deny_correlates_record_and_receipt() { + let body = r#"{"jsonrpc":"2.0","id":1,"method":"tools/list"}"#; + let exchange = Box::pin(run_denied( + mcp_fixture(), + request( + "mcp.example.test", + "/mcp", + "MCP-Protocol-Version: 2025-11-25\r\n", + body, + ), + )) + .await; + + assert!(exchange.forwarded.is_empty()); + assert!(exchange.response.starts_with(b"HTTP/1.1 403 Forbidden\r\n")); + let event = assert_primary_record(&exchange, "l7-mcp", 403, "Denied", "/mcp"); + assert!( + event["message"] + .as_str() + .is_some_and(|message| message.starts_with("JSONRPC_L7_REQUEST decision=deny")) + ); +} + +#[tokio::test] +async fn production_single_graphql_allow_correlates_record_and_response_receipt() { + let graphql = fixture( + r" +network_policies: + graphql_api: + name: graphql_api + endpoints: + - host: graphql.example.test + port: 8000 + path: /graphql + protocol: graphql + enforcement: enforce + rules: + - allow: + operation_type: query + fields: [viewer] + binaries: + - { path: /usr/bin/python3 } +", + "graphql.example.test", + "graphql_api", + ); + let body = r#"{"query":"query { viewer }"}"#; + let exchange = Box::pin(run_allowed( + graphql, + request("graphql.example.test", "/graphql", "", body), + b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 22\r\nConnection: close\r\n\r\n{\"data\":{\"viewer\":{}}}", + )) + .await; + + assert!( + exchange + .forwarded + .starts_with(b"POST /graphql HTTP/1.1\r\n") + ); + assert!(exchange.response.starts_with(b"HTTP/1.1 200 OK\r\n")); + assert_primary_record(&exchange, "l7-graphql", 200, "Allowed", "/graphql"); +} diff --git a/crates/openshell-supervisor-network/src/l7/relay/token_grant_ownership_tests.rs b/crates/openshell-supervisor-network/src/l7/relay/token_grant_ownership_tests.rs index eb34e325b9..4f22f44ba5 100644 --- a/crates/openshell-supervisor-network/src/l7/relay/token_grant_ownership_tests.rs +++ b/crates/openshell-supervisor-network/src/l7/relay/token_grant_ownership_tests.rs @@ -1418,6 +1418,7 @@ async fn transformed_jsonrpc_body_selects_final_operation_owner() { &info, "request-final-body", None, + None, ), ) .await diff --git a/crates/openshell-supervisor-network/src/l7/rest.rs b/crates/openshell-supervisor-network/src/l7/rest.rs index 27f10ae7bd..192eb7e459 100644 --- a/crates/openshell-supervisor-network/src/l7/rest.rs +++ b/crates/openshell-supervisor-network/src/l7/rest.rs @@ -21,6 +21,7 @@ use http_response::{ }; use crate::l7::EndpointObserver; +use crate::l7::correlation::RequestCorrelation; use crate::l7::http::{parse_chunk_size_line, validate_chunk_trailer_line}; use crate::l7::provider::{BodyLength, L7Provider, L7Request, RelayOutcome}; use crate::opa::PolicyGenerationGuard; @@ -158,6 +159,7 @@ pub(crate) struct DenyResponseContext<'a> { pub(crate) port: Option, pub(crate) binary: Option<&'a str>, pub(crate) agent_proposals_enabled: bool, + pub(crate) request_id: Option<&'a str>, } impl<'a> DenyResponseContext<'a> { @@ -168,11 +170,25 @@ impl<'a> DenyResponseContext<'a> { port: Some(ctx.port), binary: Some(&ctx.binary_path), agent_proposals_enabled: ctx.agent_proposals.enabled(), + request_id: None, } } + + #[must_use] + pub(crate) fn with_request_id(mut self, request_id: &'a str) -> Self { + self.request_id = Some(request_id); + self + } } impl RestProvider { + pub(crate) async fn parse_uninspected_request( + &self, + client: &mut C, + ) -> Result> { + parse_http_request_with_receipt_protection(client, &self.canonicalize_options, false).await + } + /// Deny with a redacted target for the response body. pub(crate) async fn deny_with_redacted_target( &self, @@ -198,6 +214,14 @@ impl RestProvider { async fn parse_http_request( client: &mut C, canonicalize_options: &crate::l7::path::CanonicalizeOptions, +) -> Result> { + parse_http_request_with_receipt_protection(client, canonicalize_options, true).await +} + +async fn parse_http_request_with_receipt_protection( + client: &mut C, + canonicalize_options: &crate::l7::path::CanonicalizeOptions, + protect_receipt: bool, ) -> Result> { let mut buf = Vec::with_capacity(4096); @@ -302,6 +326,10 @@ async fn parse_http_request( )?; } + if protect_receipt { + buf = strip_platform_request_id_from_head(&buf); + } + Ok(Some(L7Request { action: method, target: canonical.path, @@ -321,6 +349,18 @@ pub(crate) fn request_from_buffered_http( target: impl Into, query_target: &str, raw_header: Vec, +) -> Result { + let mut request = + request_from_buffered_http_unprotected(action, target, query_target, raw_header)?; + request.raw_header = strip_platform_request_id_from_head(&request.raw_header); + Ok(request) +} + +pub(crate) fn request_from_buffered_http_unprotected( + action: impl Into, + target: impl Into, + query_target: &str, + raw_header: Vec, ) -> Result { let header_end = raw_header .windows(4) @@ -898,6 +938,31 @@ pub(crate) async fn relay_http_request_with_response_middleware_guarded_observed response_middleware: Option>, observer: Option<&EndpointObserver>, ) -> Result +where + C: AsyncRead + AsyncWrite + Unpin, + U: AsyncRead + AsyncWrite + Unpin, +{ + relay_http_request_with_response_middleware_guarded_observed_correlated( + req, + client, + upstream, + options, + response_middleware, + observer, + None, + ) + .await +} + +pub(crate) async fn relay_http_request_with_response_middleware_guarded_observed_correlated( + req: &L7Request, + client: &mut C, + upstream: &mut U, + options: RelayRequestOptions<'_>, + response_middleware: Option>, + observer: Option<&EndpointObserver>, + correlation: Option<&RequestCorrelation>, +) -> Result where C: AsyncRead + AsyncWrite + Unpin, U: AsyncRead + AsyncWrite + Unpin, @@ -962,13 +1027,14 @@ where raw_header, body_length: req.body_length, }; - if !crate::l7::relay::enforce_final_mcp_protocol_version( + if !crate::l7::relay::enforce_final_mcp_protocol_version_correlated( validation.config, &outgoing, client, validation.ctx, validation.redacted_target, observer, + correlation, ) .await? { @@ -1155,8 +1221,14 @@ where } } BodyLength::Chunked => { - relay_chunked(client, upstream, overflow, options.generation_guard) - .await?; + relay_chunked_with_receipt_protection( + client, + upstream, + overflow, + options.generation_guard, + correlation.is_some(), + ) + .await?; } BodyLength::None => { if !overflow.is_empty() { @@ -1215,13 +1287,14 @@ where upstream.write_all(&body.body).await.into_diagnostic()?; } } else if options.deny_uninspected_credentials { - if let Err(error) = relay_request_body_with_marker_guard( + if let Err(error) = relay_request_body_with_marker_guard_correlated( req, client, upstream, &rewrite_result.rewritten, &req.raw_header[header_end..], options, + correlation.is_some(), ) .await { @@ -1250,7 +1323,14 @@ where } } BodyLength::Chunked => { - relay_chunked(client, upstream, overflow, options.generation_guard).await?; + relay_chunked_with_receipt_protection( + client, + upstream, + overflow, + options.generation_guard, + correlation.is_some(), + ) + .await?; } BodyLength::None => { if !overflow.is_empty() { @@ -1270,6 +1350,7 @@ where websocket: websocket_response, client_requested_upgrade, observer, + correlation, }, response_middleware, ) @@ -1382,6 +1463,7 @@ async fn write_guarded_chunk( write_body_bytes(writer, b"\r\n", options).await } +#[cfg(test)] async fn relay_request_body_with_marker_guard( req: &L7Request, client: &mut C, @@ -1390,6 +1472,31 @@ async fn relay_request_body_with_marker_guard( already_read: &[u8], options: RelayRequestOptions<'_>, ) -> Result<()> +where + C: AsyncRead + Unpin, + U: AsyncWrite + Unpin, +{ + relay_request_body_with_marker_guard_correlated( + req, + client, + upstream, + headers, + already_read, + options, + false, + ) + .await +} + +async fn relay_request_body_with_marker_guard_correlated( + req: &L7Request, + client: &mut C, + upstream: &mut U, + headers: &[u8], + already_read: &[u8], + options: RelayRequestOptions<'_>, + protect_receipt: bool, +) -> Result<()> where C: AsyncRead + Unpin, U: AsyncWrite + Unpin, @@ -1430,18 +1537,40 @@ where write_body_bytes(upstream, &scanner.finish()?, options).await?; } BodyLength::Chunked => { - relay_chunked_with_marker_guard(client, upstream, already_read, options).await?; + relay_chunked_with_marker_guard_protected( + client, + upstream, + already_read, + options, + protect_receipt, + ) + .await?; } } Ok(()) } +#[cfg(test)] async fn relay_chunked_with_marker_guard( client: &mut C, upstream: &mut U, already_read: &[u8], options: RelayRequestOptions<'_>, ) -> Result<()> +where + C: AsyncRead + Unpin, + U: AsyncWrite + Unpin, +{ + relay_chunked_with_marker_guard_protected(client, upstream, already_read, options, false).await +} + +async fn relay_chunked_with_marker_guard_protected( + client: &mut C, + upstream: &mut U, + already_read: &[u8], + options: RelayRequestOptions<'_>, + protect_receipt: bool, +) -> Result<()> where C: AsyncRead + Unpin, U: AsyncWrite + Unpin, @@ -1472,11 +1601,19 @@ where if contains_reserved_credential_marker_bytes(&trailer) { return Err(BodyCredentialError::Trailer.into()); } - write_body_bytes(upstream, &trailer, options).await?; - write_body_bytes(upstream, b"\r\n", options).await?; if trailer.is_empty() { + write_body_bytes(upstream, b"\r\n", options).await?; return Ok(()); } + if !protect_receipt + || !http_field_line_has_name( + &trailer, + openshell_core::token_grant_failure::REQUEST_ID_HEADER, + ) + { + write_body_bytes(upstream, &trailer, options).await?; + write_body_bytes(upstream, b"\r\n", options).await?; + } } } @@ -2292,6 +2429,89 @@ fn strip_header(headers: &[u8], strip_name: &str) -> Result> { Ok(out.into_bytes()) } +fn strip_platform_request_id_from_head(headers: &[u8]) -> Vec { + let header_end = headers + .windows(4) + .position(|window| window == b"\r\n\r\n") + .map_or(headers.len(), |position| position + 4); + let (header_block, buffered_suffix) = headers.split_at(header_end); + let protected = openshell_core::token_grant_failure::REQUEST_ID_HEADER.as_bytes(); + let mut out = Vec::with_capacity(headers.len()); + let mut position = 0usize; + + while position < header_block.len() { + let line_end = header_block[position..] + .windows(2) + .position(|window| window == b"\r\n") + .map_or(header_block.len(), |end| position + end); + let line = &header_block[position..line_end]; + if line.is_empty() { + out.extend_from_slice(b"\r\n"); + break; + } + let Some(colon) = line.iter().position(|byte| *byte == b':') else { + out.extend_from_slice(line); + out.extend_from_slice(b"\r\n"); + position = line_end.saturating_add(2); + continue; + }; + let name = trim_ascii_ows(&line[..colon]); + if name.eq_ignore_ascii_case(protected) { + position = line_end.saturating_add(2); + continue; + } + if name.eq_ignore_ascii_case(b"trailer") { + let values: Vec<_> = line[colon + 1..] + .split(|byte| *byte == b',') + .filter(|value| !trim_ascii_ows(value).eq_ignore_ascii_case(protected)) + .collect(); + if values.len() != line[colon + 1..].split(|byte| *byte == b',').count() { + if values.iter().all(|value| trim_ascii_ows(value).is_empty()) { + position = line_end.saturating_add(2); + continue; + } + out.extend_from_slice(&line[..=colon]); + for (index, value) in values.iter().enumerate() { + if index > 0 { + out.push(b','); + } + out.extend_from_slice(value); + } + out.extend_from_slice(b"\r\n"); + position = line_end.saturating_add(2); + continue; + } + } + out.extend_from_slice(line); + out.extend_from_slice(b"\r\n"); + position = line_end.saturating_add(2); + } + out.extend_from_slice(buffered_suffix); + out +} + +fn trim_ascii_ows(mut value: &[u8]) -> &[u8] { + while value + .first() + .is_some_and(|byte| matches!(byte, b' ' | b'\t')) + { + value = &value[1..]; + } + while value + .last() + .is_some_and(|byte| matches!(byte, b' ' | b'\t')) + { + value = &value[..value.len() - 1]; + } + value +} + +fn http_field_line_has_name(line: &[u8], expected: &str) -> bool { + line.iter() + .position(|byte| *byte == b':') + .is_some_and(|end| line[..end].eq_ignore_ascii_case(expected.as_bytes())) +} + pub(crate) fn connection_nominated_header_names(headers: &[u8]) -> Result> { validate_http_request_header_block(headers)?; let header_str = @@ -2815,7 +3035,15 @@ async fn send_deny_response( context: Option>, ) -> Result<()> { let body = deny_response_body(req, policy_name, reason, redacted_target, context); - send_forbidden_json(policy_name, body, client).await + send_json_response_internal( + policy_name, + body, + client, + "403 Forbidden", + None, + context.and_then(|context| context.request_id), + ) + .await } /// Send a middleware-specific 403 response without policy-advisor guidance. @@ -2831,7 +3059,15 @@ pub(crate) async fn send_middleware_deny_response( context: Option>, ) -> Result<()> { let body = middleware_deny_response_body(req, policy_name, denial, redacted_target, context); - send_forbidden_json(policy_name, body, client).await + send_json_response_internal( + policy_name, + body_with_request_id(body, context.and_then(|context| context.request_id)), + client, + "403 Forbidden", + None, + context.and_then(|context| context.request_id), + ) + .await } /// Send a platform-owned response for fail-closed middleware failures. @@ -2847,7 +3083,15 @@ pub(crate) async fn send_middleware_failure_response( context: Option>, ) -> Result<()> { let body = middleware_failure_response_body(req, policy_name, redacted_target, context); - send_forbidden_json(policy_name, body, client).await + send_json_response_internal( + policy_name, + body_with_request_id(body, context.and_then(|context| context.request_id)), + client, + "403 Forbidden", + None, + context.and_then(|context| context.request_id), + ) + .await } /// Send a platform-owned availability response when shared middleware work @@ -2860,15 +3104,25 @@ pub(crate) async fn send_middleware_unavailable_response( context: Option>, ) -> Result<()> { let body = middleware_failure_response_body(req, policy_name, redacted_target, context); - send_json_response(policy_name, body, client, "503 Service Unavailable").await + send_json_response_internal( + policy_name, + body_with_request_id(body, context.and_then(|context| context.request_id)), + client, + "503 Service Unavailable", + None, + context.and_then(|context| context.request_id), + ) + .await } -async fn send_forbidden_json( - policy_name: &str, - body: serde_json::Value, - client: &mut C, -) -> Result<()> { - send_json_response(policy_name, body, client, "403 Forbidden").await +fn body_with_request_id( + mut body: serde_json::Value, + request_id: Option<&str>, +) -> serde_json::Value { + if let (Some(request_id), serde_json::Value::Object(fields)) = (request_id, &mut body) { + fields.insert("request_id".to_string(), serde_json::json!(request_id)); + } + body } /// Send a platform-owned JSON response with an explicit HTTP status. @@ -2889,15 +3143,68 @@ pub(crate) async fn send_json_response_with_allow( client: &mut C, status: &str, allowed_methods: Option<&'static str>, +) -> Result<()> { + send_json_response_internal(policy_name, body, client, status, allowed_methods, None).await +} + +pub(crate) async fn send_json_response_with_allow_and_request_id( + policy_name: &str, + mut body: serde_json::Value, + client: &mut C, + status: &str, + allowed_methods: Option<&'static str>, + request_id: &str, +) -> Result<()> { + if let serde_json::Value::Object(fields) = &mut body { + fields.insert("request_id".to_string(), serde_json::json!(request_id)); + } + send_json_response_internal( + policy_name, + body, + client, + status, + allowed_methods, + Some(request_id), + ) + .await +} + +pub(crate) async fn send_json_response_with_request_id( + policy_name: &str, + mut body: serde_json::Value, + client: &mut C, + status: &str, + request_id: &str, +) -> Result<()> { + if let serde_json::Value::Object(fields) = &mut body { + fields.insert("request_id".to_string(), serde_json::json!(request_id)); + } + send_json_response_internal(policy_name, body, client, status, None, Some(request_id)).await +} + +async fn send_json_response_internal( + policy_name: &str, + body: serde_json::Value, + client: &mut C, + status: &str, + allowed_methods: Option<&'static str>, + request_id: Option<&str>, ) -> Result<()> { let body_bytes = body.to_string(); let allow = allowed_methods.map_or_else(String::new, |methods| format!("Allow: {methods}\r\n")); + let request_id = request_id.map_or_else(String::new, |request_id| { + format!( + "{}: {request_id}\r\n", + openshell_core::token_grant_failure::REQUEST_ID_HEADER + ) + }); let response = format!( "HTTP/1.1 {status}\r\n\ Content-Type: application/json\r\n\ Content-Length: {}\r\n\ X-OpenShell-Policy: {}\r\n\ {allow}\ + {request_id}\ Connection: close\r\n\ \r\n\ {}", @@ -3031,6 +3338,9 @@ fn deny_response_body( if let Some(binary) = binary { body.insert("binary".to_string(), serde_json::json!(binary)); } + if let Some(request_id) = context.request_id { + body.insert("request_id".to_string(), serde_json::json!(request_id)); + } body.insert( "rule_missing".to_string(), serde_json::Value::Object(rule_missing), @@ -3224,12 +3534,28 @@ where /// `already_read` are overflow bytes buffered while parsing the headers. They /// are validated here before being written so ambiguous framing cannot reach /// the peer. +#[cfg(test)] async fn relay_chunked( reader: &mut R, writer: &mut W, already_read: &[u8], generation_guard: Option<&PolicyGenerationGuard>, ) -> Result<()> +where + R: AsyncRead + Unpin, + W: AsyncWrite + Unpin, +{ + relay_chunked_with_receipt_protection(reader, writer, already_read, generation_guard, false) + .await +} + +async fn relay_chunked_with_receipt_protection( + reader: &mut R, + writer: &mut W, + already_read: &[u8], + generation_guard: Option<&PolicyGenerationGuard>, + protect_receipt: bool, +) -> Result<()> where R: AsyncRead + Unpin, W: AsyncWrite + Unpin, @@ -3253,6 +3579,7 @@ where let mut trailer_count = 0usize; let mut trailer_bytes = 0usize; loop { + let pending_start = input.pending_len(); let trailer_line = input .read_line( reader, @@ -3278,6 +3605,14 @@ where ); return Ok(()); } + if protect_receipt + && http_field_line_has_name( + &trailer_line, + openshell_core::token_grant_failure::REQUEST_ID_HEADER, + ) + { + input.pending.truncate(pending_start); + } if trailer_count == MAX_CHUNK_TRAILER_FIELDS { return Err(miette!( "Chunk trailers exceed {MAX_CHUNK_TRAILER_FIELDS} fields" @@ -4633,6 +4968,7 @@ mod tests { port: Some(443), binary: Some("/usr/bin/gh"), agent_proposals_enabled: true, + request_id: None, }), ); @@ -4697,6 +5033,7 @@ mod tests { port: Some(443), binary: Some("/usr/bin/gh"), agent_proposals_enabled: false, + request_id: None, }), ); @@ -4800,6 +5137,7 @@ mod tests { port: Some(443), binary: Some("/usr/bin/curl"), agent_proposals_enabled: true, + request_id: None, }), ); @@ -4838,6 +5176,7 @@ mod tests { port: Some(443), binary: Some("/usr/bin/gh"), agent_proposals_enabled: true, + request_id: Some("request-123"), }), ) .await @@ -4852,12 +5191,14 @@ mod tests { assert!(response.starts_with("HTTP/1.1 403 Forbidden")); assert!(response.contains("Content-Type: application/json")); assert!(response.contains("X-OpenShell-Policy: github-readonly")); + assert!(response.contains("x-openshell-request-id: request-123")); let (_, body) = response.split_once("\r\n\r\n").unwrap(); let body: serde_json::Value = serde_json::from_str(body).unwrap(); assert_eq!(body["error"], "policy_denied"); assert_eq!(body["method"], "POST"); assert_eq!(body["path"], "/user/repos"); + assert_eq!(body["request_id"], "request-123"); assert_eq!(body["rule_missing"]["host"], "api.github.com"); assert_eq!(body["next_steps"][2]["action"], "inspect_recent_denials"); assert!(body["agent_guidance"].as_str().unwrap().contains("retry")); @@ -5007,6 +5348,43 @@ mod tests { assert_eq!(forwarded, expected); } + #[tokio::test] + async fn relay_chunked_strips_platform_request_id_trailer() { + let mut reader = + b"3\r\nabc\r\n0\r\nX-OpenShell-Request-ID: forged\r\nX-Checksum: ok\r\n\r\n".as_slice(); + let mut writer = Vec::new(); + + relay_chunked_with_receipt_protection(&mut reader, &mut writer, &[], None, true) + .await + .expect("chunked body should relay"); + + assert_eq!(writer, b"3\r\nabc\r\n0\r\nX-Checksum: ok\r\n\r\n"); + } + + #[tokio::test] + async fn guarded_chunked_request_strips_platform_request_id_trailer() { + let mut reader = + b"3\r\nabc\r\n0\r\nx-openshell-request-id: forged\r\nX-Checksum: ok\r\n\r\n".as_slice(); + let mut writer = Vec::new(); + + relay_chunked_with_marker_guard_protected( + &mut reader, + &mut writer, + &[], + RelayRequestOptions::default(), + true, + ) + .await + .expect("guarded chunked body should relay"); + + assert!( + !String::from_utf8_lossy(&writer) + .to_ascii_lowercase() + .contains("x-openshell-request-id") + ); + assert!(String::from_utf8_lossy(&writer).contains("X-Checksum: ok\r\n")); + } + #[tokio::test] async fn relay_chunked_handles_tiny_chunks_without_an_aggregate_framing_limit() { let mut chunked_body = Vec::new(); @@ -5264,6 +5642,18 @@ mod tests { assert!(parse_body_length(headers).is_err()); } + #[test] + fn platform_request_id_is_stripped_without_touching_body() { + let suffix = b"4\r\nbody\r\n0\r\nX-OpenShell-Request-ID: forged-trailer\r\n\r\n"; + let mut request = b"POST /api HTTP/1.1\r\nHost: x\r\nTransfer-Encoding: chunked\r\nX-Opaque: \xff\xfe\r\nX-OpenShell-Request-ID: forged-head\r\nTrailer: X-OpenShell-Request-ID, X-Checksum\r\n\r\n".to_vec(); + request.extend_from_slice(suffix); + + let stripped = strip_platform_request_id_from_head(&request); + let mut expected = b"POST /api HTTP/1.1\r\nHost: x\r\nTransfer-Encoding: chunked\r\nX-Opaque: \xff\xfe\r\nTrailer: X-Checksum\r\n\r\n".to_vec(); + expected.extend_from_slice(suffix); + assert_eq!(stripped, expected); + } + #[test] fn strip_connection_nominated_headers_before_forwarding() { let raw = b"GET /api HTTP/1.1\r\nHost: x\r\nX-Guard: hidden\r\nConnection: keep-alive, x-guard\r\nKeep-Alive: timeout=5\r\nX-Visible: yes\r\n\r\n"; diff --git a/crates/openshell-supervisor-network/src/l7/rest/http_response.rs b/crates/openshell-supervisor-network/src/l7/rest/http_response.rs index 0113dc4323..84fecaebc0 100644 --- a/crates/openshell-supervisor-network/src/l7/rest/http_response.rs +++ b/crates/openshell-supervisor-network/src/l7/rest/http_response.rs @@ -4,6 +4,7 @@ //! HTTP response relay and pre-return middleware integration. use super::*; +use crate::l7::correlation::RequestCorrelation; /// Default wall-clock bound shared by whole-body stages in one response. pub const DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT: std::time::Duration = @@ -26,6 +27,7 @@ pub(super) struct RelayResponseOptions<'a> { pub(super) client_requested_upgrade: bool, pub(super) websocket: Option, pub(super) observer: Option<&'a EndpointObserver>, + pub(super) correlation: Option<&'a RequestCorrelation>, } impl Default for RelayResponseOptions<'_> { @@ -35,6 +37,7 @@ impl Default for RelayResponseOptions<'_> { client_requested_upgrade: true, websocket: None, observer: None, + correlation: None, } } } @@ -57,7 +60,7 @@ where // Forward interim responses unchanged, but retain the final response head // until response middleware preflight completes. let mut informational_header_bytes = 0; - let header_end = loop { + let mut header_end = loop { let header_end = buf .windows(4) .position(|w| w == b"\r\n\r\n") @@ -73,16 +76,27 @@ where } if let Some(header_end) = header_end { + if options.correlation.is_some() + && let Err(error) = validate_http_response_header_block(&buf[..header_end]) + { + if let Some(observer) = options.observer.as_ref() { + observer.observe(EndpointResult::TransportFailed); + } + record_upstream_response_error(options.correlation, "upstream_response_headers"); + return Err(error); + } let header_str = String::from_utf8_lossy(&buf[..header_end]); if matches!( parse_observed_http_status_code(&header_str), Some(100 | 102..=199) ) { informational_header_bytes += header_end; - client - .write_all(&buf[..header_end]) - .await - .into_diagnostic()?; + let sanitized = if options.correlation.is_some() { + strip_platform_request_id_from_head(&buf[..header_end]) + } else { + buf[..header_end].to_vec() + }; + client.write_all(&sanitized).await.into_diagnostic()?; client.flush().await.into_diagnostic()?; buf.drain(..header_end); continue; @@ -95,6 +109,7 @@ where if let Some(observer) = options.observer.as_ref() { observer.observe(EndpointResult::TransportFailed); } + record_upstream_response_error(options.correlation, "upstream_response_incomplete"); if !buf.is_empty() { client.write_all(&buf).await.into_diagnostic()?; } @@ -103,9 +118,22 @@ where buf.extend_from_slice(&tmp[..n]); }; + if options.correlation.is_some() { + let sanitized = strip_platform_request_id_from_head(&buf[..header_end]); + if sanitized.len() != header_end { + let overflow = buf.split_off(header_end); + buf = sanitized; + header_end = buf.len(); + buf.extend_from_slice(&overflow); + } + } + // Parse response framing let header_str = String::from_utf8_lossy(&buf[..header_end]); let observed_status = parse_observed_http_status_code(&header_str); + if let (Some(correlation), Some(status_code)) = (options.correlation, observed_status) { + correlation.record_upstream_status(status_code); + } if let Some(observer) = options.observer.as_ref() { match observed_status { Some(status_code) if status_code < 400 => { @@ -145,10 +173,8 @@ where options.websocket_extensions, options.websocket.as_ref(), )?; - client - .write_all(&buf[..header_end]) - .await - .into_diagnostic()?; + let final_head = final_response_head(&buf[..header_end], options.correlation); + client.write_all(&final_head).await.into_diagnostic()?; client.flush().await.into_diagnostic()?; let overflow = buf[header_end..].to_vec(); debug!( @@ -156,6 +182,7 @@ where overflow_bytes = overflow.len(), "101 Switching Protocols — signaling protocol upgrade" ); + record_allowed_response(options.correlation); return Ok(RelayOutcome::Upgraded { overflow, websocket_permessage_deflate, @@ -175,6 +202,7 @@ where body_length, server_wants_close || http_10_closes_by_default, event_stream, + options.correlation, )) .await? { @@ -185,13 +213,15 @@ where }; } + let final_head = final_response_head(&buf[..header_end], options.correlation); + // Bodiless responses (HEAD, 1xx, 204, 304): forward headers only, skip body if is_bodiless_response(request_method, status_code) { - client - .write_all(&buf[..header_end]) - .await - .into_diagnostic()?; - return finish_response(client, server_wants_close || http_10_closes_by_default).await; + client.write_all(&final_head).await.into_diagnostic()?; + let outcome = + finish_response(client, server_wants_close || http_10_closes_by_default).await?; + record_allowed_response(options.correlation); + return Ok(outcome); } // No explicit framing (no Content-Length, no Transfer-Encoding). @@ -200,7 +230,7 @@ where if server_wants_close || http_10_closes_by_default || event_stream { // Server indicated it will close, or this is a streaming response // such as SSE where the body is intentionally delimited by EOF. - let before_end = &buf[..header_end - 2]; + let before_end = &final_head[..final_head.len() - 2]; client.write_all(before_end).await.into_diagnostic()?; if server_wants_close || http_10_closes_by_default { client @@ -222,25 +252,21 @@ where } client.flush().await.into_diagnostic()?; client.shutdown().await.into_diagnostic()?; + record_allowed_response(options.correlation); return Ok(RelayOutcome::Consumed); } // No Connection: close — an HTTP/1.1 keep-alive server that omits // framing headers has an empty body. Forward headers and continue // the relay loop instead of blocking on relay_until_eof. debug!("BodyLength::None without Connection: close — treating body as empty"); - client - .write_all(&buf[..header_end]) - .await - .into_diagnostic()?; + client.write_all(&final_head).await.into_diagnostic()?; client.flush().await.into_diagnostic()?; + record_allowed_response(options.correlation); return Ok(RelayOutcome::Reusable); } // Hold buffered chunk framing until relay_chunked validates it. - client - .write_all(&buf[..header_end]) - .await - .into_diagnostic()?; + client.write_all(&final_head).await.into_diagnostic()?; let overflow = &buf[header_end..]; let overflow_len = (buf.len() - header_end) as u64; @@ -256,7 +282,14 @@ where } } BodyLength::Chunked => { - relay_chunked(upstream, client, overflow, None).await?; + relay_chunked_with_receipt_protection( + upstream, + client, + overflow, + None, + options.correlation.is_some(), + ) + .await?; } BodyLength::None => unreachable!(), } @@ -267,7 +300,224 @@ where "relay_response complete (explicit framing)" ); - finish_response(client, server_wants_close || http_10_closes_by_default).await + let outcome = finish_response(client, server_wants_close || http_10_closes_by_default).await?; + record_allowed_response(options.correlation); + Ok(outcome) +} + +fn record_allowed_response(correlation: Option<&RequestCorrelation>) { + if let Some(correlation) = correlation { + correlation.record_final_http_outcome("allowed"); + } +} + +fn record_denied_response(correlation: Option<&RequestCorrelation>, stage: &'static str) { + if let Some(correlation) = correlation { + correlation.record_response_status(403); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_failure_reason(stage); + } +} + +fn record_platform_response_error(correlation: Option<&RequestCorrelation>, stage: &'static str) { + if let Some(correlation) = correlation { + correlation.record_response_status(502); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("platform_error"); + correlation.record_failure_reason(stage); + } +} + +fn record_upstream_response_error(correlation: Option<&RequestCorrelation>, stage: &'static str) { + if let Some(correlation) = correlation { + correlation.record_final_http_outcome("upstream_error"); + correlation.record_failure_reason(stage); + } +} + +fn record_committed_response_error( + correlation: Option<&RequestCorrelation>, + outcome: &'static str, + stage: &'static str, +) { + if let Some(correlation) = correlation { + correlation.record_final_http_outcome(outcome); + correlation.record_failure_reason(stage); + } +} + +fn validate_http_response_header_block(headers: &[u8]) -> Result<()> { + if !headers.ends_with(b"\r\n\r\n") { + return Err(miette!("HTTP response header block is incomplete")); + } + + for (index, byte) in headers.iter().copied().enumerate() { + match byte { + b'\r' if headers.get(index + 1) != Some(&b'\n') => { + return Err(miette!("HTTP response headers contain a bare CR")); + } + b'\n' if index == 0 || headers.get(index - 1) != Some(&b'\r') => { + return Err(miette!("HTTP response headers contain a bare LF")); + } + _ => {} + } + } + + let header_lines = &headers[..headers.len() - 2]; + let mut lines = header_lines + .split(|byte| *byte == b'\n') + .map(|line| line.strip_suffix(b"\r").unwrap_or(line)); + let status_line = lines + .next() + .ok_or_else(|| miette!("HTTP response status line is missing"))?; + validate_http_response_status_line(status_line)?; + + while let Some(line) = lines.next() { + if line.is_empty() { + if lines.next().is_some() { + return Err(miette!( + "HTTP response header block contains an unexpected empty line" + )); + } + break; + } + if line + .first() + .is_some_and(|byte| matches!(byte, b' ' | b'\t')) + { + return Err(miette!( + "HTTP response header continuation lines are not supported" + )); + } + let colon = line + .iter() + .position(|byte| *byte == b':') + .ok_or_else(|| miette!("HTTP response header field is malformed"))?; + let name = &line[..colon]; + if name.last().is_some_and(|byte| matches!(byte, b' ' | b'\t')) { + return Err(miette!( + "HTTP response header field contains whitespace before the colon" + )); + } + http::HeaderName::from_bytes(name) + .map_err(|_| miette!("HTTP response field name is malformed"))?; + http::HeaderValue::from_bytes(&line[colon + 1..]) + .map_err(|_| miette!("HTTP response field value is malformed"))?; + } + Ok(()) +} + +fn validate_http_response_status_line(status_line: &[u8]) -> Result<()> { + let remainder = if let Some(remainder) = status_line.strip_prefix(b"HTTP/1.0 ") { + remainder + } else if let Some(remainder) = status_line.strip_prefix(b"HTTP/1.1 ") { + remainder + } else { + return Err(miette!( + "HTTP response status line must use HTTP/1.0 or HTTP/1.1" + )); + }; + if remainder.len() < 3 || (remainder.len() > 3 && remainder[3] != b' ') { + return Err(miette!("HTTP response status line is malformed")); + } + http::StatusCode::from_bytes(&remainder[..3]) + .map_err(|_| miette!("HTTP response status code is invalid"))?; + http::HeaderValue::from_bytes(remainder.get(4..).unwrap_or_default()) + .map_err(|_| miette!("HTTP response reason phrase is malformed"))?; + Ok(()) +} + +fn final_response_head(headers: &[u8], correlation: Option<&RequestCorrelation>) -> Vec { + let Some(correlation) = correlation else { + return headers.to_vec(); + }; + let stripped = strip_platform_request_id_from_head(headers); + let stripped = strip_platform_request_id_from_connection(&stripped); + let Some(header_end) = stripped.windows(4).position(|window| window == b"\r\n\r\n") else { + return stripped; + }; + let receipt = format!( + "{}: {}\r\n\r\n", + openshell_core::token_grant_failure::REQUEST_ID_HEADER, + correlation.request_id(), + ); + let mut output = Vec::with_capacity(stripped.len() + receipt.len()); + output.extend_from_slice(&stripped[..header_end + 2]); + output.extend_from_slice(receipt.as_bytes()); + output.extend_from_slice(&stripped[header_end + 4..]); + output +} + +fn strip_platform_request_id_from_connection(headers: &[u8]) -> Vec { + let header_end = headers + .windows(4) + .position(|window| window == b"\r\n\r\n") + .map_or(headers.len(), |position| position + 4); + let (header_block, buffered_suffix) = headers.split_at(header_end); + let protected = openshell_core::token_grant_failure::REQUEST_ID_HEADER.as_bytes(); + let mut output = Vec::with_capacity(headers.len()); + let mut position = 0usize; + + while position < header_block.len() { + let line_end = header_block[position..] + .windows(2) + .position(|window| window == b"\r\n") + .map_or(header_block.len(), |end| position + end); + let line = &header_block[position..line_end]; + if line.is_empty() { + output.extend_from_slice(b"\r\n"); + break; + } + + let Some(colon) = line.iter().position(|byte| *byte == b':') else { + output.extend_from_slice(line); + output.extend_from_slice(b"\r\n"); + position = line_end.saturating_add(2); + continue; + }; + let name = trim_ascii_ows(&line[..colon]); + if !name.eq_ignore_ascii_case(b"connection") { + output.extend_from_slice(line); + output.extend_from_slice(b"\r\n"); + position = line_end.saturating_add(2); + continue; + } + + let mut removed_protected = false; + let retained: Vec<_> = line[colon + 1..] + .split(|byte| *byte == b',') + .filter_map(|value| { + let token = trim_ascii_ows(value); + if token.eq_ignore_ascii_case(protected) { + removed_protected = true; + None + } else if token.is_empty() { + None + } else { + Some(token) + } + }) + .collect(); + if !removed_protected { + output.extend_from_slice(line); + output.extend_from_slice(b"\r\n"); + } else if !retained.is_empty() { + output.extend_from_slice(&line[..=colon]); + output.push(b' '); + for (index, token) in retained.iter().enumerate() { + if index > 0 { + output.extend_from_slice(b", "); + } + output.extend_from_slice(token); + } + output.extend_from_slice(b"\r\n"); + } + position = line_end.saturating_add(2); + } + + output.extend_from_slice(buffered_suffix); + output } /// Body framing determines when delivery finishes, not whether another @@ -299,6 +549,7 @@ async fn relay_response_through_middleware( body_length: BodyLength, server_wants_close: bool, event_stream: bool, + response_correlation: Option<&RequestCorrelation>, ) -> Result> where U: AsyncRead + Unpin, @@ -332,6 +583,7 @@ where Err(error) => { debug!(error = %error, "HTTP response head normalization failed"); emit_http_response_middleware_failure( + response_correlation, middleware.policy_name, &middleware.target, status_code, @@ -342,6 +594,8 @@ where request_method, middleware.policy_name, &middleware.target, + "response_middleware_headers", + response_correlation, ) .await?; return Ok(Some(RelayOutcome::Consumed)); @@ -377,6 +631,7 @@ where Err(error) => { debug!(error = %error, "HTTP response middleware preflight failed"); emit_http_response_middleware_failure( + response_correlation, middleware.policy_name, &middleware.target, status_code, @@ -387,6 +642,8 @@ where request_method, middleware.policy_name, &middleware.target, + "response_middleware_preflight", + response_correlation, ) .await?; return Ok(Some(RelayOutcome::Consumed)); @@ -427,7 +684,7 @@ where )); } } - crate::telemetry::emit_decision(preflight_events, None); + crate::l7::middleware::emit_request_events(response_correlation, preflight_events); if !preflight.allowed { if let Some(denial) = preflight.denial.as_ref() { send_response_middleware_denial( @@ -436,6 +693,8 @@ where middleware.policy_name, &middleware.target, denial, + "response_middleware_preflight", + response_correlation, ) .await?; } else { @@ -444,19 +703,21 @@ where request_method, middleware.policy_name, &middleware.target, + "response_middleware_preflight", + response_correlation, ) .await?; } return Ok(Some(RelayOutcome::Consumed)); } - crate::telemetry::emit_decision( + crate::l7::middleware::emit_request_events( + response_correlation, http_response_decision_events( middleware.policy_name, &middleware.target, status_code, &preflight.invocations, ), - None, ); let Some(mut session) = preflight.session else { @@ -477,8 +738,10 @@ where body_length, server_wants_close, event_stream, + response_correlation, ) .await?; + record_allowed_response(response_correlation); return Ok(Some(outcome)); }; @@ -504,12 +767,14 @@ where false, true, )); - crate::telemetry::emit_decision(events, None); + crate::l7::middleware::emit_request_events(response_correlation, events); send_response_delivery_failure( client, request_method, middleware.policy_name, &middleware.target, + "response_middleware_finish", + response_correlation, ) .await?; return Ok(Some(RelayOutcome::Consumed)); @@ -519,7 +784,8 @@ where guard.ensure_current()?; } let mut headers = preflight.headers; - crate::telemetry::emit_decision( + crate::l7::middleware::emit_request_events( + response_correlation, http_response_diagnostic_events( middleware.policy_name, &middleware.target, @@ -527,7 +793,6 @@ where &finish.invocations, &finish.findings, ), - None, ); if finish.strip_stale_integrity_headers { strip_response_integrity_headers(&mut headers); @@ -540,8 +805,10 @@ where server_wants_close, &[], ); + let head = final_response_head(&head, response_correlation); client.write_all(&head).await.into_diagnostic()?; client.flush().await.into_diagnostic()?; + record_allowed_response(response_correlation); return Ok(Some(if server_wants_close { RelayOutcome::Consumed } else { @@ -574,6 +841,7 @@ where server_wants_close || close_delimited_output, downstream_trailers, ); + let streaming_head = final_response_head(&streaming_head, response_correlation); let mut committed = !whole_body; if committed { if let Err(error) = client.write_all(&streaming_head).await { @@ -607,6 +875,7 @@ where &middleware.target, status_code, &connection_nominated_headers, + response_correlation, ) .await; let trailers = match body_result { @@ -650,8 +919,21 @@ where middleware_stop.is_some(), )); } - crate::telemetry::emit_decision(events, None); + crate::l7::middleware::emit_request_events(response_correlation, events); if committed { + if middleware_stop.is_some_and(|stop| stop.failure.denial.is_some()) { + record_committed_response_error( + response_correlation, + "denied", + "response_middleware_body", + ); + } else { + record_committed_response_error( + response_correlation, + "platform_error", + "response_middleware_body", + ); + } return Err(error); } debug!(error = %error, "HTTP response processing failed before commitment"); @@ -662,6 +944,8 @@ where middleware.policy_name, &middleware.target, denial, + "response_middleware_body", + response_correlation, ) .await?; } else { @@ -670,6 +954,8 @@ where request_method, middleware.policy_name, &middleware.target, + "response_middleware_body", + response_correlation, ) .await?; } @@ -705,8 +991,21 @@ where true, )); } - crate::telemetry::emit_decision(events, None); + crate::l7::middleware::emit_request_events(response_correlation, events); if committed { + if error.denial.is_some() { + record_committed_response_error( + response_correlation, + "denied", + "response_middleware_finish", + ); + } else { + record_committed_response_error( + response_correlation, + "platform_error", + "response_middleware_finish", + ); + } return Err(miette!( "HTTP response middleware failed after commitment: {error}" )); @@ -719,6 +1018,8 @@ where middleware.policy_name, &middleware.target, denial, + "response_middleware_finish", + response_correlation, ) .await?; } else { @@ -727,6 +1028,8 @@ where request_method, middleware.policy_name, &middleware.target, + "response_middleware_finish", + response_correlation, ) .await?; } @@ -736,7 +1039,8 @@ where if let Some(guard) = middleware.generation_guard { guard.ensure_current()?; } - crate::telemetry::emit_decision( + crate::l7::middleware::emit_request_events( + response_correlation, http_response_diagnostic_events( middleware.policy_name, &middleware.target, @@ -744,7 +1048,6 @@ where &finish.invocations, &finish.findings, ), - None, ); if whole_body && !committed { @@ -779,6 +1082,7 @@ where server_wants_close, &trailer_names, ); + let head = final_response_head(&head, response_correlation); client.write_all(&head).await.into_diagnostic()?; if matches!(framing, ResponseFraming::Chunked) { for unit in &finish.body_units { @@ -803,6 +1107,7 @@ where } } client.flush().await.into_diagnostic()?; + record_allowed_response(response_correlation); Ok(Some( if server_wants_close || (committed && close_delimited_output) @@ -829,6 +1134,7 @@ async fn relay_headers_only_response( body_length: BodyLength, server_wants_close: bool, event_stream: bool, + response_correlation: Option<&RequestCorrelation>, ) -> Result where U: AsyncRead + Unpin, @@ -842,6 +1148,7 @@ where server_wants_close, declared_trailers, ); + let head = final_response_head(&head, response_correlation); client.write_all(&head).await.into_diagnostic()?; if is_bodiless_response(request_method, status_code) { @@ -863,7 +1170,16 @@ where relay_fixed(upstream, client, remaining, None).await?; } } - BodyLength::Chunked => relay_chunked(upstream, client, overflow, None).await?, + BodyLength::Chunked => { + relay_chunked_with_receipt_protection( + upstream, + client, + overflow, + None, + response_correlation.is_some(), + ) + .await?; + } BodyLength::None if server_wants_close || event_stream => { if !overflow.is_empty() { client.write_all(overflow).await.into_diagnostic()?; @@ -926,12 +1242,14 @@ fn http_response_diagnostic_events( } fn emit_http_response_diagnostics( + correlation: Option<&RequestCorrelation>, policy_name: &str, target: &HttpRequestTarget, status_code: u16, diagnostics: &openshell_supervisor_middleware::HttpResponseDiagnostics, ) { - crate::telemetry::emit_decision( + crate::l7::middleware::emit_request_events( + correlation, http_response_diagnostic_events( policy_name, target, @@ -939,7 +1257,6 @@ fn emit_http_response_diagnostics( &diagnostics.invocations, &diagnostics.findings, ), - None, ); } @@ -959,7 +1276,7 @@ pub(super) fn http_response_middleware_invocation_events( openshell_supervisor_middleware::HttpResponseInvocationOutcome::BlockDelivery | openshell_supervisor_middleware::HttpResponseInvocationOutcome::FailClosed ); - openshell_ocsf::HttpActivityBuilder::new(ocsf_ctx()) + openshell_ocsf::HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(openshell_ocsf::ActivityId::Other) .action(if blocked { openshell_ocsf::ActionId::Denied @@ -1107,18 +1424,22 @@ pub(super) fn http_response_middleware_fail_open_finding_event( } fn emit_http_response_middleware_failure( + correlation: Option<&RequestCorrelation>, policy_name: &str, target: &HttpRequestTarget, status_code: u16, committed: bool, ) { - crate::telemetry::egress_emit!(http_response_middleware_failure_event( - policy_name, - target, - status_code, - committed, - true, - )); + crate::l7::middleware::emit_request_events( + correlation, + [http_response_middleware_failure_event( + policy_name, + target, + status_code, + committed, + true, + )], + ); } fn http_response_middleware_failure_event( @@ -1285,7 +1606,8 @@ fn validate_http_field_value(value: &str) -> Result<()> { } fn is_protected_response_field(name: &str) -> bool { - name.eq_ignore_ascii_case("content-length") + name.eq_ignore_ascii_case(openshell_core::token_grant_failure::REQUEST_ID_HEADER) + || name.eq_ignore_ascii_case("content-length") || is_hidden_response_field(name) || openshell_supervisor_middleware::headers::is_response_credential_header(name) } @@ -1468,6 +1790,7 @@ async fn relay_normalized_response_body( target: &HttpRequestTarget, status_code: u16, connection_nominated_headers: &[String], + correlation: Option<&RequestCorrelation>, ) -> Result> where R: AsyncRead + Unpin, @@ -1482,6 +1805,7 @@ where policy_name, target, status_code, + correlation, }; match body_length { BodyLength::ContentLength(mut remaining) => { @@ -1685,6 +2009,7 @@ struct ResponseOutputState<'a> { policy_name: &'a str, target: &'a HttpRequestTarget, status_code: u16, + correlation: Option<&'a RequestCorrelation>, } #[derive(Debug)] @@ -1748,6 +2073,7 @@ async fn expire_whole_body_deadline( let diagnostics = session.take_diagnostics(); if output.is_ok() { emit_http_response_diagnostics( + framing.correlation, framing.policy_name, framing.target, framing.status_code, @@ -1892,6 +2218,7 @@ async fn process_response_unit( let diagnostics = session.take_diagnostics(); if output.is_ok() { emit_http_response_diagnostics( + framing.correlation, framing.policy_name, framing.target, framing.status_code, @@ -2022,7 +2349,10 @@ async fn send_response_middleware_denial( policy_name: &str, target: &HttpRequestTarget, denial: &openshell_supervisor_middleware::MiddlewareDenial, + stage: &'static str, + correlation: Option<&RequestCorrelation>, ) -> Result<()> { + record_denied_response(correlation, stage); let mut body = serde_json::Map::new(); body.insert("error".into(), serde_json::json!("middleware_denied")); body.insert( @@ -2047,7 +2377,8 @@ async fn send_response_middleware_denial( "HTTP/1.1 403 Forbidden\r\nContent-Type: application/json\r\nContent-Length: {}\r\nX-OpenShell-Policy: {policy_name}\r\nConnection: close\r\n\r\n", body.len() ); - client.write_all(head.as_bytes()).await.into_diagnostic()?; + let head = final_response_head(head.as_bytes(), correlation); + client.write_all(&head).await.into_diagnostic()?; if !request_method.eq_ignore_ascii_case("HEAD") { client.write_all(&body).await.into_diagnostic()?; } @@ -2060,7 +2391,10 @@ async fn send_response_delivery_failure( request_method: &str, policy_name: &str, target: &HttpRequestTarget, + stage: &'static str, + correlation: Option<&RequestCorrelation>, ) -> Result<()> { + record_platform_response_error(correlation, stage); let body = serde_json::to_vec(&serde_json::json!({ "error": "response_delivery_failed", "detail": "The upstream request may have completed, but OpenShell could not deliver its response. Retrying may repeat upstream side effects.", @@ -2076,7 +2410,8 @@ async fn send_response_delivery_failure( "HTTP/1.1 502 Bad Gateway\r\nContent-Type: application/json\r\nContent-Length: {}\r\nX-OpenShell-Policy: {policy_name}\r\nConnection: close\r\n\r\n", body.len() ); - client.write_all(head.as_bytes()).await.into_diagnostic()?; + let head = final_response_head(head.as_bytes(), correlation); + client.write_all(&head).await.into_diagnostic()?; if !request_method.eq_ignore_ascii_case("HEAD") { client.write_all(&body).await.into_diagnostic()?; } @@ -2196,8 +2531,9 @@ mod tests { for committed in [false, true] { let (events, received) = exported("warn", || { super::emit_http_response_middleware_failure( + None, "response-policy", - &super::HttpRequestTarget::default(), + &HttpRequestTarget::default(), 200, committed, ); @@ -2230,8 +2566,9 @@ mod tests { ..Default::default() }; super::emit_http_response_diagnostics( + None, "response-policy", - &super::HttpRequestTarget::default(), + &HttpRequestTarget::default(), 200, &diagnostics, ); @@ -2258,7 +2595,7 @@ mod tests { #[test] fn fail_closed_invocations_and_enforced_failures_record_the_denial() { - let target = super::HttpRequestTarget::default(); + let target = HttpRequestTarget::default(); for (outcome, failed, action_id) in [ (Outcome::BlockDelivery, false, 2), (Outcome::FailClosed, true, 2), @@ -2286,12 +2623,849 @@ mod tests { } } + #[test] + fn late_fail_open_invocation_does_not_hide_block_delivery() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = test_correlation("late-response-receipt"); + let target = HttpRequestTarget { + method: "GET".into(), + scheme: "https".into(), + host: "example.test".into(), + path: "/data".into(), + port: 443, + ..Default::default() + }; + correlation.emit( + openshell_ocsf::HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(openshell_ocsf::HttpRequest::new( + "GET", + openshell_ocsf::Url::new("https", "example.test", "/data", 443), + )) + .action(openshell_ocsf::ActionId::Allowed) + .status(openshell_ocsf::StatusId::Success) + .build(), + ); + correlation.response_selected(200); + for (outcome, failed) in [(Outcome::FailOpen, true), (Outcome::BlockDelivery, false)] { + for event in super::http_response_middleware_invocation_events( + "response-policy", + &target, + 200, + &[invocation(outcome, failed)], + ) { + correlation.emit(event); + } + } + drop(correlation); + let events: Vec<_> = capture + .json_events() + .into_iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!(events.len(), 2); + assert_eq!(events[1]["action"], "Denied"); + assert_eq!(events[1]["http_response"]["code"], 200); + assert_eq!( + events[1]["unmapped"]["response_middleware_outcome"], + "blockdelivery" + ); + assert_eq!( + events[1]["metadata"]["correlation_uid"], + "late-response-receipt" + ); + } + use super::{ - ResponseFraming, parse_connection_keep_alive, parse_response_head_for_middleware, - response_is_http_10, serialize_response_head, + DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT, HttpResponseMiddlewareRelay, + RelayResponseOptions, ResponseFraming, parse_connection_keep_alive, + parse_response_head_for_middleware, relay_response, response_is_http_10, + serialize_response_head, validate_http_response_header_block, }; - use crate::l7::provider::BodyLength; - use openshell_core::proto::HttpHeader; + use crate::l7::provider::{BodyLength, L7Request, RelayOutcome}; + use crate::l7::relay::L7EvalContext; + use openshell_core::proto::{ + Decision, HttpHeader, HttpRequestResult, HttpRequestTarget, HttpResponseBlockDelivery, + HttpResponseBodyMode, HttpResponseBodyResult, HttpResponseBodyTransform, HttpResponseEvent, + HttpResponseEventResult, HttpResponsePreflightInspect, HttpResponsePreflightResult, + HttpResponseTrailersResult, MiddlewareBinding, MiddlewareManifest, RequestContext, + SupervisorMiddlewareOperation, SupervisorMiddlewarePhase, http_response_body_result, + http_response_body_transform, http_response_body_unit, http_response_event, + http_response_event_result, http_response_preflight_result, + }; + use std::collections::HashMap; + use std::sync::Arc; + use tokio::sync::mpsc; + use tokio_stream::wrappers::ReceiverStream; + + struct StreamingResponseMiddleware; + + struct BlockingResponseMiddleware; + + #[tonic::async_trait] + impl openshell_supervisor_middleware::InProcessMiddleware for BlockingResponseMiddleware { + async fn describe(&self) -> MiddlewareManifest { + MiddlewareManifest { + name: "test/blocking-response".into(), + service_version: "test".into(), + bindings: vec![MiddlewareBinding { + operation: SupervisorMiddlewareOperation::HttpResponse as i32, + phase: SupervisorMiddlewarePhase::PreReturn as i32, + max_payload_bytes: 4096, + request_timeout: None, + }], + expected_audience: String::new(), + extension: Some(openshell_core::extension_protocol::extension_metadata( + openshell_core::extension_protocol::ExtensionFamily::SupervisorMiddleware, + "openshell/test-blocking-response", + "test", + [], + )), + } + } + + async fn validate_config( + &self, + _middleware_name: &str, + _config: &prost_types::Struct, + ) -> miette::Result<()> { + Ok(()) + } + + async fn evaluate_http_request( + &self, + _request: openshell_supervisor_middleware::HttpRequestView<'_>, + ) -> miette::Result { + Ok(HttpRequestResult { + decision: Decision::Allow as i32, + ..Default::default() + }) + } + + async fn open_http_response_pre_return( + &self, + mut requests: mpsc::Receiver, + ) -> Result + { + let first = requests + .recv() + .await + .ok_or_else(|| tonic::Status::invalid_argument("missing response preflight"))?; + if !matches!(first.event, Some(http_response_event::Event::Preflight(_))) { + return Err(tonic::Status::invalid_argument( + "expected response preflight", + )); + } + let result = HttpResponseEventResult { + result: Some(http_response_event_result::Result::PreflightResult( + HttpResponsePreflightResult { + action: Some(http_response_preflight_result::Action::BlockDelivery( + HttpResponseBlockDelivery {}, + )), + reason_code: "test_block".into(), + ..Default::default() + }, + )), + }; + Ok(Box::pin(futures::stream::iter([Ok(result)]))) + } + } + + #[tonic::async_trait] + impl openshell_supervisor_middleware::InProcessMiddleware for StreamingResponseMiddleware { + async fn describe(&self) -> MiddlewareManifest { + MiddlewareManifest { + name: "test/streaming-response".into(), + service_version: "test".into(), + bindings: vec![MiddlewareBinding { + operation: SupervisorMiddlewareOperation::HttpResponse as i32, + phase: SupervisorMiddlewarePhase::PreReturn as i32, + max_payload_bytes: 4096, + request_timeout: None, + }], + expected_audience: String::new(), + extension: Some(openshell_core::extension_protocol::extension_metadata( + openshell_core::extension_protocol::ExtensionFamily::SupervisorMiddleware, + "openshell/test-streaming-response", + "test", + [], + )), + } + } + + async fn validate_config( + &self, + _middleware_name: &str, + _config: &prost_types::Struct, + ) -> miette::Result<()> { + Ok(()) + } + + async fn evaluate_http_request( + &self, + _request: openshell_supervisor_middleware::HttpRequestView<'_>, + ) -> miette::Result { + Ok(HttpRequestResult { + decision: Decision::Allow as i32, + ..Default::default() + }) + } + + async fn open_http_response_pre_return( + &self, + mut requests: mpsc::Receiver, + ) -> Result + { + let (sender, receiver) = mpsc::channel(4); + tokio::spawn(async move { + while let Some(event) = requests.recv().await { + let Some(event) = event.event else { + break; + }; + let result = match event { + http_response_event::Event::Preflight(_) => HttpResponseEventResult { + result: Some(http_response_event_result::Result::PreflightResult( + HttpResponsePreflightResult { + action: Some(http_response_preflight_result::Action::Inspect( + HttpResponsePreflightInspect { + body_mode: HttpResponseBodyMode::StreamBytes as i32, + header_mutations: Vec::new(), + }, + )), + ..Default::default() + }, + )), + }, + http_response_event::Event::Body(body) => { + let Some(http_response_body_unit::Payload::Data(data)) = body.payload + else { + break; + }; + HttpResponseEventResult { + result: Some(http_response_event_result::Result::BodyResult( + HttpResponseBodyResult { + sequence: body.sequence, + action: Some(http_response_body_result::Action::Transform( + HttpResponseBodyTransform { + replacement: Some( + http_response_body_transform::Replacement::Data( + data, + ), + ), + }, + )), + ..Default::default() + }, + )), + } + } + http_response_event::Event::Trailers(_) => HttpResponseEventResult { + result: Some(http_response_event_result::Result::TrailersResult( + HttpResponseTrailersResult::default(), + )), + }, + http_response_event::Event::SessionEnd(_) => break, + }; + if sender.send(Ok(result)).await.is_err() { + break; + } + } + }); + Ok(Box::pin(ReceiverStream::new(receiver))) + } + } + + fn test_correlation_for_method( + request_id: &str, + method: &str, + ) -> crate::l7::correlation::RequestCorrelation { + let request = L7Request { + action: method.into(), + target: "/data".into(), + query_params: HashMap::new(), + raw_header: b"GET /data HTTP/1.1\r\nHost: example.test\r\n\r\n".to_vec(), + body_length: BodyLength::None, + }; + crate::l7::correlation::RequestCorrelation::with_request_id( + &request, + &L7EvalContext { + host: "example.test".into(), + port: 443, + ..Default::default() + }, + request_id.into(), + ) + } + + fn test_correlation(request_id: &str) -> crate::l7::correlation::RequestCorrelation { + test_correlation_for_method(request_id, "GET") + } + + fn request_id_header_count(response: &[u8]) -> usize { + String::from_utf8_lossy(response) + .lines() + .filter(|line| { + line.split_once(':').is_some_and(|(name, _)| { + name.eq_ignore_ascii_case( + openshell_core::token_grant_failure::REQUEST_ID_HEADER, + ) + }) + }) + .count() + } + + #[tokio::test] + async fn allowed_final_response_carries_platform_receipt() { + let correlation = test_correlation("receipt-1"); + let mut upstream = &b"HTTP/1.1 200 OK\r\nContent-Length: 4\r\n\r\nbody"[..]; + let mut client = Vec::new(); + + let outcome = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("relay response"); + + assert!(matches!(outcome, RelayOutcome::Reusable)); + assert_eq!(request_id_header_count(&client), 1); + assert!(String::from_utf8_lossy(&client).contains("x-openshell-request-id: receipt-1\r\n")); + } + + #[tokio::test] + async fn correlated_upstream_eof_preserves_transport_close_and_audits_failure() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = test_correlation("eof-request"); + let mut upstream = tokio::io::empty(); + let mut client = Vec::new(); + let outcome = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("upstream close"); + assert!(matches!(outcome, RelayOutcome::Consumed)); + assert!( + client.is_empty(), + "EOF must not become a synthetic HTTP response" + ); + drop(correlation); + let records = capture.json_events(); + let primary: Vec<_> = records + .iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!(primary.len(), 1, "{records:?}"); + assert_eq!(primary[0]["status"], "Failure"); + assert_eq!(primary[0]["metadata"]["correlation_uid"], "eof-request"); + assert!(primary[0].get("http_response").is_none()); + } + + #[tokio::test] + async fn correlated_response_without_reason_phrase_retains_legacy_tolerance() { + let correlation = test_correlation("empty-reason"); + let mut upstream = b"HTTP/1.1 200\r\nContent-Length: 0\r\n\r\n".as_slice(); + let mut client = Vec::new(); + relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("tolerated status line"); + assert!(client.starts_with(b"HTTP/1.1 200\r\n")); + assert_eq!(request_id_header_count(&client), 1); + } + + #[tokio::test] + async fn uncorrelated_response_preserves_application_owned_request_id() { + let response = + b"HTTP/1.1 200\r\nContent-Length: 0\r\nX-OpenShell-Request-ID: application\r\n\r\n"; + let mut upstream = response.as_slice(); + let mut client = Vec::new(); + relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions::default(), + None, + ) + .await + .expect("legacy response"); + assert_eq!(client, response); + } + + #[tokio::test] + async fn response_middleware_denial_has_one_correlated_primary_without_tracing() { + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let _subscriber = capture.install(); + let correlation = test_correlation_for_method("response-denied", "POST"); + correlation.emit( + openshell_ocsf::HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(openshell_ocsf::HttpRequest::new( + "POST", + openshell_ocsf::Url::new("https", "example.test", "/mutate", 443), + )) + .firewall_rule("test-policy", "l7-rest") + .action(openshell_ocsf::ActionId::Allowed) + .status(openshell_ocsf::StatusId::Success) + .build(), + ); + let runner = + openshell_supervisor_middleware::ChainRunner::new(Arc::new(BlockingResponseMiddleware)); + let chain = vec![openshell_supervisor_middleware::ChainEntry { + name: "response".into(), + implementation: "test/blocking-response".into(), + order: 0, + config: prost_types::Struct::default(), + on_error: openshell_supervisor_middleware::OnError::FailClosed, + }]; + let middleware = HttpResponseMiddlewareRelay { + chain: &chain, + runner: &runner, + request_context: RequestContext { + request_id: "response-denied".into(), + sandbox_id: "sandbox-1".into(), + ..Default::default() + }, + target: HttpRequestTarget { + scheme: "https".into(), + host: "example.test".into(), + port: 443, + method: "POST".into(), + path: "/mutate".into(), + query: String::new(), + }, + policy_name: "test-policy", + generation_guard: None, + whole_body_timeout: DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT, + }; + let mut upstream = &b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"[..]; + let mut client = Vec::new(); + + let outcome = relay_response( + "POST", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + Some(middleware), + ) + .await + .expect("middleware denial response"); + + assert!(matches!(outcome, RelayOutcome::Consumed)); + assert!(client.starts_with(b"HTTP/1.1 403 Forbidden\r\n")); + assert!( + String::from_utf8_lossy(&client) + .contains("x-openshell-request-id: response-denied\r\n") + ); + drop(correlation); + + let records = capture.json_events(); + let primary: Vec<_> = records + .iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!(primary.len(), 1, "{records:?}"); + assert_eq!(primary[0]["action"], "Denied"); + assert_eq!(primary[0]["status"], "Failure"); + assert_eq!(primary[0]["http_response"]["code"], 403); + assert_eq!(primary[0]["http_request"]["url"]["path"], "/mutate"); + assert_eq!(primary[0]["firewall_rule"]["type"], "l7-rest"); + assert_eq!(primary[0]["metadata"]["correlation_uid"], "response-denied"); + assert!(primary[0].get("trace").is_none()); + } + + #[tokio::test] + async fn upstream_receipt_is_replaced_by_platform_receipt() { + let correlation = test_correlation("trusted"); + let mut upstream = + &b"HTTP/1.1 200 OK\r\nX-OpenShell-Request-ID: forged\r\nContent-Length: 0\r\n\r\n"[..]; + let mut client = Vec::new(); + + relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("relay response"); + + assert!(!String::from_utf8_lossy(&client).contains("forged")); + assert!(String::from_utf8_lossy(&client).contains("x-openshell-request-id: trusted\r\n")); + assert_eq!(request_id_header_count(&client), 1); + } + + #[tokio::test] + async fn ordinary_response_removes_receipt_connection_nomination() { + let correlation = test_correlation("trusted"); + let mut upstream = &b"HTTP/1.1 200 OK\r\n\ + Connection: X-OpenShell-Request-ID\r\n\ + cOnNeCtIoN: keep-alive, x-OpEnShElL-rEqUeSt-Id\r\n\ + X-OpenShell-Request-ID: forged-head\r\n\ + Trailer: x-openshell-request-id, X-Checksum\r\n\ + Transfer-Encoding: chunked\r\n\r\n\ + 4\r\nbody\r\n\ + 0\r\n\ + x-OPENshell-request-ID: forged-trailer\r\n\ + X-Checksum: ok\r\n\r\n"[..]; + let mut client = Vec::new(); + + let outcome = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("relay response"); + + assert!(matches!(outcome, RelayOutcome::Reusable)); + let delivered = String::from_utf8(client).expect("HTTP response is UTF-8"); + let head = delivered + .split_once("\r\n\r\n") + .expect("response head terminator") + .0; + let connection_headers: Vec<_> = head + .lines() + .filter(|line| { + line.split_once(':') + .is_some_and(|(name, _)| name.eq_ignore_ascii_case("connection")) + }) + .collect(); + assert_eq!(connection_headers, ["cOnNeCtIoN: keep-alive"]); + assert!(!delivered.contains("forged"), "{delivered}"); + assert!(head.contains("Trailer: X-Checksum"), "{head}"); + assert!(delivered.contains("0\r\nX-Checksum: ok\r\n\r\n")); + assert!(head.contains("x-openshell-request-id: trusted"), "{head}"); + assert_eq!(request_id_header_count(delivered.as_bytes()), 1); + } + + #[tokio::test] + async fn ordinary_response_rejects_folded_receipt_connection_nomination() { + let correlation = test_correlation("trusted"); + let mut upstream = &b"HTTP/1.1 200 OK\r\nConnection: keep-alive,\r\n X-OpenShell-Request-ID\r\nContent-Length: 0\r\n\r\n"[..]; + let mut client = Vec::new(); + + let error = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect_err("folded response fields must fail closed"); + + assert!( + error + .to_string() + .contains("response header continuation lines are not supported"), + "{error}" + ); + assert!(client.is_empty()); + } + + #[tokio::test] + async fn ordinary_response_rejects_bare_lf_receipt_smuggling() { + let correlation = test_correlation("trusted"); + let mut upstream = &b"HTTP/1.1 200 OK\r\nX-Safe: yes\nX-OpenShell-Request-ID: forged\r\nContent-Length: 0\r\n\r\n"[..]; + let mut client = Vec::new(); + + let error = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect_err("bare LF response fields must fail closed"); + + assert!(error.to_string().contains("bare LF"), "{error}"); + assert!(client.is_empty()); + } + + #[test] + fn response_head_validation_rejects_malformed_syntax() { + let cases: &[(&str, &[u8])] = &[ + ("incomplete block", b"HTTP/1.1 200 OK\r\nX-Test: yes\r\n"), + ("bare LF", b"HTTP/1.1 200 OK\r\nX-Test: yes\n\r\n"), + ("bare CR", b"HTTP/1.1 200 OK\r\nX-Test: yes\rX: no\r\n\r\n"), + ( + "obsolete fold", + b"HTTP/1.1 200 OK\r\nX-Test: yes\r\n continued\r\n\r\n", + ), + ("unsupported protocol", b"HTTP/2 200 OK\r\n\r\n"), + ("invalid status range", b"HTTP/1.1 099 Nope\r\n\r\n"), + ("non-numeric status", b"HTTP/1.1 2O0 Nope\r\n\r\n"), + ("four-digit status", b"HTTP/1.1 2000 Nope\r\n\r\n"), + ("malformed field", b"HTTP/1.1 200 OK\r\nX-Test\r\n\r\n"), + ("empty field name", b"HTTP/1.1 200 OK\r\n: value\r\n\r\n"), + ( + "non-token field name", + b"HTTP/1.1 200 OK\r\nBad Name: value\r\n\r\n", + ), + ( + "whitespace before colon", + b"HTTP/1.1 200 OK\r\nX-Test : value\r\n\r\n", + ), + ( + "field control byte", + b"HTTP/1.1 200 OK\r\nX-Test: value\x0bmore\r\n\r\n", + ), + ( + "field delete byte", + b"HTTP/1.1 200 OK\r\nX-Test: value\x7f\r\n\r\n", + ), + ("reason control byte", b"HTTP/1.1 200 O\x00K\r\n\r\n"), + ( + "early empty line", + b"HTTP/1.1 200 OK\r\n\r\nX-Smuggled: yes\r\n\r\n", + ), + ]; + + for (name, input) in cases { + assert!( + validate_http_response_header_block(input).is_err(), + "{name} unexpectedly passed validation" + ); + } + } + + #[test] + fn response_head_validation_accepts_supported_informational_and_upgrade_responses() { + for response in [ + &b"HTTP/1.0 200 OK\r\nX-Test: yes\r\n\r\n"[..], + &b"HTTP/1.1 200\r\nContent-Length: 0\r\n\r\n"[..], + &b"HTTP/1.1 103 Early Hints\r\nLink: ; rel=preload\r\n\r\n"[..], + &b"HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n"[..], + ] { + validate_http_response_header_block(response).expect("valid HTTP/1.x response head"); + } + } + + #[tokio::test] + async fn response_middleware_rejects_forged_request_id_trailer() { + let correlation = test_correlation("trusted"); + let runner = openshell_supervisor_middleware::ChainRunner::new(Arc::new( + StreamingResponseMiddleware, + )); + let chain = vec![openshell_supervisor_middleware::ChainEntry { + name: "response".into(), + implementation: "test/streaming-response".into(), + order: 0, + config: prost_types::Struct::default(), + on_error: openshell_supervisor_middleware::OnError::FailClosed, + }]; + let middleware = HttpResponseMiddlewareRelay { + chain: &chain, + runner: &runner, + request_context: RequestContext { + request_id: "middleware-request".into(), + sandbox_id: "sandbox-1".into(), + ..Default::default() + }, + target: HttpRequestTarget { + scheme: "https".into(), + host: "example.test".into(), + port: 443, + method: "GET".into(), + path: "/data".into(), + query: String::new(), + }, + policy_name: "test-policy", + generation_guard: None, + whole_body_timeout: DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT, + }; + let mut upstream = &b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\nX-OpenShell-Request-ID: forged\r\n\r\n"[..]; + let mut client = Vec::new(); + + let error = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + Some(middleware), + ) + .await + .expect_err("platform correlation trailers must fail closed"); + + assert!( + error.to_string().contains("protected field name"), + "{error}" + ); + let delivered = String::from_utf8_lossy(&client); + assert!(!delivered.contains("forged"), "{delivered}"); + assert!( + delivered.contains("x-openshell-request-id: trusted\r\n"), + "{delivered}" + ); + assert_eq!(request_id_header_count(&client), 1); + } + + #[test] + fn response_middleware_rejects_declared_request_id_trailer() { + let error = parse_response_head_for_middleware( + b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nTrailer: X-OpenShell-Request-ID\r\n\r\n", + ) + .expect_err("platform correlation trailer declarations must fail closed"); + + assert!( + error.to_string().contains("protected trailer field"), + "{error}" + ); + } + + #[tokio::test] + async fn informational_responses_are_not_decorated_but_upgrade_receives_a_receipt() { + let correlation = test_correlation("receipt-1"); + let mut upstream = + &b"HTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"[..]; + let mut client = Vec::new(); + + relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("relay response"); + assert!(client.starts_with(b"HTTP/1.1 103 Early Hints\r\n\r\nHTTP/1.1 200 OK\r\n")); + assert_eq!(request_id_header_count(&client), 1); + + let mut upstream = &b"HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nX-OpenShell-Request-ID: forged\r\n\r\n"[..]; + let mut client = Vec::new(); + let outcome = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("relay response"); + assert!(matches!(outcome, RelayOutcome::Upgraded { .. })); + assert_eq!(request_id_header_count(&client), 1); + assert!(!String::from_utf8_lossy(&client).contains("forged")); + assert!(String::from_utf8_lossy(&client).contains("x-openshell-request-id: receipt-1\r\n")); + } + + #[tokio::test] + async fn switching_protocols_removes_receipt_connection_nomination() { + let correlation = test_correlation("trusted-upgrade"); + let mut upstream = &b"HTTP/1.1 101 Switching Protocols\r\n\ + Connection: X-OpenShell-Request-ID\r\n\ + cOnNeCtIoN: keep-alive, x-OpEnShElL-rEqUeSt-Id, UpGrAdE\r\n\ + Upgrade: websocket\r\n\ + X-OpenShell-Request-ID: forged\r\n\r\n"[..]; + let mut client = Vec::new(); + + let outcome = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect("relay response"); + + assert!(matches!(outcome, RelayOutcome::Upgraded { .. })); + let delivered = String::from_utf8(client).expect("HTTP response is UTF-8"); + let connection_headers: Vec<_> = delivered + .lines() + .filter(|line| { + line.split_once(':') + .is_some_and(|(name, _)| name.eq_ignore_ascii_case("connection")) + }) + .collect(); + assert_eq!(connection_headers, ["cOnNeCtIoN: keep-alive, UpGrAdE"]); + assert!(!delivered.contains("forged"), "{delivered}"); + assert!( + delivered.contains("x-openshell-request-id: trusted-upgrade\r\n"), + "{delivered}" + ); + assert_eq!(request_id_header_count(delivered.as_bytes()), 1); + } + + #[tokio::test] + async fn switching_protocols_rejects_folded_receipt_connection_nomination() { + let correlation = test_correlation("trusted-upgrade"); + let mut upstream = &b"HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade,\r\n\tX-OpenShell-Request-ID\r\nUpgrade: websocket\r\n\r\n"[..]; + let mut client = Vec::new(); + + let error = relay_response( + "GET", + &mut upstream, + &mut client, + RelayResponseOptions { + correlation: Some(&correlation), + ..Default::default() + }, + None, + ) + .await + .expect_err("folded upgrade fields must fail closed"); + + assert!( + error + .to_string() + .contains("response header continuation lines are not supported"), + "{error}" + ); + assert!(client.is_empty()); + } #[test] fn http_10_connection_persistence() { diff --git a/crates/openshell-supervisor-network/src/l7/tls.rs b/crates/openshell-supervisor-network/src/l7/tls.rs index 5eeba7a522..df2b5c678f 100644 --- a/crates/openshell-supervisor-network/src/l7/tls.rs +++ b/crates/openshell-supervisor-network/src/l7/tls.rs @@ -620,7 +620,9 @@ mod tests { // policy-allowed, private-CA-signed destination. --- let addr = serve_one(acceptor.clone()).await; let client_without = build_upstream_client_config("").unwrap(); - let tcp = tokio::net::TcpStream::connect(addr).await.unwrap(); + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let result = tls_connect_upstream(tcp, hostname, &client_without).await; assert!( result.is_err(), @@ -633,7 +635,9 @@ mod tests { // fix. --- let addr = serve_one(acceptor).await; let client_with = build_upstream_client_config(&private_ca_pem).unwrap(); - let tcp = tokio::net::TcpStream::connect(addr).await.unwrap(); + let tcp = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let result = tls_connect_upstream(tcp, hostname, &client_with).await; assert!( result.is_ok(), diff --git a/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs b/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs index f868575097..541742b277 100644 --- a/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs +++ b/crates/openshell-supervisor-network/src/l7/token_grant_injection.rs @@ -26,9 +26,9 @@ use openshell_ocsf::{ }; use tracing::warn; +use crate::l7::correlation::RequestCorrelation; use crate::l7::provider::L7Request; use crate::l7::relay::L7EvalContext; -use crate::telemetry::egress_emit; /// Borrowed grant configuration for one endpoint-bound provider credential. pub struct TokenGrantRequest<'a> { @@ -174,15 +174,35 @@ pub fn pin_live_credentials(ctx: &mut L7EvalContext) -> Option, +) -> Result { + inject_for_selectors_correlated(req, ctx, snapshot, request_id, observation_context, None).await +} + +pub async fn inject_for_selectors_correlated( + req: L7Request, + ctx: &L7EvalContext, + snapshot: &ProviderCredentialSnapshot, + request_id: &str, + observation_context: Option<&EndpointObservationContext>, + correlation: Option<&RequestCorrelation>, ) -> Result { let selected = select_scoped(snapshot, ctx, request_path(&req.target), None); - inject_or_report(req, ctx, selected, request_id, observation_context).await + inject_or_report( + req, + ctx, + selected, + request_id, + observation_context, + correlation, + ) + .await } /// Inject only grants owned by endpoints that admitted the inspected request. @@ -192,6 +212,7 @@ pub async fn inject_for_selectors( /// Credentials come from the pinned `snapshot`, and each selected key is scoped /// to its revision and installation after selection so the one-grant-per-header /// ambiguity check still compares the original credential identities. +#[cfg(test)] pub(super) async fn inject_for_admitted_owners( req: L7Request, ctx: &L7EvalContext, @@ -199,6 +220,27 @@ pub(super) async fn inject_for_admitted_owners( admitted_owners: &HashSet, request_id: &str, observation_context: Option<&EndpointObservationContext>, +) -> Result { + inject_for_admitted_owners_correlated( + req, + ctx, + snapshot, + admitted_owners, + request_id, + observation_context, + None, + ) + .await +} + +pub(super) async fn inject_for_admitted_owners_correlated( + req: L7Request, + ctx: &L7EvalContext, + snapshot: &ProviderCredentialSnapshot, + admitted_owners: &HashSet, + request_id: &str, + observation_context: Option<&EndpointObservationContext>, + correlation: Option<&RequestCorrelation>, ) -> Result { let request_path = request_path(&req.target); let unowned = snapshot @@ -220,7 +262,15 @@ pub(super) async fn inject_for_admitted_owners( } else { select_scoped(snapshot, ctx, request_path, Some(admitted_owners)) }; - inject_or_report(req, ctx, selected, request_id, observation_context).await + inject_or_report( + req, + ctx, + selected, + request_id, + observation_context, + correlation, + ) + .await } fn select_scoped( @@ -251,6 +301,7 @@ async fn inject_or_report( selected: Result, TokenGrantInjectionError>, request_id: &str, observation_context: Option<&EndpointObservationContext>, + correlation: Option<&RequestCorrelation>, ) -> Result { if selected.as_ref().is_ok_and(Vec::is_empty) { // Most requests need no grant; keep that path free of allocation. @@ -276,11 +327,11 @@ async fn inject_or_report( Vec::new() }; let result = match selected { - Ok(credentials) => inject_selected(req, ctx, credentials, request_id).await, + Ok(credentials) => inject_selected(req, ctx, credentials, request_id, correlation).await, Err(error) => Err(error), }; if let Err(error) = &result { - report_refused_grant(ctx, &method, &request_path, request_id, error); + report_refused_grant(ctx, &method, &request_path, request_id, error, correlation); } if let Some((sender, context)) = ctx .endpoint_observation_tx @@ -323,6 +374,7 @@ fn report_refused_grant( request_path: &str, request_id: &str, error: &TokenGrantInjectionError, + correlation: Option<&RequestCorrelation>, ) { let credential = error .credential @@ -341,8 +393,9 @@ fn report_refused_grant( .map_or_else(|| "none".to_string(), |status| status.to_string()), "Token grant failed" ); - egress_emit!( - HttpActivityBuilder::new(ocsf_ctx()) + crate::l7::middleware::emit_request_events( + correlation, + [HttpActivityBuilder::new(ocsf_ctx()) .activity(ActivityId::Fail) .action(ActionId::Denied) .disposition(DispositionId::Blocked) @@ -355,7 +408,7 @@ fn report_refused_grant( ctx.host, ctx.port )) .unmapped("request_id", request_id) - .build() + .build()], ); } @@ -367,6 +420,7 @@ async fn inject_selected( ctx: &L7EvalContext, credentials: Vec<(String, ProviderProfileCredential)>, request_id: &str, + correlation: Option<&RequestCorrelation>, ) -> Result { let resolver = ctx .token_grant_resolver @@ -399,8 +453,9 @@ async fn inject_selected( inject_header(&raw_header, &name, &value).map_err(TokenGrantInjectionError::new)?; } for (provider_key, _) in credentials { - egress_emit!( - HttpActivityBuilder::new(ocsf_ctx()) + crate::l7::middleware::emit_request_events( + correlation, + [HttpActivityBuilder::new(ocsf_ctx()) .activity(ActivityId::Other) .action(ActionId::Allowed) .disposition(DispositionId::Allowed) @@ -418,7 +473,7 @@ async fn inject_selected( ctx.host, ctx.port )) - .build() + .build()], ); } Ok(L7Request { raw_header, ..req }) diff --git a/crates/openshell-supervisor-network/src/policy_dns/resolver.rs b/crates/openshell-supervisor-network/src/policy_dns/resolver.rs index 1044538ad1..34f784ade5 100644 --- a/crates/openshell-supervisor-network/src/policy_dns/resolver.rs +++ b/crates/openshell-supervisor-network/src/policy_dns/resolver.rs @@ -153,7 +153,7 @@ impl SocketTrustedResolver { async fn tcp_exchange(&self, request: &[u8]) -> Result, ResolveError> { let mut stream = timeout( self.exchange_timeout, - connect_tcp_nodelay_best_effort(&[self.server]), + connect_tcp_nodelay_best_effort(self.server), ) .await .map_err(|_| ResolveError::Timeout)??; diff --git a/crates/openshell-supervisor-network/src/procfs.rs b/crates/openshell-supervisor-network/src/procfs.rs index 000bbaa6bd..42ecb18f11 100644 --- a/crates/openshell-supervisor-network/src/procfs.rs +++ b/crates/openshell-supervisor-network/src/procfs.rs @@ -901,12 +901,13 @@ mod tests { #[cfg(target_os = "linux")] #[test] fn resolve_tcp_peer_socket_owners_returns_all_descendant_socket_holders() { - use std::net::{TcpListener, TcpStream}; + use std::net::TcpListener; use std::time::{Duration, Instant}; let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); let proxy_addr = listener.local_addr().unwrap(); - let stream = TcpStream::connect(proxy_addr).expect("connect"); + let stream = + openshell_core::net::connect_std_tcp_nodelay_best_effort(proxy_addr).expect("connect"); let workload_addr = stream.local_addr().unwrap(); let connection = WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); let (_accepted, _) = listener.accept().expect("accept"); diff --git a/crates/openshell-supervisor-network/src/proxy.rs b/crates/openshell-supervisor-network/src/proxy.rs index 96c0b8f50a..eb18f63adc 100644 --- a/crates/openshell-supervisor-network/src/proxy.rs +++ b/crates/openshell-supervisor-network/src/proxy.rs @@ -253,6 +253,7 @@ fn emit_credential_endpoint_mismatch( port: u16, policy_name: &str, connect_span: &tracing::Span, + correlation: Option<&crate::l7::correlation::RequestCorrelation>, ) { let event = build_credential_endpoint_mismatch_event(method, host, port, policy_name); let finding = crate::l7::build_credential_endpoint_mismatch_finding( @@ -261,7 +262,12 @@ fn emit_credential_endpoint_mismatch( None, "Provider credential endpoint binding mismatch; request denied", ); - crate::telemetry::emit_decision([event, finding], Some(connect_span)); + if let Some(correlation) = correlation { + correlation.emit(event); + correlation.emit(finding); + } else { + crate::telemetry::emit_decision([event, finding], Some(connect_span)); + } } /// Hostnames injected by compute drivers as `/etc/hosts` aliases for the host @@ -1913,6 +1919,7 @@ struct ForwardMiddlewarePipeline<'a> { scheme: &'a str, exchange: &'a crate::l7::middleware::HttpMiddlewareExchange, l7_reevaluation: Option>, + correlation: Option<&'a crate::l7::correlation::RequestCorrelation>, } impl ForwardMiddlewarePipeline<'_> { @@ -1936,6 +1943,7 @@ impl ForwardMiddlewarePipeline<'_> { l7.engine, self.ctx, l7.request_info, + self.correlation, ); openshell_supervisor_middleware::TransformedBodyPolicy::Reevaluate(&validate) } @@ -1943,12 +1951,13 @@ impl ForwardMiddlewarePipeline<'_> { }; self.exchange - .apply_request( + .apply_request_correlated( request, client, self.ctx, self.scheme, transformed_body_policy, + self.correlation, ) .await } @@ -2074,6 +2083,24 @@ fn build_forward_allow_ocsf_event( .build() } +fn build_forward_inspection_rejection_ocsf_event( + method: &str, + reason: &'static str, +) -> openshell_ocsf::OcsfEvent { + HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .http_request(HttpRequest { + http_method: method.parse().expect("HTTP method parsing is infallible"), + url: None, + }) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .status_detail(reason) + .message("FORWARD request rejected before policy evaluation") + .build() +} + fn build_forward_rejection_event( host: &str, port: u16, @@ -2120,13 +2147,18 @@ fn emit_forward_l7_decision( decision: openshell_ocsf::OcsfEvent, evaluation_error: Option, connect_span: &tracing::Span, + correlation: Option<&crate::l7::correlation::RequestCorrelation>, ) { // The HTTP decision carries policy context that its evaluation-error // companion lacks, so it must supply the shared deny span's attributes. - crate::telemetry::emit_decision( - std::iter::once(decision).chain(evaluation_error), - Some(connect_span), - ); + let events = std::iter::once(decision).chain(evaluation_error); + if let Some(correlation) = correlation { + for event in events { + correlation.emit(event); + } + return; + } + crate::telemetry::emit_decision(events, Some(connect_span)); } fn build_forward_parse_error_ocsf_event( @@ -2437,17 +2469,23 @@ async fn deny_forward_destination( decision: &EgressDecision, denial_tx: Option<&mpsc::UnboundedSender>, activity_tx: Option<&ActivitySender>, + correlation: Option<&crate::l7::correlation::RequestCorrelation>, ) -> Result<()> where C: TokioAsyncWrite + Unpin, { let detail = destination_denial_detail(denial.kind); - crate::telemetry::emit_decision( - [build_forward_destination_deny_ocsf_event( - denial, peer_addr, method, host, port, path, binary, pid, ancestors, cmdline, policy, - )], - Some(connect_span), + let event = build_forward_destination_deny_ocsf_event( + denial, peer_addr, method, host, port, path, binary, pid, ancestors, cmdline, policy, ); + if let Some(correlation) = correlation { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); + correlation.emit(event); + } else { + crate::telemetry::emit_decision([event], Some(connect_span)); + } emit_denial_simple( denial_tx, @@ -2465,11 +2503,12 @@ where } respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "ssrf_denied", &format!("{method} {host}:{port} blocked: {detail}"), + correlation.map(crate::l7::correlation::RequestCorrelation::request_id), ), ) .await @@ -5402,6 +5441,7 @@ struct ForwardResponseMiddleware<'a> { exchange: &'a crate::l7::middleware::HttpMiddlewareExchange, } +#[cfg(test)] async fn relay_rewritten_forward_request( method: &str, path: &str, @@ -5410,6 +5450,25 @@ async fn relay_rewritten_forward_request( upstream: &mut U, options: ForwardRelayOptions<'_>, ) -> Result +where + C: TokioAsyncRead + TokioAsyncWrite + Unpin, + U: TokioAsyncRead + TokioAsyncWrite + Unpin, +{ + relay_rewritten_forward_request_correlated( + method, path, rewritten, client, upstream, options, None, + ) + .await +} + +async fn relay_rewritten_forward_request_correlated( + method: &str, + path: &str, + rewritten: Vec, + client: &mut C, + upstream: &mut U, + options: ForwardRelayOptions<'_>, + correlation: Option<&crate::l7::correlation::RequestCorrelation>, +) -> Result where C: TokioAsyncRead + TokioAsyncWrite + Unpin, U: TokioAsyncRead + TokioAsyncWrite + Unpin, @@ -5435,7 +5494,7 @@ where .response_relay(&req, middleware.ctx, middleware.scheme) }); - crate::l7::rest::relay_http_request_with_response_middleware_guarded_observed( + crate::l7::rest::relay_http_request_with_response_middleware_guarded_observed_correlated( &req, client, upstream, @@ -5456,10 +5515,12 @@ where }, response_middleware, options.endpoint_observer, + correlation, ) .await } +#[allow(clippy::too_many_arguments)] async fn inject_token_grant_for_forward_request( method: &str, upstream_target: &str, @@ -5468,14 +5529,24 @@ async fn inject_token_grant_for_forward_request( inspection: Option>, request_id: &str, observation_context: Option<&EndpointObservationContext>, + correlation: Option<&crate::l7::correlation::RequestCorrelation>, ) -> Result> { let (path, _) = crate::l7::rest::parse_target_query(upstream_target)?; - let request = crate::l7::rest::request_from_buffered_http( - method, - path, - upstream_target, - forward_request_bytes, - )?; + let request = if correlation.is_some() { + crate::l7::rest::request_from_buffered_http( + method, + path, + upstream_target, + forward_request_bytes, + )? + } else { + crate::l7::rest::request_from_buffered_http_unprotected( + method, + path, + upstream_target, + forward_request_bytes, + )? + }; if let Some(inspection) = inspection { let (request, prepared_ctx) = crate::l7::relay::prepare_inspected_request( request, @@ -5485,6 +5556,7 @@ async fn inject_token_grant_for_forward_request( inspection.request_info, request_id, observation_context, + correlation, ) .await?; *l7_ctx = prepared_ctx; @@ -5496,15 +5568,18 @@ async fn inject_token_grant_for_forward_request( let Some(snapshot) = crate::l7::token_grant_injection::pin_live_credentials(l7_ctx) else { return Ok(request.raw_header); }; - Ok(crate::l7::token_grant_injection::inject_for_selectors( - request, - l7_ctx, - &snapshot, - request_id, - observation_context, + Ok( + crate::l7::token_grant_injection::inject_for_selectors_correlated( + request, + l7_ctx, + &snapshot, + request_id, + observation_context, + correlation, + ) + .await? + .raw_header, ) - .await? - .raw_header) } /// Handle a plain HTTP forward proxy request (non-CONNECT). @@ -5871,6 +5946,64 @@ async fn handle_forward_proxy( "https" => Some(443), _ => None, }; + let middleware_input = crate::opa::NetworkInput { + host: host_lc.clone(), + port, + binary_path: decision.binary.clone().unwrap_or_default(), + binary_sha256: String::new(), + ancestors: decision.ancestors.clone(), + cmdline_paths: decision.cmdline_paths.clone(), + }; + let middleware_chain = opa_engine.query_middleware_chain_with_generation(&middleware_input); + let l7_inspected = decision + .endpoint + .l7_route + .as_ref() + .is_some_and(|route| !route.configs.is_empty()); + let inspected = l7_inspected + || middleware_chain + .as_ref() + .is_ok_and(|(chain, _)| !chain.is_empty()); + let correlation_request = crate::l7::rest::request_from_buffered_http_unprotected( + method, + path.clone(), + &upstream_target, + forward_request_bytes, + )?; + let correlation = if inspected { + crate::l7::correlation::RequestCorrelation::new(&correlation_request, &l7_ctx) + } else { + crate::l7::correlation::RequestCorrelation::uninspected() + }; + let request_correlation = inspected.then_some(&correlation); + let receipt_id = + request_correlation.map(crate::l7::correlation::RequestCorrelation::request_id); + let mut audited_client = correlation.observe_response(client); + let client = &mut audited_client; + correlation.record_enforcement_action("allow"); + let mut primary = build_forward_allow_ocsf_event( + workload_addr, + method, + &host_lc, + port, + &telemetry_path, + &binary_str, + &pid_str, + &ancestors_str, + &cmdline_str, + policy_str, + ); + if l7_inspected && let openshell_ocsf::OcsfEvent::HttpActivity(event) = &mut primary { + event.action = None; + event.disposition = None; + event.base.status = None; + event.firewall_rule = None; + event.base.message = Some("FORWARD request awaiting inspection".into()); + } + if inspected { + correlation.emit(primary); + } + forward_request_bytes = correlation_request.raw_header; if let Some(route) = decision .endpoint .l7_route @@ -5897,13 +6030,17 @@ async fn handle_forward_proxy( ), ); emit_activity_simple(activity_tx, true, "policy_stale"); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!("{method} {host_lc}:{port}{telemetry_path} not permitted by policy"), + receipt_id, ), ) .await?; @@ -5922,15 +6059,19 @@ async fn handle_forward_proxy( ); emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); emit_activity_simple(activity_tx, true, "policy_stale"); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!( "{method} {host_lc}:{port}{telemetry_path} not permitted by policy" ), + receipt_id, ), ) .await?; @@ -5939,29 +6080,62 @@ async fn handle_forward_proxy( }; let Ok(redacted_path) = secrets::redact_target_for_policy(&path) else { + correlation.record_final_http_outcome("platform_error"); + correlation.record_response_status(400); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 400, "Bad Request", "invalid_credential_placeholder", "request-target contains an invalid credential placeholder", + receipt_id, ), ) .await?; return Ok(()); }; let Some(l7_config) = select_l7_config_for_path(&route.configs, &redacted_path) else { + let reason = "no L7 endpoint path matched request"; + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); + correlation.emit( + HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(ActivityId::for_http_method(method)) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .http_request(HttpRequest::new( + method, + OcsfUrl::new("http", &host_lc, &telemetry_path, port), + )) + .http_response(HttpResponse { code: 403 }) + .dst_endpoint(Endpoint::from_domain(&host_lc, port)) + .src_endpoint(Endpoint::from_ip(workload_addr.ip(), workload_addr.port())) + .actor_process( + Process::from_bypass(&binary_str, &pid_str, &ancestors_str) + .with_cmd_line(&cmdline_str), + ) + .firewall_rule(policy_str, "l7") + .message(format!( + "FORWARD_L7 denied {method} {host_lc}:{port}{telemetry_path}" + )) + .status_detail(reason) + .build(), + ); emit_activity_simple(activity_tx, true, "l7_policy"); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!( "{method} {host_lc}:{port}{telemetry_path} did not match an L7 endpoint path" ), + receipt_id, ), ) .await?; @@ -5984,30 +6158,35 @@ async fn handle_forward_proxy( if !l7_config.config.allow_encoded_slash && crate::l7::path::canonical_path_has_encoded_slash(&path) { - crate::telemetry::emit_decision( - [build_forward_l7_parse_rejection_ocsf_event( - workload_addr, - method, - &host_lc, - port, - &telemetry_path, - &binary_str, - &pid_str, - &ancestors_str, - &cmdline_str, - policy_str, - FORWARD_ENCODED_SLASH_REJECTION_DETAIL, - )], - Some(&connect_span), - ); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(400); + correlation.emit(build_forward_inspection_rejection_ocsf_event( + method, + "invalid_request_target", + )); + correlation.emit(build_forward_l7_parse_rejection_ocsf_event( + workload_addr, + method, + &host_lc, + port, + &telemetry_path, + &binary_str, + &pid_str, + &ancestors_str, + &cmdline_str, + policy_str, + FORWARD_ENCODED_SLASH_REJECTION_DETAIL, + )); emit_activity_simple(activity_tx, true, "forward_parse_rejection"); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 400, "Bad Request", "invalid_request_target", "request-target must be canonical", + receipt_id, ), ) .await?; @@ -6019,6 +6198,9 @@ async fn handle_forward_proxy( &forward_request_bytes, l7_config.config.protocol, ) { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); if let Some(observer) = endpoint_observer.as_ref() { observer.observe(EndpointResult::PolicyDenied); } @@ -6044,7 +6226,7 @@ async fn handle_forward_proxy( )) .status_detail(upgrade_detail) .build(); - crate::telemetry::emit_decision([event], Some(&connect_span)); + correlation.emit(event); emit_activity_simple(activity_tx, true, "l7_parse_rejection"); emit_denial_simple( denial_tx, @@ -6057,11 +6239,12 @@ async fn handle_forward_proxy( ); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "unsupported_l7_protocol", upgrade_detail, + receipt_id, ), ) .await?; @@ -6102,20 +6285,28 @@ async fn handle_forward_proxy( { Ok(info) => info, Err(e) => { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(400); + correlation.emit(build_forward_inspection_rejection_ocsf_event( + method, + "invalid_graphql_request", + )); let event = build_forward_rejection_event( &host_lc, port, format!("FORWARD_GRAPHQL_L7 request rejected: {e}"), ); - egress_emit!(event); + correlation.emit(event); emit_activity_simple(activity_tx, true, "l7_parse_rejection"); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 400, "Bad Request", "invalid_graphql_request", &format!("GraphQL request rejected before policy evaluation: {e}"), + receipt_id, ), ) .await?; @@ -6154,20 +6345,28 @@ async fn handle_forward_proxy( { Ok(body) => body, Err(e) => { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(400); + correlation.emit(build_forward_inspection_rejection_ocsf_event( + method, + "invalid_jsonrpc_request", + )); let event = build_forward_rejection_event( &host_lc, port, format!("FORWARD_JSONRPC_L7 request rejected: {e}"), ); - egress_emit!(event); + correlation.emit(event); emit_activity_simple(activity_tx, true, "l7_parse_rejection"); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 400, "Bad Request", "invalid_jsonrpc_request", &format!("JSON-RPC request rejected before policy evaluation: {e}"), + receipt_id, ), ) .await?; @@ -6181,7 +6380,7 @@ async fn handle_forward_proxy( }; // Policy evaluation must use the selected revision's inspection, // including method classification, just as the CONNECT relays do. - let Some(info) = crate::l7::relay::enforce_mcp_protocol_version( + let Some(info) = crate::l7::relay::enforce_mcp_protocol_version_correlated( &l7_config.config, &jsonrpc_request, info, @@ -6189,6 +6388,7 @@ async fn handle_forward_proxy( &l7_ctx, &telemetry_path, endpoint_observer.as_ref(), + request_correlation, ) .await? else { @@ -6233,6 +6433,13 @@ async fn handle_forward_proxy( (false, crate::l7::EnforcementMode::Audit) => "audit", (false, crate::l7::EnforcementMode::Enforce) => "deny", }; + correlation.record_enforcement_action(if allowed { + "allow" + } else if l7_config.config.enforcement == crate::l7::EnforcementMode::Audit && !force_deny { + "audit" + } else { + "block" + }); { let (action_id, disposition_id, severity) = match decision_str { @@ -6296,13 +6503,15 @@ async fn handle_forward_proxy( .firewall_rule(policy_str, engine_type) .message(log_message) .build(); - emit_forward_l7_decision(event, evaluation_error, &connect_span); + emit_forward_l7_decision(event, evaluation_error, &connect_span, request_correlation); } let effectively_denied = force_deny || (!allowed && l7_config.config.enforcement == crate::l7::EnforcementMode::Enforce); if effectively_denied { + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); emit_activity_simple(activity_tx, true, "l7_policy"); emit_denial_simple( denial_tx, @@ -6315,13 +6524,14 @@ async fn handle_forward_proxy( ); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!( "{method} {host_lc}:{port}{telemetry_path} denied by L7 policy: {reason}" ), + receipt_id, ), ) .await?; @@ -6362,6 +6572,7 @@ async fn handle_forward_proxy( &decision, denial_tx, activity_tx, + request_correlation, ) .await?; return Ok(()); @@ -6401,6 +6612,7 @@ async fn handle_forward_proxy( &decision, denial_tx, activity_tx, + request_correlation, ) .await?; return Ok(()); @@ -6418,13 +6630,17 @@ async fn handle_forward_proxy( ); emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); emit_activity_simple(activity_tx, true, "policy_stale"); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!("{method} {host_lc}:{port}{telemetry_path} not permitted by policy"), + receipt_id, ), ) .await?; @@ -6432,16 +6648,7 @@ async fn handle_forward_proxy( } let middleware_path = path.split_once('?').map_or(path.as_str(), |(path, _)| path); - let middleware_input = crate::opa::NetworkInput { - host: host_lc.clone(), - port, - binary_path: decision.binary.clone().unwrap_or_default(), - binary_sha256: String::new(), - ancestors: decision.ancestors.clone(), - cmdline_paths: decision.cmdline_paths.clone(), - }; - let (chain, generation) = - opa_engine.query_middleware_chain_with_generation(&middleware_input)?; + let (chain, generation) = middleware_chain?; if generation != forward_generation_guard.captured_generation() { emit_l7_tunnel_close_after_policy_change( &host_lc, @@ -6452,19 +6659,23 @@ async fn handle_forward_proxy( generation, ), ); + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!("{method} {host_lc}:{port}{telemetry_path} not permitted by policy"), + receipt_id, ), ) .await?; return Ok(()); } - let request_id = uuid::Uuid::new_v4().to_string(); + let request_id = correlation.request_id().to_string(); let websocket_chain = forward_websocket_request.then(|| chain.clone()); let response_selection = if chain.is_empty() { None @@ -6495,6 +6706,7 @@ async fn handle_forward_proxy( scheme: &scheme, exchange: &middleware_exchange, l7_reevaluation, + correlation: request_correlation, }; // Middleware denials join the connect trace, like other FORWARD denials. forward_request_bytes = match pipeline @@ -6505,16 +6717,29 @@ async fn handle_forward_proxy( crate::l7::middleware::MiddlewareApplyResult::Allowed(request) => request.raw_header, crate::l7::middleware::MiddlewareApplyResult::Denied { denial, .. } => { emit_activity_simple(activity_tx, true, "middleware"); + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if denial.is_some() { + "denied" + } else { + "platform_error" + }); let response = denial.as_ref().map_or_else( - || build_middleware_failure_response(&l7_ctx.policy_name), - |denial| build_middleware_deny_response(&l7_ctx.policy_name, denial), + || build_middleware_failure_response(&l7_ctx.policy_name, receipt_id), + |denial| { + build_middleware_deny_response(&l7_ctx.policy_name, denial, receipt_id) + }, ); respond(client, &response).await?; return Ok(()); } crate::l7::middleware::MiddlewareApplyResult::AdmissionExhausted => { emit_activity_simple(activity_tx, true, "middleware"); - let response = build_middleware_unavailable_response(&l7_ctx.policy_name); + correlation.record_enforcement_action("block"); + correlation.record_response_status(503); + correlation.record_final_http_outcome("platform_error"); + let response = + build_middleware_unavailable_response(&l7_ctx.policy_name, receipt_id); respond(client, &response).await?; return Ok(()); } @@ -6541,13 +6766,17 @@ async fn handle_forward_proxy( Ok(preflight) => preflight, Err(error) => { warn!(error = %error, "Plaintext WebSocket middleware preflight failed"); + correlation.record_enforcement_action("block"); + correlation.record_response_status(502); + correlation.record_final_http_outcome("platform_error"); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 502, "Bad Gateway", "middleware_failed", "WebSocket middleware preflight failed", + receipt_id, ), ) .await?; @@ -6556,9 +6785,16 @@ async fn handle_forward_proxy( }; crate::l7::middleware::emit_websocket_preflight_events(&l7_ctx, &preflight); if preflight.terminal_reason.is_some() { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome(if preflight.denial.is_some() { + "denied" + } else { + "platform_error" + }); let response = preflight.denial.as_ref().map_or_else( - || build_middleware_failure_response(&l7_ctx.policy_name), - |denial| build_middleware_deny_response(&l7_ctx.policy_name, denial), + || build_middleware_failure_response(&l7_ctx.policy_name, receipt_id), + |denial| build_middleware_deny_response(&l7_ctx.policy_name, denial, receipt_id), ); respond(client, &response).await?; return Ok(()); @@ -6585,6 +6821,7 @@ async fn handle_forward_proxy( }), &request_id, endpoint_observation_context.as_ref(), + request_correlation, ) .await { @@ -6592,6 +6829,9 @@ async fn handle_forward_proxy( Err(e) => { let refusal = e.downcast_ref::(); + correlation.record_enforcement_action("block"); + correlation.record_response_status(502); + correlation.record_final_http_outcome("platform_error"); if refusal.is_none() { warn!( dst_host = %host_lc, @@ -6607,15 +6847,15 @@ async fn handle_forward_proxy( } let response = refusal.map_or_else( || { - build_json_error_response( + build_json_error_response_with_request_id( 502, "Bad Gateway", "token_grant_failed", "dynamic token grant failed", + receipt_id, ) }, |refusal| { - // The refusal event already names the credential and request ID. if let Some(observer) = endpoint_observer.as_ref() { observer.observe_token_grant_refusal(refusal, &request_id); } @@ -6657,7 +6897,10 @@ async fn handle_forward_proxy( _ => None, }; if let Some(guard) = credential_generation { - guard.ensure_current()?; + guard.ensure_current().inspect_err(|_| { + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("platform_error"); + })?; } // 9. Rewrite request and forward to upstream @@ -6682,11 +6925,20 @@ async fn handle_forward_proxy( .await; } if e.is_placement_mismatch() { - crate::l7::relay::reject_credential_resolution(client, &l7_ctx, method, &e) - .instrument(connect_span.clone()) - .await?; + crate::l7::relay::reject_credential_resolution( + client, + &l7_ctx, + method, + &e, + request_correlation, + ) + .instrument(connect_span.clone()) + .await?; client.shutdown().await.into_diagnostic()?; } else if e.is_endpoint_mismatch() { + correlation.record_enforcement_action("block"); + correlation.record_response_status(403); + correlation.record_final_http_outcome("denied"); if let Some(observer) = endpoint_observer.as_ref() { observer.observe_credential_failure(true); } @@ -6696,14 +6948,16 @@ async fn handle_forward_proxy( port, policy_str, &connect_span, + request_correlation, ); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "credential_endpoint_mismatch", "credential is not authorized for this request endpoint", + receipt_id, ), ) .await?; @@ -6719,13 +6973,17 @@ async fn handle_forward_proxy( }) .await; } else { + correlation.record_enforcement_action("block"); + correlation.record_response_status(500); + correlation.record_final_http_outcome("platform_error"); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 500, "Internal Server Error", "credential_injection_failed", "unresolved credential placeholder in request", + receipt_id, ), ) .await?; @@ -6745,13 +7003,14 @@ async fn handle_forward_proxy( &upstream_target, rewritten, )?; - if !crate::l7::relay::enforce_final_mcp_protocol_version( + if !crate::l7::relay::enforce_final_mcp_protocol_version_correlated( config, &request, client, &l7_ctx, &telemetry_path, endpoint_observer.as_ref(), + request_correlation, ) .await? { @@ -6782,13 +7041,17 @@ async fn handle_forward_proxy( .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; } + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!("{method} {host_lc}:{port}{telemetry_path} not permitted by policy"), + receipt_id, ), ) .await?; @@ -6823,19 +7086,22 @@ async fn handle_forward_proxy( "FORWARD upstream connect failed for {host_lc}:{port}: {e}" )) .build(); - egress_emit!(event); + correlation.emit(event); if let Some(session) = middleware_session.take() { session .end(openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure) .await; } + correlation.record_response_status(502); + correlation.record_final_http_outcome("upstream_error"); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 502, "Bad Gateway", "upstream_unreachable", &format!("connection to {host_lc}:{port} failed"), + receipt_id, ), ) .await?; @@ -6860,13 +7126,17 @@ async fn handle_forward_proxy( .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; } + correlation.record_enforcement_action("block"); + correlation.record_final_http_outcome("denied"); + correlation.record_response_status(403); respond( client, - &build_json_error_response( + &build_json_error_response_with_request_id( 403, "Forbidden", "policy_denied", &format!("{method} {host_lc}:{port}{path} not permitted by policy"), + receipt_id, ), ) .await?; @@ -6884,7 +7154,7 @@ async fn handle_forward_proxy( let signing_region = forward_upgrade_config .as_ref() .map_or("", |config| config.signing_region.as_str()); - let outcome_result = relay_rewritten_forward_request( + let outcome_result = relay_rewritten_forward_request_correlated( method, &upstream_target, rewritten, @@ -6912,6 +7182,7 @@ async fn handle_forward_proxy( }), endpoint_observer: endpoint_observer.as_ref(), }, + request_correlation, ) .await; let outcome_result = match outcome_result { @@ -6923,7 +7194,13 @@ async fn handle_forward_proxy( .await; } let _ = upstream.shutdown().await; - crate::l7::relay::reject_body_credential(client, *error).await?; + crate::l7::relay::reject_body_credential( + client, + *error, + &l7_ctx, + request_correlation, + ) + .await?; return Ok(()); } if let Some(error) = report.downcast_ref::() { @@ -6932,8 +7209,14 @@ async fn handle_forward_proxy( .end(openshell_core::proto::MiddlewareSessionEndReason::Cancellation) .await; } - crate::l7::relay::reject_credential_resolution(client, &l7_ctx, method, error) - .await?; + crate::l7::relay::reject_credential_resolution( + client, + &l7_ctx, + method, + error, + request_correlation, + ) + .await?; return Ok(()); } Err(report) @@ -6950,21 +7233,20 @@ async fn handle_forward_proxy( ) .await?; - // The request has now survived middleware, token grant, credential - // rewriting, generation checks, and the HTTP relay. Only now record the - // final allowed outcome. - egress_emit!(build_forward_allow_ocsf_event( - workload_addr, - method, - &host_lc, - port, - &telemetry_path, - &binary_str, - &pid_str, - &ancestors_str, - &cmdline_str, - policy_str, - )); + if !inspected { + correlation.emit(build_forward_allow_ocsf_event( + workload_addr, + method, + &host_lc, + port, + &telemetry_path, + &binary_str, + &pid_str, + &ancestors_str, + &cmdline_str, + policy_str, + )); + } emit_forward_success_activity(activity_tx, l7_activity_pending); match outcome { @@ -7103,6 +7385,47 @@ fn build_json_error_response(status: u16, status_text: &str, error: &str, detail .into_bytes() } +fn build_json_error_response_with_request_id( + status: u16, + status_text: &str, + error: &str, + detail: &str, + request_id: Option<&str>, +) -> Vec { + request_id.map_or_else( + || build_json_error_response(status, status_text, error, detail), + |id| build_correlated_json_error_response(status, status_text, error, detail, id), + ) +} + +fn build_correlated_json_error_response( + status: u16, + status_text: &str, + error: &str, + detail: &str, + request_id: &str, +) -> Vec { + let body = serde_json::json!({ + "error": error, + "detail": detail, + "request_id": request_id, + }) + .to_string(); + format!( + "HTTP/1.1 {status} {status_text}\r\n\ + Content-Type: application/json\r\n\ + {}: {request_id}\r\n\ + Content-Length: {}\r\n\ + Connection: close\r\n\ + \r\n\ + {}", + openshell_core::token_grant_failure::REQUEST_ID_HEADER, + body.len(), + body, + ) + .into_bytes() +} + /// Build the forward proxy's refusal for a failed dynamic token grant. /// /// This is the generic grant refusal plus a request ID header, which lets the @@ -7157,6 +7480,7 @@ fn build_json_error_response_with_reason( fn build_middleware_deny_response( policy_name: &str, denial: &openshell_supervisor_middleware::MiddlewareDenial, + request_id: Option<&str>, ) -> Vec { let mut body = serde_json::Map::new(); body.insert("error".to_string(), serde_json::json!("middleware_denied")); @@ -7172,11 +7496,21 @@ fn build_middleware_deny_response( if let Some(reason_code) = &denial.reason_code { body.insert("reason_code".to_string(), serde_json::json!(reason_code)); } + if let Some(request_id) = request_id { + body.insert("request_id".to_string(), serde_json::json!(request_id)); + } let body_str = serde_json::Value::Object(body).to_string(); + let request_id_header = request_id.map_or_else(String::new, |request_id| { + format!( + "{}: {request_id}\r\n", + openshell_core::token_grant_failure::REQUEST_ID_HEADER + ) + }); format!( "HTTP/1.1 403 Forbidden\r\n\ Content-Type: application/json\r\n\ Content-Length: {}\r\n\ + {request_id_header}\ Connection: close\r\n\ \r\n\ {}", @@ -7186,25 +7520,39 @@ fn build_middleware_deny_response( .into_bytes() } -fn build_middleware_failure_response(policy_name: &str) -> Vec { - build_middleware_platform_response(policy_name, "403 Forbidden") +fn build_middleware_failure_response(policy_name: &str, request_id: Option<&str>) -> Vec { + build_middleware_platform_response(policy_name, "403 Forbidden", request_id) } -fn build_middleware_unavailable_response(policy_name: &str) -> Vec { - build_middleware_platform_response(policy_name, "503 Service Unavailable") +fn build_middleware_unavailable_response(policy_name: &str, request_id: Option<&str>) -> Vec { + build_middleware_platform_response(policy_name, "503 Service Unavailable", request_id) } -fn build_middleware_platform_response(policy_name: &str, status: &str) -> Vec { - let body = serde_json::json!({ +fn build_middleware_platform_response( + policy_name: &str, + status: &str, + request_id: Option<&str>, +) -> Vec { + let mut body = serde_json::json!({ "error": "middleware_failed", "detail": "Request could not be processed by configured middleware", "policy": policy_name, }); + if let (Some(request_id), serde_json::Value::Object(fields)) = (request_id, &mut body) { + fields.insert("request_id".to_string(), serde_json::json!(request_id)); + } let body_str = body.to_string(); + let request_id_header = request_id.map_or_else(String::new, |request_id| { + format!( + "{}: {request_id}\r\n", + openshell_core::token_grant_failure::REQUEST_ID_HEADER + ) + }); format!( "HTTP/1.1 {status}\r\n\ Content-Type: application/json\r\n\ Content-Length: {}\r\n\ + {request_id_header}\ Connection: close\r\n\ \r\n\ {}", @@ -8625,7 +8973,7 @@ network_policies: .await .unwrap(); workload.shutdown().await.unwrap(); - let handler = tokio::spawn(handle_mediated_connection( + let mut handler = tokio::spawn(handle_mediated_connection( tokio::io::BufReader::new(stream), supplied_identity, socket_addrs, @@ -8647,10 +8995,11 @@ network_policies: None, None, )); - let request = tokio::time::timeout(std::time::Duration::from_secs(5), request_rx) - .await - .expect("upstream never received the workload's request") - .unwrap(); + let request = tokio::select! { + result = tokio::time::timeout(std::time::Duration::from_secs(5), request_rx) => + result.expect("upstream never received the workload's request").unwrap(), + result = &mut handler => panic!("proxy ended before forwarding the workload request: {result:?}"), + }; assert!( request.starts_with(b"GET /ping HTTP/1.1\r\n"), "upstream received {:?} instead of the workload's request", @@ -9539,7 +9888,7 @@ network_policies: {} let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let address = listener.local_addr().unwrap(); let client = tokio::spawn(async move { - let mut socket = TcpStream::connect(address).await.unwrap(); + let mut socket = connect_tcp_nodelay_best_effort(address).await.unwrap(); socket.write_all(&raw).await.unwrap(); let mut response = Vec::new(); socket.read_to_end(&mut response).await.unwrap(); @@ -9630,6 +9979,122 @@ network_policies: {} } } + #[tokio::test] + async fn plaintext_forward_allow_retains_path_and_process_in_primary_record() { + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + let Some(upstream_ip) = non_loopback_test_ipv4() else { + eprintln!("skipping: no routable non-loopback IPv4 test address"); + return; + }; + let listener = TcpListener::bind((upstream_ip, 0)).await.unwrap(); + let upstream_port = listener.local_addr().unwrap().port(); + let engine = Arc::new( + native_identity_engine( + include_str!("../data/sandbox-policy.rego"), + &format!( + r#" +network_policies: + plain_api: + name: plain_api + endpoints: + - host: "{upstream_ip}" + port: {upstream_port} + allowed_ips: ["{upstream_ip}/32"] + binaries: + - path: /usr/bin/python3 +"# + ), + ) + .unwrap(), + ); + let binary = executable_path("/usr/bin/python3"); + let command = executable_path("/workspace/investigate.py"); + let identity = Ok(ContractBinaryIdentity { + executable: ContractExecutableIdentity { + path: binary.clone(), + digest: Some("00".repeat(32).parse().unwrap()), + }, + ancestors: Vec::new(), + cmdline_paths: vec![command.clone()], + }); + let capture = openshell_ocsf::test_support::OcsfCapture::default(); + let subscriber = tracing_subscriber::registry().with(capture.clone()); + let (mut app, proxy) = tokio::io::duplex(8192); + let mut proxy: ProxyClient = tokio::io::BufReader::new(Box::new(proxy)); + let target = format!("http://{upstream_ip}:{upstream_port}/v1/incidents?token=not-logged"); + let request = format!( + "GET {target} HTTP/1.1\r\nHost: {upstream_ip}:{upstream_port}\r\nx-openshell-request-id: app-owned-in\r\nConnection: close\r\n\r\n" + ); + let run = async { + tokio::join!( + async { + let result = Box::pin(handle_forward_proxy( + "GET", &target, request.as_bytes(), request.len(), &mut proxy, + Some(&identity), + Some(("192.0.2.20:42123".parse().unwrap(), "192.0.2.10:3128".parse().unwrap())), + engine, Arc::new(BinaryIdentityCache::new()), Arc::new(AtomicU32::new(0)), + None, AgentProposals::default(), Arc::new(None), Arc::new(None), + None, None, None, None, None, + )).await; + drop(proxy); + result + }, + async { + let (mut upstream, _) = listener.accept().await.unwrap(); + let mut request = Vec::new(); + while !request.ends_with(b"\r\n\r\n") { + request.push(upstream.read_u8().await.unwrap()); + } + assert!(request.starts_with(b"GET /v1/incidents?token=not-logged HTTP/1.1\r\n")); + assert!(String::from_utf8_lossy(&request).contains("x-openshell-request-id: app-owned-in\r\n")); + upstream.write_all(b"HTTP/1.1 200 OK\r\nx-openshell-request-id: app-owned-out\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok").await.unwrap(); + }, + async { + let mut response = Vec::new(); + app.read_to_end(&mut response).await.unwrap(); + response + }, + ) + }.with_subscriber(subscriber); + let (result, (), response) = + tokio::time::timeout(std::time::Duration::from_secs(5), Box::pin(run)) + .await + .unwrap(); + result.unwrap(); + assert!(response.starts_with(b"HTTP/1.1 200 OK\r\n")); + assert!( + String::from_utf8_lossy(&response) + .contains("x-openshell-request-id: app-owned-out\r\n") + ); + let records: Vec<_> = capture + .json_events() + .into_iter() + .filter(|event| event["class_uid"] == 4002) + .collect(); + assert_eq!(records.len(), 1, "one primary FORWARD record: {records:?}"); + let record = &records[0]; + assert_eq!(record["action"], "Allowed"); + assert_eq!(record["status"], "Success"); + assert_eq!(record["http_request"]["url"]["path"], "/v1/incidents"); + assert_eq!( + record["actor"]["process"]["name"], + binary.to_string_lossy().as_ref() + ); + assert_eq!( + record["actor"]["process"]["cmd_line"], + command.to_string_lossy().as_ref() + ); + assert!( + record["message"] + .as_str() + .unwrap() + .starts_with("FORWARD allowed") + ); + assert!(!record.to_string().contains("not-logged")); + } + #[tokio::test] async fn plaintext_mcp_forwarding_preserves_initialization_and_selected_revision() { if !cfg!(target_os = "linux") { @@ -9767,7 +10232,9 @@ network_policies: body.len(), ); let client = tokio::spawn(async move { - let mut socket = TcpStream::connect(proxy_address).await.unwrap(); + let mut socket = connect_tcp_nodelay_best_effort(proxy_address) + .await + .unwrap(); let mut response = Vec::new(); socket.read_to_end(&mut response).await.unwrap(); response @@ -9893,7 +10360,7 @@ network_policies: "GET {target} HTTP/1.1\r\nHost: {upstream_ip}:{upstream_port}\r\nAccept: text/event-stream\r\nMCP-Protocol-Version: 2025-11-25\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n\r\n" ); let client = tokio::spawn(async move { - let mut socket = TcpStream::connect(proxy_address) + let mut socket = connect_tcp_nodelay_best_effort(proxy_address) .await .expect("connect proxy"); let mut response = Vec::new(); @@ -10011,7 +10478,7 @@ network_policies: "GET {target} HTTP/1.1\r\nHost: {upstream_ip}:{upstream_port}\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n\r\n" ); let client = tokio::spawn(async move { - let mut socket = TcpStream::connect(proxy_address) + let mut socket = connect_tcp_nodelay_best_effort(proxy_address) .await .expect("connect proxy"); let mut response = Vec::new(); @@ -10138,7 +10605,7 @@ network_policies: "GET {target} HTTP/1.1\r\nHost: {upstream_ip}:{upstream_port}\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n\r\n" ); let client = tokio::spawn(async move { - let mut socket = TcpStream::connect(proxy_address) + let mut socket = connect_tcp_nodelay_best_effort(proxy_address) .await .expect("connect proxy"); let mut response = Vec::new(); @@ -10277,7 +10744,7 @@ network_policies: "GET {target} HTTP/1.1\r\nHost: {upstream_ip}:{upstream_port}\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Extensions: permessage-deflate; client_no_context_takeover\r\n\r\n" ); let client = tokio::spawn(async move { - let mut socket = TcpStream::connect(proxy_address) + let mut socket = connect_tcp_nodelay_best_effort(proxy_address) .await .expect("connect proxy"); let response = read_http_headers_unbounded(&mut socket).await; @@ -10510,7 +10977,7 @@ network_policies: #[test] fn middleware_failure_response_uses_platform_text_without_policy_guidance() { - let response = build_middleware_failure_response("api-policy"); + let response = build_middleware_failure_response("api-policy", None); let response = String::from_utf8(response).expect("UTF-8 error response"); let (_, body) = response.split_once("\r\n\r\n").expect("HTTP response"); let body: serde_json::Value = serde_json::from_str(body).expect("JSON response"); @@ -10528,7 +10995,7 @@ network_policies: #[test] fn middleware_unavailable_response_is_complete_and_has_no_retry_hint() { - let response = build_middleware_unavailable_response("api-policy"); + let response = build_middleware_unavailable_response("api-policy", None); let response = String::from_utf8(response).expect("UTF-8 error response"); assert!(response.starts_with("HTTP/1.1 503 Service Unavailable\r\n")); assert!(!response.to_ascii_lowercase().contains("retry-after")); @@ -10973,7 +11440,7 @@ network_policies: let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); - let mut client = TcpStream::connect(addr).await.unwrap(); + let mut client = connect_tcp_nodelay_best_effort(addr).await.unwrap(); let (mut server, _) = listener.accept().await.unwrap(); client @@ -11285,6 +11752,7 @@ network_policies: engine: &tunnel_engine, request_info: &request_info, }), + correlation: None, }; let (_app, mut client) = tokio::io::duplex(8192); @@ -11384,6 +11852,7 @@ network_policies: scheme: "http", exchange: &exchange, l7_reevaluation: None, + correlation: None, }; let (_app, mut client) = tokio::io::duplex(8192); let revoke = async { @@ -11821,6 +12290,7 @@ network_policies: None, FORWARD_REQUEST_ID, None, + None, ) .await } @@ -14289,6 +14759,7 @@ network_policies: scheme: "http", exchange: &exchange, l7_reevaluation: None, + correlation: None, }; let (_app, mut client) = tokio::io::duplex(8192); @@ -15107,6 +15578,7 @@ network_policies: config_name: "prototype-content-guard".into(), reason_code: Some("content_match".into()), }, + None, ); let resp_str = String::from_utf8(resp).unwrap(); let body_start = resp_str.find("\r\n\r\n").unwrap() + 4; @@ -15156,7 +15628,7 @@ network_policies: async fn connect_without_tls_termination_refused_with_503_before_200() { let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); - let mut client = TcpStream::connect(addr).await.unwrap(); + let mut client = connect_tcp_nodelay_best_effort(addr).await.unwrap(); let (mut server, _) = listener.accept().await.unwrap(); // Terminating route (tls: skip = false) with no termination state. @@ -15196,7 +15668,7 @@ network_policies: for (tls_present, tls_skip) in [(true, false), (false, true), (true, true)] { let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); - let mut client = TcpStream::connect(addr).await.unwrap(); + let mut client = connect_tcp_nodelay_best_effort(addr).await.unwrap(); let (mut server, _) = listener.accept().await.unwrap(); let refused = refuse_connect_when_tls_unavailable(&mut server, tls_present, tls_skip) @@ -15266,7 +15738,9 @@ network_policies: // dropped at `budget`, which closes the socket and ends the read. let target = connect_target.to_string(); let client = tokio::spawn(async move { - let mut sock = TcpStream::connect(("127.0.0.1", proxy_port)).await.unwrap(); + let mut sock = connect_tcp_nodelay_best_effort(("127.0.0.1", proxy_port)) + .await + .unwrap(); let req = format!("CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n\r\n"); sock.write_all(req.as_bytes()).await.unwrap(); let mut buf = Vec::new(); @@ -15340,7 +15814,9 @@ network_policies: let proxy_port = listener.local_addr().unwrap().port(); let target = target.to_string(); let client = tokio::spawn(async move { - let mut socket = TcpStream::connect(("127.0.0.1", proxy_port)).await.unwrap(); + let mut socket = connect_tcp_nodelay_best_effort(("127.0.0.1", proxy_port)) + .await + .unwrap(); let request = format!("GET {target} HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"); socket.write_all(request.as_bytes()).await.unwrap(); let mut response = Vec::new(); @@ -15757,7 +16233,7 @@ network_policies: // the same crate so it inherits the user_home_t label. fn resolve_process_identity_denies_exec_shared_socket_ambiguity() { use crate::identity::BinaryIdentityCache; - use std::net::{TcpListener, TcpStream}; + use std::net::TcpListener; use std::time::{Duration, Instant}; if !std::path::Path::new("/bin/sleep").exists() { @@ -15777,7 +16253,8 @@ network_policies: let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); let proxy_addr = listener.local_addr().unwrap(); - let stream = TcpStream::connect(proxy_addr).expect("connect"); + let stream = + openshell_core::net::connect_std_tcp_nodelay_best_effort(proxy_addr).expect("connect"); let workload_addr = stream.local_addr().unwrap(); let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); let (_accepted, _) = listener.accept().expect("accept"); diff --git a/crates/openshell-supervisor-network/src/proxy/relay.rs b/crates/openshell-supervisor-network/src/proxy/relay.rs index 495c22241b..f49f0367ca 100644 --- a/crates/openshell-supervisor-network/src/proxy/relay.rs +++ b/crates/openshell-supervisor-network/src/proxy/relay.rs @@ -404,6 +404,65 @@ mod tests { } } + async fn read_http_response(reader: &mut R) -> Vec + where + R: AsyncRead + Unpin, + { + use tokio::io::AsyncReadExt; + + let mut response = Vec::new(); + while !response.ends_with(b"\r\n\r\n") { + response.push(reader.read_u8().await.unwrap()); + assert!(response.len() < 4096); + } + let headers = String::from_utf8_lossy(&response); + let content_length = headers + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("content-length") + .then(|| value.trim().parse::().unwrap()) + }) + .unwrap_or_default(); + for _ in 0..content_length { + response.push(reader.read_u8().await.unwrap()); + } + response + } + + fn assert_passthrough_response( + response: &[u8], + expected_body: &[u8], + expect_connection_close: bool, + ) { + let header_end = response + .windows(4) + .position(|bytes| bytes == b"\r\n\r\n") + .expect("response headers") + + 4; + let headers = String::from_utf8_lossy(&response[..header_end]); + assert!(headers.starts_with("HTTP/1.1 200 OK\r\n"), "{headers}"); + assert!( + headers.contains(&format!("Content-Length: {}\r\n", expected_body.len())), + "{headers}" + ); + assert_eq!( + headers + .lines() + .any(|line| line.eq_ignore_ascii_case("Connection: close")), + expect_connection_close, + "{headers}" + ); + assert!( + !headers + .lines() + .any(|line| line.split_once(':').is_some_and(|(name, _)| name + .eq_ignore_ascii_case(openshell_core::token_grant_failure::REQUEST_ID_HEADER))), + "uninspected passthrough must not claim an inspected request receipt" + ); + assert_eq!(&response[header_end..], expected_body); + } + async fn assert_response_lifecycle(mut caller: C, mut client: P) where C: AsyncRead + AsyncWrite + Unpin, @@ -443,16 +502,15 @@ mod tests { let request = b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"; caller.write_all(request).await.unwrap(); caller.flush().await.unwrap(); - let mut first = vec![0; persistent.len()]; - caller.read_exact(&mut first).await.unwrap(); - assert_eq!(first, persistent); + let first = read_http_response(&mut caller).await; + assert_passthrough_response(&first, b"hello", false); // A real second exchange verifies reuse, without a timing assertion. caller.write_all(request).await.unwrap(); caller.flush().await.unwrap(); let mut second = Vec::new(); // TLS must return clean EOF, not UnexpectedEof from a dropped socket. caller.read_to_end(&mut second).await.unwrap(); - assert_eq!(second, closing); + assert_passthrough_response(&second, &body, true); }; tokio::time::timeout(std::time::Duration::from_secs(5), async { let (result, (), ()) = tokio::join!(relay, serve, receive); diff --git a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs index 3623b1dbb7..455f177a5f 100644 --- a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs +++ b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs @@ -25,7 +25,7 @@ fn allowed_decision(intent: EgressIntent) -> EgressDecision { async fn tcp_pair() -> (TcpStream, TcpStream) { let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let client = TcpStream::connect(listener.local_addr().unwrap()) + let client = connect_tcp_nodelay_best_effort(listener.local_addr().unwrap()) .await .unwrap(); let (server, _) = listener.accept().await.unwrap(); @@ -37,6 +37,7 @@ fn assert_json_response( expected_status: &str, expected_error: &str, expected_detail: &str, + expected_request_id: Option<&str>, ) { let (headers, body) = response .windows(4) @@ -48,12 +49,19 @@ fn assert_json_response( assert!(headers.contains("Content-Type: application/json\r\n")); assert!(headers.contains(&format!("Content-Length: {}\r\n", body.len()))); assert!(headers.contains("Connection: close\r\n")); + if let Some(request_id) = expected_request_id { + assert!(headers.contains(&format!("x-openshell-request-id: {request_id}\r\n"))); + } + let mut expected = serde_json::json!({ + "error": expected_error, + "detail": expected_detail, + }); + if let Some(request_id) = expected_request_id { + expected["request_id"] = serde_json::json!(request_id); + } assert_eq!( serde_json::from_slice::(body).unwrap(), - serde_json::json!({ - "error": expected_error, - "detail": expected_detail, - }) + expected ); } @@ -111,9 +119,26 @@ async fn destination_denials_preserve_adapter_specific_wire_contracts() { "HTTP/1.1 403 Forbidden\r\n", "ssrf_denied", &format!("CONNECT target.example:8443 blocked: {detail}"), + None, ); let (mut app, mut proxy) = tcp_pair().await; + let request = crate::l7::provider::L7Request { + action: "POST".to_string(), + target: "/v1/items".to_string(), + query_params: std::collections::HashMap::new(), + raw_header: b"POST /v1/items HTTP/1.1\r\nHost: target.example:8080\r\n\r\n".to_vec(), + body_length: crate::l7::provider::BodyLength::None, + }; + let correlation = crate::l7::correlation::RequestCorrelation::with_request_id( + &request, + &crate::l7::relay::L7EvalContext { + host: "target.example".to_string(), + port: 8080, + ..Default::default() + }, + "test-request-id".to_string(), + ); deny_forward_destination( &mut proxy, &tracing::Span::none(), @@ -134,6 +159,7 @@ async fn destination_denials_preserve_adapter_specific_wire_contracts() { )), None, None, + Some(&correlation), ) .await .unwrap(); @@ -145,6 +171,7 @@ async fn destination_denials_preserve_adapter_specific_wire_contracts() { "HTTP/1.1 403 Forbidden\r\n", "ssrf_denied", &format!("POST target.example:8080 blocked: {detail}"), + Some("test-request-id"), ); } } @@ -471,7 +498,7 @@ async fn forward_https_absolute_form_rejection_is_snapshotted() { } async fn exercise_benchmark_request(proxy_addr: SocketAddr, target: SocketAddr, connect: bool) { - let mut client = TcpStream::connect(proxy_addr).await.unwrap(); + let mut client = connect_tcp_nodelay_best_effort(proxy_addr).await.unwrap(); let authority = target.to_string(); let request = if connect { format!("CONNECT {authority} HTTP/1.1\r\nHost: {authority}\r\n\r\n") diff --git a/crates/openshell-supervisor-network/src/proxy/tests/token_grants.rs b/crates/openshell-supervisor-network/src/proxy/tests/token_grants.rs index 0d26319749..b1dbeb00d4 100644 --- a/crates/openshell-supervisor-network/src/proxy/tests/token_grants.rs +++ b/crates/openshell-supervisor-network/src/proxy/tests/token_grants.rs @@ -80,6 +80,7 @@ async fn forward_inspection_selects_only_the_owner_admitting_the_request() { }), "5c4b3a29-1807-4f6e-8d5c-4b3a29180716", None, + None, ) .await .expect("prepare admitted forward request"); @@ -130,6 +131,7 @@ async fn forward_inspection_retains_installation_through_guarded_write() { }), "5c4b3a29-1807-4f6e-8d5c-4b3a29180716", None, + None, ) .await .expect("prepare authenticated request"); diff --git a/crates/openshell-supervisor-network/src/proxy/trace_tests.rs b/crates/openshell-supervisor-network/src/proxy/trace_tests.rs index 1602913885..d6871ca941 100644 --- a/crates/openshell-supervisor-network/src/proxy/trace_tests.rs +++ b/crates/openshell-supervisor-network/src/proxy/trace_tests.rs @@ -16,6 +16,7 @@ async fn paired_credential_denial_shares_one_deny_span() { 443, "credential-policy", &tracing::Span::none(), + None, ); }) .await; @@ -38,6 +39,7 @@ async fn repeated_credential_denials_keep_decision_spans_separate() { 443, "credential-policy", &connect, + None, ); } let allowed = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) @@ -133,6 +135,7 @@ async fn forward_denials_use_the_saved_connect_parent() { &decision, None, None, + None, )) .unwrap(); assert!(output.starts_with(b"HTTP/1.1 403 Forbidden")); @@ -194,7 +197,7 @@ async fn forward_evaluation_error_keeps_the_http_decision_policy_context() { "evaluation unavailable", ); // Use the same grouping as the forward-proxy production path. - emit_forward_l7_decision(decision, Some(error), &connect); + emit_forward_l7_decision(decision, Some(error), &connect, None); }) .await; diff --git a/crates/openshell-supervisor-network/src/sigv4.rs b/crates/openshell-supervisor-network/src/sigv4.rs index 79e4c32e18..fb44920bb0 100644 --- a/crates/openshell-supervisor-network/src/sigv4.rs +++ b/crates/openshell-supervisor-network/src/sigv4.rs @@ -2,14 +2,25 @@ // SPDX-License-Identifier: Apache-2.0 use aws_credential_types::Credentials; +use aws_sigv4::SigningOutput; use aws_sigv4::http_request::{ - PayloadChecksumKind, SignableBody, SignableRequest, SigningSettings, sign, + PayloadChecksumKind, SignableBody, SignableRequest, SigningInstructions, SigningSettings, sign, }; use aws_sigv4::sign::v4; use aws_smithy_runtime_api::client::identity::Identity; use miette::{Result, miette}; use std::time::SystemTime; +/// Keep AWS signer diagnostics out of supervisor log sinks: the canonical +/// request can contain the raw session token added during signing. +fn sign_without_tracing<'a>( + request: SignableRequest<'a>, + params: &'a aws_sigv4::http_request::SigningParams<'a>, +) -> std::result::Result, aws_sigv4::http_request::SigningError> +{ + tracing::dispatcher::with_default(&tracing::Dispatch::none(), || sign(request, params)) +} + /// AWS regions contain a hyphen followed by a digit (e.g., `us-east-1`). /// Service names like `s3` or `bedrock-runtime` do not. fn looks_like_region(s: &str) -> bool { @@ -233,7 +244,7 @@ fn build_identity(access_key: &str, secret_key: &str, session_token: Option<&str fn rebuild_request( parts: &RequestParts<'_>, - instructions: &aws_sigv4::http_request::SigningInstructions, + instructions: &SigningInstructions, body: &[u8], ) -> Vec { let mut output = Vec::with_capacity(256 + body.len()); @@ -297,7 +308,7 @@ pub fn apply_sigv4_to_request( ) .map_err(|e| miette!("SigV4 signable request: {e}"))?; - let (instructions, _signature) = sign(signable_request, &signing_params) + let (instructions, _signature) = sign_without_tracing(signable_request, &signing_params) .map_err(|e| miette!("SigV4 signing failed: {e}"))? .into_parts(); @@ -364,7 +375,7 @@ pub fn apply_sigv4_headers_only_with_body( ) .map_err(|e| miette!("SigV4 signable request: {e}"))?; - let (instructions, _signature) = sign(signable_request, &signing_params) + let (instructions, _signature) = sign_without_tracing(signable_request, &signing_params) .map_err(|e| miette!("SigV4 signing failed: {e}"))? .into_parts(); @@ -375,6 +386,55 @@ pub fn apply_sigv4_headers_only_with_body( mod tests { use super::*; + #[test] + fn signing_does_not_emit_the_canonical_request_to_tracing() { + let log = tempfile::NamedTempFile::new().unwrap(); + let subscriber = tracing_subscriber::fmt() + .with_max_level(tracing::Level::TRACE) + .with_ansi(false) + .with_writer(log.reopen().unwrap()) + .finish(); + let session_token = "SYNTHETIC_SIGV4_SESSION_TOKEN"; + let raw = b"GET /bucket/key HTTP/1.1\r\nHost: s3.us-east-1.amazonaws.com\r\n\r\n"; + + tracing::subscriber::with_default(subscriber, || { + tracing::trace!(target: "test_capture", "before signing"); + apply_sigv4_to_request( + raw, + "s3.us-east-1.amazonaws.com", + "us-east-1", + "s3", + "ASIATEST", + "synthetic-secret", + Some(session_token), + ) + .unwrap(); + for body in [ + SignableBody::UnsignedPayload, + SignableBody::StreamingUnsignedPayloadTrailer, + ] { + apply_sigv4_headers_only_with_body( + raw, + "s3.us-east-1.amazonaws.com", + "us-east-1", + "s3", + "ASIATEST", + "synthetic-secret", + Some(session_token), + body, + ) + .unwrap(); + } + tracing::trace!(target: "test_capture", "after signing"); + }); + + let events = std::fs::read_to_string(log.path()).unwrap(); + assert!(events.contains("before signing")); + assert!(events.contains("after signing")); + assert!(!events.contains("aws_sigv4")); + assert!(!events.contains(session_token)); + } + #[test] fn extract_region_from_hostname() { let region = extract_aws_region("bedrock-runtime.us-east-2.amazonaws.com").unwrap(); diff --git a/crates/openshell-supervisor-network/src/upstream_proxy.rs b/crates/openshell-supervisor-network/src/upstream_proxy.rs index dd0360f7a9..86b1ddf770 100644 --- a/crates/openshell-supervisor-network/src/upstream_proxy.rs +++ b/crates/openshell-supervisor-network/src/upstream_proxy.rs @@ -63,7 +63,6 @@ use std::task::{Context, Poll}; use std::time::Duration; use base64::Engine as _; -use openshell_core::net::set_tcp_nodelay_best_effort; use rustls::ClientConfig; use rustls::pki_types::ServerName; use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, ReadBuf}; @@ -1016,10 +1015,18 @@ async fn connect_via_inner( target: ConnectTarget, ) -> std::io::Result { let tcp = match endpoint.dial_ip { - Some(ip) => TcpStream::connect(SocketAddr::new(ip, endpoint.port)).await?, - None => TcpStream::connect((endpoint.host.as_str(), endpoint.port)).await?, + Some(ip) => { + openshell_core::net::connect_tcp_nodelay_best_effort(SocketAddr::new(ip, endpoint.port)) + .await? + } + None => { + openshell_core::net::connect_tcp_nodelay_best_effort(( + endpoint.host.as_str(), + endpoint.port, + )) + .await? + } }; - set_tcp_nodelay_best_effort(&tcp); // For an `https://` proxy, wrap the connection in TLS (verifying the proxy // certificate against the configured roots) before the CONNECT handshake. let mut stream = match &endpoint.tls { @@ -2062,7 +2069,9 @@ mod tests { .write_all(b"HTTP/1.1 200 Connection established\r\n\r\n") .await .unwrap(); - if let Ok(mut upstream) = TcpStream::connect(tls_addr).await { + if let Ok(mut upstream) = + openshell_core::net::connect_tcp_nodelay_best_effort(tls_addr).await + { let _ = copy_bidirectional(&mut client, &mut upstream).await; } }); @@ -2150,7 +2159,11 @@ mod tests { async fn prefixed_stream_replays_prefix_before_socket_bytes() { let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); - let connect = tokio::spawn(async move { TcpStream::connect(addr).await.unwrap() }); + let connect = tokio::spawn(async move { + openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap() + }); let (mut accepted, _) = listener.accept().await.unwrap(); let dialed = connect.await.unwrap(); diff --git a/crates/openshell-supervisor-network/tests/accept_fd_exhaustion.rs b/crates/openshell-supervisor-network/tests/accept_fd_exhaustion.rs index eba9ef051f..f3d39c2a38 100644 --- a/crates/openshell-supervisor-network/tests/accept_fd_exhaustion.rs +++ b/crates/openshell-supervisor-network/tests/accept_fd_exhaustion.rs @@ -61,7 +61,8 @@ fn accept_fd_exhaustion_child() { let addr = listener.local_addr().unwrap(); // Place a connection in the kernel backlog before exhausting FDs. - let backlog_conn = std::net::TcpStream::connect(addr).expect("backlog connect"); + let backlog_conn = + openshell_core::net::connect_std_tcp_nodelay_best_effort(addr).expect("backlog connect"); // Exhaust remaining FDs. #[allow(clippy::collection_is_never_read)] @@ -86,7 +87,8 @@ fn accept_fd_exhaustion_child() { drop(backlog_conn); // Make a fresh connection now that FDs are available. - let client = std::net::TcpStream::connect(addr).expect("connect after FD release"); + let client = openshell_core::net::connect_std_tcp_nodelay_best_effort(addr) + .expect("connect after FD release"); // Accept should succeed, proving that an accept loop retrying on // EMFILE (like the proxy does) will recover once FDs are available. diff --git a/crates/openshell-supervisor-network/tests/sigv4_localstack.rs b/crates/openshell-supervisor-network/tests/sigv4_localstack.rs index 2d76b7e2f7..bc33314def 100644 --- a/crates/openshell-supervisor-network/tests/sigv4_localstack.rs +++ b/crates/openshell-supervisor-network/tests/sigv4_localstack.rs @@ -8,7 +8,6 @@ use openshell_supervisor_network::sigv4::{apply_sigv4_headers_only, apply_sigv4_to_request}; use std::sync::atomic::{AtomicU32, Ordering}; use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::net::TcpStream; const LOCALSTACK: &str = "127.0.0.1:4566"; const ACCESS_KEY: &str = "test"; @@ -28,7 +27,7 @@ fn s3_host() -> String { } async fn send_raw(raw: &[u8]) -> (u16, String) { - let mut stream = TcpStream::connect(LOCALSTACK) + let mut stream = openshell_core::net::connect_tcp_nodelay_best_effort(LOCALSTACK) .await .expect("connect to LocalStack"); stream.write_all(raw).await.expect("write request"); diff --git a/crates/openshell-supervisor-network/tests/websocket_upgrade.rs b/crates/openshell-supervisor-network/tests/websocket_upgrade.rs index 322d6709cd..3806e27f3b 100644 --- a/crates/openshell-supervisor-network/tests/websocket_upgrade.rs +++ b/crates/openshell-supervisor-network/tests/websocket_upgrade.rs @@ -31,7 +31,7 @@ use openshell_supervisor_network::l7::rest::RestProvider; use std::collections::HashMap; use std::net::SocketAddr; use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::net::{TcpListener, TcpStream}; +use tokio::net::TcpListener; use tokio_tungstenite::accept_async; use tokio_tungstenite::tungstenite::Message; @@ -100,7 +100,9 @@ async fn websocket_upgrade_through_l7_relay_exchanges_message() { let ws_addr = start_ws_echo_server().await; // Open a real TCP connection to the WebSocket server (simulates upstream) - let mut upstream = TcpStream::connect(ws_addr).await.unwrap(); + let mut upstream = openshell_core::net::connect_tcp_nodelay_best_effort(ws_addr) + .await + .unwrap(); // In-memory duplex for the client side of the relay let (mut client_app, mut client_proxy) = tokio::io::duplex(8192); @@ -227,7 +229,9 @@ async fn normal_http_request_still_works_after_relay_changes() { stream.flush().await.unwrap(); }); - let mut upstream = TcpStream::connect(addr).await.unwrap(); + let mut upstream = openshell_core::net::connect_tcp_nodelay_best_effort(addr) + .await + .unwrap(); let (mut client_read, mut client_proxy) = tokio::io::duplex(8192); let raw_header = format!( diff --git a/crates/openshell-supervisor-process/src/ssh.rs b/crates/openshell-supervisor-process/src/ssh.rs index 795838da17..a11eddbd21 100644 --- a/crates/openshell-supervisor-process/src/ssh.rs +++ b/crates/openshell-supervisor-process/src/ssh.rs @@ -1779,13 +1779,14 @@ mod tests { openshell_isolation_interface::contract::BoundaryDuplexStream, openshell_isolation_interface::contract::BackendError, > { - let stream = tokio::net::TcpStream::connect((target.host(), target.port())) - .await - .map_err(|error| { - openshell_isolation_interface::contract::BackendError::Process( - error.to_string(), - ) - })?; + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(( + target.host(), + target.port(), + )) + .await + .map_err(|error| { + openshell_isolation_interface::contract::BackendError::Process(error.to_string()) + })?; Ok(Box::new(stream)) } } diff --git a/crates/openshell-supervisor/src/app_listener.rs b/crates/openshell-supervisor/src/app_listener.rs index ce63c9bd0b..9b7098aed6 100644 --- a/crates/openshell-supervisor/src/app_listener.rs +++ b/crates/openshell-supervisor/src/app_listener.rs @@ -1261,7 +1261,7 @@ mod tests { host: &str, extra: &str, ) -> String { - let stream = openshell_core::net::connect_tcp_nodelay_best_effort(&[f.address]) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(f.address) .await .unwrap(); let server_name = rustls::pki_types::ServerName::try_from(sni.to_string()).unwrap(); @@ -1290,7 +1290,7 @@ mod tests { use http_body_util::BodyExt as _; let f = fixture(); - let stream = openshell_core::net::connect_tcp_nodelay_best_effort(&[f.address]) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(f.address) .await .unwrap(); let tls = tokio_rustls::TlsConnector::from(f.tls.clone()) @@ -1589,7 +1589,7 @@ mod tests { #[tokio::test] async fn websocket_echo_closes_on_route_withdrawal() { let f = fixture(); - let stream = openshell_core::net::connect_tcp_nodelay_best_effort(&[f.address]) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(f.address) .await .unwrap(); let name = rustls::pki_types::ServerName::try_from(HOST.to_string()).unwrap(); @@ -1639,7 +1639,7 @@ mod tests { async fn sandbox_websocket(f: &Fixture, sandbox: &str, service: &str) -> TestWebSocket { let host = format!("default--{sandbox}--{service}.svc.openshell.internal"); - let stream = openshell_core::net::connect_tcp_nodelay_best_effort(&[f.address]) + let stream = openshell_core::net::connect_tcp_nodelay_best_effort(f.address) .await .unwrap(); let name = rustls::pki_types::ServerName::try_from(host.clone()).unwrap(); diff --git a/crates/openshell-supervisor/src/backend_setup.rs b/crates/openshell-supervisor/src/backend_setup.rs index a0cd52ea87..eacec32962 100644 --- a/crates/openshell-supervisor/src/backend_setup.rs +++ b/crates/openshell-supervisor/src/backend_setup.rs @@ -38,6 +38,7 @@ pub struct BackendServices { pub ca_file_paths: Arc>>, pub provider_credentials: ProviderCredentialState, pub sandbox_bearer: SessionBearerTokenSlot, + pub attach_request_timeout: std::time::Duration, /// Gateway-granted authority to replace the generation's current owner. pub takeover: Option, /// Recovery deadline (seconds) offered to the runtime at every attach. @@ -75,6 +76,7 @@ pub trait PreparedBackend: Send + Sync { async fn discover_policy( &self, bearer: SessionBearerTokenSlot, + attach_request_timeout: std::time::Duration, ) -> std::result::Result<(Option, bool), BackendError>; /// Consume prepared launch data and retain the supervisor's live handles. @@ -159,9 +161,10 @@ impl SelectedBackend { pub async fn discover_policy( &self, bearer: SessionBearerTokenSlot, + attach_request_timeout: std::time::Duration, ) -> Result<(Option, bool)> { self.prepared - .discover_policy(bearer) + .discover_policy(bearer, attach_request_timeout) .await .map_err(|error| miette::miette!("discover workload image policy: {error}")) } @@ -314,11 +317,13 @@ impl PreparedBackend for OpenShellLaunch { async fn discover_policy( &self, bearer: SessionBearerTokenSlot, + attach_request_timeout: std::time::Duration, ) -> std::result::Result<(Option, bool), BackendError> { openshell_sandbox_backend::OpenShellRuntimeBackend::discover_policy( self.descriptor.clone(), bearer, self.transport.clone(), + attach_request_timeout, ) .await } @@ -334,6 +339,7 @@ impl PreparedBackend for OpenShellLaunch { services.sandbox_bearer, ) .with_transport(self.transport) + .with_attach_request_timeout(services.attach_request_timeout) .with_recovery_timeout_secs(services.supervisor_recovery_timeout_secs); if let Some(takeover) = services.takeover { backend = backend.with_takeover(takeover); diff --git a/crates/openshell-supervisor/src/backend_setup/registration.rs b/crates/openshell-supervisor/src/backend_setup/registration.rs index df337cc7f1..978ef65033 100644 --- a/crates/openshell-supervisor/src/backend_setup/registration.rs +++ b/crates/openshell-supervisor/src/backend_setup/registration.rs @@ -99,12 +99,14 @@ impl PreparedBackend for RegisteredLaunch { async fn discover_policy( &self, bearer: SessionBearerTokenSlot, + attach_request_timeout: std::time::Duration, ) -> std::result::Result<(Option, bool), BackendError> { OpenShellRuntimeBackend::discover_registered_policy( self.name.clone(), self.descriptor.clone(), self.payload.clone(), bearer, + attach_request_timeout, ) .await } @@ -119,14 +121,17 @@ impl PreparedBackend for RegisteredLaunch { )); } let payload = self.payload.clone(); - let backend = Arc::new(OpenShellRuntimeBackend::registered( - self.name, - self.descriptor, - self.payload, - services.ca_file_paths, - services.provider_credentials, - services.sandbox_bearer, - )?); + let backend = Arc::new( + OpenShellRuntimeBackend::registered( + self.name, + self.descriptor, + self.payload, + services.ca_file_paths, + services.provider_credentials, + services.sandbox_bearer, + )? + .with_attach_request_timeout(services.attach_request_timeout), + ); Ok(BuiltBackend { backend, payload, diff --git a/crates/openshell-supervisor/src/backend_setup/tests.rs b/crates/openshell-supervisor/src/backend_setup/tests.rs index 42626fbe7c..822cf1ae26 100644 --- a/crates/openshell-supervisor/src/backend_setup/tests.rs +++ b/crates/openshell-supervisor/src/backend_setup/tests.rs @@ -200,6 +200,7 @@ impl TestSetup { HashMap::from([("TEST_CONFIG".into(), "initial".into())]), ), sandbox_bearer: self.auth.sandbox_bearer_slot().unwrap(), + attach_request_timeout: std::time::Duration::from_mins(5), takeover: None, supervisor_recovery_timeout_secs: None, } @@ -277,6 +278,7 @@ impl PreparedBackend for TestLaunch { async fn discover_policy( &self, bearer: SessionBearerTokenSlot, + _attach_request_timeout: std::time::Duration, ) -> std::result::Result<(Option, bool), BackendError> { self.observed.record("discover"); bearer.authorization_metadata().unwrap(); @@ -853,6 +855,7 @@ async fn shared_startup_uses_selected_backend_through_readiness_and_shutdown() { auth_bundle: setup.auth.clone(), admitted_isolation_backend: Some(TEST_BACKEND.into()), main_exit_marker: Some(marker.clone()), + attach_request_timeout: Duration::from_mins(5), trace_context: None, provider_workload_api_socket: None, // The binary loads the proxy trust directory from its environment. @@ -897,9 +900,12 @@ async fn shared_startup_uses_selected_backend_through_readiness_and_shutdown() { assert!(!readiness.exists()); assert!(!marker.exists()); assert!( - tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) - .await - .is_err() + openshell_core::net::connect_tcp_nodelay_best_effort(( + std::net::Ipv4Addr::LOCALHOST, + health_port + )) + .await + .is_err() ); setup .observed @@ -925,9 +931,12 @@ async fn shared_startup_uses_selected_backend_through_readiness_and_shutdown() { assert!(!readiness.exists()); assert!(!marker.exists()); assert!( - tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) - .await - .is_err() + openshell_core::net::connect_tcp_nodelay_best_effort(( + std::net::Ipv4Addr::LOCALHOST, + health_port + )) + .await + .is_err() ); setup @@ -939,7 +948,7 @@ async fn shared_startup_uses_selected_backend_through_readiness_and_shutdown() { loop { if marker.exists() && tokio::net::UnixStream::connect(&readiness).await.is_ok() - && tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) + && openshell_core::net::connect_tcp_nodelay_best_effort((std::net::Ipv4Addr::LOCALHOST, health_port)) .await .is_ok() { @@ -976,9 +985,12 @@ async fn shared_startup_uses_selected_backend_through_readiness_and_shutdown() { // Listener tasks are aborted on drop; let the runtime reap their sockets. tokio::task::yield_now().await; assert!( - tokio::net::TcpStream::connect((std::net::Ipv4Addr::LOCALHOST, health_port)) - .await - .is_err() + openshell_core::net::connect_tcp_nodelay_best_effort(( + std::net::Ipv4Addr::LOCALHOST, + health_port + )) + .await + .is_err() ); } @@ -1061,6 +1073,7 @@ async fn suspend_signal_exits_without_terminating_boundary() { auth_bundle: setup.auth.clone(), admitted_isolation_backend: Some(TEST_BACKEND.into()), main_exit_marker: Some(marker.clone()), + attach_request_timeout: Duration::from_mins(5), trace_context: None, provider_workload_api_socket: None, // The binary loads the proxy trust directory from its environment. diff --git a/crates/openshell-supervisor/src/legacy_startup_regression_tests.rs b/crates/openshell-supervisor/src/legacy_startup_regression_tests.rs index a9dd93b5c2..a28e0cf4bf 100644 --- a/crates/openshell-supervisor/src/legacy_startup_regression_tests.rs +++ b/crates/openshell-supervisor/src/legacy_startup_regression_tests.rs @@ -235,6 +235,7 @@ impl PreparedBackend for HeldPreparedBackend { async fn discover_policy( &self, _bearer: openshell_core::jwt::SessionBearerTokenSlot, + _attach_request_timeout: Duration, ) -> Result<(Option, bool), BackendError> { self.0.discovered.store(true, Ordering::SeqCst); Ok((None, false)) @@ -310,6 +311,7 @@ async fn config_push_regression_legacy_gateway_never_observes_ready_before_workl auth_bundle, admitted_isolation_backend: Some("test-held".into()), main_exit_marker: None, + attach_request_timeout: Duration::from_mins(5), trace_context: None, provider_workload_api_socket: None, proxy_tls: openshell_supervisor_network::run::ProxyTlsFiles::default(), diff --git a/crates/openshell-supervisor/src/lib.rs b/crates/openshell-supervisor/src/lib.rs index b9a5546bf2..589c87fffe 100644 --- a/crates/openshell-supervisor/src/lib.rs +++ b/crates/openshell-supervisor/src/lib.rs @@ -920,6 +920,8 @@ pub struct SandboxInstanceConfig { pub auth_bundle: openshell_core::jwt::SupervisorAuthBundle, pub admitted_isolation_backend: Option, pub main_exit_marker: Option, + /// Initial boundary attachment and image-policy discovery budget. + pub attach_request_timeout: Duration, /// Parent of the startup span. pub trace_context: Option, /// SPIFFE Workload API socket for provider token grants; absent fails closed. @@ -1092,6 +1094,7 @@ pub async fn run_sandbox_instance( auth_bundle, admitted_isolation_backend, main_exit_marker, + attach_request_timeout, trace_context, provider_workload_api_socket, proxy_tls, @@ -1181,7 +1184,7 @@ pub async fn run_sandbox_instance( } let startup_status = startup.in_scope(openshell_otel::ErrorStatusGuard::current); let (image_yaml, invalid_image) = selected_backend - .discover_policy(sandbox_bearer.clone()) + .discover_policy(sandbox_bearer.clone(), attach_request_timeout) .instrument(tracing::info_span!(parent: &startup, "supervisor.boundary.discover_policy")) .await?; let image_discovery = if invalid_image { @@ -1337,6 +1340,7 @@ pub async fn run_sandbox_instance( ca_file_paths: ca_file_paths.clone(), provider_credentials: provider_credentials.clone(), sandbox_bearer, + attach_request_timeout, takeover, supervisor_recovery_timeout_secs, }, @@ -4707,6 +4711,7 @@ mod tests { auth_bundle, admitted_isolation_backend: None, main_exit_marker: None, + attach_request_timeout: Duration::from_mins(5), trace_context: None, provider_workload_api_socket: None, proxy_tls: openshell_supervisor_network::run::ProxyTlsFiles::default(), @@ -4845,7 +4850,7 @@ mod tests { "net.ipv6.bindv6only=1 must not make the wildcard listener IPv6-only" ); } - std::net::TcpStream::connect(("127.0.0.1", port)) + openshell_core::net::connect_std_tcp_nodelay_best_effort(("127.0.0.1", port)) .expect("IPv4 kubelet probe reaches the readiness listener"); } @@ -4861,7 +4866,7 @@ mod tests { let connects = || async { timeout( Duration::from_millis(100), - tokio::net::TcpStream::connect(("127.0.0.1", port)), + openshell_core::net::connect_tcp_nodelay_best_effort(("127.0.0.1", port)), ) .await .is_ok_and(|result| result.is_ok()) @@ -4908,7 +4913,7 @@ mod tests { assert!( !timeout( Duration::from_millis(100), - tokio::net::TcpStream::connect(("127.0.0.1", port)) + openshell_core::net::connect_tcp_nodelay_best_effort(("127.0.0.1", port)) ) .await .is_ok_and(|result| result.is_ok()) @@ -4917,7 +4922,7 @@ mod tests { timeout(Duration::from_secs(2), async { while !timeout( Duration::from_millis(100), - tokio::net::TcpStream::connect(("127.0.0.1", port)), + openshell_core::net::connect_tcp_nodelay_best_effort(("127.0.0.1", port)), ) .await .is_ok_and(|result| result.is_ok()) diff --git a/crates/openshell-supervisor/src/main.rs b/crates/openshell-supervisor/src/main.rs index 4c1bcac7a0..6c54c03ee9 100644 --- a/crates/openshell-supervisor/src/main.rs +++ b/crates/openshell-supervisor/src/main.rs @@ -237,6 +237,17 @@ struct Args { /// Stdin is a driver-owned liveness pipe. EOF means the driver exited. #[arg(long, hide = true)] parent_liveness_stdin: bool, + + /// Attach and policy-discover request timeout, in seconds. Slow + /// VM-backed runtimes (CoCo/peer-pods) need longer than the default for + /// the remote boundary to boot before the supervisor gives up. + #[arg( + long, + env = "OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS", + default_value = "300", + value_parser = clap::value_parser!(u64).range(1..=86_400) + )] + attach_request_timeout_seconds: u64, } #[cfg(unix)] @@ -660,6 +671,9 @@ fn main() -> Result<()> { auth_bundle, admitted_isolation_backend: Some(admitted_backend), main_exit_marker: args.main_exit_marker, + attach_request_timeout: std::time::Duration::from_secs( + args.attach_request_timeout_seconds, + ), trace_context: openshell_supervisor::trace_context_from_environment(), provider_workload_api_socket: std::env::var_os( openshell_core::sandbox_env::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET, @@ -873,6 +887,37 @@ fn supervisor_exit(result: Result) -> Result { #[cfg(test)] mod tests { + #[test] + fn bootstrap_timeout_arguments_reject_invalid_values() { + assert_eq!( + Args::try_parse_from(["openshell-supervisor"]) + .unwrap() + .attach_request_timeout_seconds, + 300 + ); + for value in ["0", "86401", "invalid"] { + assert!( + Args::try_parse_from([ + "openshell-supervisor", + "--attach-request-timeout-seconds", + value + ]) + .is_err() + ); + } + for value in ["1", "900", "86400"] { + let args = Args::try_parse_from([ + "openshell-supervisor", + "--attach-request-timeout-seconds", + value, + ]) + .unwrap(); + assert_eq!( + args.attach_request_timeout_seconds, + value.parse::().unwrap() + ); + } + } #[cfg(unix)] #[test] diff --git a/crates/openshell-supervisor/src/shared.rs b/crates/openshell-supervisor/src/shared.rs index 243f181653..aa35b527a8 100644 --- a/crates/openshell-supervisor/src/shared.rs +++ b/crates/openshell-supervisor/src/shared.rs @@ -125,6 +125,9 @@ from sandbox runtimes)", instance_root, runtime_listen, removal_grace: openshell_supervisor_control::INSTANCE_REMOVAL_GRACE, + attach_request_timeout: std::time::Duration::from_secs( + args.attach_request_timeout_seconds, + ), pool, }, )?; diff --git a/crates/openshell-supervisor/src/shared_host.rs b/crates/openshell-supervisor/src/shared_host.rs index 9e6ae7d241..e4a2202d46 100644 --- a/crates/openshell-supervisor/src/shared_host.rs +++ b/crates/openshell-supervisor/src/shared_host.rs @@ -175,6 +175,8 @@ pub struct SharedHostConfig { pub runtime_listen: SocketAddr, /// Time a withdrawn sandbox instance may take to stop its workload. pub removal_grace: Duration, + /// Initial boundary attachment and image-policy discovery budget. + pub attach_request_timeout: Duration, /// This process's identity as a member of the supervisor pool. pub pool: PoolIdentity, } @@ -187,6 +189,7 @@ pub struct SharedHost { router: Arc, gateway_endpoint: String, removal_grace: Duration, + attach_request_timeout: Duration, /// Assignments of the latest snapshot that could not be admitted, reported /// as failed instances until the gateway withdraws them. rejected: Mutex>, @@ -305,6 +308,7 @@ impl SharedHost { router, gateway_endpoint: config.gateway_endpoint, removal_grace: config.removal_grace, + attach_request_timeout: config.attach_request_timeout, rejected: Mutex::new(BTreeMap::new()), desired_routes: Mutex::new(None), pool: config.pool, @@ -816,6 +820,7 @@ impl SharedHost { auth_bundle: registration.auth, admitted_isolation_backend: Some(openshell_sandbox_backend::BACKEND_NAME.to_string()), main_exit_marker: None, + attach_request_timeout: self.attach_request_timeout, trace_context: registration.trace_context, // One process cannot hold a per-sandbox workload identity, so // providers that need one fail closed. @@ -1621,6 +1626,7 @@ mod tests { instance_root: root.path().join("instances"), runtime_listen: "127.0.0.1:0".parse().unwrap(), removal_grace: Duration::from_secs(10), + attach_request_timeout: Duration::from_mins(5), pool: PoolIdentity { member_id: "member-1".to_string(), runtime_address: "127.0.0.1:5500".parse().unwrap(), @@ -1657,6 +1663,7 @@ mod tests { instance_root: root.path().join("instances"), runtime_listen: "127.0.0.1:0".parse().unwrap(), removal_grace: Duration::from_secs(10), + attach_request_timeout: Duration::from_mins(5), pool: PoolIdentity { member_id: "member-1".to_string(), runtime_address: "127.0.0.1:5500".parse().unwrap(), @@ -1778,6 +1785,7 @@ mod tests { instance_root: root.path().join("instances"), runtime_listen: "127.0.0.1:0".parse().unwrap(), removal_grace: Duration::from_secs(10), + attach_request_timeout: Duration::from_mins(5), pool: PoolIdentity { member_id: "member-1".to_string(), runtime_address: "127.0.0.1:5500".parse().unwrap(), @@ -1874,6 +1882,7 @@ mod tests { instance_root: root.path().join("instances"), runtime_listen: "127.0.0.1:0".parse().unwrap(), removal_grace: Duration::from_secs(10), + attach_request_timeout: Duration::from_mins(5), pool: PoolIdentity { member_id: "member-1".to_string(), runtime_address: "127.0.0.1:5500".parse().unwrap(), @@ -1938,9 +1947,10 @@ mod tests { Some(SandboxInstancePhaseReport::Supervising) ); assert!(kept_stops.try_recv().is_err()); - let mut stream = tokio::net::TcpStream::connect(host.router.local_addr()) - .await - .unwrap(); + let mut stream = + openshell_core::net::connect_tcp_nodelay_best_effort(host.router.local_addr()) + .await + .unwrap(); assert_eq!( openshell_sandbox_backend::runtime_routed::negotiate_route(&mut stream, elsewhere) .await @@ -2094,6 +2104,7 @@ mod tests { instance_root: root.path().join("instances"), runtime_listen: "127.0.0.1:0".parse().unwrap(), removal_grace: Duration::from_secs(10), + attach_request_timeout: Duration::from_mins(5), pool: PoolIdentity { member_id: "member-1".to_string(), runtime_address: "127.0.0.1:5500".parse().unwrap(), @@ -2447,6 +2458,7 @@ mod tests { instance_root: root.path().join("instances"), runtime_listen: "127.0.0.1:0".parse().unwrap(), removal_grace: Duration::from_secs(10), + attach_request_timeout: Duration::from_mins(5), pool: PoolIdentity { member_id: "member-1".to_string(), runtime_address: "127.0.0.1:5500".parse().unwrap(), diff --git a/crates/openshell-tui/src/lib.rs b/crates/openshell-tui/src/lib.rs index 133d7d5c0a..0e807cd452 100644 --- a/crates/openshell-tui/src/lib.rs +++ b/crates/openshell-tui/src/lib.rs @@ -1315,6 +1315,16 @@ fn render_policy_lines( // L7 allow rules. for l7 in &ep.rules { if let Some(allow) = &l7.allow { + if ep.protocol.eq_ignore_ascii_case("mcp") { + render_mcp_rule( + &mut lines, + "Allow", + &allow.method, + allow.params.get("name"), + t, + ); + continue; + } let method = if allow.method.is_empty() { "*" } else { @@ -1334,6 +1344,18 @@ fn render_policy_lines( } } + if ep.protocol.eq_ignore_ascii_case("mcp") { + for deny in &ep.deny_rules { + render_mcp_rule( + &mut lines, + "Deny", + &deny.method, + deny.params.get("name"), + t, + ); + } + } + // Access preset (if set instead of explicit rules). if ep.access != 0 && ep.rules.is_empty() { lines.push(Line::from(vec![ @@ -1371,6 +1393,37 @@ fn render_policy_lines( lines } +fn render_mcp_rule( + lines: &mut Vec>, + action: &str, + method: &str, + tool: Option<&openshell_core::proto::L7QueryMatcher>, + theme: &theme::Theme, +) { + use ratatui::text::{Line, Span}; + + let method = if method.is_empty() { + if tool.is_some() { "tools/call" } else { "*" } + } else { + method + }; + lines.push(Line::from(vec![ + Span::styled(format!(" {action}: "), theme.muted), + Span::styled(method.to_string(), theme.text), + ])); + if let Some(tool) = tool { + for name in std::iter::once(&tool.glob) + .filter(|name| !name.is_empty()) + .chain(&tool.any) + { + lines.push(Line::from(vec![ + Span::styled(" Tool: ", theme.muted), + Span::styled(name.clone(), theme.text), + ])); + } + } +} + // --------------------------------------------------------------------------- // Animation helper // --------------------------------------------------------------------------- @@ -3444,3 +3497,173 @@ mod sandbox_notes_tests { assert_eq!(sandbox_notes(&sandbox, String::new()), ""); } } + +#[cfg(test)] +mod policy_render_tests { + use super::*; + + fn policy_lines(yaml: &str) -> Vec> { + let policy = openshell_policy::parse_sandbox_policy(yaml).expect("valid policy"); + render_policy_lines(&policy, &theme::Theme::dark()) + } + + #[test] + fn mcp_tool_allowlist_is_visible_one_tool_per_line() { + let lines = policy_lines( + r" +version: 1 +network_policies: + review: + endpoints: + - host: host.openshell.internal + port: 32443 + protocol: mcp + enforcement: enforce + rules: + - allow: + method: tools/call + tool: + any: + - local_gh_issue_view + - local_gh_pr_checks + - local_gh_pr_details + - local_gh_pr_diff + - local_gh_pr_list + - local_gh_pr_view + - local_gh_repo_view + - local_gh_run_failed_logs + - local_gh_run_view + - local_gh_source_blob + - local_gh_source_tree + - local_gh_version +", + ); + let tools: Vec<_> = lines + .iter() + .map(ToString::to_string) + .filter_map(|line| line.strip_prefix(" Tool: ").map(str::to_string)) + .collect(); + assert_eq!( + tools, + [ + "local_gh_issue_view", + "local_gh_pr_checks", + "local_gh_pr_details", + "local_gh_pr_diff", + "local_gh_pr_list", + "local_gh_pr_view", + "local_gh_repo_view", + "local_gh_run_failed_logs", + "local_gh_run_view", + "local_gh_source_blob", + "local_gh_source_tree", + "local_gh_version", + ] + ); + assert!( + lines + .iter() + .any(|line| line.to_string() == " Allow: tools/call") + ); + assert!( + !lines + .iter() + .any(|line| line.to_string().contains("tools/call *")) + ); + + let mut buffer = ratatui::buffer::Buffer::empty(ratatui::layout::Rect::new(0, 0, 60, 30)); + ratatui::widgets::Widget::render( + ratatui::widgets::Paragraph::new(lines), + buffer.area, + &mut buffer, + ); + for tool in tools { + assert!(buffer.content.chunks(60).any(|row| { + row.iter() + .map(ratatui::buffer::Cell::symbol) + .collect::() + .contains(&tool) + })); + } + } + + #[test] + fn mcp_methods_and_allow_deny_selectors_preserve_their_scope() { + let lines = policy_lines( + r" +version: 1 +network_policies: + review: + endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: + method: tools/list + - allow: + tool: read_source + - allow: + method: tools/call + params: + name: + any: [read_*, list_*] + deny_rules: + - tool: read_private + - method: tools/call + params: + name: + any: [read_secret_*, list_private] + - method: resources/read + profile: + endpoints: + - host: profile.example.com + port: 443 + protocol: mcp + mcp: + allow_all_known_mcp_methods: true + rules: + - allow: {} + deny_rules: + - {} + rest: + endpoints: + - host: api.example.com + port: 443 + protocol: rest + rules: + - allow: {method: GET, path: /repos/**} +", + ); + let text: Vec<_> = lines.iter().map(ToString::to_string).collect(); + assert!(text.iter().any(|line| line == " Allow: *")); + assert!(text.iter().any(|line| line == " Deny: *")); + let mcp_start = text + .iter() + .position(|line| line == " Allow: tools/list") + .unwrap(); + assert_eq!( + &text[mcp_start..mcp_start + 12], + [ + " Allow: tools/list", + " Allow: tools/call", + " Tool: read_source", + " Allow: tools/call", + " Tool: read_*", + " Tool: list_*", + " Deny: tools/call", + " Tool: read_private", + " Deny: tools/call", + " Tool: read_secret_*", + " Tool: list_private", + " Deny: resources/read", + ] + ); + assert!( + text.iter() + .any(|line| line == " Allow: GET /repos/**") + ); + } +} diff --git a/deny.toml b/deny.toml index b813b44dd4..08ac2d3606 100644 --- a/deny.toml +++ b/deny.toml @@ -15,6 +15,9 @@ unmaintained = "workspace" maximum-db-staleness = "P30D" ignore = [ { id = "RUSTSEC-2023-0071", reason = "rsa Marvin attack — required by russh/ssh-key and RSA test fixtures; no patched release is available" }, + # Temporary: SQLx's SQLite dependency-bound update is merged but unreleased. + # https://github.com/transact-rs/sqlx/pull/4502 + { id = "RUSTSEC-2026-0459", reason = "sqlx-sqlite 0.9.0 requires libsqlite3-sys <0.38.0; remove this exception when a released SQLx accepts patched libsqlite3-sys >=0.39.0" }, ] # -- Licenses ------------------------------------------------------------------ diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 58aa132eb6..bc6676b9f1 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -615,6 +615,7 @@ discovery endpoint or its TLS CA. | podLabels | object | `{}` | Extra labels to add to the gateway pod. | | podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. | | podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. | +| priorityClassName | string | `""` | PriorityClass for the gateway pod. The gateway is a control-plane component; when it shares nodes with the workloads it serves, the default priority gives it no advantage under node pressure. Empty = omit the field. | | probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. | | probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. | | probes.liveness.periodSeconds | int | `5` | Liveness probe period, in seconds. | @@ -758,6 +759,7 @@ discovery endpoint or its TLS CA. | supervisor.replicas | int | `1` | Shared supervisor pool members, used only when gatewayConfig selects supervisor_topology = "gateway". The gateway always coordinates the shared supervisor as a high-availability pool: when a member fails or leaves, another member takes over its sandboxes without restarting their workloads. 1 has no failover; use 2 or more to survive losing a member. Only the supervisor Deployment's replica count depends on this value, so changing it neither restarts the gateway nor replaces running members. Ignored when supervisor.autoscaling.enabled is true. | | supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. | | tolerations | list | `[]` | Tolerations for the gateway pod. | +| topologySpreadConstraints | list | `[]` | Topology spread constraints for the gateway pod, used to spread replicas across zones or nodes. Empty = omit the field. | | upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. | | upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. | | upstreamProxy.authSecret.key | string | `""` | Secret key containing the proxy credential. | diff --git a/deploy/helm/openshell/ci/values-local-tracing.yaml b/deploy/helm/openshell/ci/values-local-tracing.yaml new file mode 100644 index 0000000000..68d1f23500 --- /dev/null +++ b/deploy/helm/openshell/ci/values-local-tracing.yaml @@ -0,0 +1,10 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Export gateway traces to the collector that `mise run helm:k3s:create` +# installs. Layer only onto clusters that run that collector. +gatewayConfig: + openshell.gateway.otlp: + endpoint: http://openshell-collector.observability.svc.cluster.local:4317 + openshell.gateway.otlp.agent: + enabled: true diff --git a/deploy/helm/openshell/ci/values-skaffold.yaml b/deploy/helm/openshell/ci/values-skaffold.yaml index 1ff2bb2fc4..76e0167e79 100644 --- a/deploy/helm/openshell/ci/values-skaffold.yaml +++ b/deploy/helm/openshell/ci/values-skaffold.yaml @@ -5,16 +5,10 @@ gatewayConfig: openshell.drivers.kubernetes: image_pull_policy: if_not_present - openshell.gateway.otlp: - endpoint: http://openshell-collector.observability.svc.cluster.local:4317 - openshell.gateway.otlp.agent: - enabled: true openshell.gateway.auth: allow_unauthenticated_users: true server: - otlp: - endpoint: http://openshell-collector.observability.svc.cluster.local:4317 # Comment out to enforce mTLS (uses PKI secrets generated by pkiInitJob). disableTls: true auth: diff --git a/deploy/helm/openshell/skaffold.yaml b/deploy/helm/openshell/skaffold.yaml index 3ad4d3b8bf..f505c479fa 100644 --- a/deploy/helm/openshell/skaffold.yaml +++ b/deploy/helm/openshell/skaffold.yaml @@ -124,6 +124,7 @@ deploy: valuesFiles: - values.yaml - ci/values-skaffold.yaml + - ci/values-local-tracing.yaml # Add ci/values-cert-manager.yaml here (and uncomment the cert-manager # release above) to switch TLS generation to cert-manager. #- ci/values-cert-manager.yaml diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 6b73241e6c..b0814a40b8 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -358,6 +358,9 @@ spec: {{- end }} - name: gateway-state emptyDir: {} + {{- with .Values.priorityClassName }} + priorityClassName: {{ . }} + {{- end }} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 4 }} @@ -370,4 +373,8 @@ spec: tolerations: {{- toYaml . | nindent 4 }} {{- end }} + {{- with .Values.topologySpreadConstraints }} + topologySpreadConstraints: + {{- toYaml . | nindent 4 }} + {{- end }} {{- end }} diff --git a/deploy/helm/openshell/tests/gateway_scheduling_test.yaml b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml new file mode 100644 index 0000000000..3a1a755d02 --- /dev/null +++ b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway scheduling fields +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml + - templates/deployment.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: omits scheduling overrides by default on the Deployment + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + asserts: + - notExists: + path: spec.template.spec.priorityClassName + - notExists: + path: spec.template.spec.topologySpreadConstraints + + # priorityClassName — shared pod template, so both workload shapes + - it: omits priorityClassName by default on the StatefulSet + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.priorityClassName + + - it: sets priorityClassName on the StatefulSet when configured + template: templates/statefulset.yaml + set: + priorityClassName: system-cluster-critical + asserts: + - equal: + path: spec.template.spec.priorityClassName + value: system-cluster-critical + - contains: + path: spec.template.spec.volumes + content: + name: gateway-state + emptyDir: {} + + - it: sets priorityClassName on the Deployment when configured + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + priorityClassName: system-cluster-critical + asserts: + - equal: + path: spec.template.spec.priorityClassName + value: system-cluster-critical + + # topologySpreadConstraints — shared pod template, so both workload shapes + - it: omits topologySpreadConstraints by default on the StatefulSet + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.topologySpreadConstraints + + - it: sets topologySpreadConstraints on the StatefulSet when configured + template: templates/statefulset.yaml + set: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell + asserts: + - equal: + path: spec.template.spec.topologySpreadConstraints[0].maxSkew + value: 1 + - equal: + path: spec.template.spec.topologySpreadConstraints[0].topologyKey + value: topology.kubernetes.io/zone + - equal: + path: spec.template.spec.topologySpreadConstraints[0].whenUnsatisfiable + value: DoNotSchedule + - equal: + path: spec.template.spec.topologySpreadConstraints[0].labelSelector.matchLabels + value: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell + + - it: sets topologySpreadConstraints on the Deployment when configured + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell + asserts: + - equal: + path: spec.template.spec.topologySpreadConstraints[0].topologyKey + value: kubernetes.io/hostname diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 0bbb2cd6b2..987744afbb 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -389,6 +389,15 @@ tolerations: [] # -- Affinity rules for the gateway pod. affinity: {} +# -- PriorityClass for the gateway pod. The gateway is a control-plane +# component; when it shares nodes with the workloads it serves, the default +# priority gives it no advantage under node pressure. Empty = omit the field. +priorityClassName: "" + +# -- Topology spread constraints for the gateway pod, used to spread replicas +# across zones or nodes. Empty = omit the field. +topologySpreadConstraints: [] + # -- Non-secret gateway application configuration. Top-level keys name TOML # tables and are rendered into the mounted gateway.toml file. Kubernetes # resource inputs remain outside this map; template expressions derive the diff --git a/python/openshell/rpm_gateway_config_migration_test.py b/deploy/rpm/migrate_gateway_config_test.py similarity index 100% rename from python/openshell/rpm_gateway_config_migration_test.py rename to deploy/rpm/migrate_gateway_config_test.py diff --git a/docs/CONTRIBUTING.mdx b/docs/CONTRIBUTING.mdx index bd3e66350f..155770b5a2 100644 --- a/docs/CONTRIBUTING.mdx +++ b/docs/CONTRIBUTING.mdx @@ -1,10 +1,10 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Contributing to NVIDIA OpenShell Documentation" +title: "Contributing to OpenShell Documentation" description: "" --- -This guide covers how to write, edit, and review documentation for NVIDIA OpenShell. If you change code that affects user-facing behavior, update the relevant docs in the same PR. +This guide covers how to write, edit, and review documentation for OpenShell. If you change code that affects user-facing behavior, update the relevant docs in the same PR. The published docs live in `docs/`, navigation is defined in `docs/index.yml`, and `fern/` contains the site config, components, and theme assets. diff --git a/docs/about/architecture.mdx b/docs/about/architecture.mdx index a63e5257ed..320ed9d700 100644 --- a/docs/about/architecture.mdx +++ b/docs/about/architecture.mdx @@ -12,7 +12,7 @@ position: 2 OpenShell governs what agents can do in two ways: it instruments the kernel to enforce policy on every file access, system call, and network connection at runtime, and it uses formal verification to check what a policy change would allow before it is applied. -- **Kernel-level enforcement.** Each agent runs in an isolated sandbox. Kernel controls confine which files it can access and which system calls it can make, and every network connection passes through a policy check before it leaves the sandbox. Agents never see real credentials; OpenShell adds them only to requests bound for approved endpoints. +- **Kernel-level enforcement.** Each agent runs in an isolated sandbox. Kernel controls confine which files it can access and which system calls it can make, and every network connection passes through a policy check before it leaves the sandbox. OpenShell supplies credential placeholders to agents and adds real credentials only to requests bound for approved endpoints. - **Formally verified policy changes.** Before a policy change is approved, OpenShell uses formal verification to flag risky new access it would grant, such as reaching a new host with credentials or calling a new API method, so those changes wait for human review. The **gateway** is the control plane: it manages the lifecycle of sandboxes and @@ -55,7 +55,7 @@ trusted process observations. ### Mediated Channel The **supervisor** and **sandbox** talk over the OpenShell Sandbox Protocol: -one mutually authenticated HTTP/2 connection that carries many independent +one HTTP/2 connection authenticated with TLS and a sandbox JWT that carries many independent streams. The compute driver picks the transport: a Unix socket for Docker and Podman, TCP for Kubernetes, or vsock for MicroVM. Authentication and protocol behavior are the same on all of them. The **supervisor** presents a @@ -112,7 +112,7 @@ network. |---|---|---|---| | Docker | Its own container | Authenticated Unix socket on a driver-owned volume | Workload container has networking turned off | | Podman | Its own container | Authenticated Unix socket on a driver-owned volume | Workload container has networking turned off | -| Kubernetes | Its own pod | Private service with mutual TLS | NetworkPolicy allows only the **supervisor** service | +| Kubernetes | Its own pod | Private service with server-authenticated TLS and a sandbox JWT | NetworkPolicy allows only the **supervisor** service | | VM | A process on the host | Authenticated vsock | Guest has no network device | The runtime's job is to build the boundary and prove it's in place. It never @@ -126,15 +126,15 @@ do. ## How Components Authenticate -Three connections tie a sandbox together. The gateway is the only component -that signs credentials, and every credential names exactly one sandbox. +Three connections tie a sandbox session together. The gateway signs the +session's gateway and sandbox JWTs, each bound to that sandbox and generation. -![OpenShell sandbox authentication showing the compute driver giving the supervisor a bootstrap credential, the gateway issuing a gateway JWT and sandbox JWT, and the supervisor presenting the sandbox JWT over mutual TLS to the sandbox, which holds only the gateway's public key.](../images/openshell-sandbox-authentication.svg) +![OpenShell sandbox authentication showing the gateway preissuing session JWTs, delivery to dedicated supervisors through the compute driver or to shared supervisors through a host-JWT-authenticated assignment stream, and the supervisor presenting the sandbox JWT over server-authenticated TLS to the sandbox.](../images/openshell-sandbox-authentication.svg) | Connection | Who connects | How it's protected | |---|---|---| | **Supervisor** to gateway | The **supervisor** dials out to the gateway. | A gateway JWT, over TLS when the gateway has TLS enabled. | -| **Supervisor** to **sandbox** | The **supervisor** dials into the workload over the driver's private channel. | Mutual TLS, plus a sandbox JWT. | +| **Supervisor** to **sandbox** | The **supervisor** acts as the TLS and gRPC client over the driver's private channel. The runtime can initiate the underlying transport connection. | TLS authenticates the **sandbox** server; the sandbox JWT authenticates the **supervisor**. No TLS client certificate is used. | | Agent to **supervisor** | The agent never connects directly. The **sandbox** relays its traffic over the connection above. | Covered by the supervisor-to-sandbox channel. | The **supervisor** verifies the gateway's TLS certificate with its CA. User @@ -143,27 +143,32 @@ workload; the gateway JWT authenticates supervisor RPCs. ### Getting the first credential -The **supervisor** needs a starting credential to prove which sandbox it belongs -to. How it gets one depends on the runtime: +The gateway prepares session credentials for each sandbox generation and +delivers them through trusted launch provisioning: -- **Docker, Podman, and MicroVM.** The driver hands the **supervisor** its initial - tokens directly, in files only the **supervisor** can read. -- **Kubernetes.** The **supervisor** presents its pod's ServiceAccount token. The - gateway asks the Kubernetes driver to verify the token and confirm that the - pod belongs to the expected sandbox before it issues any JWTs. +- **Docker, Podman, and MicroVM.** The driver supplies the dedicated + **supervisor** with private credential files. +- **Kubernetes, dedicated supervisor.** The driver mounts the preissued + credential bundle from a Secret into the **supervisor** pod. +- **Kubernetes, shared supervisor.** Each sandbox's credential bundle arrives + in its assignment over the authenticated supervisor-host stream. -Either way, the gateway checks the claim against its own record of the sandbox -before returning credentials. +A separate `IssueSandboxToken` exchange accepts a compute-driver bootstrap +identity. On Kubernetes, the driver validates a projected ServiceAccount token +and its pod and owner identity. The gateway compares the driver and runtime +identity with its stored sandbox record before returning a gateway JWT. This +exchange is not required for supervisors provisioned with a credential bundle. ### Gateway and Sandbox JWTs The gateway issues a pair of JWTs for each run of a sandbox: -- **Gateway JWT.** Sent with every **supervisor** call to the gateway. It allows - only the calls a **supervisor** needs, such as fetching policy, pushing logs, and +- **Gateway JWT.** Authenticates the **supervisor**'s sandbox-specific calls to + the gateway. It allows only the calls a **supervisor** needs, such as fetching policy, pushing logs, and relaying sessions. It is not a user credential and can't manage other sandboxes. -- **Sandbox JWT.** Sent with every **supervisor** call to the **sandbox**. +- **Sandbox JWT.** Authenticates boundary opening and the **supervisor**'s control + and mediation streams to the **sandbox**. The **sandbox** holds only the gateway's public key, so it can verify the token but can never create one. @@ -172,11 +177,17 @@ one run of that sandbox, called a generation. Restarting a sandbox starts a new generation with fresh tokens and fresh TLS certificates, and the old ones stop working. +Shared Kubernetes supervisors also use a separate, gateway-scoped supervisor-host +JWT to receive assignments. The pool's replicas share this token, which +authorizes only the host assignment stream. The gateway checks assignment +ownership separately, and each assigned sandbox still uses its own session JWTs. +See [Multi-tenant Supervisor](/kubernetes/high-availability#experimental-multi-tenant-supervisor). + ### Renewing and revoking -The **supervisor** keeps its tokens in memory and renews both together before they -expire. Renewal works only while the sandbox still exists, so deleting a -sandbox cuts off its **supervisor**. +The **supervisor** keeps its session tokens in memory and renews both together +before they expire. Renewal works only while the sandbox still exists, so +deleting a sandbox ends that sandbox's session. Shared deployments, such as Kubernetes, should set `gateway_jwt.ttl_secs` so tokens expire. Local single-user gateways can leave it unset, which issues @@ -189,9 +200,15 @@ the **sandbox** holds nothing worth stealing: no gateway signing key, no gateway JWT, and no provider credentials. It can verify that it's talking to the right **supervisor**, but it can't impersonate one. -If the **supervisor** disconnects, the **sandbox** freezes the agent. Only the -same **supervisor** process can reconnect and resume it. A new **supervisor** can't -take over a running sandbox, even with valid credentials. +If the **supervisor** disconnects, the **sandbox** freezes the agent. The same +**supervisor** process can reconnect and resume it within the recovery window. +In a [shared Kubernetes supervisor pool](/kubernetes/high-availability#recover-from-a-supervisor-failure), +the gateway can authorize a replacement **supervisor** with a signed takeover +grant for that sandbox generation and a higher ownership epoch. When the +replacement confirms the boundary, a runtime that supports takeover revokes the +previous owner's access and resumes the agent. Valid session credentials alone do not +authorize takeover. If recovery does not complete before the deadline, the +runtime terminates the workload. ## Working With Your Existing Infrastructure diff --git a/docs/about/overview.mdx b/docs/about/overview.mdx index 011bb6d55b..d77febbf27 100644 --- a/docs/about/overview.mdx +++ b/docs/about/overview.mdx @@ -1,14 +1,14 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Overview of NVIDIA OpenShell" +title: "Overview of OpenShell" sidebar-title: "Overview" description: "OpenShell is the safe, private runtime for fleets of autonomous AI agents. Run agents in sandboxed environments that protect your data, credentials, and infrastructure." keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Provider Credentials" position: 1 --- -NVIDIA OpenShell is an open-source runtime for executing fleets of autonomous AI agents in sandboxed environments with kernel-level isolation. It combines sandbox runtime controls and a declarative YAML policy so teams can run agents without giving them unrestricted access to local files, credentials, and external networks. +OpenShell is an open-source runtime for executing fleets of autonomous AI agents in sandboxed environments with kernel-level isolation. It combines sandbox runtime controls and a declarative YAML policy so teams can run agents without giving them unrestricted access to local files, credentials, and external networks. New in OpenShell 0.1.x: a [stable release cadence](/about/support-matrix#releases), new [isolation primitives](/about/architecture), and an expanded [extension surface](/extensibility/overview), along with much more. diff --git a/docs/extensibility/gateway-interceptors.mdx b/docs/extensibility/gateway-interceptors.mdx index 20600fabf6..459661c235 100644 --- a/docs/extensibility/gateway-interceptors.mdx +++ b/docs/extensibility/gateway-interceptors.mdx @@ -105,6 +105,8 @@ Registration is static. Restart the gateway after adding, removing, or changing Start with the invariant the interceptor must preserve, then identify every gateway RPC that can establish or weaken it. For example, an interceptor that owns sandbox policy can use `modify_operation` on `CreateSandbox` to apply an approved initial policy and `validate` on `UpdateConfig` to reject unauthorized changes. +To gate sandbox activation, bind `validate` to `CreateSandbox`, `StartSandbox`, and `ResumeSandbox`. Bind `post_commit` to `StopSandbox` and `SuspendSandbox` to observe successful stop and suspend requests. + The gateway maintains the canonical [interceptable route allowlist](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-gateway-interceptors/src/routes.rs). Read the interceptor manifest and gateway startup diagnostics when selecting bindings. Unknown, streaming, read-only, and non-allowlisted RPCs cannot be intercepted. Choose a phase by intent: diff --git a/docs/how-it-works/gateways/authentication.mdx b/docs/how-it-works/gateways/authentication.mdx index 232cd0f5a0..84579be562 100644 --- a/docs/how-it-works/gateways/authentication.mdx +++ b/docs/how-it-works/gateways/authentication.mdx @@ -190,7 +190,7 @@ openshell gateway add https://gateway.example.com \ --oidc-audience openshell-cli ``` -When you register or log in to an OIDC gateway, the CLI uses the Authorization Code flow with PKCE. It opens a browser, receives the authorization code on a localhost callback, exchanges the code for tokens, and stores the token bundle under the gateway credential directory. After `openshell gateway logout`, the next browser login asks the identity provider for a fresh login prompt so you can choose a different browser user instead of silently reusing the previous session. If `OPENSHELL_OIDC_CLIENT_SECRET` is set, the CLI uses the client credentials flow instead. Use that mode for CI and other non-interactive automation. +When you register or log in to an OIDC gateway, the CLI uses the Authorization Code flow with PKCE. It opens a browser, receives the authorization code on a localhost callback, exchanges the code for tokens, and stores the token bundle under the gateway credential directory. After `openshell gateway logout`, the next browser login asks the identity provider for a fresh login prompt so you can choose a different browser user instead of silently reusing the previous session. If `OPENSHELL_OIDC_CLIENT_SECRET` is set, the CLI uses the client credentials flow instead. Use that mode for CI and other non-interactive automation. The CLI renews an expired client credentials token by repeating the grant, so keep the variable set for as long as the commands run. Official Python, TypeScript, and Go SDKs can perform renewable client-credentials authentication directly. Configure the service account at the identity provider diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 424efa6016..408ac0977a 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -105,6 +105,27 @@ version = 2 The gateway selector is `compute_driver = ""`. It accepts one scalar driver name. Omit it to retain auto-detection. +## Slow sandbox startup + +For slow isolation backends, increase these startup windows. Each defaults to +`300` seconds (5 minutes) and accepts `1` through `86400` seconds. Environment +variables override TOML values. + +| Setting | Scope | Environment variable | +|---|---|---| +| `provisioning_repair_window_seconds` | `[openshell.gateway]`: configuration repair after admission starts | `OPENSHELL_PROVISIONING_REPAIR_WINDOW_SECONDS` | +| `runtime_bootstrap_grace_seconds` | `[openshell.drivers.kubernetes]`: sandbox startup | `OPENSHELL_RUNTIME_BOOTSTRAP_GRACE_SECONDS` | +| `sandbox_attach_request_timeout_seconds` | `[openshell.drivers.kubernetes]`: dedicated supervisor startup | `OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS` | + +For a 15-minute Kubernetes startup window, set both driver settings to `900`. +On shared supervisor deployments, set `OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS=900` +directly; the driver setting does not configure shared hosts. Directly launched +supervisors also accept `--attach-request-timeout-seconds`. + +Restart affected gateways and supervisors to apply changes. Existing admission +deadlines stay unchanged. After a timeout, wait for cleanup before starting the +sandbox again. CLI wait and image-preparation timeouts are separate settings. + ## Additional Destination CAs Use the global supervisor network table to trust private certificate authorities @@ -278,10 +299,18 @@ credential_drivers = ["kubernetes-secrets"] ssh_session_ttl_secs = 3600 +# Per-sandbox caps on concurrent relays. SSH sessions and service forwards +# have separate budgets. Each defaults to 20 and must be at least 1. +max_ssh_connections_per_sandbox = 20 +max_service_connections_per_sandbox = 20 + # Absolute image preparation and initial supervisor startup budget. # Applies to new attempts only; allowed values are 1 through 86400 seconds. image_preparation_timeout_seconds = 1800 +# Admission repair window in seconds; existing deadlines are preserved. +provisioning_repair_window_seconds = 300 + # Reject invalid policy generations securely by default. Set # "retain_last_valid" only when availability takes priority. policy_validation_failure_mode = "fail_closed" @@ -610,7 +639,7 @@ that the previous process has stopped, not that sharing a SQLite file enables HA ## Database Pool -`[openshell.gateway] database_max_connections` sets the connection ceiling for the gateway's database pool. One pool is shared by every database-backed RPC, so this value is also the gateway's ceiling on concurrent database work. When the pool is saturated, callers queue on connection acquisition and sqlx logs that the time to acquire a connection exceeded its slow threshold. The background readiness monitor draws from the same pool, so sustained saturation can push its database ping past the check timeout and flip `/readyz` to not-ready while the gateway is otherwise serving. +`[openshell.gateway] database_max_connections` sets the connection ceiling for the gateway's database pool. One pool is shared by every database-backed RPC, so this value is also the gateway's ceiling on concurrent database work. When the pool is saturated, callers queue on connection acquisition and sqlx logs that the time to acquire a connection exceeded its slow threshold. The background readiness monitor draws from the same pool, so sustained saturation can push its database ping past the check timeout and flip `/readyz` to not-ready while the gateway is otherwise serving. On Postgres, a caller that gets no connection within 5 seconds fails instead of waiting for the database to answer, usually with a retryable `UNAVAILABLE` error whose reason is `DATABASE_UNAVAILABLE`. Omit the key to use the backend default, which is unchanged from earlier releases: @@ -1277,6 +1306,9 @@ service_account_name = "openshell-sandbox" default_image = "nvcr.io/nvidia/base/ubuntu:24.04" image_pull_policy = "if_not_present" image_pull_secrets = ["regcred"] +# Startup windows in seconds; defaults are 300. +# runtime_bootstrap_grace_seconds = 900 +# sandbox_attach_request_timeout_seconds = 900 # Defaults to the gateway version; override to pin a specific build. # sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" sandbox_runtime_image_pull_policy = "if_not_present" @@ -1338,6 +1370,9 @@ workspace_default_storage_size = "10Gi" # Kubernetes RuntimeClass applied to sandbox pods when the API request does # not specify one. Empty (default) = omit the field, using the cluster default. # default_runtime_class_name = "kata-containers" +# Kubernetes RuntimeClass for dedicated supervisor pods. Only the administrator +# sets it. It never inherits the workload class. Empty (default) = omit. +# supervisor_runtime_class_name = "kata-supervisor" # Pod annotation keys that sandbox templates may set. Each entry is an exact # key or a `/*` wildcard. Empty (default) rejects every template # annotation. Keys under openshell.ai (OpenShell sandbox identity) and @@ -1836,6 +1871,8 @@ Without `--path`, the command validates a nonempty `OPENSHELL_GATEWAY_CONFIG`. O Config preflight validates configuration without loading launch-signing keys, so a successful result does not confirm that the key files exist or form a usable pair. Gateway startup validates configured or discovered signing bundles. Sandbox creation checks the selected driver's launch-signing requirement before image preparation. +When `allow_unauthenticated_users = true`, preflight prints a warning to stderr and still exits 0. + To validate the exact daemon arguments that a wrapper will pass, place them after `--` instead of using `--path`: diff --git a/docs/how-it-works/gateways/overview.mdx b/docs/how-it-works/gateways/overview.mdx index 428679fc2a..093f73517a 100644 --- a/docs/how-it-works/gateways/overview.mdx +++ b/docs/how-it-works/gateways/overview.mdx @@ -128,6 +128,15 @@ passed the gateway authentication layer. If authentication fails while status remains connected, run `openshell gateway login ` to refresh the stored credentials. +When you select a gateway by name with `--gateway ` or +`OPENSHELL_GATEWAY`, `status` exits nonzero if that name is not registered. +With no gateway configured and no selector, `status` exits 0 and prints a +registration hint. + +Exit 0 alone does not guarantee a successful health check. A registered gateway +can return 0 even when its health endpoint reports an error. Check the `status` +field in structured output to determine reachability. + For automation or scripting, use `--output json` or `--output yaml` to get machine-readable output: ```shell diff --git a/docs/how-it-works/inference.mdx b/docs/how-it-works/inference.mdx index db2335320a..d9fe96177f 100644 --- a/docs/how-it-works/inference.mdx +++ b/docs/how-it-works/inference.mdx @@ -192,7 +192,8 @@ response = client.chat.completions.create( For an authenticated alternate endpoint, declare a credential in the custom profile, bind it to that endpoint, and create the provider from the original -credential source. OpenShell never exports stored credential values. +credential source. The workload receives an opaque placeholder. Separately, an +explicitly enabled [mTLS operator can retrieve runtime credential values](/how-it-works/providers/overview#retrieve-runtime-credentials). ## Verify Access @@ -264,8 +265,8 @@ openshell provider create \ openshell sandbox provider attach inference-demo nvidia-native-prod --wait ``` -OpenShell does not export stored credential values or change a provider's -profile type in place. If the original credential source is unavailable, import +Provider inspection commands do not export stored credential values, and you +cannot change a provider's profile type in place. If the original credential source is unavailable, import the original profile at its own ID so the preserved provider keeps resolving while you arrange credential rotation into the replacement provider. @@ -308,8 +309,8 @@ Special cases require additional work: ## Security Differences -Provider attachment preserves credential non-disclosure: workloads receive -opaque placeholders, and the proxy substitutes a real credential only after +Provider attachment delivers opaque placeholders to workloads, and the proxy +substitutes a real credential only after network policy and endpoint binding both pass. Native requests are no longer filtered or rewritten by a model-specific router. The provider profile's L7 rules therefore define the allowed API surface, and the workload controls diff --git a/docs/how-it-works/policies/default-policy.mdx b/docs/how-it-works/policies/default-policy.mdx index 3f148dcf70..4c9ab29ff8 100644 --- a/docs/how-it-works/policies/default-policy.mdx +++ b/docs/how-it-works/policies/default-policy.mdx @@ -93,12 +93,16 @@ additional paths when the corresponding GPU device is present: | Access | Paths | |---|---| | Read-only | `/run/nvidia-persistenced`, `/usr/lib/wsl` | -| Read-write | `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools`, `/dev/nvidia-modeset`, `/dev/dxg`, numbered `/dev/nvidia` device nodes, and `/proc` | +| Read-write | `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools`, `/dev/nvidia-modeset`, `/dev/dxg`, numbered `/dev/nvidia` device nodes | -CUDA writes thread names under `/proc` during initialization, so GPU enrichment -moves `/proc` from read-only to read-write. OpenShell adds each path only when -it exists in the workload. These paths apply at runtime and are not saved in -the sandbox's policy. +OpenShell adds each path only when it exists in the workload. These paths apply +at runtime and are not saved in the sandbox's policy. GPU enrichment does not +change the filesystem policy's access to `/proc`. + +CUDA can write thread names during initialization through the runtime's narrow +`/proc` exception. The runtime brokers supported `comm` writes only for threads +in the calling process's own thread group. This exception also applies without +a GPU and does not grant general write access to `/proc`. ### Protected Paths diff --git a/docs/how-it-works/policies/manage-policies.mdx b/docs/how-it-works/policies/manage-policies.mdx index c20af7b67b..e24cf05f22 100644 --- a/docs/how-it-works/policies/manage-policies.mdx +++ b/docs/how-it-works/policies/manage-policies.mdx @@ -368,11 +368,11 @@ openshell sandbox get my-sandbox --output json The same repair window applies when the gateway refuses an image-policy upload or a policy update that adds baseline filesystem paths with `FAILED_PRECONDITION` or `INVALID_ARGUMENT`. After the gateway acknowledges the rejection report, startup waits two seconds, reads a fresh configuration, and retries. If your repair reaches the gateway before that report, startup immediately reads the repaired configuration. The sandbox log records the gateway's reason; repeated refusals of the same write, status code, and configuration are logged once. Authentication and authorization failures still stop startup. -You have 300 seconds to fix the configuration after the supervisor starts admission. Image preparation has its own deadline and does not consume that repair time. Replace the policy with `openshell policy set`, or fix the provider configuration. Until the workload first starts, you can also change filesystem, Landlock, and process settings. Each effective change to the policy, providers, or settings restarts the 300 seconds; writing an unchanged value does not. +You have the configured repair window (300 seconds by default) to fix the configuration after the supervisor starts admission. Image preparation has its own deadline and does not consume that repair time. Replace the policy with `openshell policy set`, or fix the provider configuration. Until the workload first starts, you can also change filesystem, Landlock, and process settings. Each effective change to the policy, providers, or settings restarts the repair window; writing an unchanged value does not. If the time runs out, the sandbox moves to `Error` with the reason `ProvisioningTimedOut`. Fixing the configuration does not restart it. After you -fix it, start the sandbox again. Admission gets a new 300-second window after preparation: +fix it, start the sandbox again. Admission gets a new repair window after preparation: ```shell openshell sandbox start my-sandbox diff --git a/docs/how-it-works/policies/network-rules.mdx b/docs/how-it-works/policies/network-rules.mdx index d7dec792a6..119818a395 100644 --- a/docs/how-it-works/policies/network-rules.mdx +++ b/docs/how-it-works/policies/network-rules.mdx @@ -343,6 +343,10 @@ request, which the read-only preset denies. Run `npm install --no-audit`, or replace `access: read-only` with explicit rules that also allow the audit's `POST` requests. An endpoint cannot combine `access` and `rules`. +To set the option on a live sandbox without replacing the policy, add +`allow-encoded-slash` to the `--add-endpoint` options segment of +`openshell policy update`, which requires a `rest` or `websocket` endpoint. + Verify that a scoped package lookup succeeds: ```shell diff --git a/docs/how-it-works/providers/aws.mdx b/docs/how-it-works/providers/aws.mdx index 087161faa9..86cf7dde7a 100644 --- a/docs/how-it-works/providers/aws.mdx +++ b/docs/how-it-works/providers/aws.mdx @@ -3,11 +3,11 @@ # SPDX-License-Identifier: Apache-2.0 title: "AWS SigV4 Credential Signing" sidebar-title: "AWS" -description: "Configure proxy-side AWS SigV4 request signing so sandbox agents can reach AWS services through CONNECT tunnels without holding real credentials." +description: "Configure proxy-side AWS SigV4 request signing so sandbox agents can reach AWS services through CONNECT tunnels using placeholder credentials." keywords: "Generative AI, Cybersecurity, AI Agents, AWS, SigV4, Bedrock, S3, Credential Signing, Sandbox" --- -AWS SigV4 credential signing lets sandbox agents call AWS services (Bedrock, S3, STS, and others) through the proxy's CONNECT tunnel. The proxy intercepts outbound requests, strips the sandbox client's placeholder `Authorization` header, and re-signs the request with real AWS credentials from the provider. The sandbox never sees the real credentials. +AWS SigV4 credential signing lets sandbox agents call AWS services (Bedrock, S3, STS, and others) through the proxy's CONNECT tunnel. The proxy intercepts outbound requests, strips the sandbox client's placeholder `Authorization` header, and re-signs the request with real AWS credentials from the provider. The sandbox uses placeholder credentials; signing with the provider's credentials happens in the proxy. See [How Credential Injection Works](/how-it-works/providers/overview#how-credential-injection-works) for credential delivery and response-handling limits. ## Prerequisites @@ -166,10 +166,11 @@ endpoints: - OpenShell rejects a signed sandbox policy before activation unless the endpoint has a resolvable AWS credential source. An endpoint-bearing profile must declare `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` and cover the signed host, port, and path. An endpointless profile must declare those keys and be selected with `credential_binding.provider` on the signed endpoint. - The `sigv4:body` mode buffers at most 10 MiB. Requests with larger bodies are rejected. Use `sigv4:no_body` or `sigv4` (auto-detect) for large payloads. - The active provider must contain current `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` values. If either becomes unavailable after policy activation, the request fails closed. +- AWS signer diagnostics are omitted from supervisor logs, including at `TRACE` level, because they can contain session tokens. ## Use from a Sandbox -Inside a sandbox, configure the AWS SDK with placeholder credentials. The proxy replaces them with real credentials during re-signing: +Inside a sandbox, configure the AWS SDK with placeholder credentials. The proxy uses the provider's real credentials to re-sign requests: ```shell export AWS_ACCESS_KEY_ID=placeholder diff --git a/docs/how-it-works/providers/overview.mdx b/docs/how-it-works/providers/overview.mdx index aba6269546..5cb1f82e94 100644 --- a/docs/how-it-works/providers/overview.mdx +++ b/docs/how-it-works/providers/overview.mdx @@ -228,7 +228,16 @@ The update records which sandboxes are attached before saving the new credential Credential refresh status tells you whether OpenShell obtained credentials. [Provider readiness](/how-it-works/providers/profiles#inspect-provider-readiness) tells you whether the sandbox applied the credentials, policy, and environment for new processes. It does not test access to a backend model or cancel requests already sent upstream. -After a static credential update completes, launch a new client process to use the updated reference. An existing process keeps its revision-scoped reference; readiness does not make that old reference resolve the new value. Acknowledged detach revokes retained references and removes them from environments for future launches. +After an ordinary static credential update completes, launch a new client +process to use the updated reference. Existing processes keep their environment +references; readiness does not guarantee that they use the replacement value. +Credentials configured with +[`stable_placeholder: true`](/how-it-works/providers/profiles#keep-an-externally-updated-placeholder-stable) +let processes already holding a stable reference use a value-only update after +the supervisor installs it. See +[Runtime Limitations](/how-it-works/providers/profiles#runtime-limitations) for +retained revision behavior. Acknowledged detach revokes retained references and +removes them from environments for future launches. Set or clear a credential expiry timestamp: @@ -437,10 +446,13 @@ state. ## How Credential Injection Works -The agent process inside the sandbox never sees real credential values. At -startup, OpenShell replaces each credential with an opaque placeholder token in +At startup, OpenShell supplies each eligible provider credential as an opaque placeholder token in the agent's environment. When the agent sends an HTTP request containing a placeholder, the proxy resolves it immediately before forwarding the request. +Credential injection does not automatically scrub credential values from +upstream responses, so send credentialed requests only to trusted endpoints. +Operator retrieval is a separate export path described in +[Retrieve Runtime Credentials](#retrieve-runtime-credentials). Static credential resolution has three independent authorization boundaries: diff --git a/docs/how-it-works/providers/profiles.mdx b/docs/how-it-works/providers/profiles.mdx index 359b59c486..94d668751d 100644 --- a/docs/how-it-works/providers/profiles.mdx +++ b/docs/how-it-works/providers/profiles.mdx @@ -1231,9 +1231,23 @@ Running sandboxes periodically check for provider and policy changes. Use `--wai The policy effect applies to future effective policy reads after the sandbox observes the update. The credential environment effect applies only to new process launches after the update is observed, such as later SSH, exec, or SFTP sessions. -Already-running processes keep the placeholder environment they started with. For ordinary static credentials, an existing reference retains its selected revision after a value update; readiness does not retarget that reference to the replacement value. After an update wait succeeds, launch a new process to receive the updated credential reference. Gateway-managed refresh follows its own reference lifecycle. Expiry, endpoint authorization, and acknowledged detach continue to apply when the proxy resolves retained references. - -For a static provider, the sequence is attach, wait, then launch client A; update, wait, then launch client B. B's first request uses the newly installed credential. Readiness does not establish that A has stopped using its retained revision or that an old upstream key can be retired. +Already-running processes keep the placeholder environment they started with. +For ordinary static credentials, a retained revision can continue to resolve its +previous value. After that revision leaves retained history, its reference can +fall back to the current value only while it remains authorized for the same +provider identity epoch. A revision-scoped reference is therefore not a +permanent pin to one credential value. After an update wait succeeds, launch a +new process to receive the updated credential reference. Expiry, endpoint +authorization, and acknowledged detach continue to apply to retained references. + +For credentials configured with +[`stable_placeholder: true`](#keep-an-externally-updated-placeholder-stable), a +process already holding a stable reference uses the current value after the +supervisor installs a value-only update. Enabling the setting does not convert +references in an existing process's environment. Gateway-managed refresh +follows its own reference lifecycle. + +For an ordinary static provider, the sequence is attach, wait, then launch client A; update, wait, then launch client B. B's first request uses the newly installed credential. Readiness does not establish that A has stopped using the previous credential value or that an old upstream key can be retired. An acknowledged detachment removes its provider policy layer from the active effective policy, revokes future resolution for its existing placeholders, and removes its credential placeholders from future process environments. It does not remove strings from already-running process environments or undo requests already forwarded upstream. diff --git a/docs/how-it-works/sandboxes/overview.mdx b/docs/how-it-works/sandboxes/overview.mdx index 2ac6a4f69d..6365f9e684 100644 --- a/docs/how-it-works/sandboxes/overview.mdx +++ b/docs/how-it-works/sandboxes/overview.mdx @@ -1171,7 +1171,7 @@ stopped sandbox repeats configuration admission before launching its workload. Image preparation and initial supervisor startup have an absolute deadline of 1800 seconds by default, independently of the CLI wait timeout. Operators can set `image_preparation_timeout_seconds` in `[openshell.gateway]` to any value from 1 through 86400. Each new create or explicit start stores its deadline. Progress events, configuration edits, and gateway or driver restarts cannot extend it. Changing the gateway setting affects new attempts only. -The first authenticated configuration report from the current supervisor ends preparation and starts a separate 300-second admission repair window. A slow image download therefore does not consume time reserved for fixing policy. During admission, an effective policy, settings, provider, profile, or attachment change resets the window from its stored change time. The first failed configuration load for that change grants another full window. Repeated failures, duplicate reports, and reconnects do not extend it; reaching `Ready` clears it. +The first authenticated configuration report from the current supervisor ends preparation and starts a separate admission repair window (300 seconds by default). A slow image download therefore does not consume time reserved for fixing policy. During admission, an effective policy, settings, provider, profile, or attachment change resets the window from its stored change time. The first failed configuration load for that change grants another full window. Repeated failures, duplicate reports, and reconnects do not extend it; reaching `Ready` clears it. Preparation expiry sets the sandbox to `Error` with reason `ImagePreparationTimedOut`; admission repair expiry uses `ProvisioningTimedOut`. The gateway stops its workload and supervisor compute, retaining the sandbox record, diagnostic, and restartable storage. Cleanup can remain pending if the backend is unavailable; the gateway retries it. Inspect `provisioning` in JSON output for the active `phase`, `deadline`, original `preparation_deadline`, `admission_start_time`, and cleanup timestamps. TUI NOTES identifies preparation timeout and distinguishes pending cleanup from reclaimed compute. diff --git a/docs/images/openshell-sandbox-authentication.svg b/docs/images/openshell-sandbox-authentication.svg index ecebf17c27..6d6a460570 100644 --- a/docs/images/openshell-sandbox-authentication.svg +++ b/docs/images/openshell-sandbox-authentication.svg @@ -3,7 +3,7 @@ OpenShell sandbox authentication - The compute driver gives the supervisor a bootstrap credential. The supervisor presents it to the gateway, which issues a gateway JWT and a sandbox JWT. The supervisor uses the gateway JWT with the gateway and the sandbox JWT over mutual TLS with the OpenShell Sandbox, which holds only the gateway's public key. + The gateway preissues a gateway JWT and a sandbox JWT for each sandbox generation. Dedicated supervisors receive the credential bundle through the compute driver; shared supervisors receive it in an assignment over a host stream authenticated with a separate supervisor-host JWT. The supervisor uses the gateway JWT for sandbox-specific gateway RPCs, verifies the sandbox's TLS server certificate, and presents the sandbox JWT to authenticate itself. The sandbox holds only the gateway's public key for JWT verification. @@ -37,12 +37,12 @@ API Server - signs every JWT + signs the JWT pair checks the sandbox record Compute driver - delivers or verifies the - bootstrap credential + delivers the dedicated + supervisor credential bundle @@ -64,22 +64,27 @@ Fresh TLS certs per run Agent - sees no credentials + receives placeholders + + PREISSUED + JWT PAIR + - 1 · BOOTSTRAP + JWT PAIR - 2 · AUTHENTICATES + GATEWAY JWT - 3 · ISSUES JWT PAIR + SHARED ASSIGNMENT + HOST JWT STREAM - 4 · SANDBOX JWT - MUTUAL TLS + SANDBOX JWT + SERVER TLS - Every JWT names one sandbox and one run of it. Restarting a sandbox issues new tokens and TLS certificates. + Each session JWT names one sandbox and one run of it. Restarting a sandbox issues new tokens and TLS certificates. diff --git a/docs/index.mdx b/docs/index.mdx index 45f99288a6..5846c828e6 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -1,7 +1,7 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "NVIDIA OpenShell Developer Guide" +title: "OpenShell Developer Guide" description: "OpenShell is the safe, private runtime for fleets of autonomous AI agents. Run agents in sandboxed environments that protect your data, credentials, and infrastructure." keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Provider Credentials" position: 1 @@ -30,7 +30,7 @@ import { CommandTerminal } from "./_components/CommandTerminal"; ]} /> -NVIDIA OpenShell is the safe, private runtime for fleets of autonomous AI agents. It provides sandboxed execution environments +OpenShell is the safe, private runtime for fleets of autonomous AI agents. It provides sandboxed execution environments that protect your data, credentials, and infrastructure. Agents run with exactly the permissions they need and nothing more, governed by declarative policies that prevent unauthorized file access, data exfiltration, and uncontrolled network activity. diff --git a/docs/index.yml b/docs/index.yml index 596ac87172..9f6b4084eb 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -6,7 +6,7 @@ landing-page: path: index.mdx navigation: -- section: "About NVIDIA OpenShell" +- section: "About OpenShell" slug: about contents: - page: "Overview" diff --git a/docs/kubernetes/access-control.mdx b/docs/kubernetes/access-control.mdx index 70e5f7d32a..dbc5e99e48 100644 --- a/docs/kubernetes/access-control.mdx +++ b/docs/kubernetes/access-control.mdx @@ -22,9 +22,19 @@ For how the CLI resolves gateways and stores credentials, refer to [Gateway Auth ## Sandbox Supervisor Identity -Kubernetes sandbox supervisors authenticate back to the gateway as sandbox workloads. By default, the Kubernetes compute driver validates each projected ServiceAccount token and returns the authenticated sandbox ID plus a stable runtime identity to the gateway. The gateway requires that identity to match the namespace, immutable Sandbox resource UID, and supervisor Pod UID recorded during provisioning before it mints its own sandbox JWT. +Kubernetes sandbox supervisors use gateway-issued session JWTs for sandbox-specific +RPCs. A dedicated supervisor reads its preissued credential bundle from a mounted +Secret. A shared supervisor receives each sandbox's bundle through its assignment +stream, authenticated with a separate supervisor-host JWT. See +[Getting the first credential](/about/architecture#getting-the-first-credential). -Dynamic provider token grants can use SPIFFE without changing supervisor-to-gateway authentication. Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path` to mount the SPIFFE CSI Workload API socket into gateway and sandbox pods while keeping the projected ServiceAccount token bootstrap and gateway-minted sandbox JWT path. +The separate `IssueSandboxToken` bootstrap exchange remains available. When used, +the Kubernetes driver validates the projected ServiceAccount token and returns +the authenticated sandbox ID and runtime identity. The gateway requires the +driver and runtime identity to match its stored sandbox record before issuing +a gateway JWT. + +Dynamic provider token grants can use SPIFFE without changing supervisor-to-gateway authentication. Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path` to mount the SPIFFE CSI Workload API socket into gateway and sandbox pods while keeping gateway-issued session JWTs for sandbox-specific RPCs. Provider token grants require a SPIFFE implementation such as SPIRE and identities for the gateway and sandbox pods. The repository's local SPIRE overlay assigns sandbox IDs from the pod's `openshell.ai/sandbox-id` annotation, but the gateway validation path only requires the supervisor SVID to be valid and in the same SPIFFE trust domain as the gateway SVID. Provider profiles with `token_grant` metadata cause the sandbox supervisor to request JWT-SVIDs and exchange them for upstream OAuth2 access tokens. Token-exchange profiles also require a gateway SPIFFE identity because the gateway brokers the intermediate token exchange with its own JWT-SVID. @@ -55,9 +65,9 @@ The `audience` value must match the client ID configured in your identity provid | `gatewayConfig.openshell.gateway.oidc.issuer` | `""` | OIDC issuer URL. Empty disables OIDC. | | `gatewayConfig.openshell.gateway.oidc.audience` | `openshell-cli` | Expected `aud` claim in the JWT. | | `gatewayConfig.openshell.gateway.oidc.jwks_ttl_secs` | `3600` | JWKS key cache TTL in seconds. Must be greater than zero. | -| `gatewayConfig.openshell.gateway.oidc.roles_claim` | `""` | Dot-separated path to the roles array in JWT claims. | -| `gatewayConfig.openshell.gateway.oidc.admin_role` | `""` | Role name that grants admin access. | -| `gatewayConfig.openshell.gateway.oidc.user_role` | `""` | Role name that grants standard user access. | +| `gatewayConfig.openshell.gateway.oidc.roles_claim` | `realm_access.roles` | Dot-separated path to the roles array in JWT claims. | +| `gatewayConfig.openshell.gateway.oidc.admin_role` | `openshell-admin` | Role name that grants admin access. | +| `gatewayConfig.openshell.gateway.oidc.user_role` | `openshell-user` | Role name that grants standard user access. | | `gatewayConfig.openshell.gateway.oidc.scopes_claim` | `""` | Dot-separated path to the scopes array in JWT claims. | The issuer must use HTTPS. The gateway rejects discovery and JWKS redirects, @@ -67,9 +77,15 @@ a different origin unless that origin appears in `jwks_allowed_origins`. Use ### Auth-only mode vs. RBAC mode -Leave both `admin_role` and `user_role` empty to use auth-only mode: any request with a valid JWT from the configured issuer is accepted, but no role distinction is enforced. +When OIDC is enabled, omitting the role fields uses the defaults above and +enables RBAC. To use auth-only mode, explicitly set both `admin_role` and +`user_role` to empty strings (`""`) under `gatewayConfig.openshell.gateway.oidc`. +Authenticated users are treated as Platform Admins and bypass workspace membership +checks. Configured OAuth scope checks still apply. -Set both values to enable RBAC mode, where the gateway checks the role claim and enforces access based on the assigned role: +Keep the default role names, or configure both names for your identity provider, +to use RBAC mode. The gateway checks the role claim and enforces access based on +the assigned role: ```shell helm upgrade openshell \ @@ -84,9 +100,17 @@ helm upgrade openshell \ --set 'gatewayConfig.openshell\.gateway\.oidc.user_role=openshell-user' ``` -Both `admin_role` and `user_role` must be set, or both must be empty. Setting only one is not supported. - -OIDC RBAC is method-level authorization. It controls which API operations a caller can perform, but provider and sandbox records are not owned by individual OIDC subjects. In shared clusters, treat provider credentials as gateway-wide resources and use separate gateways or external tenancy controls when users must not see or attach each other's providers and sandboxes. +The effective role names must either both be non-empty or both be empty. The +gateway rejects a configuration with exactly one empty role name. + +OIDC RBAC combines method-level role and optional scope checks with workspace +membership and role checks. Sandboxes and providers belong to a workspace, +rather than to individual OIDC subjects. Workspace Users can use sandboxes and +attach providers within their assigned workspaces; Workspace Admins manage +resources in those workspaces. Platform Admins bypass workspace membership +checks. Configure OIDC roles and separate workspace memberships when users must +not access each other's resources. See +[Workspaces](/how-it-works/workspaces#understand-the-role-model). ### Provider-specific rolesClaim paths diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index bbc223e32f..a053a8f709 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -300,6 +300,8 @@ The most commonly changed values are: | `global.image.registry` / `global.image.tag` / `global.image.pullPolicy` | Shared registry, tag, and pull policy for the gateway, supervisor, and trusted sandbox runtime. Individual image settings take precedence. | | `gateway.image.registry` / `gateway.image.repository` / `gateway.image.tag` / `gateway.image.digest` | Gateway container image. Defaults to `ghcr.io/nvidia/openshell/gateway:latest`. | | `replicaCount` | Number of gateway replicas. Values above `1` require shared PostgreSQL through `server.externalDbSecret`. | +| `priorityClassName` | Existing Kubernetes PriorityClass for gateway pods. Empty by default. | +| `topologySpreadConstraints` | Kubernetes topology spread constraints for gateway replicas. Empty by default. Set selectors to match the gateway pod labels. | | `workload.kind` | Gateway workload controller. Use `statefulset` for SQLite or `deployment` with `server.externalDbSecret`. | | `workload.allowMultiReplicaStatefulSet` | Allow `replicaCount > 1` with `workload.kind=statefulset`. Prefer Deployment for external database-backed multi-replica gateways. | | `autoscaling.enabled` / `autoscaling.minReplicas` / `autoscaling.maxReplicas` | Render a HorizontalPodAutoscaler instead of a fixed `replicaCount`. Refer to [High Availability](/kubernetes/high-availability#scale-the-gateway). | @@ -566,6 +568,8 @@ The gateway exposes `/healthz` for process liveness and `/readyz` for dependency - `startupProbe` and `livenessProbe` use `/healthz`. - `readinessProbe` uses `/readyz`, which reflects the latest result of an in-process background database check. +A gateway that starts while its database is unreachable keeps retrying the connection. Meanwhile `/healthz` succeeds and `/readyz` reports not ready, so Kubernetes keeps the pod without routing traffic to it. + Each sandbox supervisor Pod uses a `tcpSocket` readiness probe on port 5501. The supervisor accepts connections on that port only while its gateway session is ready. If you add a default-deny ingress policy to sandbox namespaces, allow TCP port 5501 from the node so kubelet can reach supervisor Pods. ## Next Steps diff --git a/docs/observability/accessing-logs.mdx b/docs/observability/accessing-logs.mdx index 9e19c637bf..a7ebc50bc7 100644 --- a/docs/observability/accessing-logs.mdx +++ b/docs/observability/accessing-logs.mdx @@ -51,9 +51,9 @@ When the container runtime falls behind reading the log, the main process blocks ## Gateway Log Storage -The supervisor pushes logs to the gateway over gRPC in real time. The gateway stores up to 2,000 recent log lines per sandbox in memory. This buffer is not persisted to disk and is lost when the gateway restarts or the sandbox's buffers are removed. The push channel is bounded and can drop events under load or during disconnection. +The supervisor pushes logs to the gateway over gRPC in real time. The gateway stores up to 2,000 recent log lines per sandbox in memory. This buffer is not persisted to disk. It is lost when the gateway restarts and when the sandbox is deleted, so a `--no-keep` sandbox leaves no log in the gateway after it exits. Each gateway replica keeps its own buffer. The push channel is bounded and can drop events under load or during disconnection. -For durable retention, collect supervisor output or files and workload container logs into an external log store. The local files rotate and use temporary storage on container drivers; they are not a durable archive. See [Direct Filesystem Access](#direct-filesystem-access) for locations and collection limits, and [OCSF JSON export](/observability/ocsf-json-export) for structured file output. +For durable retention, collect supervisor output or files and workload container logs into an external log store, or export OCSF events with [OTLP log export](/observability/overview#otlp-log-export). The local files rotate and use temporary storage on container drivers; they are not a durable archive. See [Direct Filesystem Access](#direct-filesystem-access) for locations and collection limits, and [OCSF JSON export](/observability/ocsf-json-export) for structured file output. The gateway's separate `ocsf_log` configuration writes gateway-produced OCSF events to JSONL. It does not persist supervisor log lines received over gRPC or main process output. @@ -108,6 +108,8 @@ Shorthand and diagnostics follow the supervisor's diagnostic log filter. With `o The default supervisor image has no shell or `tar`, so shell-based `kubectl exec` and `kubectl cp` examples do not work with that image. Reading Kubernetes supervisor files requires operator-managed access to its log volume, such as cluster debugging tooling. OpenShell does not configure a file collector for you. +The shorthand and JSONL files each rotate daily in UTC and keep the three most recent files, so retention is a file count rather than a number of days. The Docker supervisor's `/var/log` is a 64 MiB tmpfs, and the Docker driver removes the supervisor container when the sandbox stops or is deleted or when the supervisor exits. A Kubernetes `emptyDir` has no size limit and is removed with the supervisor pod, which the driver also deletes when it stops the sandbox. OpenShell sets no log options for container stdout and stderr, so their retention follows the Docker daemon or kubelet configuration. + Ship files before rotation or resource teardown and configure your collector to handle daily rotation. Local file writers are non-blocking and can also drop events under pressure. If the supervisor cannot open `/var/log`, it falls back to stderr-only logging and does not produce these files, even when `ocsf_json_enabled` is true. Neither the local files nor the gateway stream guarantees complete audit history. ## Filtering by Event Type diff --git a/docs/observability/ocsf-json-export.mdx b/docs/observability/ocsf-json-export.mdx index 2f4b52c10f..120bb1c679 100644 --- a/docs/observability/ocsf-json-export.mdx +++ b/docs/observability/ocsf-json-export.mdx @@ -317,6 +317,32 @@ To bound telemetry volume when a workload retries blocked destinations, the supe Gateway records and their OTLP trace context can inherit the caller's incoming `traceparent`, including before authentication succeeds. The caller also decides whether they are correlated at all, because an unsampled `traceparent` suppresses `trace.uid`. Treat gateway trace IDs as correlation hints, not evidence of the caller's identity, and do not treat a missing `trace.uid` as meaningful. +## Join Inspected HTTP Requests + +The supervisor assigns each inspected HTTP request a UUID shared by middleware +`RequestContext.request_id`, OCSF `metadata.correlation_uid`, and the +`x-openshell-request-id` response header. It also exports the ID as the OTLP log +attribute `ocsf.metadata.correlation_uid`. + +This header is reserved for OpenShell: workload and upstream values are removed, +and middleware cannot set or remove it. Use a different header for +application-owned correlation. The receipt is a join key, not proof of identity, +authorization or downstream execution. L4 and uninspected traffic do not provide +this receipt guarantee. + +One primary HTTP Activity record is emitted when the final response head is +written. If the request ends earlier, a guard emits the available outcome. +Routine middleware outcomes share that record; security findings remain separate. +A failure observed after the primary record can produce an additional correlated +record. The response header does not prove that the body was delivered. + +The request ID join works without tracing, when spans are not sampled, and when +the collector is unavailable. This feature adds no request spans and does not +adopt workload trace headers into supervisor traces. Existing bounded deny spans +are unchanged. An instrumented client can record the receipt on its own trace +and use the request ID to find OpenShell's security records; it does not need to +receive those records inside the sandbox. + ## OCSF Event Classes in JSON The `class_uid` field identifies the event type: diff --git a/docs/observability/traces.mdx b/docs/observability/traces.mdx index 0cf8ca3c27..3cbdba70a3 100644 --- a/docs/observability/traces.mdx +++ b/docs/observability/traces.mdx @@ -71,6 +71,8 @@ Give each installation that reports to the same collector its own name, includin The gateway continues a valid W3C `traceparent` sent by the caller, so a client's trace can include the gateway's spans for its requests. Without one, the gateway starts a new trace. +The CLI forwards `TRACEPARENT` and `TRACESTATE` from its environment on gateway requests from its standard commands. A CI pipeline instrumented with an OpenTelemetry integration that exports these variables (for example the Jenkins OpenTelemetry plugin or an OpenTelemetry GitHub Action) therefore parents the gateway's spans for the commands it runs, with no collector or provider configuration on the CI runner. + The gateway also passes its trace context to compute drivers, so driver spans appear in the same trace as the gateway request that caused them. Built-in drivers export their spans through the gateway's process. Managed external drivers receive the trace endpoint, gateway name, and compute driver from the gateway. Drivers that you run yourself own their telemetry configuration. ## Next Steps diff --git a/docs/resources/license.mdx b/docs/resources/license.mdx index 56281bea1c..78e5fc7c41 100644 --- a/docs/resources/license.mdx +++ b/docs/resources/license.mdx @@ -2,12 +2,12 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 title: "License" -description: "NVIDIA OpenShell is licensed under the Apache License, Version 2.0." +description: "OpenShell is licensed under the Apache License, Version 2.0." keywords: "Legal, License, Apache 2.0" position: 1 --- -NVIDIA OpenShell is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). +OpenShell is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Apache License Version 2.0, January 2004 diff --git a/docs/sdk/api-errors.mdx b/docs/sdk/api-errors.mdx index 5d45369e68..7f11c9ea45 100644 --- a/docs/sdk/api-errors.mdx +++ b/docs/sdk/api-errors.mdx @@ -31,6 +31,7 @@ Recognized gateway reasons include the following. | `INVALID_ARGUMENT` | `INVALID_ARGUMENT` | Correct the fields listed in `BadRequest`. | | `RESOURCE_VERSION_CONFLICT` | `ABORTED` | Read the resource again and construct a new conditional write. `metadata.recovery` is `REFRESH_STATE`; `current_resource_version` is included when known. | | `PROFILE_SOURCE_UNAVAILABLE` | `UNAVAILABLE` | Retry a profile snapshot read after at least the supplied delay. | +| `DATABASE_UNAVAILABLE` | `UNAVAILABLE` | The gateway could not get a database connection within 5 seconds. Retry after at least the supplied delay. | | `MUTATION_LOCK_TIMEOUT` | `UNAVAILABLE` | The request waited more than 10 seconds for a concurrent mutation to release a sandbox, workspace, or gateway-wide lock. Retry after at least the supplied delay. A request that carried a `request_id` leaves its admission unresolved, like any other error, so a retry with that ID returns `REQUEST_OUTCOME_UNCERTAIN`. Observe resource state and reconcile effects before you start a new request with a new ID. | | `REQUEST_ID_PAYLOAD_MISMATCH` | `FAILED_PRECONDITION` | Keep the original payload for that request ID. Inspect the original operation before submitting a different one. | | `REQUEST_OUTCOME_UNCERTAIN` | `FAILED_PRECONDITION` | An attempt is admitted but has no confirmed replayable success. Observe resource state and reconcile effects. Do not switch to a new ID to bypass the admission. | diff --git a/docs/sdk/python.mdx b/docs/sdk/python.mdx index 165fd7ac8a..ac806ef496 100644 --- a/docs/sdk/python.mdx +++ b/docs/sdk/python.mdx @@ -148,6 +148,11 @@ workspace for repeated exec, stop, start, and delete operations. List methods return lazy `Pager` instances; use `list_all()` only when you want to fetch the complete collection. +Delete calls use the client timeout, 30 seconds by default. Kubernetes gateways +can hold a delete open until the pod terminates, so pass a larger +`timeout_seconds` to `delete`, or `delete_timeout_seconds` (for example, 120) to +`Sandbox`. + ## Call Generated RPCs The generated gRPC stubs include concrete request and response types for synchronous `grpc.Channel` and asynchronous `grpc.aio.Channel` connections. Install the dependency type stubs in your development environment for editor signatures and type checking: diff --git a/docs/security/best-practices.mdx b/docs/security/best-practices.mdx index 10a17c0cea..b432124ce0 100644 --- a/docs/security/best-practices.mdx +++ b/docs/security/best-practices.mdx @@ -30,17 +30,20 @@ You can update dynamic controls on a running sandbox with `openshell policy upda | --- | --- | --- | --- | | Network | Unauthorized outbound connections and data exfiltration. | CONNECT proxy + OPA policy engine | Yes. Use `openshell policy update`, `openshell policy set`, or operator approval in the TUI. | | Filesystem | System binary tampering, credential theft, config manipulation. | Landlock LSM (kernel level) | No. Requires sandbox re-creation. | -| Process | Privilege escalation, fork bombs, dangerous syscalls. | Seccomp BPF + privilege drop (`setuid`/`setgid`) | No. Requires sandbox re-creation. | +| Process | Privilege escalation, fork bombs, dangerous syscalls. | Seccomp BPF + non-root identity without Linux capabilities | No. Requires sandbox re-creation. | | Provider credentials | Credential exposure and unauthorized service access. | Network policy plus profile endpoint binding and proxy substitution | Yes. Attach or detach providers and update dynamic policy. | ## Network Controls -The CONNECT proxy and OPA policy engine enforce all network controls at the gateway level. +The trusted supervisor's proxy and OPA policy engine enforce network policy. +The sandbox runtime mediates network operations, and the compute driver provides +an outer network fence. ### Deny-by-Default Egress -Every outbound connection from the sandbox goes through the CONNECT proxy. -The proxy evaluates each connection against the OPA policy engine. +The sandbox runtime sends outbound connection requests over the authenticated +Sandbox Protocol channel to the supervisor. The supervisor evaluates each +connection against the OPA policy engine. If no `network_policies` entry matches the destination host, port, and calling binary, the proxy denies the connection. | Aspect | Detail | @@ -52,16 +55,19 @@ If no `network_policies` entry matches the destination host, port, and calling b ### Network Namespace Isolation -The sandbox runs in a dedicated Linux network namespace with a veth pair. -All traffic routes through the host-side veth IP (`10.200.0.1`) where the proxy listens. -Even if a process ignores proxy environment variables, it can only reach the proxy. +Each compute driver blocks direct workload egress using its runtime's network +controls. Docker and Podman disable networking on the workload container and use +a private Unix socket for the Sandbox Protocol. Kubernetes uses `NetworkPolicy` +to permit only the protected control channel. VM guests have no network device +and use vsock. Ignoring proxy environment variables does not bypass mediation. +See [Runtime Boundaries](/about/architecture#how-each-runtime-builds-the-boundary). | Aspect | Detail | |---|---| | Default | Always active. The sandbox cannot bypass the proxy at the network level. | -| What you can change | This is not a user-facing knob. OpenShell always enforces it in proxy mode. | -| Risk if bypassed | Without network namespace isolation, a process could connect directly to the internet, bypassing all policy enforcement. | -| Recommendation | No action needed. OpenShell enforces this automatically. | +| What you can change | The compute driver establishes the fence. Kubernetes operators must provide a CNI that enforces ingress and egress `NetworkPolicy`. | +| Risk if bypassed | A missing or ineffective outer fence removes an independent barrier against direct connections that bypass mediation. | +| Recommendation | Keep the driver-managed network restrictions intact. On Kubernetes, verify CNI enforcement and restrict who can modify sandbox network policies and role labels. | ### User Namespace Isolation @@ -200,26 +206,34 @@ OpenShell validates policies before they take effect. ## Process Controls -The sandbox supervisor drops privileges, applies seccomp filters, and enforces process-level restrictions during startup. +The compute driver establishes the workload identity and outer boundary. The +sandbox runtime validates that environment and applies process controls before +launching agent code. ### Privilege Drop -The sandbox process runs as a non-root user after explicit privilege dropping. +The sandbox runtime starts as a non-root user with no Linux capabilities and +`no_new_privs` set. It rejects an environment that does not meet these +requirements. VM startup includes trusted privileged guest preparation, which +drops UID, GID, and capabilities before entering the qualified runtime. | Aspect | Detail | |---|---| -| Default | The compute driver selects a non-root identity. Docker and Podman use the image's OCI `USER` as a per-field fallback. The supervisor calls `setuid()`/`setgid()` with post-condition verification, disables core dumps with `RLIMIT_CORE=0`, and on Linux sets `PR_SET_DUMPABLE=0`. | +| Default | The compute driver selects a non-root identity. Docker and Podman use the image's OCI `USER` as a per-field fallback. Agent children inherit the runtime identity. Before exec, the runtime disables core dumps with `RLIMIT_CORE=0` and sets `PR_SET_DUMPABLE=0`. | | What you can change | Set either or both `run_as_user` and `run_as_group` fields in the `process` section. Each explicit field takes precedence and must be `sandbox` or a numeric UID/GID from `1` through `4294967294`. Docker and Podman may use named identities through OCI `USER` fallback. Root and the invalid identity sentinel are always rejected. | | Risk if relaxed | A numeric identity inherits every permission granted to the same UID/GID on image files, mounted volumes, or devices. Low IDs often collide with system accounts and groups. | | Recommendation | Use a dedicated non-root image identity or explicit numeric policy identity. Do not attempt to set root. | ### Seccomp Filters -OpenShell applies seccomp in two phases. A narrow supervisor-startup prelude runs before CLI parsing and async runtime initialization, then the child process receives the broader runtime seccomp filter after privilege drop. +OpenShell installs a startup seccomp prelude in the already qualified sandbox +runtime. Workload children inherit network-mediation filters from the process +launcher. Before exec, each child applies Landlock, seccomp self-protection, +and the runtime socket and syscall filters. | Aspect | Detail | |---|---| -| Startup prelude | After privileged bootstrap helpers complete, including network setup and provider-token SPIFFE child mount-namespace preparation, the supervisor sets `PR_SET_NO_NEW_PRIVS` and synchronizes a seccomp filter across all runtime threads that blocks `mount`, the new mount API syscalls, `pivot_root`, `umount2`, `bpf`, `perf_event_open`, `userfaultfd`, module-loading syscalls, and kexec. This closes the long-lived privileged remount and kernel-surface window while leaving required setup syscalls such as `setns` available. | +| Startup prelude | After runtime qualification and bootstrap validation, the sandbox installs a synchronized filter before creating the process launcher and async runtime. It blocks `mount`, the new mount API syscalls, `pivot_root`, `umount2`, `bpf`, `perf_event_open`, `userfaultfd`, module-loading syscalls, and kexec. The runtime already has `PR_SET_NO_NEW_PRIVS` and no Linux capabilities. | | Socket domains | The filter allows only `AF_UNIX`, `AF_NETLINK`, and the brokered `AF_INET` and `AF_INET6` families. Every other family, including `AF_PACKET`, `AF_VSOCK`, `AF_RXRPC`, `AF_SMC`, `AF_KCM`, and `AF_ALG`, fails with `EPERM`, and `socketpair(2)` is limited to `AF_UNIX`. `AF_NETLINK` is partially allowed: only `NETLINK_ROUTE` (protocol 0) is permitted so that `getifaddrs(3)` works; all other netlink protocols are blocked. Write operations via `NETLINK_ROUTE` still require `CAP_NET_ADMIN`, which the sandbox does not grant. | | Runtime unconditional syscall blocks | `memfd_create`, `ptrace`, `bpf`, `process_vm_readv`, `process_vm_writev`, `pidfd_open`, `pidfd_getfd`, `pidfd_send_signal`, `io_uring_setup`, `mount`, `fsopen`, `fsconfig`, `fsmount`, `fspick`, `move_mount`, `open_tree`, `setns`, `umount2`, `pivot_root`, `userfaultfd`, `perf_event_open`. | | Conditional syscall blocks | `execveat` with `AT_EMPTY_PATH`, `unshare` and `clone` with `CLONE_NEWUSER`, and `seccomp(SECCOMP_SET_MODE_FILTER)` are denied with `EPERM`. | @@ -229,16 +243,21 @@ OpenShell applies seccomp in two phases. A narrow supervisor-startup prelude run ### Enforcement Application Order -The sandbox supervisor applies enforcement in a specific order during process startup. -This ordering is intentional: named network-namespace setup still relies on privileged helpers, and privilege dropping still needs `/etc/group` and `/etc/passwd`, which Landlock subsequently restricts. - -1. Privileged supervisor bootstrap helpers, including network-namespace setup, provider-token SPIFFE child mount-namespace setup, and optional `nft` probes. -2. Supervisor startup prelude seccomp (`PR_SET_NO_NEW_PRIVS` plus the early syscall denylist) synchronized across runtime threads. -3. Network and child-only mount namespace entry (`setns`) in child `pre_exec`. -4. Privilege drop (`initgroups` + `setgid` + `setuid`). -5. Core-dump hardening (`RLIMIT_CORE=0`, plus `PR_SET_DUMPABLE=0` on Linux). -6. Landlock filesystem restrictions. -7. Runtime seccomp socket domain and syscall filters. +The built-in Linux sandbox runtime applies enforcement before agent execution. + +1. The compute driver prepares the outer boundary and non-root runtime identity. + VM guest preparation drops privileges before starting the runtime. +2. The runtime verifies its identity, empty capability sets, `no_new_privs`, and + required kernel enforcement features. +3. The runtime validates its bootstrap, installs the startup seccomp prelude, + and creates the process launcher with network-mediation filters that its + children inherit. +4. The supervisor verifies and confirms the boundary before allowing launch. +5. The child applies core-dump hardening (`RLIMIT_CORE=0` and `PR_SET_DUMPABLE=0`). +6. The child applies the mandatory Landlock baseline and configured filesystem + restrictions. +7. The child installs seccomp self-protection and runtime socket-domain and + syscall filters, then executes the agent. ## Inference Controls @@ -273,12 +292,12 @@ Gateway transport uses TLS, with client certificate checks available where the d SSH connections to sandboxes travel through the gateway over the sandbox supervisor's authenticated control path. Each SSH connect call also carries a short-lived session token scoped to a specific sandbox. The sandbox never exposes an SSH port on the network. Its SSH daemon listens on a local Unix socket that only the sandbox's own supervisor process can reach. -Service forwards (`openshell forward`) use the same relay path but carry no session token. The gateway authorizes each forwarded connection against the caller's gateway identity and workspace membership, and the per-sandbox connection limit applies. +Service forwards (`openshell forward`) use the same relay path but carry no session token. The gateway authorizes each forwarded connection against the caller's gateway identity and workspace membership, and the per-sandbox service forward limit applies. | Aspect | Detail | |---|---| -| Default | Session tokens expire after 24 hours. Concurrent connections are limited to 3 per token and 20 per sandbox. | -| What you can change | Configure `ssh_session_ttl_secs`. Set to 0 for no expiry. | +| Default | Session tokens expire after 24 hours. Concurrent connections are limited to 3 per token, 20 SSH relays per sandbox, and 20 service forwards per sandbox. | +| What you can change | Configure `ssh_session_ttl_secs` (set to 0 for no expiry), `max_ssh_connections_per_sandbox`, and `max_service_connections_per_sandbox`. | | Risk if relaxed | Longer TTLs or no expiry increase the window for stolen token reuse. Higher connection limits increase the blast radius of a compromised token. | | Recommendation | Keep the 24-hour default. Monitor connection counts through the TUI. | diff --git a/docs/tutorials/first-network-policy.mdx b/docs/tutorials/first-network-policy.mdx index 5cdeef2200..0fe3dbf104 100644 --- a/docs/tutorials/first-network-policy.mdx +++ b/docs/tutorials/first-network-policy.mdx @@ -23,7 +23,7 @@ After completing this tutorial, you understand: - A user-owned workload image with `curl` installed. -To run every step of this tutorial, you can also use the automated demo script at the [examples/sandbox-policy-quickstart](https://github.com/NVIDIA/OpenShell/blob/main/examples/sandbox-policy-quickstart) directory in the NVIDIA OpenShell repository. It runs the full walkthrough in under a minute but without any user interaction. +To run every step of this tutorial, you can also use the automated demo script at the [examples/sandbox-policy-quickstart](https://github.com/NVIDIA/OpenShell/blob/main/examples/sandbox-policy-quickstart) directory in the OpenShell repository. It runs the full walkthrough in under a minute but without any user interaction. ```shell bash examples/sandbox-policy-quickstart/demo.sh diff --git a/docs/tutorials/run-pi-with-openrouter.mdx b/docs/tutorials/run-pi-with-openrouter.mdx index b53fc2a686..9ba215c57e 100644 --- a/docs/tutorials/run-pi-with-openrouter.mdx +++ b/docs/tutorials/run-pi-with-openrouter.mdx @@ -11,8 +11,8 @@ inside an OpenShell sandbox. Pi works with many model providers out of the box, including [OpenRouter](https://openrouter.ai), which we use here. We'll build a Pi image, give the sandbox access to OpenRouter, and start Pi. -The sandbox lets Pi reach OpenRouter but nothing else on the network, and Pi -never sees your real API key. +The sandbox lets Pi reach OpenRouter but nothing else on the network. OpenShell +supplies Pi with an opaque placeholder for your API key. ## Prerequisites @@ -76,8 +76,10 @@ use that image reference with `--from`. A sandbox has no network access until you grant it. To let Pi reach OpenRouter, we'll create a provider, which stores your API key and opens access -to the service. Pi only ever sees a placeholder for the key. OpenShell swaps in -the real key only when Pi calls OpenRouter. +to the service. Pi receives an opaque placeholder in `OPENROUTER_API_KEY`. +OpenShell substitutes the real key in requests authorized for OpenRouter. See +[How Credential Injection Works](/how-it-works/providers/overview#how-credential-injection-works) +for credential delivery and response-handling limits. A provider is built from a provider profile, which describes the service. Save this one as `provider-openrouter.yaml`: @@ -117,8 +119,7 @@ Here's what the main sections do: - `credentials` declares the API key. Pi finds the placeholder in `OPENROUTER_API_KEY`. -- `endpoints` allows only `openrouter.ai`, and OpenShell sends the key nowhere - else. +- `endpoints` authorizes requests and credential injection for `openrouter.ai`. - `binaries` allows only `node` to use this access. Pi runs on Node.js, so that includes Pi. diff --git a/docs/upgrade/0-1-0.mdx b/docs/upgrade/0-1-0.mdx index 2e7ea7a37a..6a787b1639 100644 --- a/docs/upgrade/0-1-0.mdx +++ b/docs/upgrade/0-1-0.mdx @@ -1,7 +1,7 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Upgrade to NVIDIA OpenShell 0.1.0" +title: "Upgrade to OpenShell 0.1.0" sidebar-title: "0.1.0" description: "Prepare OpenShell operators and users for the breaking changes in version 0.1.0." keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Upgrade, Migration" diff --git a/python/openshell/gateway_config_fixture_test.py b/e2e/configs/gateway/gateway_config_fixture_test.py similarity index 98% rename from python/openshell/gateway_config_fixture_test.py rename to e2e/configs/gateway/gateway_config_fixture_test.py index d7c6b8297b..9fc40a32c9 100644 --- a/python/openshell/gateway_config_fixture_test.py +++ b/e2e/configs/gateway/gateway_config_fixture_test.py @@ -11,7 +11,7 @@ import pytest import yaml -REPO_ROOT = Path(__file__).resolve().parents[2] +REPO_ROOT = Path(__file__).resolve().parents[3] FIXTURE_DIR = REPO_ROOT / "e2e/configs/gateway" TEST_GUEST_ROLE_PATH = ( REPO_ROOT diff --git a/e2e/helm-plugins/openshell-external-compute-driver/host-gateway/driver-env-patch.yaml b/e2e/helm-plugins/openshell-external-compute-driver/host-gateway/driver-env-patch.yaml new file mode 100644 index 0000000000..3c829a6cb8 --- /dev/null +++ b/e2e/helm-plugins/openshell-external-compute-driver/host-gateway/driver-env-patch.yaml @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: openshell +spec: + template: + spec: + containers: + - name: kubernetes-compute-driver + env: + - name: OPENSHELL_HOST_GATEWAY_IP + value: replaced-by-kustomize diff --git a/e2e/helm-plugins/openshell-external-compute-driver/host-gateway/kustomization.yaml b/e2e/helm-plugins/openshell-external-compute-driver/host-gateway/kustomization.yaml new file mode 100644 index 0000000000..2afb0e5262 --- /dev/null +++ b/e2e/helm-plugins/openshell-external-compute-driver/host-gateway/kustomization.yaml @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# The in-process driver reads server.hostGatewayIP from the rendered gateway +# TOML. The external driver reads it from its own environment, so copy the +# chart's rendered value into that container. +apiVersion: kustomize.config.k8s.io/v1alpha1 +kind: Component +patches: + - path: driver-env-patch.yaml + target: + group: apps + version: v1 + kind: StatefulSet + name: openshell +replacements: + - source: + group: apps + version: v1 + kind: StatefulSet + name: openshell + fieldPath: spec.template.spec.hostAliases.0.ip + targets: + - select: + group: apps + version: v1 + kind: StatefulSet + name: openshell + fieldPaths: + - spec.template.spec.containers.[name=kubernetes-compute-driver].env.[name=OPENSHELL_HOST_GATEWAY_IP].value diff --git a/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh b/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh index 43ee52140a..24faf26b62 100755 --- a/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh +++ b/e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh @@ -16,4 +16,18 @@ cp "${plugin_dir}/kustomization.yaml" "${work_dir}/kustomization.yaml" cp "${plugin_dir}/workload-patch.yaml" "${work_dir}/workload-patch.yaml" tee "${work_dir}/rendered.yaml" >/dev/null +# server.hostGatewayIP renders the gateway pod's only hostAliases entry +# (deploy/helm/openshell/templates/_gateway-workload.tpl), which the +# host-gateway component copies from hostAliases.0.ip. The lane exports +# OPENSHELL_E2E_EXPECTED_HOST_GATEWAY_IP when it sets server.hostGatewayIP, so +# a chart change that moves the alias fails here instead of leaving the +# external driver without the IP. +if grep -q '^ *- host\.openshell\.internal$' "${work_dir}/rendered.yaml"; then + cp -R "${plugin_dir}/host-gateway" "${work_dir}/host-gateway" + printf 'components:\n - host-gateway\n' >>"${work_dir}/kustomization.yaml" +elif [ -n "${OPENSHELL_E2E_EXPECTED_HOST_GATEWAY_IP:-}" ]; then + echo "ERROR: server.hostGatewayIP is set, but the rendered chart has no host.openshell.internal hostAliases entry." >&2 + exit 1 +fi + kubectl kustomize "${work_dir}" diff --git a/e2e/policy-advisor/README.md b/e2e/policy-advisor/README.md index adb700306f..a04a8bc996 100644 --- a/e2e/policy-advisor/README.md +++ b/e2e/policy-advisor/README.md @@ -66,7 +66,7 @@ candidate, run the `policy-advisor` suite of the `conformance` crate through the test runner: ```bash -nix run .#openshell-test-runner -- \ +nix run .#openshell-test-runner -- run \ --environment ubuntu-docker-rootful --installer deb \ --family conformance --package conformance \ --filter 'test(/^policy_advisor::/)' diff --git a/e2e/python/conftest.py b/e2e/python/conftest.py index 3c2bea236a..fb27b1e810 100644 --- a/e2e/python/conftest.py +++ b/e2e/python/conftest.py @@ -56,7 +56,10 @@ def cluster_name() -> str | None: @pytest.fixture(scope="session") def sandbox_client(cluster_name: str | None) -> Iterator[SandboxClient]: - with SandboxClient.from_active_cluster(cluster=cluster_name) as client: + # Direct deletes can wait out the Kubernetes pod grace period. + with SandboxClient.from_active_cluster( + cluster=cluster_name, timeout=120.0 + ) as client: yield client @@ -95,6 +98,9 @@ def _create(*, spec: object | None = None, delete_on_exit: bool = True) -> Sandb delete_on_exit=delete_on_exit, # Allow time to pull an explicitly supplied workload fixture. ready_timeout_seconds=300.0, + # Kubernetes holds DeleteSandbox open until the pod terminates + # (30s grace plus API headroom), which can exceed the 30s default. + delete_timeout_seconds=120.0, ) return _create diff --git a/e2e/rust/Cargo.lock b/e2e/rust/Cargo.lock index 20c13456fc..e664206f5c 100644 --- a/e2e/rust/Cargo.lock +++ b/e2e/rust/Cargo.lock @@ -710,7 +710,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -1002,9 +1002,9 @@ dependencies = [ [[package]] name = "h2" -version = "0.4.15" +version = "0.4.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155" +checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27" dependencies = [ "atomic-waker", "bytes", @@ -2641,7 +2641,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] @@ -3301,7 +3301,7 @@ dependencies = [ "getrandom 0.4.3", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.52.0", ] [[package]] diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index b0938d9625..2af824018c 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -204,11 +204,6 @@ name = "websocket_conformance" path = "tests/websocket_conformance.rs" required-features = ["e2e-host-gateway"] -[[test]] -name = "credential_gating" -path = "tests/credential_gating.rs" -required-features = ["e2e-host-gateway"] - [[test]] name = "user_namespaces" path = "tests/user_namespaces.rs" diff --git a/e2e/rust/e2e-docker.sh b/e2e/rust/e2e-docker.sh index 8db220ca1b..68d48e7370 100755 --- a/e2e/rust/e2e-docker.sh +++ b/e2e/rust/e2e-docker.sh @@ -54,9 +54,9 @@ if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then exit 0 fi -if [ "${E2E_FEATURES}" != "e2e-conformance-only" ]; then - CONTAINER_ENGINE=docker bash "${ROOT}/tasks/scripts/e2e-build-workload.sh" -fi +# The conformance suite runs on every lane, including e2e-conformance-only, +# and creates sandboxes from this image (testkit::python_image()). +CONTAINER_ENGINE=docker bash "${ROOT}/tasks/scripts/e2e-build-workload.sh" exec "${ROOT}/e2e/with-docker-gateway.sh" \ bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}" diff --git a/e2e/rust/e2e-kubernetes.sh b/e2e/rust/e2e-kubernetes.sh index fc90a827ee..7ef5576eab 100755 --- a/e2e/rust/e2e-kubernetes.sh +++ b/e2e/rust/e2e-kubernetes.sh @@ -80,6 +80,7 @@ run_conformance() { fi OPENSHELL_CONFORMANCE_JUNIT="${OPENSHELL_CONFORMANCE_JUNIT:-${ROOT}/results/${OPENSHELL_E2E_REPORT_NAME:-e2e-kubernetes}-conformance.xml}" \ + OPENSHELL_TEST_PYTHON_IMAGE="${OPENSHELL_TEST_PYTHON_IMAGE:-${E2E_PULLABLE_PYTHON_IMAGE}}" \ e2e_run_openshell_conformance "Kubernetes" } diff --git a/e2e/rust/e2e-podman.sh b/e2e/rust/e2e-podman.sh index 0d6cd5008c..e718f0afea 100755 --- a/e2e/rust/e2e-podman.sh +++ b/e2e/rust/e2e-podman.sh @@ -25,7 +25,6 @@ PODMAN_CI_TESTS=( bypass_detection ephemeral_cleanup core_dump_hardening - credential_gating default_image driver_config_volume edge_tunnel_e2e @@ -117,9 +116,9 @@ if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ] && [ -z "${OPENSHELL_BIN:-}" ]; then export OPENSHELL_BIN="${ROOT}/target/debug/openshell" fi -if [ "${E2E_FEATURES}" != "e2e-conformance-only" ]; then - CONTAINER_ENGINE=podman bash "${ROOT}/tasks/scripts/e2e-build-workload.sh" -fi +# The conformance suite runs on every lane, including e2e-conformance-only, +# and creates sandboxes from this image (testkit::python_image()). +CONTAINER_ENGINE=podman bash "${ROOT}/tasks/scripts/e2e-build-workload.sh" exec "${ROOT}/e2e/with-podman-gateway.sh" \ bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}" diff --git a/e2e/rust/e2e-vm.sh b/e2e/rust/e2e-vm.sh index 0cb2c46858..bafaadbc71 100755 --- a/e2e/rust/e2e-vm.sh +++ b/e2e/rust/e2e-vm.sh @@ -463,7 +463,8 @@ e2e_export_gateway_restart_metadata \ # preparation; allow 180s for slower CI runners. export OPENSHELL_PROVISION_TIMEOUT="${SANDBOX_PROVISION_TIMEOUT}" -e2e_run_openshell_conformance "VM" +OPENSHELL_TEST_PYTHON_IMAGE="${OPENSHELL_TEST_PYTHON_IMAGE:-${E2E_PULLABLE_PYTHON_IMAGE}}" \ + e2e_run_openshell_conformance "VM" if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then echo "::notice::Skipping Rust e2e tests (e2e-conformance-only)" exit 0 diff --git a/e2e/rust/src/harness/port.rs b/e2e/rust/src/harness/port.rs index e6c82ceb5b..6127438b5e 100644 --- a/e2e/rust/src/harness/port.rs +++ b/e2e/rust/src/harness/port.rs @@ -17,6 +17,10 @@ use tokio::time::{interval, timeout}; /// # Errors /// /// Returns an error if the port does not accept a connection within `max_wait`. +#[allow( + clippy::disallowed_methods, + reason = "the e2e workspace does not depend on openshell-core" +)] pub async fn wait_for_port(host: &str, port: u16, max_wait: Duration) -> Result<(), String> { let addr = format!("{host}:{port}"); diff --git a/e2e/rust/tests/credential_gating.rs b/e2e/rust/tests/credential_gating.rs deleted file mode 100644 index 110e02cb1e..0000000000 --- a/e2e/rust/tests/credential_gating.rs +++ /dev/null @@ -1,1057 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -#![cfg(feature = "e2e")] - -//! E2E coverage for credentialed endpoint admission and REST body backstops. -//! -//! Each test installs its own provider and profile under a unique name, so the -//! tests can run in parallel. Each test prints its steps to stderr with the -//! elapsed time, so a timeout shows which step was slow. - -use std::fmt::Display; -use std::io::Write; -use std::process::Stdio; -use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Mutex}; -use std::time::{Duration, Instant}; - -use base64::Engine as _; -use openshell_e2e::harness::binary::{openshell_bin, openshell_cmd}; -use openshell_e2e::harness::sandbox::SandboxGuard; -use sha1::{Digest, Sha1}; -use tempfile::NamedTempFile; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::net::{TcpListener, TcpStream}; -use tokio::task::JoinHandle; - -const TEST_HOST: &str = "host.openshell.internal"; -const TOKEN_ENV: &str = "E2E_GATING_TOKEN"; -const TEST_SECRET: &str = "e2e-gating-secret-value"; -const PLACEHOLDER_PREFIX: &str = "openshell:resolve:env:"; -const KEY_PRESENCE_SCRIPT: &str = r#"if [ "${E2E_GATING_TOKEN+x}" != x ]; then - echo TOKEN_ABSENT -else - case "$E2E_GATING_TOKEN" in - openshell:resolve:env:*) echo TOKEN_PLACEHOLDER ;; - *) echo TOKEN_UNSAFE ;; - esac -fi"#; -const WEBSOCKET_GUID: &str = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; - -async fn run_cli(args: &[&str]) -> (bool, String) { - let mut command = openshell_cmd(); - command - .args(args) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()); - let output = command.output().await.expect("spawn openshell CLI"); - ( - output.status.success(), - format!( - "{}{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ), - ) -} - -/// Prints a test's steps to stderr with the time since the test started. -#[derive(Clone, Copy)] -struct Progress { - scenario: &'static str, - started: Instant, -} - -impl Progress { - fn start(scenario: &'static str) -> Self { - let progress = Self { - scenario, - started: Instant::now(), - }; - progress.step("start"); - progress - } - - fn step(self, message: impl Display) { - eprintln!( - "credential_gating[{}] +{:.1}s {message}", - self.scenario, - self.started.elapsed().as_secs_f64() - ); - } -} - -fn utf8_path(file: &NamedTempFile) -> Result<&str, String> { - file.path() - .to_str() - .ok_or_else(|| format!("{} is not UTF-8", file.path().display())) -} - -/// Retry a delete until it takes effect. -/// -/// The gateway refuses to delete a provider or a profile that a sandbox still -/// references, and sandbox teardown drains asynchronously, so a single attempt -/// can be rejected right after a sandbox was removed. This runs from `Drop`, -/// so it blocks instead of awaiting. -fn delete_until_gone(args: &[&str]) -> Result<(), String> { - const ATTEMPTS: u32 = 40; - let binary = openshell_bin(); - let mut last_output = String::new(); - for _ in 0..ATTEMPTS { - match std::process::Command::new(&binary) - .args(args) - .stdin(Stdio::null()) - .output() - { - Ok(output) => { - let combined = format!( - "{}{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ); - if output.status.success() || combined.to_lowercase().contains("not found") { - return Ok(()); - } - last_output = combined; - } - Err(error) => last_output = format!("spawn openshell CLI: {error}"), - } - std::thread::sleep(Duration::from_millis(500)); - } - Err(format!( - "'{}' still failing after {ATTEMPTS} attempts:\n{last_output}", - args.join(" ") - )) -} - -/// A provider and the profile it uses, both under one name unique to the -/// test. Dropping it deletes both, also when the test fails. -struct GatingProvider { - name: String, - progress: Progress, -} - -impl GatingProvider { - async fn install_with_endpoints( - progress: Progress, - rest_port: u16, - websocket_port: u16, - ) -> Result { - let name = unique_name(progress.scenario); - let profile = write_provider_profile(&name, rest_port, websocket_port)?; - Self::install(progress, name, &profile).await - } - - async fn install_endpointless(progress: Progress) -> Result { - let name = unique_name(progress.scenario); - let profile = write_endpointless_provider_profile(&name)?; - Self::install(progress, name, &profile).await - } - - async fn install( - progress: Progress, - name: String, - profile: &NamedTempFile, - ) -> Result { - progress.step(format_args!("importing profile {name}")); - let (imported, output) = - run_cli(&["profile", "import", "--file", utf8_path(profile)?]).await; - if !imported { - return Err(format!("profile import failed:\n{output}")); - } - let provider = Self { name, progress }; - progress.step(format_args!("creating provider {}", provider.name)); - let credential = format!("{TOKEN_ENV}={TEST_SECRET}"); - let (created, output) = run_cli(&[ - "provider", - "create", - "--name", - &provider.name, - "--type", - &provider.name, - "--credential", - &credential, - ]) - .await; - if !created { - return Err(format!("provider create failed:\n{output}")); - } - Ok(provider) - } -} - -impl Drop for GatingProvider { - fn drop(&mut self) { - self.progress - .step(format_args!("deleting provider and profile {}", self.name)); - for kind in ["provider", "profile"] { - if let Err(error) = delete_until_gone(&[kind, "delete", &self.name]) { - self.progress - .step(format_args!("{kind} cleanup did not settle: {error}")); - } - } - self.progress.step("cleanup finished"); - } -} - -/// A provider and profile name that does not collide with parallel tests or -/// earlier runs against the same gateway. -fn unique_name(scenario: &str) -> String { - format!( - "e2e-cg-{scenario}-{}-{:08x}", - std::process::id(), - rand::random::() - ) -} - -fn write_provider_profile( - profile_id: &str, - rest_port: u16, - websocket_port: u16, -) -> Result { - let mut file = tempfile::Builder::new() - .suffix(".yaml") - .tempfile() - .map_err(|error| format!("create profile: {error}"))?; - let profile = format!( - r"id: {profile_id} -display_name: E2E Credential Gating -category: other -credentials: - - name: token - env_vars: [{TOKEN_ENV}] - required: true - auth_style: bearer - header_name: authorization -endpoints: - - host: {TEST_HOST} - port: {rest_port} - protocol: rest - access: full - - host: {TEST_HOST} - port: {websocket_port} - protocol: websocket - access: read-write -binaries: - - path: /usr/bin/python* - - path: /usr/local/bin/python* - - path: /sandbox/.uv/python/*/bin/python* -", - ); - file.write_all(profile.as_bytes()) - .map_err(|error| format!("write profile: {error}"))?; - file.flush() - .map_err(|error| format!("flush profile: {error}"))?; - Ok(file) -} - -fn write_endpointless_provider_profile(profile_id: &str) -> Result { - let mut file = tempfile::Builder::new() - .suffix(".yaml") - .tempfile() - .map_err(|error| format!("create endpointless profile: {error}"))?; - let profile = format!( - r"id: {profile_id} -display_name: E2E Endpointless Credential Gating -category: other -credentials: - - name: token - env_vars: [{TOKEN_ENV}] - required: true - auth_style: bearer - header_name: authorization -binaries: - - path: /usr/bin/python* - - path: /usr/local/bin/python* - - path: /sandbox/.uv/python/*/bin/python* -", - ); - file.write_all(profile.as_bytes()) - .map_err(|error| format!("write endpointless profile: {error}"))?; - file.flush() - .map_err(|error| format!("flush endpointless profile: {error}"))?; - Ok(file) -} - -#[derive(Clone, Copy)] -enum EndpointMode { - L4, - TlsSkip, - L4OptIn, - RestBody { rewrite: bool }, - WebSocket, -} - -#[derive(Clone, Copy)] -enum CredentialSource { - ProviderProfile, - PolicyBinding, -} - -fn write_policy( - port: u16, - mode: EndpointMode, - credential_source: CredentialSource, - provider: &str, -) -> Result { - let mut file = NamedTempFile::new().map_err(|error| format!("create policy: {error}"))?; - let endpoint_options = match mode { - EndpointMode::L4 => String::new(), - EndpointMode::TlsSkip => { - " protocol: rest\n access: full\n tls: skip\n".to_string() - } - EndpointMode::L4OptIn => " allow_uninspected_credentials: true\n".to_string(), - EndpointMode::RestBody { rewrite } => format!( - " protocol: rest\n access: full\n request_body_credential_rewrite: {rewrite}\n" - ), - EndpointMode::WebSocket => { - " protocol: websocket\n access: read-write\n".to_string() - } - }; - let credential_binding = match credential_source { - CredentialSource::ProviderProfile => String::new(), - CredentialSource::PolicyBinding => { - format!(" credential_binding:\n provider: {provider}\n") - } - }; - let policy = format!( - r"version: 1 -filesystem_policy: - include_workdir: true - read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log] - read_write: [/sandbox, /tmp, /dev/null] -landlock: - compatibility: best_effort -process: - run_as_user: sandbox - run_as_group: sandbox -network_policies: - credential_gating: - name: credential_gating - endpoints: - - host: {TEST_HOST} - port: {port} -{endpoint_options}{credential_binding} binaries: - - path: /usr/bin/python* - - path: /usr/local/bin/python* - - path: /sandbox/.uv/python/*/bin/python* -", - ); - file.write_all(policy.as_bytes()) - .map_err(|error| format!("write policy: {error}"))?; - file.flush() - .map_err(|error| format!("flush policy: {error}"))?; - Ok(file) -} - -#[derive(Debug, Default, Clone, Copy)] -struct BodyObservation { - saw_placeholder: bool, - saw_secret: bool, -} - -struct HttpProbeServer { - port: u16, - observations: Arc>>, - task: JoinHandle<()>, -} - -impl HttpProbeServer { - async fn start() -> Result { - let listener = TcpListener::bind(("0.0.0.0", 0)) - .await - .map_err(|error| format!("bind HTTP probe: {error}"))?; - let port = listener - .local_addr() - .map_err(|error| format!("read HTTP probe address: {error}"))? - .port(); - let observations = Arc::new(Mutex::new(Vec::new())); - let task_observations = Arc::clone(&observations); - let task = tokio::spawn(async move { - loop { - let Ok((stream, _)) = listener.accept().await else { - break; - }; - let observations = Arc::clone(&task_observations); - tokio::spawn(async move { - let _ = handle_http_probe(stream, observations).await; - }); - } - }); - Ok(Self { - port, - observations, - task, - }) - } - - async fn wait_for_observations(&self, count: usize) -> Vec { - for _ in 0..100 { - let observations = self.observations.lock().unwrap().clone(); - if observations.len() >= count { - return observations; - } - tokio::time::sleep(Duration::from_millis(100)).await; - } - self.observations.lock().unwrap().clone() - } -} - -impl Drop for HttpProbeServer { - fn drop(&mut self) { - self.task.abort(); - } -} - -struct BinaryWebSocketProbeServer { - port: u16, - handshake_seen: Arc, - binary_seen: Arc, - task: JoinHandle<()>, -} - -impl BinaryWebSocketProbeServer { - async fn start() -> Result { - let listener = TcpListener::bind(("0.0.0.0", 0)) - .await - .map_err(|error| format!("bind WebSocket probe: {error}"))?; - let port = listener - .local_addr() - .map_err(|error| format!("read WebSocket probe address: {error}"))? - .port(); - let handshake_seen = Arc::new(AtomicBool::new(false)); - let binary_seen = Arc::new(AtomicBool::new(false)); - let task_handshake_seen = Arc::clone(&handshake_seen); - let task_binary_seen = Arc::clone(&binary_seen); - let task = tokio::spawn(async move { - loop { - let Ok((stream, _)) = listener.accept().await else { - break; - }; - let handshake_seen = Arc::clone(&task_handshake_seen); - let binary_seen = Arc::clone(&task_binary_seen); - tokio::spawn(async move { - let _ = - handle_binary_websocket_probe(stream, handshake_seen, binary_seen).await; - }); - } - }); - Ok(Self { - port, - handshake_seen, - binary_seen, - task, - }) - } - - async fn wait_for_handshake(&self) -> bool { - for _ in 0..100 { - if self.handshake_seen.load(Ordering::Acquire) { - return true; - } - tokio::time::sleep(Duration::from_millis(50)).await; - } - false - } -} - -impl Drop for BinaryWebSocketProbeServer { - fn drop(&mut self) { - self.task.abort(); - } -} - -async fn recv_until(stream: &mut TcpStream, marker: &[u8]) -> std::io::Result> { - let mut received = Vec::new(); - let mut buffer = [0_u8; 1024]; - loop { - let read = stream.read(&mut buffer).await?; - if read == 0 { - return Ok(received); - } - received.extend_from_slice(&buffer[..read]); - if received - .windows(marker.len()) - .any(|window| window == marker) - { - return Ok(received); - } - } -} - -fn websocket_header_value(request: &str, name: &str) -> Option { - request.lines().find_map(|line| { - let (header, value) = line.split_once(':')?; - header - .trim() - .eq_ignore_ascii_case(name) - .then(|| value.trim().to_string()) - }) -} - -fn websocket_accept(key: &str) -> String { - let mut hasher = Sha1::new(); - hasher.update(key.as_bytes()); - hasher.update(WEBSOCKET_GUID.as_bytes()); - base64::engine::general_purpose::STANDARD.encode(hasher.finalize()) -} - -async fn handle_binary_websocket_probe( - mut stream: TcpStream, - handshake_seen: Arc, - binary_seen: Arc, -) -> std::io::Result<()> { - let request_bytes = recv_until(&mut stream, b"\r\n\r\n").await?; - let request = String::from_utf8_lossy(&request_bytes); - let key = websocket_header_value(&request, "Sec-WebSocket-Key").ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidData, "missing WebSocket key") - })?; - let accept = websocket_accept(&key); - let response = format!( - "HTTP/1.1 101 Switching Protocols\r\n\ - Upgrade: websocket\r\n\ - Connection: Upgrade\r\n\ - Sec-WebSocket-Accept: {accept}\r\n\ - \r\n" - ); - stream.write_all(response.as_bytes()).await?; - handshake_seen.store(true, Ordering::Release); - - let mut header = [0_u8; 2]; - if tokio::time::timeout(Duration::from_secs(5), stream.read_exact(&mut header)) - .await - .is_ok_and(|result| result.is_ok()) - && header[0] & 0x0f == 0x02 - { - binary_seen.store(true, Ordering::Release); - } - Ok(()) -} - -async fn handle_http_probe( - mut stream: TcpStream, - observations: Arc>>, -) -> std::io::Result<()> { - let mut received = Vec::new(); - let mut buffer = [0_u8; 4096]; - let mut expected_total = None; - loop { - let read = tokio::time::timeout(Duration::from_secs(10), stream.read(&mut buffer)).await; - let Ok(Ok(read)) = read else { - break; - }; - if read == 0 { - break; - } - received.extend_from_slice(&buffer[..read]); - if expected_total.is_none() - && let Some(header_end) = received.windows(4).position(|window| window == b"\r\n\r\n") - { - let header_end = header_end + 4; - let headers = String::from_utf8_lossy(&received[..header_end]); - let content_length = headers - .lines() - .find_map(|line| { - let (name, value) = line.split_once(':')?; - name.eq_ignore_ascii_case("content-length") - .then(|| value.trim().parse::().ok()) - .flatten() - }) - .unwrap_or(0); - expected_total = Some(header_end + content_length); - } - if expected_total.is_some_and(|expected| received.len() >= expected) { - break; - } - } - - let observation = BodyObservation { - saw_placeholder: received - .windows(PLACEHOLDER_PREFIX.len()) - .any(|window| window == PLACEHOLDER_PREFIX.as_bytes()), - saw_secret: received - .windows(TEST_SECRET.len()) - .any(|window| window == TEST_SECRET.as_bytes()), - }; - observations.lock().unwrap().push(observation); - if expected_total.is_some_and(|expected| received.len() >= expected) { - let result = if observation.saw_secret && !observation.saw_placeholder { - "BODY_REWRITTEN" - } else if observation.saw_placeholder && !observation.saw_secret { - "BODY_TEXT" - } else { - "BODY_BAD" - }; - let response = format!( - "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{result}", - result.len() - ); - stream.write_all(response.as_bytes()).await?; - } - Ok(()) -} - -fn body_client_script(port: u16) -> String { - format!( - r#" -import os -import socket - -host = {TEST_HOST:?} -port = {port} -token = os.environ[{TOKEN_ENV:?}] - -with socket.create_connection((host, port), timeout=10) as sock: - target = f"{{host}}:{{port}}" - body = ("prefix-" + token + "-suffix").encode("utf-8") - request = ( - f"POST /token HTTP/1.1\r\nHost: {{target}}\r\n" - f"Content-Type: text/plain\r\nContent-Length: {{len(body)}}\r\nConnection: close\r\n\r\n" - ).encode("ascii") + body - sock.sendall(request) - sock.settimeout(3) - response = b"" - while True: - try: - chunk = sock.recv(4096) - except socket.timeout: - break - if not chunk: - break - response += chunk - print("BODY_REWRITTEN" if b"BODY_REWRITTEN" in response else "BODY_TEXT" if b"BODY_TEXT" in response else "BODY_DENIED") -"# - ) -} - -fn binary_websocket_client_script(port: u16) -> String { - format!( - r#" -import base64 -import os -import socket -import struct - -host = {TEST_HOST:?} -port = {port} - -def recv_until(sock, marker): - data = b"" - while marker not in data: - chunk = sock.recv(4096) - if not chunk: - break - data += chunk - return data - -def recv_exact(sock, size): - data = b"" - while len(data) < size: - chunk = sock.recv(size - len(data)) - if not chunk: - break - data += chunk - return data - -with socket.create_connection((host, port), timeout=10) as sock: - target = f"{{host}}:{{port}}" - key = base64.b64encode(os.urandom(16)).decode("ascii") - request = ( - f"GET /ws HTTP/1.1\r\nHost: {{target}}\r\n" - "Upgrade: websocket\r\nConnection: Upgrade\r\n" - f"Sec-WebSocket-Key: {{key}}\r\nSec-WebSocket-Version: 13\r\n\r\n" - ) - sock.sendall(request.encode("ascii")) - if not recv_until(sock, b"\r\n\r\n").startswith(b"HTTP/1.1 101"): - raise RuntimeError("upgrade failed") - payload = b"binary-credential-channel" - mask = os.urandom(4) - masked = bytes(byte ^ mask[index % 4] for index, byte in enumerate(payload)) - frame = bytes([0x82, 0x80 | len(payload)]) + mask + masked - sock.sendall(frame) - sock.settimeout(3) - denied = False - try: - header = recv_exact(sock, 2) - if not header: - denied = True - elif len(header) == 2: - opcode = header[0] & 0x0f - masked = bool(header[1] & 0x80) - payload_length = header[1] & 0x7f - if opcode == 0x08 and not masked and payload_length < 126: - close_payload = recv_exact(sock, payload_length) - denied = ( - len(close_payload) >= 2 - and struct.unpack("!H", close_payload[:2])[0] == 1008 - ) - except socket.timeout: - denied = True - print("BINARY_DENIED" if denied else "BINARY_FORWARDED") -"# - ) -} - -async fn sandbox_create_failure( - progress: Progress, - label: &str, - provider: &str, - policy: &NamedTempFile, -) -> String { - progress.step(format_args!( - "{label}: creating sandbox, expecting rejection" - )); - let policy_path = policy.path().to_str().expect("policy path is UTF-8"); - let (success, output) = run_cli(&[ - "sandbox", - "create", - "--policy", - policy_path, - "--provider", - provider, - "--", - "echo", - "must-not-run", - ]) - .await; - assert!(!success, "sandbox create unexpectedly succeeded:\n{output}"); - progress.step(format_args!("{label}: rejected")); - output -} - -/// Create a sandbox, run one command in it, and delete it. -/// -/// Returns the create output followed by the command output. -async fn run_in_sandbox( - progress: Progress, - label: &str, - create_args: &[&str], - command: &[&str], -) -> Result { - progress.step(format_args!("{label}: creating sandbox")); - let mut sandbox = SandboxGuard::create(create_args).await?; - progress.step(format_args!( - "{label}: sandbox {} ready, running command", - sandbox.name - )); - let output = sandbox.exec(command).await; - progress.step(format_args!("{label}: deleting sandbox {}", sandbox.name)); - sandbox.cleanup().await; - progress.step(format_args!("{label}: sandbox deleted")); - Ok(format!("{}{}", sandbox.create_output, output?)) -} - -async fn assert_gateway_admission( - progress: Progress, - provider: &str, - port: u16, - credential_source: CredentialSource, -) -> Result<(), String> { - let l4 = write_policy(port, EndpointMode::L4, credential_source, provider)?; - let l4_error = sandbox_create_failure(progress, "L4-only policy", provider, &l4).await; - assert!(l4_error.contains("credentialed endpoint"), "{l4_error}"); - assert!(l4_error.contains("L4-only"), "{l4_error}"); - - let tls_skip = write_policy(port, EndpointMode::TlsSkip, credential_source, provider)?; - let tls_error = sandbox_create_failure(progress, "tls: skip policy", provider, &tls_skip).await; - assert!(tls_error.contains("credentialed endpoint"), "{tls_error}"); - assert!( - tls_error.contains("tls:") && tls_error.contains("skip"), - "{tls_error}" - ); - - let opt_in = write_policy(port, EndpointMode::L4OptIn, credential_source, provider)?; - let accepted_marker = match credential_source { - CredentialSource::ProviderProfile => "OPT_IN_ACCEPTED", - CredentialSource::PolicyBinding => "ENDPOINTLESS_OPT_IN_ACCEPTED", - }; - let output = run_in_sandbox( - progress, - "opt-in policy", - &["--policy", utf8_path(&opt_in)?, "--provider", provider], - &["echo", accepted_marker], - ) - .await?; - assert!(output.contains(accepted_marker), "{output}"); - Ok(()) -} - -async fn wait_for_provider_key_presence( - sandbox: &SandboxGuard, - expected_present: bool, -) -> Result<(), String> { - let expected = if expected_present { - "TOKEN_PLACEHOLDER" - } else { - "TOKEN_ABSENT" - }; - let mut last_output = String::new(); - for _ in 0..60 { - last_output = sandbox.exec(&["sh", "-lc", KEY_PRESENCE_SCRIPT]).await?; - if last_output.contains(expected) { - return Ok(()); - } - tokio::time::sleep(Duration::from_millis(500)).await; - } - Err(format!( - "provider environment did not converge to {expected}; last key-presence output:\n{last_output}" - )) -} - -async fn set_policy(progress: Progress, sandbox: &str, policy_path: &str) -> Result<(), String> { - let (success, output) = run_cli(&[ - "policy", - "set", - sandbox, - "--policy", - policy_path, - "--wait", - "--timeout", - "120", - ]) - .await; - if !success { - return Err(format!("policy update failed:\n{output}")); - } - progress.step("policy applied"); - Ok(()) -} - -async fn assert_endpointless_provider_env_live_update( - progress: Progress, - provider: &str, - port: u16, -) -> Result<(), String> { - let unbound = write_policy( - port, - EndpointMode::RestBody { rewrite: false }, - CredentialSource::ProviderProfile, - provider, - )?; - let bound = write_policy( - port, - EndpointMode::RestBody { rewrite: false }, - CredentialSource::PolicyBinding, - provider, - )?; - let unbound_path = utf8_path(&unbound)?; - let bound_path = utf8_path(&bound)?; - - progress.step("creating long-running sandbox with the unbound policy"); - let mut sandbox = SandboxGuard::create_keep_with_args( - &["--policy", unbound_path, "--provider", provider, "--no-tty"], - &["sh", "-c", "echo PROVIDER_UPDATE_READY && sleep infinity"], - "PROVIDER_UPDATE_READY", - ) - .await?; - - let result = async { - progress.step(format_args!( - "sandbox {} ready, waiting for the token to be absent", - sandbox.name - )); - wait_for_provider_key_presence(&sandbox, false).await?; - - progress.step("binding the provider with policy set --wait"); - set_policy(progress, &sandbox.name, bound_path) - .await - .map_err(|error| format!("binding {error}"))?; - progress.step("waiting for the token placeholder"); - wait_for_provider_key_presence(&sandbox, true).await?; - - progress.step("unbinding the provider with policy set --wait"); - set_policy(progress, &sandbox.name, unbound_path) - .await - .map_err(|error| format!("unbinding {error}"))?; - progress.step("waiting for the token to be absent"); - wait_for_provider_key_presence(&sandbox, false).await - } - .await; - - progress.step(format_args!("deleting sandbox {}", sandbox.name)); - sandbox.cleanup().await; - progress.step("sandbox deleted"); - result -} - -async fn run_body_sandbox( - progress: Progress, - provider: &str, - port: u16, - mode: EndpointMode, - credential_source: CredentialSource, -) -> Result { - let policy = write_policy(port, mode, credential_source, provider)?; - let script = body_client_script(port); - run_in_sandbox( - progress, - "REST body client", - &["--policy", utf8_path(&policy)?, "--provider", provider], - &["python3", "-c", &script], - ) - .await -} - -async fn assert_websocket_binary_denied( - progress: Progress, - provider: &str, - server: &BinaryWebSocketProbeServer, -) -> Result<(), String> { - let policy = write_policy( - server.port, - EndpointMode::WebSocket, - CredentialSource::ProviderProfile, - provider, - )?; - let script = binary_websocket_client_script(server.port); - let output = run_in_sandbox( - progress, - "WebSocket client", - &["--policy", utf8_path(&policy)?, "--provider", provider], - &["python3", "-c", &script], - ) - .await?; - assert!(output.contains("BINARY_DENIED"), "{output}"); - assert!( - server.wait_for_handshake().await, - "upstream should receive the WebSocket handshake" - ); - assert!( - !server.binary_seen.load(Ordering::Acquire), - "credentialed WebSocket binary frame reached upstream" - ); - Ok(()) -} - -/// Start both probe servers and install a provider whose profile declares -/// their endpoints. -async fn profile_endpoint_fixture( - progress: Progress, -) -> (HttpProbeServer, BinaryWebSocketProbeServer, GatingProvider) { - let server = HttpProbeServer::start().await.expect("start HTTP probe"); - let websocket_server = BinaryWebSocketProbeServer::start() - .await - .expect("start WebSocket probe"); - let provider = - GatingProvider::install_with_endpoints(progress, server.port, websocket_server.port) - .await - .expect("install credentialed provider"); - (server, websocket_server, provider) -} - -/// Start the HTTP probe server and install a provider whose profile declares -/// no endpoints, so policies must bind it explicitly. -async fn endpointless_fixture(progress: Progress) -> (HttpProbeServer, GatingProvider) { - let server = HttpProbeServer::start().await.expect("start HTTP probe"); - let provider = GatingProvider::install_endpointless(progress) - .await - .expect("install endpointless provider"); - (server, provider) -} - -#[tokio::test] -async fn profile_rest_endpoint_keeps_body_placeholder() { - let progress = Progress::start("profile-rest"); - let (server, _websocket_server, provider) = profile_endpoint_fixture(progress).await; - - let output = run_body_sandbox( - progress, - &provider.name, - server.port, - EndpointMode::RestBody { rewrite: false }, - CredentialSource::ProviderProfile, - ) - .await - .expect("run REST body client"); - assert!(output.contains("BODY_TEXT"), "{output}"); - let observations = server.wait_for_observations(1).await; - assert_eq!(observations.len(), 1, "observations: {observations:?}"); - assert!(observations[0].saw_placeholder); - assert!(!observations[0].saw_secret); - progress.step("checks passed"); -} - -#[tokio::test] -async fn profile_websocket_endpoint_denies_binary_frames() { - let progress = Progress::start("profile-websocket"); - let (_server, websocket_server, provider) = profile_endpoint_fixture(progress).await; - - assert_websocket_binary_denied(progress, &provider.name, &websocket_server) - .await - .expect("run WebSocket client"); - progress.step("checks passed"); -} - -#[tokio::test] -async fn policy_binding_requires_inspection_or_opt_in() { - let progress = Progress::start("binding-admission"); - let (server, provider) = endpointless_fixture(progress).await; - - assert_gateway_admission( - progress, - &provider.name, - server.port, - CredentialSource::PolicyBinding, - ) - .await - .expect("gateway admission"); - progress.step("checks passed"); -} - -#[tokio::test] -async fn policy_binding_rest_body_keeps_placeholder_without_rewrite() { - let progress = Progress::start("binding-body"); - let (server, provider) = endpointless_fixture(progress).await; - - let literal = run_body_sandbox( - progress, - &provider.name, - server.port, - EndpointMode::RestBody { rewrite: false }, - CredentialSource::PolicyBinding, - ) - .await - .expect("run REST body client"); - assert!(literal.contains("BODY_TEXT"), "{literal}"); - let observations = server.wait_for_observations(1).await; - assert_eq!(observations.len(), 1, "observations: {observations:?}"); - assert!(observations[0].saw_placeholder); - assert!(!observations[0].saw_secret); - progress.step("checks passed"); -} - -#[tokio::test] -async fn policy_binding_rest_body_rewrite_sends_secret_upstream() { - let progress = Progress::start("binding-rewrite"); - let (server, provider) = endpointless_fixture(progress).await; - - let rewritten = run_body_sandbox( - progress, - &provider.name, - server.port, - EndpointMode::RestBody { rewrite: true }, - CredentialSource::PolicyBinding, - ) - .await - .expect("run REST body client"); - assert!(rewritten.contains("BODY_REWRITTEN")); - assert!(!rewritten.contains(TEST_SECRET)); - assert!(!rewritten.contains(PLACEHOLDER_PREFIX)); - let observations = server.wait_for_observations(1).await; - assert_eq!(observations.len(), 1, "observations: {observations:?}"); - assert!(!observations[0].saw_placeholder); - assert!(observations[0].saw_secret); - progress.step("checks passed"); -} - -#[tokio::test] -async fn policy_binding_updates_provider_environment_live() { - let progress = Progress::start("binding-live-update"); - let (server, provider) = endpointless_fixture(progress).await; - - assert_endpointless_provider_env_live_update(progress, &provider.name, server.port) - .await - .expect("provider environment live update"); - progress.step("checks passed"); -} diff --git a/e2e/rust/tests/kubernetes_shared_supervisor.rs b/e2e/rust/tests/kubernetes_shared_supervisor.rs index 4835327ddc..a908b7eb8c 100644 --- a/e2e/rust/tests/kubernetes_shared_supervisor.rs +++ b/e2e/rust/tests/kubernetes_shared_supervisor.rs @@ -544,6 +544,10 @@ impl RuntimeListener { /// answered. A challenged connection is then sent `reply` in place of the /// pinned TLS identity and must be closed. Silence fails the test: a /// listener that hangs has not rejected anything. + #[allow( + clippy::disallowed_methods, + reason = "the e2e workspace does not depend on openshell-core" + )] async fn probe(&mut self, route: &str, reply: &[u8]) -> RouteAnswer { let mut stream = TcpStream::connect(("127.0.0.1", self.port)) .await diff --git a/e2e/rust/tests/kubernetes_shared_supervisor_ha.rs b/e2e/rust/tests/kubernetes_shared_supervisor_ha.rs index 36b16cf698..bfa3dc6675 100644 --- a/e2e/rust/tests/kubernetes_shared_supervisor_ha.rs +++ b/e2e/rust/tests/kubernetes_shared_supervisor_ha.rs @@ -513,6 +513,10 @@ impl GatewayForward { Self { port, child: None } } + #[allow( + clippy::disallowed_methods, + reason = "the e2e workspace does not depend on openshell-core" + )] async fn reachable(&self) -> bool { tokio::net::TcpStream::connect(("127.0.0.1", self.port)) .await diff --git a/e2e/rust/tests/port_forward.rs b/e2e/rust/tests/port_forward.rs index 89d9d86e1c..e91919d177 100644 --- a/e2e/rust/tests/port_forward.rs +++ b/e2e/rust/tests/port_forward.rs @@ -112,6 +112,10 @@ async fn port_forward_echo() { } /// Attempt to send `hello-nav\n` to the echo server and read the response. +#[allow( + clippy::disallowed_methods, + reason = "the e2e workspace does not depend on openshell-core" +)] async fn try_echo(port: u16) -> Result { let mut stream = TcpStream::connect(format!("127.0.0.1:{port}")) .await diff --git a/e2e/rust/tests/service_bearer_passthrough.rs b/e2e/rust/tests/service_bearer_passthrough.rs index aa64432802..75415c7df2 100644 --- a/e2e/rust/tests/service_bearer_passthrough.rs +++ b/e2e/rust/tests/service_bearer_passthrough.rs @@ -204,6 +204,10 @@ where Ok((status, body)) } +#[allow( + clippy::disallowed_methods, + reason = "the e2e workspace does not depend on openshell-core" +)] async fn request_service( target: &ServiceTarget, authorization: &str, diff --git a/e2e/support/conformance.sh b/e2e/support/conformance.sh index 56ce5a50f1..eef41ecb29 100644 --- a/e2e/support/conformance.sh +++ b/e2e/support/conformance.sh @@ -23,6 +23,12 @@ e2e_require_openshell_bin() { export OPENSHELL_BIN="${binary_dir}/$(basename "${OPENSHELL_BIN}")" } +# A pullable image for testkit::python_image() on lanes that can't load the +# locally built openshell/e2e-python:dev into their compute platform. It +# matches the Rust harness's E2E_WORKLOAD_IMAGE for those lanes. +# shellcheck disable=SC2034 # Used by the scripts that source this file. +E2E_PULLABLE_PYTHON_IMAGE="ghcr.io/astral-sh/uv:0.12.17-python3.12-trixie-slim@sha256:9a59bb7206905ccaae4f7dab222fbac47c125a21e5fc16f43f427cd6c940ade3" + # Print the path of the openshell-conformance program: OPENSHELL_CONFORMANCE_BIN # if set, else a debug build from this checkout. e2e_openshell_conformance_bin() { diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh index 31c0771dc8..b85e72b579 100755 --- a/e2e/with-kube-gateway.sh +++ b/e2e/with-kube-gateway.sh @@ -1331,6 +1331,7 @@ if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then exit 2 fi export HELM_PLUGINS="${ROOT}/e2e/helm-plugins" + export OPENSHELL_E2E_EXPECTED_HOST_GATEWAY_IP="${HOST_GATEWAY_IP}" helm_post_renderer_args+=( --post-renderer openshell-external-compute-driver ) diff --git a/fern/docs.yml b/fern/docs.yml index a36a9ac1e3..c28a70d1cd 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -8,7 +8,7 @@ instances: custom-domain: docs.nvidia.com/openshell multi-source: true -title: NVIDIA OpenShell +title: OpenShell global-theme: nvidia diff --git a/install.sh b/install.sh index 10dd62f59a..02b0e22e9c 100755 --- a/install.sh +++ b/install.sh @@ -792,24 +792,18 @@ find_rpm_asset() { _package="$3" case "$_package" in - openshell) - _dev_name="openshell-dev-${_arch}.rpm" - _fallback_re="^openshell-[0-9].*\\.${_arch}\\.rpm$" - ;; - openshell-gateway) - _dev_name="openshell-gateway-dev-${_arch}.rpm" - _fallback_re="^openshell-gateway-[0-9].*\\.${_arch}\\.rpm$" - ;; - openshell-prover) - _dev_name="openshell-prover-dev-${_arch}.rpm" - _fallback_re="^openshell-prover-[0-9].*\\.${_arch}\\.rpm$" - ;; + openshell|openshell-gateway|openshell-prover) ;; *) error "unknown RPM package selector: ${_package}" ;; esac - awk -v dev_name="$_dev_name" -v fallback_re="$_fallback_re" ' + # Build the regex inside awk: -v values have their escapes processed. + awk -v package="$_package" -v arch="$_arch" ' + BEGIN { + dev_name = package "-dev-" arch ".rpm" + fallback_re = "^" package "-[0-9].*\\." arch "\\.rpm$" + } { name = $2 sub("^\\*", "", name) diff --git a/mise.toml b/mise.toml index 10c964e956..7a99d5f50b 100644 --- a/mise.toml +++ b/mise.toml @@ -74,7 +74,7 @@ DOCKER_BUILDKIT = "1" [vars] # Python paths to include in formatting/linting -python_paths = "python/ tasks/scripts/*.py e2e/mcp-conformance/*.py e2e/kubernetes/app-listener/*.py deploy/sbom/*.py" +python_paths = "python/ tasks/scripts/*.py e2e/mcp-conformance/*.py e2e/kubernetes/app-listener/*.py deploy/sbom/*.py deploy/rpm/*.py e2e/configs/gateway/*.py" [task_config] includes = ["tasks/*.toml"] diff --git a/python/openshell/sandbox.py b/python/openshell/sandbox.py index c02a33cdbf..0670f1d3a6 100644 --- a/python/openshell/sandbox.py +++ b/python/openshell/sandbox.py @@ -680,9 +680,16 @@ def exec_python( timeout_seconds=timeout_seconds, ) - def delete(self, *, allow_missing: bool = False) -> DeletionResult: + def delete( + self, *, allow_missing: bool = False, timeout_seconds: float | None = None + ) -> DeletionResult: + # Forward `timeout_seconds` only on opt-in so older clients still work. + extra = {} if timeout_seconds is None else {"timeout_seconds": timeout_seconds} return self._client.delete( - self.sandbox.name, workspace=self._workspace, allow_missing=allow_missing + self.sandbox.name, + workspace=self._workspace, + allow_missing=allow_missing, + **extra, ) def stop(self) -> SandboxRef: @@ -1138,15 +1145,25 @@ def list_ids_for_all_workspaces( ] def delete( - self, name: str, *, workspace: str, allow_missing: bool = False + self, + name: str, + *, + workspace: str, + allow_missing: bool = False, + timeout_seconds: float | None = None, ) -> DeletionResult: + """Delete a sandbox. + + `timeout_seconds` is the gRPC deadline for this call (default: the + client timeout). + """ response = self._stub.DeleteSandbox( openshell_pb2.DeleteSandboxRequest( workspace_scope=_workspace_scope(workspace), name=name, allow_missing=allow_missing, ), - timeout=self._timeout, + timeout=self._timeout if timeout_seconds is None else timeout_seconds, ) return DeletionResult( DeletionOutcome(response.outcome), response.sandbox_id or None @@ -1751,6 +1768,7 @@ def __init__( workload_template: str | None = None, timeout: float = 30.0, ready_timeout_seconds: float = 120.0, + delete_timeout_seconds: float | None = None, auto_refresh: bool = True, write_back: bool = True, insecure: bool = False, @@ -1758,6 +1776,10 @@ def __init__( ) -> None: """Bind a Sandbox context to the active gateway. + `delete_timeout_seconds` bounds the delete RPC on exit and defaults to + `timeout`. Raise it for drivers that hold the delete open while the + workload terminates. + OIDC kwargs (`auto_refresh`, `write_back`, `insecure`, and `client_credentials`) forward directly to `SandboxClient.from_active_cluster` and have the @@ -1777,6 +1799,7 @@ def __init__( self._workload_template = workload_template self._timeout = timeout self._ready_timeout_seconds = ready_timeout_seconds + self._delete_timeout_seconds = delete_timeout_seconds self._auto_refresh = auto_refresh self._write_back = write_back self._insecure = insecure @@ -1872,7 +1895,9 @@ def __exit__(self, *args: object) -> None: and self._session is not None and self._client is not None ): - result = self._session.delete(allow_missing=True) + result = self._session.delete( + allow_missing=True, timeout_seconds=self._delete_timeout_seconds + ) if result.outcome == DeletionOutcome.ACCEPTED: self._client.wait_deleted( self._session.sandbox.name, diff --git a/python/openshell/sandbox_cleanup_test.py b/python/openshell/sandbox_cleanup_test.py index 4cf558251c..0450877b51 100644 --- a/python/openshell/sandbox_cleanup_test.py +++ b/python/openshell/sandbox_cleanup_test.py @@ -3,6 +3,7 @@ """Cleanup status handling through the curated client's real gRPC interceptor.""" +import time from concurrent import futures from contextlib import nullcontext from types import SimpleNamespace @@ -20,6 +21,7 @@ def cleanup_client(): state = SimpleNamespace( exists=False, code=grpc.StatusCode.NOT_FOUND, + delete_delay=0.0, calls=[], closed=[], ) @@ -46,6 +48,7 @@ def delete(request, context): assert request.name == "cleanup-test" assert request.workspace_scope.workspace == "default" assert request.allow_missing + time.sleep(state.delete_delay) if state.code == grpc.StatusCode.NOT_FOUND: return openshell_pb2.DeleteSandboxResponse( outcome=openshell_pb2.DELETION_OUTCOME_ALREADY_ABSENT @@ -158,6 +161,44 @@ def test_context_cleanup_handles_absence_and_intercepted_errors( assert managed._session is None +@pytest.mark.parametrize( + ("client_timeout", "delete_timeout_seconds", "expect_deadline"), + [(0.2, None, True), (0.2, 5, False), (5, 0.2, True)], +) +def test_context_cleanup_delete_timeout_is_opt_in( + cleanup_client, + monkeypatch, + client_timeout, + delete_timeout_seconds, + expect_deadline, +): + client, state = cleanup_client + state.exists = True + state.delete_delay = 0.5 + monkeypatch.setattr( + SandboxClient, + "from_active_cluster", + classmethod(lambda _cls, **_kwargs: client), + ) + + managed = Sandbox( + workspace="default", + sandbox="cleanup-test", + delete_timeout_seconds=delete_timeout_seconds, + ) + expected = pytest.raises(GatewayError) if expect_deadline else nullcontext() + with expected as caught, managed: + # Only the exit delete runs under the short deadline. + client._timeout = client_timeout + state.exists = False + if caught is not None: + assert caught.value.code() == grpc.StatusCode.DEADLINE_EXCEEDED + assert state.calls == ["GetSandbox", "GetSandbox", "DeleteSandbox"] + assert state.closed == [True] + assert managed._client is None + assert managed._session is None + + class _EntryFailureClient: """Fails the named client call and records close().""" diff --git a/python/openshell/sandbox_test.py b/python/openshell/sandbox_test.py index 3fed31734e..b9b059f00b 100644 --- a/python/openshell/sandbox_test.py +++ b/python/openshell/sandbox_test.py @@ -3276,3 +3276,25 @@ def test_sandbox_session_delete_passes_workspace() -> None: assert stub.delete_request is not None assert _request_workspace(stub.delete_request) == "staging" + + +def test_sandbox_session_delete_omits_timeout_for_legacy_clients() -> None: + from openshell.sandbox import SandboxSession + + calls: list[tuple[str, str, bool]] = [] + + class LegacyClient: + def delete(self, name, *, workspace, allow_missing=False): + calls.append((name, workspace, allow_missing)) + + ref = SandboxRef( + id="sandbox-1", + name="job-1", + workspace="staging", + status=SandboxStatusRef(phase=2, current_policy_version=0), + ) + session = SandboxSession(cast("Any", LegacyClient()), ref) + + session.delete(allow_missing=True) + + assert calls == [("job-1", "staging", True)] diff --git a/python/wheel_verifier_test.py b/python/wheel_verifier_test.py deleted file mode 100644 index 0ade8cd918..0000000000 --- a/python/wheel_verifier_test.py +++ /dev/null @@ -1,157 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -from __future__ import annotations - -import importlib.util -import sys -from pathlib import Path -from zipfile import ZipFile - -import pytest - - -def _load_verifier_module(): - path = Path(__file__).resolve().parents[1] / "tasks/scripts/verify-python-wheel.py" - spec = importlib.util.spec_from_file_location("openshell_wheel_verifier", path) - assert spec is not None - assert spec.loader is not None - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - return module - - -verifier = _load_verifier_module() - - -def _wheel_files() -> set[str]: - files = { - "openshell/__init__.py", - "openshell/sandbox.py", - "openshell/py.typed", - "openshell/_proto/__init__.py", - } - for stem in ( - "credential_driver", - "datamodel", - "extension", - "openshell", - "options", - "sandbox", - ): - files.add(f"openshell/_proto/{stem}_pb2.py") - files.add(f"openshell/_proto/{stem}_pb2.pyi") - files.add(f"openshell/_proto/{stem}_pb2_grpc.py") - files.add(f"openshell/_proto/{stem}_pb2_grpc.pyi") - return files - - -def _write_wheel( - directory: Path, - *, - version: str = "1.2.3", - tag: str = "py3-none-any", - files: set[str] | None = None, - entry_points: str | None = None, -) -> Path: - wheel = directory / f"openshell-{version}-{tag}.whl" - dist_info = f"openshell-{version}.dist-info" - with ZipFile(wheel, "w") as archive: - for name in files if files is not None else _wheel_files(): - archive.writestr(name, "") - archive.writestr( - f"{dist_info}/METADATA", - f"Metadata-Version: 2.4\nName: openshell\nVersion: {version}\n", - ) - archive.writestr( - f"{dist_info}/WHEEL", - f"Wheel-Version: 1.0\nRoot-Is-Purelib: true\nTag: {tag}\n", - ) - if entry_points is not None: - archive.writestr(f"{dist_info}/entry_points.txt", entry_points) - return wheel - - -def test_accepts_pure_python_sdk_wheel(tmp_path: Path) -> None: - wheel = _write_wheel(tmp_path) - - assert verifier.verify_wheel(wheel, "1.2.3") == wheel - - -def test_requires_exactly_one_wheel_in_directory(tmp_path: Path) -> None: - _write_wheel(tmp_path, version="1.2.3") - _write_wheel(tmp_path, version="1.2.4") - - with pytest.raises(ValueError, match="expected exactly one wheel"): - verifier.verify_wheel(tmp_path) - - -@pytest.mark.parametrize("stem", verifier.PROTO_STEMS) -@pytest.mark.parametrize( - "suffix", ["_pb2.py", "_pb2.pyi", "_pb2_grpc.py", "_pb2_grpc.pyi"] -) -def test_rejects_missing_generated_proto( - tmp_path: Path, stem: str, suffix: str -) -> None: - files = _wheel_files() - {f"openshell/_proto/{stem}{suffix}"} - wheel = _write_wheel(tmp_path, files=files) - - with pytest.raises(ValueError, match="wheel is missing required files"): - verifier.verify_wheel(wheel) - - -def test_rejects_bundled_cli(tmp_path: Path) -> None: - files = _wheel_files() | {"openshell-1.2.3.data/scripts/openshell"} - wheel = _write_wheel(tmp_path, files=files) - - with pytest.raises(ValueError, match="contains an openshell executable"): - verifier.verify_wheel(wheel) - - -def test_rejects_bundled_windows_cli(tmp_path: Path) -> None: - files = _wheel_files() | {"openshell-1.2.3.data/scripts/openshell.exe"} - wheel = _write_wheel(tmp_path, files=files) - - with pytest.raises(ValueError, match="contains an openshell executable"): - verifier.verify_wheel(wheel) - - -def test_rejects_openshell_console_script(tmp_path: Path) -> None: - wheel = _write_wheel( - tmp_path, - entry_points="[console_scripts]\nopenshell = openshell.cli:main\n", - ) - - with pytest.raises(ValueError, match="defines an openshell console script"): - verifier.verify_wheel(wheel) - - -def test_rejects_native_extension(tmp_path: Path) -> None: - files = _wheel_files() | {"openshell/_native.so"} - wheel = _write_wheel(tmp_path, files=files) - - with pytest.raises(ValueError, match="contains native files"): - verifier.verify_wheel(wheel) - - -def test_rejects_bytecode(tmp_path: Path) -> None: - files = _wheel_files() | {"openshell/__pycache__/sandbox.cpython-314.pyc"} - wheel = _write_wheel(tmp_path, files=files) - - with pytest.raises(ValueError, match="contains Python bytecode"): - verifier.verify_wheel(wheel) - - -def test_rejects_platform_wheel(tmp_path: Path) -> None: - wheel = _write_wheel(tmp_path, tag="cp311-cp311-manylinux_2_28_x86_64") - - with pytest.raises(ValueError, match="not platform-independent"): - verifier.verify_wheel(wheel) - - -def test_rejects_unexpected_version(tmp_path: Path) -> None: - wheel = _write_wheel(tmp_path) - - with pytest.raises(ValueError, match=r"expected version 1\.2\.4"): - verifier.verify_wheel(wheel, "1.2.4") diff --git a/sdk/typescript/src/client.test.ts b/sdk/typescript/src/client.test.ts index 86fdfa3972..756ff6e44d 100644 --- a/sdk/typescript/src/client.test.ts +++ b/sdk/typescript/src/client.test.ts @@ -12,6 +12,7 @@ import type { MessageInitShape } from '@bufbuild/protobuf'; import { Code, ConnectError, createRouterTransport, type ServiceImpl, type Transport } from '@connectrpc/connect'; import { afterEach, describe, expect, it } from 'vitest'; import { + __setSshCleanupTimeoutMsForTests, __setWatchLogsBackoffSleepForTests, errorCode, Pager, @@ -2236,6 +2237,81 @@ describe('forward', () => { expect(revokedToken).toBe('fwd-tok'); }); + it('revokes the token of a connection still open at close()', async () => { + const revoked: string[] = []; + const sandbox = client({ + getSandbox: () => readySandbox('sb', 'sb-id-open'), + createSshSession: () => ({ + sandboxId: 'sb-id-open', + token: 'open-tok', + gatewayHost: 'gw', + gatewayPort: 443, + gatewayScheme: 'https', + hostKeyFingerprint: '', + expirationTime: undefined, + }), + // A real transport never delivers a call whose signal is already aborted. + revokeSshSession: (req, ctx) => { + if (!ctx.signal.aborted) revoked.push(req.token); + return { outcome: 1 }; + }, + forwardTcp: async function* (requests) { + for await (const frame of requests) { + if (frame.payload.case === 'data') yield { payload: { case: 'data', value: frame.payload.value } }; + } + }, + }); + + const handle = await sandbox.forward('sb', { targetPort: 9000 }); + const socket = net.connect(handle.localPort, handle.localHost); + socket.on('error', () => {}); + await new Promise((resolve) => { + socket.once('data', () => resolve()); + socket.write('ping'); + }); + + await handle.close(); + expect(revoked).toEqual(['open-tok']); + }); + + it('close() resolves within the cleanup bound when revoke never resolves', async () => { + __setSshCleanupTimeoutMsForTests(100); + try { + const sandbox = client({ + getSandbox: () => readySandbox('sb', 'sb-id-stall'), + createSshSession: () => ({ + sandboxId: 'sb-id-stall', + token: 'stall-tok', + gatewayHost: 'gw', + gatewayPort: 443, + gatewayScheme: 'https', + hostKeyFingerprint: '', + expirationTime: undefined, + }), + revokeSshSession: () => new Promise(() => {}), + forwardTcp: async function* (requests) { + for await (const frame of requests) { + if (frame.payload.case === 'data') yield { payload: { case: 'data', value: frame.payload.value } }; + } + }, + }); + + const handle = await sandbox.forward('sb', { targetPort: 9000 }); + const socket = net.connect(handle.localPort, handle.localHost); + socket.on('error', () => {}); + await new Promise((resolve) => { + socket.once('data', () => resolve()); + socket.write('ping'); + }); + + const start = Date.now(); + await handle.close(); + expect(Date.now() - start).toBeLessThan(2_000); + } finally { + __setSshCleanupTimeoutMsForTests(); + } + }); + it('rejects when the sandbox is not ready', async () => { const sandbox = client({ getSandbox: () => ({ diff --git a/sdk/typescript/src/client.ts b/sdk/typescript/src/client.ts index 205796f1bd..18eeb2f698 100644 --- a/sdk/typescript/src/client.ts +++ b/sdk/typescript/src/client.ts @@ -486,7 +486,8 @@ export interface ForwardOptions extends SandboxWorkspaceOptions { // A process-lifetime local listener that tunnels each accepted connection into // the sandbox. Call `close()` on teardown; `closed` resolves once the listener -// is fully torn down. An in-process forward cannot outlive the Node process. +// is fully torn down. `close()` may wait up to 5s for best-effort revokes of +// open connections' tokens. An in-process forward cannot outlive the Node process. export interface ForwardHandle { localHost: string; localPort: number; @@ -724,6 +725,15 @@ function versionPin(value: string | undefined): bigint { const FORWARD_CHUNK = 64 * 1024; +// Bound for revoking a forward's SSH session token during cleanup. +const SSH_CLEANUP_TIMEOUT_MS = 5_000; +let sshCleanupTimeoutMs = SSH_CLEANUP_TIMEOUT_MS; + +// Test hook; not re-exported from index.ts. No argument restores the default. +export function __setSshCleanupTimeoutMsForTests(ms?: number): void { + sshCleanupTimeoutMs = ms ?? SSH_CLEANUP_TIMEOUT_MS; +} + function workspaceName(options?: SandboxWorkspaceOptions | null): string { const workspace = options?.workspace ?? 'default'; if (workspace.trim() === '') throw new SdkError('invalid_config', 'workspace must be non-empty'); @@ -1934,7 +1944,11 @@ export class SandboxClient { input.end(); if (token !== undefined) { try { - await this.grpc.revokeSshSession({ token, allowMissing: true }, { signal }); + // Not the connection signal: close() aborts it before this runs. + await this.grpc.revokeSshSession( + { token, allowMissing: true }, + { signal: AbortSignal.timeout(sshCleanupTimeoutMs) }, + ); } catch { // Best-effort revoke; the token expires on its own regardless. } diff --git a/skills/debug-inference/SKILL.md b/skills/debug-inference/SKILL.md index 8629499476..4fad1480eb 100644 --- a/skills/debug-inference/SKILL.md +++ b/skills/debug-inference/SKILL.md @@ -73,7 +73,12 @@ Launch the client after the attachment wait succeeds so it receives the updated openshell sandbox exec -- ``` -After updating an ordinary static provider, wait for that change and launch a new client. An existing process keeps its revision-scoped reference; readiness does not make the old reference resolve the replacement value. Diagnose managed-refresh credentials according to their own lifecycle. +After updating an ordinary static provider, wait for that change and launch a +new client. An existing process keeps its environment reference; readiness does +not guarantee that it uses the replacement value. Retained revisions and +`stable_placeholder: true` have different resolution behavior, as described in +[Runtime Limitations](https://docs.nvidia.com/openshell/latest/how-it-works/providers/profiles#runtime-limitations). +Diagnose managed-refresh credentials according to their own lifecycle. Keep credentials and issued references out of diagnostic output. Acknowledged detach revokes future credential resolution and removes the reference from future process environments. Requests already forwarded may still finish: diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index ec8c9dee6f..a2a075091c 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -121,6 +121,14 @@ For Kubernetes mount-path conflicts, move caller PVC mounts away from These driver-managed paths remain reserved even for approved resources and `sub_path` mounts; changing approval labels does not resolve a path conflict. +For dedicated Kubernetes supervisor placement, inspect the administrator's +`supervisor_runtime_class_name` in the Kubernetes driver configuration and the +supervisor Pod's `spec.runtimeClassName`. The setting is independent of the +workload RuntimeClass and defaults to unset; it does not configure shared +supervisor Deployments. If startup fails, check that the named RuntimeClass +exists and its handler is available on eligible nodes. OpenShell does not retry +with an ordinary supervisor Pod when the configured class is unavailable. + For out-of-tree compute drivers, also check that their versioned admission-policy acknowledgement matches the gateway's policy. Configure standalone driver policy through its administrator-owned `--admission-config-json` option. @@ -249,7 +257,7 @@ the image/effective policy or provider configuration. The supervisor remains alive while the workload stays unstarted. Inspect `openshell sandbox get` and repair the desired configuration with a complete policy replacement or provider change; do not treat a healthy container as proof that the workload is ready. -If the 300-second provisioning repair window expires, the gateway records +If the provisioning repair window (default 300 seconds) expires, the gateway records `ProvisioningTimedOut` and stops workload and supervisor compute. Inspect `provisioning` in sandbox JSON and TUI NOTES to distinguish cleanup pending from complete. Repairing configuration after expiry does not restart compute: wait @@ -263,6 +271,14 @@ fails discovery within its control-request deadline. Permanent gateway errors and exhausted transient retries terminate startup; inspect those errors as connectivity, authorization, or lifecycle failures. +For slow Kubernetes isolation backends, increase `runtime_bootstrap_grace_seconds` +and the dedicated supervisor's `sandbox_attach_request_timeout_seconds` together. +Shared supervisor deployments instead need `OPENSHELL_ATTACH_REQUEST_TIMEOUT_SECONDS` +on each shared host. The default is 300 seconds; allowed values are 1 through +86400. Restart affected components and explicitly start a timed-out sandbox +after cleanup; changing settings does not reset a persisted admission deadline. +See [gateway configuration](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration#slow-sandbox-startup). + ### Step 4: Check Docker-Backed Gateways The sandbox container's log holds the sandbox runtime's warnings and the @@ -530,6 +546,14 @@ kubectl -n openshell logs -c openshell-gateway --tail=200 `server.telemetryEnabled` renders `OPENSHELL_TELEMETRY_ENABLED` on the gateway pod, and the gateway propagates the effective value to sandbox supervisors. +Gateway pod scheduling accepts top-level `nodeSelector`, `affinity`, +`tolerations`, `priorityClassName`, and `topologySpreadConstraints` values for +both StatefulSet and Deployment workloads. The last two default to unset. +If a gateway pod stays Pending, inspect its scheduling events, confirm the +named PriorityClass exists (`kubectl get priorityclass`), and check that topology +spread selectors match the gateway pod labels and that eligible nodes have the +required topology labels. These values apply only to gateway pods. + #### Additional destination CA ConfigMap When `supervisor.network.additionalCaConfigMapName` is set, Helm must mount the @@ -1582,7 +1606,7 @@ credential failures. | `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs | | Collector receives no OCSF log records, or `openshell.ocsf.gap` records | `[openshell.gateway.otlp.ocsf]` missing or without `enabled = true`, an unreachable endpoint or one that does not speak the table's `protocol`, or export failures and queue pressure | Check `[openshell.gateway.otlp.ocsf]` for `enabled = true` and the endpoint it resolves to, the gateway's `OTLP exporting enabled` startup log for `ocsf_endpoint` and `ocsf_protocol`, export warnings, and `openshell_ocsf_otlp_*` metrics. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). | | Gateway OCSF JSONL stops growing, has gaps, or is rejected by a SIEM | File errors, queue pressure, or shipper falling behind retention | Inspect gateway warnings and `openshell_ocsf_log_*` metrics; check `[openshell.gateway.ocsf_log]`, directory permissions, free space, per-replica paths, and shipper rotation checkpoints. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). | -| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart `gateway:docker`, `gateway:podman`, or `gateway:vm` so it re-detects the listener; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` | +| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with `OPENSHELL_DEV_OTLP_ENDPOINT` set but no collector is listening, for example because `helm:k3s:forward` stopped | Start the collector forwarding again, or restart the gateway task without `OPENSHELL_DEV_OTLP_ENDPOINT`; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` | | Sandbox log shows `OTLP agent trace relay inactive: not enabled by the gateway` at startup, then `agent OTLP export attempted but the agent trace relay is not enabled` when the agent exports | The gateway does not enable `[openshell.gateway.otlp.agent]`, so the supervisor refuses the workload's OTLP relay address | Set `enabled = true` in `[openshell.gateway.otlp.agent]` (Helm: `gatewayConfig."openshell.gateway.otlp.agent".enabled`) with an endpoint from that table or `[openshell.gateway.otlp]`, and recreate the sandbox; `openshell logs --source sandbox` then shows `OTLP agent trace relay enabled` | | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs | | Docker or Podman sandbox never registers | Wrong gateway endpoint, unavailable host networking, or supervisor startup failure | Gateway logs and supervisor container logs | @@ -1593,6 +1617,8 @@ credential failures. | Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` | | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | | Mutating RPCs return `UNAVAILABLE` with "timed out waiting for a concurrent mutation" | Advisory-lock contention, a full lock pool on one replica (4 connections unless `database_lock_max_connections` is set), a slow PostgreSQL, or older gateways still running during an upgrade | `openshell_server_mutation_lock_timeouts_total`, `openshell_server_mutation_lock_connections_in_use` against its capacity, the `detail` field of `mutation lock acquisition timed out` in gateway logs, `mutation lock held longer than the lock wait timeout` warnings and `openshell_server_mutation_lock_hold_seconds`, the `pg_locks` query in Step 6 | +| RPCs fail after about 5 seconds with `UNAVAILABLE` `DATABASE_UNAVAILABLE` or "database unavailable: pool timed out" | PostgreSQL unreachable or not answering, or every data-pool connection busy | `database health check` warnings and `/readyz` on the gateway, PostgreSQL logs, `database_max_connections` and the connection sizing in the High Availability guide | +| Kubernetes gateway pod Running but never Ready, logging `database unreachable; retrying` | The gateway started while PostgreSQL was unreachable and waits for it | PostgreSQL pod and Service endpoints, the host and port in the database URL, network policy between the gateway and PostgreSQL | | Mutating RPCs return `INTERNAL` with "could not open a mutation lock connection" | PostgreSQL out of connection slots, restarting, or unreachable | `openshell_server_mutation_lock_errors_total`, PostgreSQL logs, `SELECT count(*) FROM pg_stat_activity` against `max_connections`, `database_max_connections` and `database_lock_max_connections`, the connection sizing in the High Availability guide | | `helm upgrade` fails with an `autoscaling.*` message | HPA values invalid: missing `resources.requests` (or `resources.limits`), `maxReplicas` above 1 without `server.externalDbSecret` (or on a StatefulSet without `workload.allowMultiReplicaStatefulSet`), no metric target, or min/max out of order. "`minReplicas` and `maxReplicas` are not set" means `--reuse-values` kept a release without the chart's autoscaling defaults | Fix the values named in the error; upgrade with `--reset-then-reuse-values` instead of `--reuse-values` | | HPA shows `` targets | No metrics-server for CPU/memory, or the metrics adapter does not serve the custom metric | `kubectl -n openshell describe hpa openshell`, `kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1` | diff --git a/skills/generate-sandbox-policy/SKILL.md b/skills/generate-sandbox-policy/SKILL.md index 864e9d61f3..b1b98235ec 100644 --- a/skills/generate-sandbox-policy/SKILL.md +++ b/skills/generate-sandbox-policy/SKILL.md @@ -527,9 +527,12 @@ remain supported. A policy with no usable filesystem paths also fails launch. Process identity is omitted so the selected compute driver can choose it. For Docker and Podman, each omitted identity field falls back to the image's OCI `USER`. Tell the user these are defaults and may need adjustment for their -environment. Gateway inference is configured separately through `openshell -inference set/get`. The generated `network_policies` block is the primary -output. +environment. Inference access uses provider profiles and sandbox attachments; +the workload calls the provider's native endpoint and selects its model and +timeout. Inspect the effective policy with `openshell policy get +--full`, including provider-derived rules, and follow the published +[inference guide](https://docs.nvidia.com/openshell/latest/how-it-works/inference). +The generated `network_policies` block is the primary output. When the user explicitly requests `process.run_as_user` or `process.run_as_group`, accept `sandbox` or a numeric UID/GID from `1` through diff --git a/skills/openshell-cli/SKILL.md b/skills/openshell-cli/SKILL.md index b757172b7a..1265b8a610 100644 --- a/skills/openshell-cli/SKILL.md +++ b/skills/openshell-cli/SKILL.md @@ -633,7 +633,7 @@ repair the complete policy or provider set through the gateway. The workload has not started on its first activation, so static fields can also be replaced during this initial repair. A previously activated sandbox retains static-field restrictions while restart admission is pending or rejected. -Before the gateway's 300-second repair window expires, successful validation +Before the gateway's configured repair window expires (300 seconds by default), successful validation completes startup in place. Effective stored configuration changes and their first failed load reset that window; repeated failures do not. After `ProvisioningTimedOut`, inspect the retained record and cleanup status, repair @@ -938,7 +938,12 @@ openshell sandbox exec work-session -- The application owns the native base URL, model, request shape, and timeout. Launch a new process after attachment readiness so it inherits the installed provider environment. Use the `debug-inference` skill for endpoint, policy, credential-binding, or migration failures. -For an ordinary static provider update, wait for the update and launch a new client process to obtain the new reference. Do not claim that readiness updates the environment of an existing process or retargets its old reference. Acknowledged detach revokes retained references and removes them from future process environments. +For an ordinary static provider update, wait for the update and launch a new +client process to obtain the new reference. Readiness does not update an +existing process's environment or guarantee that it uses the replacement value. +See [Runtime Limitations](https://docs.nvidia.com/openshell/latest/how-it-works/providers/profiles#runtime-limitations) +for retained revision and stable-placeholder behavior. Acknowledged detach +revokes retained references and removes them from future process environments. ## Workflow 8: Gateway Management diff --git a/tasks/ci.toml b/tasks/ci.toml index 298fc1f7df..c4835ac24c 100644 --- a/tasks/ci.toml +++ b/tasks/ci.toml @@ -5,7 +5,7 @@ [build] description = "Build the whole project" -depends = ["build:rust:workspace", "build:docker", "build:python:wheel"] +depends = ["build:rust:workspace", "build:docker"] ["build:rust"] description = "Alias for build:rust:workspace" diff --git a/tasks/python.toml b/tasks/python.toml index 8d3a142147..2df00fb757 100644 --- a/tasks/python.toml +++ b/tasks/python.toml @@ -10,32 +10,6 @@ env = { SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL = "0.0.0" } run = "uv sync --group dev" hide = true -["build:python:wheel"] -description = "Build the pure-Python SDK wheel" -depends = ["python:proto"] -run = """ -#!/usr/bin/env bash -set -euo pipefail - -WHEEL_OUTPUT_DIR=${WHEEL_OUTPUT_DIR:-target/wheels} -rm -rf "$WHEEL_OUTPUT_DIR" -mkdir -p "$WHEEL_OUTPUT_DIR" - -uv build --wheel --out-dir "$WHEEL_OUTPUT_DIR" - -VERIFY_ARGS=() -if [ -n "${SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL:-}" ]; then - VERIFY_ARGS+=(--expected-version "$SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL") -fi -uv run --no-project python tasks/scripts/verify-python-wheel.py "$WHEEL_OUTPUT_DIR" ${VERIFY_ARGS[@]+"${VERIFY_ARGS[@]}"} -""" -hide = true - -["python:build"] -description = "Alias for build:python:wheel" -depends = ["build:python:wheel"] -hide = true - ["python:lint"] description = "Lint Python code with ruff" depends = ["python:proto"] diff --git a/tasks/scripts/gateway-docker.sh b/tasks/scripts/gateway-docker.sh index c7e71abfed..5a951eb28d 100644 --- a/tasks/scripts/gateway-docker.sh +++ b/tasks/scripts/gateway-docker.sh @@ -18,6 +18,7 @@ # OPENSHELL_SANDBOX_IMAGE=ghcr.io/... mise run gateway:docker # OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/... mise run gateway:docker # OPENSHELL_SANDBOX_RUNTIME_IMAGE=ghcr.io/... mise run gateway:docker +# OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317 mise run gateway:docker # # After the gateway is running, point the CLI at it with either: # openshell --gateway docker-dev @@ -83,27 +84,6 @@ ensure_docker_runtime_image() { fi } -append_local_otlp_config_if_available() { - local config_path=$1 - if ! port_is_in_use 4317; then - echo "OTLP collector not detected on 127.0.0.1:4317; trace export, OCSF log export, and the agent trace relay disabled." - return - fi - - cat >>"${config_path}" <<'EOF' - -[openshell.gateway.otlp] -endpoint = "http://127.0.0.1:4317" - -[openshell.gateway.otlp.ocsf] -enabled = true - -[openshell.gateway.otlp.agent] -enabled = true -EOF - echo "OTLP trace export, OCSF log export, and the agent trace relay enabled for http://127.0.0.1:4317." -} - register_gateway_metadata() { local name=$1 local endpoint=$2 @@ -231,7 +211,7 @@ if [[ -n "${OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET+x}" ]]; then printf 'provider_spiffe_workload_api_socket = "%s"\n' "$(toml_escape "${OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET}")" >>"${CONFIG_PATH}" fi -append_local_otlp_config_if_available "${CONFIG_PATH}" +append_dev_otlp_config "${CONFIG_PATH}" GATEWAY_ENDPOINT="http://127.0.0.1:${PORT}" register_gateway_metadata "${GATEWAY_NAME}" "${GATEWAY_ENDPOINT}" "${PORT}" diff --git a/tasks/scripts/gateway-podman.sh b/tasks/scripts/gateway-podman.sh index c032cdad80..bb837e464a 100644 --- a/tasks/scripts/gateway-podman.sh +++ b/tasks/scripts/gateway-podman.sh @@ -19,6 +19,7 @@ # OPENSHELL_SANDBOX_IMAGE=ghcr.io/... mise run gateway:podman # OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/... mise run gateway:podman # OPENSHELL_SANDBOX_RUNTIME_IMAGE=ghcr.io/... mise run gateway:podman +# OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317 mise run gateway:podman set -euo pipefail @@ -112,27 +113,6 @@ port_is_in_use() { (echo >/dev/tcp/127.0.0.1/"${port}") >/dev/null 2>&1 } -append_local_otlp_config_if_available() { - local config_path=$1 - if ! port_is_in_use 4317; then - echo "OTLP collector not detected on 127.0.0.1:4317; trace export, OCSF log export, and the agent trace relay disabled." - return - fi - - cat >>"${config_path}" <<'EOF' - -[openshell.gateway.otlp] -endpoint = "http://127.0.0.1:4317" - -[openshell.gateway.otlp.ocsf] -enabled = true - -[openshell.gateway.otlp.agent] -enabled = true -EOF - echo "OTLP trace export, OCSF log export, and the agent trace relay enabled for http://127.0.0.1:4317." -} - register_gateway_metadata() { local name=$1 local endpoint=$2 @@ -273,7 +253,7 @@ if [[ -n "${OPENSHELL_SANDBOX_PROXY_CA_BUNDLE+x}" ]]; then printf 'proxy_ca_bundle = "%s"\n' "$(toml_escape "${OPENSHELL_SANDBOX_PROXY_CA_BUNDLE}")" >>"${CONFIG_PATH}" fi -append_local_otlp_config_if_available "${CONFIG_PATH}" +append_dev_otlp_config "${CONFIG_PATH}" GATEWAY_ENDPOINT="http://${CLI_ENDPOINT_HOST}:${PORT}" register_gateway_metadata "${GATEWAY_NAME}" "${GATEWAY_ENDPOINT}" "${PORT}" diff --git a/tasks/scripts/gateway-toml.sh b/tasks/scripts/gateway-toml.sh index f9b1e1916b..ce53b639a9 100644 --- a/tasks/scripts/gateway-toml.sh +++ b/tasks/scripts/gateway-toml.sh @@ -17,3 +17,27 @@ toml_escape() { value=${value//$'\t'/\\t} printf '%s' "${value}" } + +# Append OTLP export to a gateway config when OPENSHELL_DEV_OTLP_ENDPOINT is +# set. Export stays off otherwise so no collector is assumed. +append_dev_otlp_config() { + local config_path=$1 + local endpoint=${OPENSHELL_DEV_OTLP_ENDPOINT:-} + if [[ -z "${endpoint}" ]]; then + echo "OTLP export disabled; set OPENSHELL_DEV_OTLP_ENDPOINT to enable it." + return + fi + + cat >>"${config_path}" </dev/tcp/127.0.0.1/"${port}") >/dev/null 2>&1 } -append_local_otlp_config_if_available() { - local config_path=$1 - if ! port_is_in_use 4317; then - echo "OTLP collector not detected on 127.0.0.1:4317; trace export, OCSF log export, and the agent trace relay disabled." - return - fi - - cat >>"${config_path}" <<'EOF' - -[openshell.gateway.otlp] -endpoint = "http://127.0.0.1:4317" - -[openshell.gateway.otlp.ocsf] -enabled = true - -[openshell.gateway.otlp.agent] -enabled = true -EOF - echo "OTLP trace export, OCSF log export, and the agent trace relay enabled for http://127.0.0.1:4317." -} - invoking_user() { if [ -n "${SUDO_USER:-}" ] && [ "${SUDO_USER}" != "root" ]; then printf '%s\n' "${SUDO_USER}" @@ -397,7 +377,7 @@ if [[ -n "${OPENSHELL_PROVIDER_SPIFFE_ALLOW_GUEST_TCP+x}" ]]; then printf 'provider_spiffe_allow_guest_tcp = %s\n' "${OPENSHELL_PROVIDER_SPIFFE_ALLOW_GUEST_TCP}" >>"${CONFIG_PATH}" fi -append_local_otlp_config_if_available "${CONFIG_PATH}" +append_dev_otlp_config "${CONFIG_PATH}" GATEWAY_ENDPOINT="http://127.0.0.1:${PORT}" register_gateway_metadata "${GATEWAY_NAME}" "${GATEWAY_ENDPOINT}" "${PORT}" "${VM_DRIVER_STATE_DIR}" diff --git a/tasks/scripts/gateway.sh b/tasks/scripts/gateway.sh index 3b04ee2684..1fc9d9e9f8 100644 --- a/tasks/scripts/gateway.sh +++ b/tasks/scripts/gateway.sh @@ -18,6 +18,8 @@ set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" # shellcheck source=tasks/scripts/gateway-pull-policy.sh source "${ROOT}/tasks/scripts/gateway-pull-policy.sh" +# shellcheck source=tasks/scripts/gateway-toml.sh +source "${ROOT}/tasks/scripts/gateway-toml.sh" GATEWAY_BIN="${OPENSHELL_GATEWAY_BIN:-${ROOT}/target/debug/openshell-gateway}" usage() { @@ -41,6 +43,9 @@ Environment: or ::1 for Podman Machine on macOS. OPENSHELL_SERVER_PORT Gateway port. Defaults to 8080 for Kubernetes, 18080 for Podman/Docker, and 18081 for VM. + OPENSHELL_DEV_OTLP_ENDPOINT + OTLP collector for trace export, OCSF log export, + and the agent trace relay. Unset disables export. Docker, Podman, and VM runs delegate to their gateway: setup scripts. EOF } @@ -262,15 +267,6 @@ name = "${GATEWAY_NAME}" compute_driver = "${DRIVER}" disable_tls = true -[openshell.gateway.otlp] -endpoint = "http://127.0.0.1:4317" - -[openshell.gateway.otlp.ocsf] -enabled = true - -[openshell.gateway.otlp.agent] -enabled = true - [openshell.gateway.auth] allow_unauthenticated_users = true @@ -293,6 +289,8 @@ if [[ -n "${GRPC_ENDPOINT}" ]]; then printf 'grpc_endpoint = "%s"\n' "${GRPC_ENDPOINT}" >>"${CONFIG_PATH}" fi +append_dev_otlp_config "${CONFIG_PATH}" + GATEWAY_ENDPOINT="http://127.0.0.1:${PORT}" register_gateway_metadata "${GATEWAY_NAME}" "${GATEWAY_ENDPOINT}" "${PORT}" diff --git a/python/openshell/release_formula_test.py b/tasks/scripts/release_formula_test.py similarity index 100% rename from python/openshell/release_formula_test.py rename to tasks/scripts/release_formula_test.py diff --git a/python/release_tooling_test.py b/tasks/scripts/release_tooling_test.py similarity index 97% rename from python/release_tooling_test.py rename to tasks/scripts/release_tooling_test.py index d170839cca..cf913e31a9 100644 --- a/python/release_tooling_test.py +++ b/tasks/scripts/release_tooling_test.py @@ -9,7 +9,7 @@ def _load_release_module(): - path = Path(__file__).resolve().parents[1] / "tasks/scripts/release.py" + path = Path(__file__).resolve().parent / "release.py" spec = importlib.util.spec_from_file_location("openshell_release_tooling", path) assert spec is not None assert spec.loader is not None diff --git a/tasks/scripts/test-install-sh.sh b/tasks/scripts/test-install-sh.sh index 9021f45921..0c126d438c 100755 --- a/tasks/scripts/test-install-sh.sh +++ b/tasks/scripts/test-install-sh.sh @@ -872,6 +872,32 @@ if [ "$(find_rpm_asset "${tmpdir}/checksums" x86_64 openshell-prover)" != "opens exit 1 fi +cat >"${tmpdir}/checksums" <<'EOF' +aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa openshell-0.1.5-1xx86_64xrpm +bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb openshell-0.1.5-1.x86_64.rpm +cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc openshell-gateway-0.1.5-1.x86_64.rpm +EOF + +# gawk warns on bad regex escapes; mawk does not. Prefer gawk when present. +rpm_path="$PATH" +if gawk_bin="$(command -v gawk)"; then + mkdir "${tmpdir}/gawk-bin" + ln -s "$gawk_bin" "${tmpdir}/gawk-bin/awk" + rpm_path="${tmpdir}/gawk-bin:${PATH}" +fi + +rpm_err="${tmpdir}/rpm_err" +if [ "$(PATH="$rpm_path" find_rpm_asset "${tmpdir}/checksums" x86_64 openshell 2>"$rpm_err")" != "openshell-0.1.5-1.x86_64.rpm" ] || + [ "$(PATH="$rpm_path" find_rpm_asset "${tmpdir}/checksums" x86_64 openshell-gateway 2>>"$rpm_err")" != "openshell-gateway-0.1.5-1.x86_64.rpm" ]; then + echo "FAIL: RPM fallback must match a literal dot" >&2 + exit 1 +fi +if [ -s "$rpm_err" ]; then + echo "FAIL: RPM package selection must not print to stderr" >&2 + cat "$rpm_err" >&2 + exit 1 +fi + mock_gh_log="${tmpdir}/gh.log" PLATFORM=linux linux_package_method() { diff --git a/tasks/scripts/verify-python-wheel.py b/tasks/scripts/verify-python-wheel.py deleted file mode 100644 index b145d1fd48..0000000000 --- a/tasks/scripts/verify-python-wheel.py +++ /dev/null @@ -1,159 +0,0 @@ -#!/usr/bin/env python3 - -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -from __future__ import annotations - -import argparse -import sys -from configparser import ConfigParser -from configparser import Error as ConfigParserError -from email.parser import Parser -from pathlib import Path -from zipfile import BadZipFile, ZipFile - -PROTO_STEMS = ( - "credential_driver", - "datamodel", - "extension", - "openshell", - "options", - "sandbox", -) -BYTECODE_SUFFIXES = (".pyc", ".pyo") -NATIVE_SUFFIXES = (".dll", ".dylib", ".exe", ".pyd", ".so") - - -def _find_wheel(path: Path) -> Path: - if path.is_file(): - return path - - wheels = sorted(path.glob("*.whl")) - if len(wheels) != 1: - raise ValueError(f"expected exactly one wheel in {path}, found {len(wheels)}") - return wheels[0] - - -def _required_files() -> set[str]: - files = { - "openshell/__init__.py", - "openshell/sandbox.py", - "openshell/py.typed", - "openshell/_proto/__init__.py", - } - for stem in PROTO_STEMS: - files.add(f"openshell/_proto/{stem}_pb2.py") - files.add(f"openshell/_proto/{stem}_pb2.pyi") - files.add(f"openshell/_proto/{stem}_pb2_grpc.py") - files.add(f"openshell/_proto/{stem}_pb2_grpc.pyi") - return files - - -def _single_member(names: set[str], suffix: str) -> str: - matches = [name for name in names if name.endswith(suffix)] - if len(matches) != 1: - raise ValueError(f"expected exactly one {suffix} file, found {len(matches)}") - return matches[0] - - -def verify_wheel(path: Path, expected_version: str | None = None) -> Path: - wheel = _find_wheel(path) - if not wheel.name.endswith("-py3-none-any.whl"): - raise ValueError(f"wheel is not platform-independent: {wheel.name}") - - with ZipFile(wheel) as archive: - names = set(archive.namelist()) - missing = sorted(_required_files() - names) - if missing: - raise ValueError(f"wheel is missing required files: {', '.join(missing)}") - - scripts = sorted( - name - for name in names - if not name.endswith("/") - and Path(name).name in {"openshell", "openshell.exe"} - ) - if scripts: - raise ValueError( - f"wheel contains an openshell executable: {', '.join(scripts)}" - ) - - entry_points = [ - name for name in names if name.endswith(".dist-info/entry_points.txt") - ] - if len(entry_points) > 1: - raise ValueError( - f"expected at most one entry_points.txt file, found {len(entry_points)}" - ) - if entry_points: - parser = ConfigParser(interpolation=None) - try: - parser.read_string(archive.read(entry_points[0]).decode()) - except (ConfigParserError, UnicodeDecodeError) as error: - raise ValueError( - "wheel contains invalid entry point metadata" - ) from error - if parser.has_option("console_scripts", "openshell"): - raise ValueError("wheel defines an openshell console script") - - bytecode_files = sorted( - name for name in names if name.endswith(BYTECODE_SUFFIXES) - ) - if bytecode_files: - raise ValueError( - f"wheel contains Python bytecode: {', '.join(bytecode_files)}" - ) - - native_files = sorted(name for name in names if name.endswith(NATIVE_SUFFIXES)) - if native_files: - raise ValueError(f"wheel contains native files: {', '.join(native_files)}") - - metadata_name = _single_member(names, ".dist-info/METADATA") - metadata = Parser().parsestr(archive.read(metadata_name).decode()) - if metadata["Name"] != "openshell": - raise ValueError(f"unexpected distribution name: {metadata['Name']}") - - version = metadata["Version"] - filename_version = wheel.name.removesuffix("-py3-none-any.whl").removeprefix( - "openshell-" - ) - if version != filename_version: - raise ValueError( - f"wheel filename version {filename_version} does not match metadata {version}" - ) - if expected_version is not None and version != expected_version: - raise ValueError(f"expected version {expected_version}, found {version}") - - wheel_name = _single_member(names, ".dist-info/WHEEL") - wheel_metadata = Parser().parsestr(archive.read(wheel_name).decode()) - if wheel_metadata.get_all("Tag", []) != ["py3-none-any"]: - raise ValueError( - f"unexpected wheel tags: {wheel_metadata.get_all('Tag', [])}" - ) - if wheel_metadata["Root-Is-Purelib"] != "true": - raise ValueError("wheel is not marked as pure Python") - - return wheel - - -def main() -> int: - parser = argparse.ArgumentParser(description="Verify the OpenShell Python wheel") - parser.add_argument( - "path", type=Path, help="Wheel file or directory containing one wheel" - ) - parser.add_argument("--expected-version") - args = parser.parse_args() - - try: - wheel = verify_wheel(args.path, args.expected_version) - except (BadZipFile, OSError, ValueError) as error: - print(f"ERROR: {error}", file=sys.stderr) - return 1 - - print(f"Verified {wheel}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tasks/scripts/vm/build-libkrun.sh b/tasks/scripts/vm/build-libkrun.sh index 036b3bbdc9..2e6f1d1138 100755 --- a/tasks/scripts/vm/build-libkrun.sh +++ b/tasks/scripts/vm/build-libkrun.sh @@ -8,8 +8,9 @@ # with OpenShell's custom kernel configuration for sandbox enforcement. # # In addition to the platform's native .so artifacts, this script exports -# kernel.c and ABI_VERSION metadata so that other platforms (e.g. macOS) can -# compile their own libkrunfw wrapper without rebuilding the kernel. +# kernel.c, ABI_VERSION, and provenance.json (libkrunfw commit, kernel version, +# final kernel config hash) so that other platforms (e.g. macOS) can compile +# their own libkrunfw wrapper without rebuilding the kernel. # # Prerequisites: # - Linux (aarch64 or x86_64) @@ -273,6 +274,14 @@ for artifact in kernel.c ABI_VERSION; do done echo " Built: $(ls "$OUTPUT_DIR"/libkrunfw.so* | xargs -n1 basename | tr '\n' ' ')" +# Record the kernel inputs for package-vm-runtime.sh. The macOS job reuses +# kernel.c without a libkrunfw checkout, so it reads this file instead. +kernel_config_sha256="$(sha256sum "${KERNEL_SOURCES}/.config" | awk '{print $1}')" +printf '{"libkrunfw_commit":"%s","kernel_version":"%s","kernel_config_sha256":"%s"}\n' \ + "$(git rev-parse HEAD)" "$KERNEL_VERSION" "$kernel_config_sha256" \ + > "$OUTPUT_DIR/provenance.json" +echo " Kernel config sha256: ${kernel_config_sha256}" + cd "$BUILD_DIR" # ── Build libkrun (VMM) ───────────────────────────────────────────────── diff --git a/tasks/scripts/vm/package-vm-runtime.sh b/tasks/scripts/vm/package-vm-runtime.sh index ce05dc07c2..e3c8a10df6 100755 --- a/tasks/scripts/vm/package-vm-runtime.sh +++ b/tasks/scripts/vm/package-vm-runtime.sh @@ -20,8 +20,9 @@ # Environment (optional, for provenance): # GITHUB_SHA - Git commit SHA # GITHUB_RUN_ID - GitHub Actions run ID -# CUSTOM_PROVENANCE_DIR - Directory containing provenance.json from custom -# libkrunfw build (macOS only) +# CUSTOM_PROVENANCE_DIR - Directory containing the provenance.json written by +# build-libkrun.sh, when it is not in --build-dir +# (macOS reuses the Linux ARM64 kernel build) set -euo pipefail @@ -120,38 +121,29 @@ download_umoci_binary "${PACKAGE_DIR}/umoci" "${UMOCI_VERSION}" "${UMOCI_GUEST_A echo "==> Writing provenance metadata..." -LIBKRUNFW_COMMIT="unknown" -KERNEL_VERSION="unknown" - -# Try custom provenance first (macOS builds produce this) -if [ -n "$CUSTOM_PROVENANCE_DIR" ] && [ -f "${CUSTOM_PROVENANCE_DIR}/provenance.json" ]; then - LIBKRUNFW_COMMIT="$(jq -r '.libkrunfw_commit // "unknown"' "${CUSTOM_PROVENANCE_DIR}/provenance.json" 2>/dev/null || echo unknown)" - KERNEL_VERSION="$(jq -r '.kernel_version // "unknown"' "${CUSTOM_PROVENANCE_DIR}/provenance.json" 2>/dev/null || echo unknown)" -fi - -# Fall back to inspecting the build directory (Linux builds) -if [ "$LIBKRUNFW_COMMIT" = "unknown" ] && [ -d "${BUILD_DIR}/libkrunfw/.git" ]; then - LIBKRUNFW_COMMIT="$(git -C "${BUILD_DIR}/libkrunfw" rev-parse HEAD 2>/dev/null || echo unknown)" -fi -if [ "$KERNEL_VERSION" = "unknown" ] && [ -f "${BUILD_DIR}/libkrunfw/Makefile" ]; then - KERNEL_VERSION="$(grep -oE 'KERNEL_VERSION\s*=\s*linux-[^\s]+' "${BUILD_DIR}/libkrunfw/Makefile" | head -1 | sed 's/.*= *//' || echo unknown)" -fi - if ! command -v jq &>/dev/null; then echo "Error: jq is required for provenance generation" >&2 exit 1 fi +# Kernel inputs come from the provenance.json written by build-libkrun.sh, +# in the build dir or, for macOS, the directory holding the Linux kernel.c. +KERNEL_PROVENANCE="${CUSTOM_PROVENANCE_DIR:-${BUILD_DIR}}/provenance.json" +kernel_field() { + jq -r --arg k "$1" '.[$k] // "unknown"' "$KERNEL_PROVENANCE" 2>/dev/null || echo unknown +} + jq -n \ --arg artifact "vm-runtime" \ --arg platform "$PLATFORM" \ --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg kfw_commit "$LIBKRUNFW_COMMIT" \ - --arg kver "$KERNEL_VERSION" \ + --arg kfw_commit "$(kernel_field libkrunfw_commit)" \ + --arg kver "$(kernel_field kernel_version)" \ + --arg kcfg "$(kernel_field kernel_config_sha256)" \ --arg umoci "$UMOCI_VERSION" \ --arg sha "${GITHUB_SHA:-unknown}" \ --arg run "${GITHUB_RUN_ID:-unknown}" \ - '{artifact: $artifact, platform: $platform, build_timestamp: $ts, libkrunfw_commit: $kfw_commit, kernel_version: $kver, umoci_version: $umoci, github_sha: $sha, github_run_id: $run}' \ + '{artifact: $artifact, platform: $platform, build_timestamp: $ts, libkrunfw_commit: $kfw_commit, kernel_version: $kver, kernel_config_sha256: $kcfg, umoci_version: $umoci, github_sha: $sha, github_run_id: $run}' \ > "${PACKAGE_DIR}/provenance.json" # ── Create tarball ────────────────────────────────────────────────────── diff --git a/tasks/test.toml b/tasks/test.toml index 2839c1cf0c..89eb4709b7 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -8,6 +8,7 @@ description = "Run all tests (Rust + Python + TypeScript SDK)" depends = [ "test:rust", "test:python", + "test:python-scripts", "sdk:ts:test", "test:sbom", "test:install-sh", @@ -25,8 +26,13 @@ depends = [ "test:maintainer-approval", "test:cargo-lockfile-alignment", "test:review-readiness", + "test:merge-gateway-toml", ] +["test:merge-gateway-toml"] +description = "Test the tmachine gateway TOML merge script" +run = "uv run --no-project --with pytest pytest tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py" + ["test:cargo-lockfile-alignment"] description = "Test the Cargo lockfile alignment check" run = "uv run --no-project --with pytest pytest tasks/scripts/check_cargo_lockfile_alignment_test.py" @@ -156,6 +162,11 @@ run = "tasks/scripts/test-postgres-test-runner.sh" run_windows = "echo Skipping test:postgres-runner: the Unix PostgreSQL test runner does not apply on Windows." hide = true +["test:python-scripts"] +description = "Test the release, RPM, and E2E gateway config tooling" +run = "uv run --no-project --with pytest --with pyyaml pytest -o \"python_files=*_test.py\" tasks/scripts/release_formula_test.py tasks/scripts/release_tooling_test.py deploy/rpm/ e2e/configs/gateway/" +hide = true + ["test:python"] description = "Run Python tests" depends = ["python:proto"] @@ -177,6 +188,7 @@ run = "OPENSHELL_TEST_CONTAINER_ENGINE=podman e2e/with-podman-gateway.sh cargo t ["e2e:testkit"] description = "Run the testkit suites in tests/suites/conformance against a Docker-backed gateway" +depends = ["e2e:workload:build"] run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --test-threads 1 --no-fail-fast" ["e2e:rust:push"] @@ -312,7 +324,12 @@ run = 'OPENSHELL_E2E_KUBE_EXTRA_VALUES="deploy/helm/openshell/ci/values-supervis description = "Run Kubernetes e2e for a two-replica shared supervisor pool behind two gateway replicas: a killed owning member's sandboxes are adopted, not restarted, and each pool sandbox's session stays on its supervisor's gateway through gateway and supervisor failures" env = { OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET = "openshell-ha-pg", OPENSHELL_E2E_KUBE_TEST = "kubernetes_shared_supervisor_ha", OPENSHELL_E2E_KUBE_TLS = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-shared-supervisor-ha" } # The shared supervisor requires gateway TLS, which the Skaffold values turn off. -run = 'OPENSHELL_E2E_KUBE_EXTRA_VALUES="deploy/helm/openshell/ci/values-high-availability.yaml:deploy/helm/openshell/ci/values-supervisor-ha.yaml:deploy/helm/openshell/ci/values-tls-port-forward.yaml${OPENSHELL_E2E_KUBE_EXTRA_VALUES:+:$OPENSHELL_E2E_KUBE_EXTRA_VALUES}" e2e/rust/e2e-kubernetes.sh' +run = """ + extra="deploy/helm/openshell/ci/values-high-availability.yaml" + extra="${extra}:deploy/helm/openshell/ci/values-supervisor-ha.yaml" + extra="${extra}:deploy/helm/openshell/ci/values-tls-port-forward.yaml" + OPENSHELL_E2E_KUBE_EXTRA_VALUES="${extra}${OPENSHELL_E2E_KUBE_EXTRA_VALUES:+:$OPENSHELL_E2E_KUBE_EXTRA_VALUES}" e2e/rust/e2e-kubernetes.sh +""" ["e2e:kubernetes:shared-supervisor-scaling"] description = "Run Kubernetes e2e that scales a one-member shared supervisor pool to two members and back to one with Helm, checking sandboxes are handed off, not restarted" diff --git a/tests/runner/src/bundle.rs b/tests/runner/src/bundle.rs index 08e66738c6..95d0b914ea 100644 --- a/tests/runner/src/bundle.rs +++ b/tests/runner/src/bundle.rs @@ -8,12 +8,21 @@ //! `nextest` config with the runner's `JUnit` profile, and `testkit-ready`, //! which waits for the gateway before the tests run. The guest needs only the //! `cargo-nextest` binary to run it. +//! +//! `build` writes a bundle to a directory, so CI can build it once and every +//! row can run it (`run --bundle`). A `bundle.json` in it names the family +//! and the packages it was built with. +//! +//! A family that ships a prebuilt test program, such as `openshell-conformance`, +//! can run it instead: the bundle then holds that program alone, built once +//! elsewhere for the guest target, and the runner builds nothing. +use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; use std::process::Command; use anyhow::{Context, Result, bail, ensure}; -use serde::Deserialize; +use serde::{Deserialize, Serialize}; use tempfile::TempDir; /// The nextest profile the runner uses in the guest. @@ -34,18 +43,64 @@ slow-timeout = { period = "30s", terminate-after = 4 } path = "junit.xml" "#; +/// The prebuilt program's name in a program bundle. +pub const PROGRAM_BINARY: &str = "test-program"; + +/// The nextest archive in a nextest bundle. +const ARCHIVE: &str = "tests.tar.zst"; +/// The nextest config in a nextest bundle. +const CONFIG: &str = "nextest.toml"; +/// What a nextest bundle was built from. +const MANIFEST: &str = "bundle.json"; + +/// The family and packages a nextest bundle holds. +#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] +struct Manifest { + family: String, + /// Empty when the bundle holds every package in the family. + packages: Vec, +} + +/// What a bundle runs the tests with. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Kind { + /// A nextest archive, run with `cargo-nextest`. + Nextest, + /// A prebuilt testkit program (`list`, `ready` and `run`), run directly. + Program, +} + /// The bundle directory, removed when this is dropped. -pub struct Bundle(TempDir); +pub struct Bundle(TempDir, Kind); impl Bundle { pub fn path(&self) -> &Path { self.0.path() } + + pub const fn kind(&self) -> Kind { + self.1 + } +} + +/// A bundle holding only `program`, a testkit program already built for the +/// guest target. `program` must be a static binary, as the guest's libc is +/// not known. +pub fn program(program: &Path) -> Result { + ensure!( + program.is_file(), + "--program {} is not a file", + program.display() + ); + let dir = tempfile::tempdir().context("create bundle directory")?; + std::fs::copy(program, dir.path().join(PROGRAM_BINARY)) + .with_context(|| format!("copy {}", program.display()))?; + Ok(Bundle(dir, Kind::Program)) } /// The Rust target for guests. tmachine only boots guests of the host's /// architecture, and static musl binaries don't depend on the guest's libc. -pub fn guest_target() -> Result<&'static str> { +fn guest_target() -> Result<&'static str> { if cfg!(target_arch = "x86_64") { Ok("x86_64-unknown-linux-musl") } else if cfg!(target_arch = "aarch64") { @@ -56,7 +111,7 @@ pub fn guest_target() -> Result<&'static str> { } /// Build `testkit-ready` for the guest and return its path. -pub fn build_ready(root: &Path, target: &str) -> Result { +fn build_ready(root: &Path, target: &str) -> Result { let output = Command::new("cargo") .args(["build", "--locked", "--manifest-path"]) .arg(root.join("tests/testkit/Cargo.toml")) @@ -105,26 +160,41 @@ fn built_executable(messages: &[u8], name: &str) -> Option { .find_map(|message| message.executable) } -/// Build the bundle for a family workspace (`tests/suites/`). With no -/// `packages`, the archive holds every package in the workspace. `ready` is -/// the guest's `testkit-ready`. -pub fn build(workspace: &Path, packages: &[String], target: &str, ready: &Path) -> Result { +/// Build the bundle for `family` into a new temporary directory. +pub fn build_temp(root: &Path, family: &str, packages: &[String]) -> Result { let dir = tempfile::tempdir().context("create bundle directory")?; - let stage = dir.path(); + build(root, family, packages, dir.path())?; + Ok(Bundle(dir, Kind::Nextest)) +} + +/// Build the bundle for `family`, the path of a workspace under +/// `tests/suites/`, into `out`, an empty or missing directory. With no +/// `packages`, the archive holds every package in the workspace. +pub fn build(root: &Path, family: &str, packages: &[String], out: &Path) -> Result<()> { + let workspace = crate::family_workspace(root, family)?; let manifest = workspace.join("Cargo.toml"); ensure!( manifest.is_file(), "{} is not a test family workspace", workspace.display() ); - let archive = stage.join("tests.tar.zst"); + std::fs::create_dir_all(out).with_context(|| format!("create {}", out.display()))?; + ensure!( + std::fs::read_dir(out)?.next().is_none(), + "{} is not empty", + out.display() + ); + let target = guest_target()?; + eprintln!("==> building {READY_BINARY} for {target}"); + let ready = build_ready(root, target)?; + eprintln!("==> building the {family} tests for {target}"); let mut command = Command::new("cargo"); command .args(["nextest", "archive", "--locked", "--manifest-path"]) .arg(&manifest) .args(["--target", target, "--archive-file"]) - .arg(&archive); + .arg(out.join(ARCHIVE)); if packages.is_empty() { command.arg("--workspace"); } @@ -133,11 +203,108 @@ pub fn build(workspace: &Path, packages: &[String], target: &str, ready: &Path) } run(&mut command).with_context(|| format!("build the {} test archive", workspace.display()))?; - copy_manifests(workspace, stage)?; - std::fs::write(stage.join("nextest.toml"), NEXTEST_CONFIG)?; - std::fs::copy(ready, stage.join(READY_BINARY)) + copy_manifests(&workspace, out)?; + std::fs::write(out.join(CONFIG), NEXTEST_CONFIG)?; + std::fs::copy(&ready, out.join(READY_BINARY)) .with_context(|| format!("copy {}", ready.display()))?; - Ok(Bundle(dir)) + let manifest = Manifest { + family: family.to_string(), + packages: packages.to_vec(), + }; + std::fs::write(out.join(MANIFEST), serde_json::to_vec_pretty(&manifest)?)?; + Ok(()) +} + +/// A bundle that `build` wrote to `dir`, for running `packages` of `family` +/// (every package in it when empty). The bundle is copied, so `dir` may have +/// lost the executable bits, as a downloaded CI artifact does. +pub fn prebuilt(dir: &Path, family: &str, packages: &[String]) -> Result { + let read = std::fs::read(dir.join(MANIFEST)) + .with_context(|| format!("{} is not a test bundle: read {MANIFEST}", dir.display()))?; + let manifest: Manifest = serde_json::from_slice(&read) + .with_context(|| format!("parse {}", dir.join(MANIFEST).display()))?; + check_selection(&manifest, family, packages) + .with_context(|| format!("--bundle {}", dir.display()))?; + for name in [ARCHIVE, CONFIG, READY_BINARY] { + ensure!( + dir.join(name).is_file(), + "{} is not a test bundle: it has no {name}", + dir.display() + ); + } + + let copy = tempfile::tempdir().context("create bundle directory")?; + copy_tree(dir, copy.path())?; + let ready = copy.path().join(READY_BINARY); + std::fs::set_permissions(&ready, std::fs::Permissions::from_mode(0o755)) + .with_context(|| format!("make {} executable", ready.display()))?; + Ok(Bundle(copy, Kind::Nextest)) +} + +/// Check that a bundle built from `manifest` holds `packages` of `family`. +fn check_selection(manifest: &Manifest, family: &str, packages: &[String]) -> Result<()> { + ensure!( + manifest.family == family, + "the bundle holds the {} tests, not {family}", + manifest.family + ); + if manifest.packages.is_empty() { + return Ok(()); + } + let missing: Vec<_> = if packages.is_empty() { + vec!["every package"] + } else { + packages + .iter() + .filter(|package| !manifest.packages.contains(package)) + .map(String::as_str) + .collect() + }; + ensure!( + missing.is_empty(), + "the bundle holds only {}, not {}", + manifest.packages.join(", "), + missing.join(", ") + ); + Ok(()) +} + +/// The nextest filtersets that run the tests of `packages` that match any of +/// `filters`. nextest runs the tests that match any `-E`, so the packages and +/// the filters go in one expression. +pub fn narrow(packages: &[String], filters: &[String]) -> Vec { + if packages.is_empty() { + return filters.to_vec(); + } + let packages = packages + .iter() + .map(|package| format!("package(={package})")) + .collect::>() + .join(" | "); + if filters.is_empty() { + return vec![packages]; + } + let filters = filters + .iter() + .map(|filter| format!("({filter})")) + .collect::>() + .join(" | "); + vec![format!("({packages}) & ({filters})")] +} + +/// Copy the files under `src` into `dst`, keeping relative paths. +fn copy_tree(src: &Path, dst: &Path) -> Result<()> { + for entry in std::fs::read_dir(src).with_context(|| format!("read {}", src.display()))? { + let path = entry?.path(); + let dest = dst.join(path.file_name().unwrap_or_default()); + if path.is_dir() { + std::fs::create_dir_all(&dest)?; + copy_tree(&path, &dest)?; + } else { + std::fs::copy(&path, &dest).with_context(|| format!("copy {}", path.display()))?; + } + } + Ok(()) } /// Copy every `Cargo.toml` under `workspace` into `stage`, keeping relative @@ -176,6 +343,91 @@ fn run(command: &mut Command) -> Result<()> { mod tests { use super::*; + #[test] + fn program_bundle_holds_only_the_program() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("openshell-conformance"); + std::fs::write(&path, b"binary").unwrap(); + let bundle = program(&path).unwrap(); + assert_eq!(bundle.kind(), Kind::Program); + let names: Vec<_> = std::fs::read_dir(bundle.path()) + .unwrap() + .map(|entry| entry.unwrap().file_name()) + .collect(); + assert_eq!(names, [PROGRAM_BINARY]); + assert!(program(&dir.path().join("missing")).is_err()); + } + + fn write_bundle(dir: &Path, family: &str, packages: &[&str]) { + std::fs::create_dir_all(dir.join("host-gateway")).unwrap(); + for name in [ARCHIVE, CONFIG, READY_BINARY, "Cargo.toml"] { + std::fs::write(dir.join(name), name).unwrap(); + } + std::fs::write(dir.join("host-gateway/Cargo.toml"), "member").unwrap(); + let manifest = Manifest { + family: family.to_string(), + packages: packages.iter().map(ToString::to_string).collect(), + }; + std::fs::write(dir.join(MANIFEST), serde_json::to_vec(&manifest).unwrap()).unwrap(); + } + + #[test] + fn prebuilt_bundle_is_copied_with_an_executable_ready_check() { + let dir = tempfile::tempdir().unwrap(); + write_bundle(dir.path(), "features", &[]); + let bundle = prebuilt(dir.path(), "features", &["host-gateway".to_string()]).unwrap(); + assert_eq!(bundle.kind(), Kind::Nextest); + assert_ne!(bundle.path(), dir.path()); + assert_eq!( + std::fs::read(bundle.path().join("host-gateway/Cargo.toml")).unwrap(), + b"member" + ); + let mode = std::fs::metadata(bundle.path().join(READY_BINARY)) + .unwrap() + .permissions() + .mode(); + assert_eq!(mode & 0o111, 0o111); + } + + #[test] + fn prebuilt_bundle_must_be_complete_and_of_the_family() { + let dir = tempfile::tempdir().unwrap(); + assert!(prebuilt(dir.path(), "features", &[]).is_err()); + write_bundle(dir.path(), "features", &[]); + assert!(prebuilt(dir.path(), "drivers/podman", &[]).is_err()); + std::fs::remove_file(dir.path().join(ARCHIVE)).unwrap(); + assert!(prebuilt(dir.path(), "features", &[]).is_err()); + } + + #[test] + fn selection_must_be_in_the_bundle() { + let some = Manifest { + family: "features".to_string(), + packages: vec!["oci-image".to_string()], + }; + let check = |packages: &[&str]| { + let packages: Vec<_> = packages.iter().map(ToString::to_string).collect(); + check_selection(&some, "features", &packages) + }; + assert!(check(&["oci-image"]).is_ok()); + assert!(check(&["host-gateway"]).is_err()); + assert!(check(&[]).is_err()); + } + + #[test] + fn narrow_combines_packages_and_filters() { + let strings = |items: &[&str]| items.iter().map(ToString::to_string).collect::>(); + assert_eq!(narrow(&[], &strings(&["test(a)"])), strings(&["test(a)"])); + assert_eq!( + narrow(&strings(&["oci-image", "host-gateway"]), &[]), + strings(&["package(=oci-image) | package(=host-gateway)"]) + ); + assert_eq!( + narrow(&strings(&["oci-image"]), &strings(&["test(a)", "test(b)"])), + strings(&["(package(=oci-image)) & ((test(a)) | (test(b)))"]) + ); + } + #[test] fn built_executable_finds_the_named_binary() { let messages = br#"{"reason":"compiler-artifact","target":{"name":"testkit"},"executable":null} diff --git a/tests/runner/src/execute.rs b/tests/runner/src/execute.rs index 8c46dbe103..2f477134f9 100644 --- a/tests/runner/src/execute.rs +++ b/tests/runner/src/execute.rs @@ -20,11 +20,12 @@ use anyhow::{Context, Result, ensure}; use tmachine::Session; use tmachine::config::Config; -use crate::bundle::{Bundle, PROFILE, READY_BINARY}; +use crate::bundle::{Bundle, Kind, PROFILE, PROGRAM_BINARY, READY_BINARY}; use crate::test_env; const GUEST_DIR: &str = "/var/tmp/openshell-tests"; -/// The nextest log's name, in the guest directory and in the results. +/// The test log's name, in the guest directory and in the results. A program +/// bundle's log keeps the name, so consumers of the results see one layout. const NEXTEST_LOG: &str = "nextest.log"; /// How long the gateway has to become healthy after the environment is up. // Matches the installers' own wait: an HA k3s gateway booted from a cached @@ -41,6 +42,9 @@ const READY_SLACK: Duration = Duration::from_mins(1); const NEXTEST_TEST_FAILED: i32 = 100; /// nextest's exit code when `--no-tests fail` found no tests to run. const NEXTEST_NO_TESTS_RUN: i32 = 4; +/// A testkit program's exit code when a test failed. It exits 2 when tests +/// could not run. +const PROGRAM_TEST_FAILED: i32 = 1; /// How much of the end of the nextest log to print after a failure. nextest /// ends its output with a summary that names the failed tests. const LOG_TAIL_LINES: usize = 40; @@ -91,6 +95,18 @@ impl Status { )), } } + + /// The status of a testkit program's `run` that exited with `status`. + fn of_program(status: ExitStatus, log: &Path) -> Self { + match status.code() { + Some(0) => Self::Passed, + Some(PROGRAM_TEST_FAILED) => Self::Failed, + _ => Self::Error(format!( + "the test program exited with {status}; see {}", + log.display() + )), + } + } } impl fmt::Display for Status { @@ -112,8 +128,9 @@ pub struct Run<'a> { /// The gateway name the installer registers with the CLI. pub gateway: &'a str, pub bundle: &'a Bundle, - /// A nextest filterset (`-E`) narrowing the tests in the bundle. - pub filter: Option<&'a str>, + /// Narrowing the tests in the bundle: nextest filtersets (`-E`) for a + /// nextest bundle, parts of test names for a program bundle. + pub filters: &'a [String], /// Where `junit.xml`, `nextest.log` and, after a failure, the guest's /// `diagnostics/` go. pub results: &'a Path, @@ -244,15 +261,20 @@ async fn run_tests(session: &Session, run: &Run<'_>) -> Result { .exec(&["mkdir", "-p", &format!("{guest}/extract")]) .await?, )?; + let kind = run.bundle.kind(); // A booted guest restarts its gateway, so wait for it once here rather // than in every test. Failing here is an environment error, not a test // failure. + let program = format!("{guest}/{PROGRAM_BINARY}"); let ready_binary = format!("{guest}/{READY_BINARY}"); let ready_timeout = READY_TIMEOUT_SECS.to_string(); let mut ready = vec!["env"]; ready.extend(env.iter().map(String::as_str)); - ready.extend([ready_binary.as_str(), ready_timeout.as_str()]); + match kind { + Kind::Nextest => ready.extend([ready_binary.as_str(), ready_timeout.as_str()]), + Kind::Program => ready.extend([program.as_str(), "ready", "--timeout", &ready_timeout]), + } checked( "wait for the gateway", session @@ -282,36 +304,52 @@ async fn run_tests(session: &Session, run: &Run<'_>) -> Result { )?; } + let junit = format!("{guest}/junit.xml"); let archive = format!("{guest}/tests.tar.zst"); let extract = format!("{guest}/extract"); let config = format!("{guest}/nextest.toml"); - let mut nextest = vec!["env"]; - nextest.extend(env.iter().map(String::as_str)); - nextest.extend([ - "cargo-nextest", - "nextest", - "run", - "--archive-file", - &archive, - "--workspace-remap", - guest, - "--extract-to", - &extract, - "--extract-overwrite", - "--config-file", - &config, - "--profile", - PROFILE, - "--no-capture", - // Tests share one gateway and the guest has 4 GiB; run them serially - // whatever --no-capture implies. - "--test-threads", - "1", - "--no-tests", - "fail", - ]); - if let Some(filter) = run.filter { - nextest.extend(["-E", filter]); + let mut tests = vec!["env"]; + tests.extend(env.iter().map(String::as_str)); + match kind { + Kind::Nextest => { + tests.extend([ + "cargo-nextest", + "nextest", + "run", + "--archive-file", + &archive, + "--workspace-remap", + guest, + "--extract-to", + &extract, + "--extract-overwrite", + "--config-file", + &config, + "--profile", + PROFILE, + "--no-capture", + // Tests share one gateway and the guest has 4 GiB; run them + // serially whatever --no-capture implies. + "--test-threads", + "1", + "--no-tests", + "fail", + ]); + for filter in run.filters { + tests.extend(["-E", filter]); + } + } + // The program runs its tests one at a time, and writes the report + // where `collect_results` looks for it. + Kind::Program => { + tests.extend([program.as_str(), "run", "--junit", &junit]); + // `run` takes name filters as arguments, after `--` so that one + // starting with a dash isn't read as an option. + if !run.filters.is_empty() { + tests.push("--"); + tests.extend(run.filters.iter().map(String::as_str)); + } + } } // nextest writes its output to a file in the guest, so a run that times // out or loses SSH still leaves a log for `collect_results`. The guest @@ -323,12 +361,13 @@ async fn run_tests(session: &Session, run: &Run<'_>) -> Result { test_env::export_script(test_env::ENV_DIR) ); let mut command = vec!["sh", "-c", &script, &log]; - command.extend(nextest); + command.extend(tests); let output = session.exec_timeout(&command, RUN_TIMEOUT).await?; - Ok(Status::of_nextest( - output.status, - &run.results.join(NEXTEST_LOG), - )) + let log = run.results.join(NEXTEST_LOG); + Ok(match kind { + Kind::Nextest => Status::of_nextest(output.status, &log), + Kind::Program => Status::of_program(output.status, &log), + }) } /// Copy the nextest log and the JUnit report out of the guest, whether or not @@ -488,6 +527,15 @@ mod tests { assert!(matches!(status(1), Status::Error(_))); } + #[test] + fn status_follows_the_program_exit_code() { + let status = |code| Status::of_program(ExitStatus::from_raw(code << 8), Path::new("log")); + assert_eq!(status(0), Status::Passed); + assert_eq!(status(PROGRAM_TEST_FAILED), Status::Failed); + // Exit 2: tests could not run. + assert!(matches!(status(2), Status::Error(_))); + } + #[test] fn k3s_diagnostics_keep_everything_the_old_playbook_collected() { for wanted in [ diff --git a/tests/runner/src/main.rs b/tests/runner/src/main.rs index 9bac397824..bf07210261 100644 --- a/tests/runner/src/main.rs +++ b/tests/runner/src/main.rs @@ -1,9 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! `openshell-test-runner`: run a test family in one tmachine environment. +//! `openshell-test-runner`: build a test family for tmachine guests, and run +//! it in one tmachine environment. //! -//! Run it through `nix run .#openshell-test-runner -- `, which +//! Run it through `nix run .#openshell-test-runner -- `, which //! provides the guest cross-compilation toolchain, QEMU, Ansible, and the //! tmachine configuration. @@ -11,7 +12,7 @@ use std::path::{Path, PathBuf}; use std::process::ExitCode; use anyhow::{Context, Result, ensure}; -use clap::Parser; +use clap::{Args, Parser, Subcommand}; use crate::execute::{Run, Status}; @@ -19,28 +20,69 @@ mod bundle; mod execute; mod test_env; -/// Build a test family for the guest, boot a tmachine environment with -/// OpenShell installed, run the tests in it, and check the results. #[derive(Parser)] #[command(name = "openshell-test-runner")] struct Cli { - /// The tmachine environment to boot, e.g. `ubuntu-docker-rootful`. - #[arg(long)] - environment: String, - /// The tmachine installer that installs OpenShell, e.g. `deb`. - #[arg(long)] - installer: String, + #[command(subcommand)] + command: Command, +} + +#[derive(Subcommand)] +enum Command { + /// Build a test family's bundle for the guest into a directory, for + /// `run --bundle`. + Build(BuildArgs), + /// Boot a tmachine environment with OpenShell installed, run a test + /// family in it, and check the results. + Run(RunArgs), +} + +/// The tests to build or run. +#[derive(Args)] +struct Selection { /// The test family: the path of a workspace under `tests/suites/`, e.g. /// `conformance` or `drivers/podman`. #[arg(long)] family: String, - /// Only build these packages of the family [default: every package]. + /// Only these packages of the family [default: every package]. #[arg(long = "package", short = 'p', value_name = "PACKAGE")] packages: Vec, - /// A nextest filterset narrowing the tests that run, e.g. - /// `test(/^smoke::/)`. - #[arg(long, value_name = "FILTERSET")] - filter: Option, +} + +#[derive(Args)] +struct BuildArgs { + #[command(flatten)] + selection: Selection, + /// The directory to write the bundle to. It must be empty or missing. + #[arg(long, value_name = "DIR")] + out: PathBuf, +} + +#[derive(Args)] +struct RunArgs { + /// The tmachine environment to boot, e.g. `ubuntu-docker-rootful`. + #[arg(long)] + environment: String, + /// The tmachine installer that installs OpenShell, e.g. `deb`. + #[arg(long)] + installer: String, + #[command(flatten)] + selection: Selection, + /// Narrow the tests that run; repeat to run the tests that match any + /// filter. Without `--program` a filter is a nextest filterset, e.g. + /// `test(/^smoke::/)`. With it, a filter is a part of a test's name, e.g. + /// `smoke::`, as the program takes them. + #[arg(long, value_name = "FILTER")] + filter: Vec, + /// Run the family's bundle that `build` wrote to this directory instead + /// of building it. `--package` then selects the tests to run from it. + #[arg(long, value_name = "DIR", conflicts_with = "program")] + bundle: Option, + /// Run this prebuilt testkit program, such as `openshell-conformance`, + /// instead of building the family's tests. It must be a static binary for + /// the guest's architecture (a `*-unknown-linux-musl` build). + #[arg(long, value_name = "PATH", conflicts_with = "packages")] + program: Option, /// The gateway name the installer registers with the CLI. The packaged /// installers register `openshell`. #[arg(long, default_value = "openshell")] @@ -58,7 +100,11 @@ struct Cli { async fn main() -> ExitCode { let cli = Cli::parse(); // A SIGTERM from CI cancellation must not leave QEMU holding the SSH port. - match tmachine::until_signal(run(cli)).await { + let result = match cli.command { + Command::Build(args) => tmachine::until_signal(build(args)).await, + Command::Run(args) => tmachine::until_signal(run(args)).await, + }; + match result { // Exit at once: a blocking build or stdin read would otherwise hold // up the runtime's shutdown. Err(status) => std::process::exit(status), @@ -71,44 +117,66 @@ async fn main() -> ExitCode { } } -async fn run(cli: Cli) -> Result { +async fn build(args: BuildArgs) -> Result { let root = repo_root()?; - let workspace = family_workspace(&root, &cli.family)?; - // Absolute, because `load_tmachine` changes the working directory. - let results = std::path::absolute(&cli.results) - .with_context(|| format!("resolve {}", cli.results.display()))?; - - let target = bundle::guest_target()?; - // The builds block, so they run off the runtime thread to keep the - // signal handlers responsive. - let bundle = tokio::task::spawn_blocking({ - let (root, family, packages) = (root.clone(), cli.family.clone(), cli.packages.clone()); - move || { - eprintln!("==> building {} for {target}", bundle::READY_BINARY); - let ready = bundle::build_ready(&root, target)?; - eprintln!("==> building the {family} tests for {target}"); - bundle::build(&workspace, &packages, target, &ready) - } + let Selection { family, packages } = args.selection; + let out = args.out; + // The build blocks, so it runs off the runtime thread to keep the signal + // handlers responsive. + tokio::task::spawn_blocking(move || { + bundle::build(&root, &family, &packages, &out)?; + eprintln!("==> wrote the {family} bundle to {}", out.display()); + anyhow::Ok(()) }) .await .context("build task panicked")??; + Ok(ExitCode::SUCCESS) +} + +async fn run(args: RunArgs) -> Result { + let root = repo_root()?; + let Selection { family, packages } = &args.selection; + family_workspace(&root, family)?; + // Absolute, because `load_tmachine` changes the working directory. + let results = std::path::absolute(&args.results) + .with_context(|| format!("resolve {}", args.results.display()))?; + + let mut filters = args.filter.clone(); + let bundle = if let Some(program) = &args.program { + eprintln!("==> using the prebuilt {}", program.display()); + bundle::program(program)? + } else if let Some(dir) = &args.bundle { + eprintln!("==> using the prebuilt bundle {}", dir.display()); + // A prebuilt bundle may hold more packages than were asked for. + filters = bundle::narrow(packages, &filters); + bundle::prebuilt(dir, family, packages)? + } else { + // The builds block, so they run off the runtime thread to keep the + // signal handlers responsive. + tokio::task::spawn_blocking({ + let (root, family, packages) = (root.clone(), family.clone(), packages.clone()); + move || bundle::build_temp(&root, &family, &packages) + }) + .await + .context("build task panicked")?? + }; let tmachine = load_tmachine(&root)?; let status = execute::run( &tmachine, &Run { - environment: &cli.environment, - installer: &cli.installer, - gateway: &cli.gateway, + environment: &args.environment, + installer: &args.installer, + gateway: &args.gateway, bundle: &bundle, - filter: cli.filter.as_deref(), + filters: &filters, results: &results, - keep: cli.keep, + keep: args.keep, }, ) .await; - let label = format!("{} / {} / {}", cli.environment, cli.installer, cli.family); + let label = format!("{} / {} / {family}", args.environment, args.installer); eprintln!("\n{label}: {status}"); eprintln!("Results: {}", results.display()); match &status { diff --git a/tests/suites/conformance/Cargo.lock b/tests/suites/conformance/Cargo.lock index ea04cd150d..0d4ab62225 100644 --- a/tests/suites/conformance/Cargo.lock +++ b/tests/suites/conformance/Cargo.lock @@ -493,6 +493,16 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + [[package]] name = "concurrent-queue" version = "2.5.0" @@ -506,6 +516,7 @@ dependencies = [ name = "conformance" version = "0.0.0" dependencies = [ + "host-server-support", "openshell-testkit", "serde_json", "tempfile", @@ -561,6 +572,12 @@ version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + [[package]] name = "crossbeam-queue" version = "0.3.12" @@ -1138,6 +1155,25 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hickory-proto" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bab31817bfb44672a252e97fe81cd0c18d1b2cf892108922f6818820df8c643" +dependencies = [ + "data-encoding", + "idna", + "ipnet", + "jni", + "once_cell", + "rand 0.10.2", + "ring", + "thiserror 2.0.20", + "tinyvec", + "tracing", + "url", +] + [[package]] name = "hkdf" version = "0.13.0" @@ -1165,6 +1201,15 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "host-server-support" +version = "0.0.0" +dependencies = [ + "openshell-testkit", + "tokio", + "tungstenite", +] + [[package]] name = "hostname" version = "0.4.2" @@ -1518,6 +1563,55 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror 2.0.20", + "walkdir", + "windows-link", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -2021,6 +2115,10 @@ name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "critical-section", + "portable-atomic", +] [[package]] name = "once_cell_polyfill" @@ -2109,6 +2207,19 @@ dependencies = [ "tower", ] +[[package]] +name = "openshell-policy" +version = "0.0.0" +dependencies = [ + "hickory-proto", + "miette", + "openshell-core", + "openshell-policy-schema", + "prost-types", + "serde", + "serde_json", +] + [[package]] name = "openshell-policy-schema" version = "0.0.0" @@ -2155,15 +2266,19 @@ dependencies = [ "libc", "libtest-mimic", "linkme", + "noyalib", "openshell-bootstrap", "openshell-core", + "openshell-policy", "openshell-sdk", "openshell-testkit-macros", "quick-junit", "rand 0.9.4", "serde", "serde_json", + "tempfile", "tokio", + "tonic", ] [[package]] @@ -2337,6 +2452,12 @@ version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + [[package]] name = "potential_utf" version = "0.1.5" @@ -2755,6 +2876,15 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rusticata-macros" version = "4.1.0" @@ -2837,6 +2967,15 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + [[package]] name = "schannel" version = "0.1.29" @@ -3083,6 +3222,22 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version", + "simdutf8", +] + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + [[package]] name = "simple_asn1" version = "0.6.4" @@ -3951,6 +4106,16 @@ dependencies = [ "memchr", ] +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + [[package]] name = "want" version = "0.3.1" @@ -4098,6 +4263,15 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "626c4bac6755d76ffc12cb01b2eac751db1996b9e0041de9aa02c8c211ddc82c" +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "windows-core" version = "0.62.2" diff --git a/tests/suites/conformance/Cargo.toml b/tests/suites/conformance/Cargo.toml index 99274d8dfa..7201953c9d 100644 --- a/tests/suites/conformance/Cargo.toml +++ b/tests/suites/conformance/Cargo.toml @@ -21,6 +21,7 @@ path = "src/main.rs" test = false [dependencies] +host-server-support = { path = "../../support/host-server" } openshell-testkit = { path = "../../testkit/testkit" } serde_json = "1" tempfile = "3" diff --git a/tests/suites/conformance/README.md b/tests/suites/conformance/README.md index c756107cc1..c51df37c16 100644 --- a/tests/suites/conformance/README.md +++ b/tests/suites/conformance/README.md @@ -14,6 +14,7 @@ nextest run the tests as the library's unit tests. The crate also builds the | `persistent-workspace` | Files in `/sandbox` survive stop and start | | `policy-advisor` | `policy.local` accepts a proposed rule; a denied address or hostname yields one draft rule | | `file-transfer` | `sandbox upload` and `download`: round trips, `.gitignore` filtering, workspace-escape and option-looking names | +| `credential-gating` | Provider credentials: admission of credentialed endpoints, REST body placeholders and rewriting, WebSocket binary frames, live policy binding | `smoke` is the smallest end-to-end check that the runner can run a real test in a tmachine environment on CI. The other suites are ports of the old @@ -21,7 +22,10 @@ tmachine environment on CI. The other suites are ports of the old (credited in each file). `file-transfer` runs `git` on the machine that runs the tests, so a test environment needs it installed (`tests/tmachine/ansible/playbooks/test-tools.yaml` does that for the tmachine -environments). +environments). `credential-gating`, ported from NVIDIA/OpenShell#5264, starts +servers (`tests/support/host-server`) on the machine that runs the tests and +reaches them from sandboxes created from `testkit::python_image()` through +`host.openshell.internal`. ## Running it @@ -42,7 +46,10 @@ tests/suites/conformance/target/debug/openshell-conformance run --junit conforma ``` The program runs only this suite: passing it means a gateway meets the -conformance contract below. Feature and driver suites are not part of it. The +conformance contract below. The machine that runs it must be reachable from +sandboxes as `host.openshell.internal`, which rules out a gateway on a remote +Kubernetes cluster ([#5409](https://github.com/NVIDIA/OpenShell/issues/5409)). +Feature and driver suites are not part of it. The [testkit README](../../testkit/README.md#test-programs) describes the commands and exit codes. `tasks/scripts/check-conformance-program.sh` checks that it lists the same tests as nextest. diff --git a/tests/suites/conformance/src/credential_gating.rs b/tests/suites/conformance/src/credential_gating.rs new file mode 100644 index 0000000000..ea67464e35 --- /dev/null +++ b/tests/suites/conformance/src/credential_gating.rs @@ -0,0 +1,425 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Credential gating: which endpoints a provider credential may reach, and +//! what the supervisor forwards upstream in REST bodies and WebSocket frames. +//! +//! Ported from the e2e tests added in NVIDIA/OpenShell#5264. The upstream +//! servers run on the test host ([`host_server_support`]), so the suite +//! requires sandboxes to reach the test host as `host.openshell.internal`. + +use std::time::Duration; + +use host_server_support::{HttpRecorder, Request, WebSocketRecorder}; +use testkit::policy::{self, Access, Endpoint as _}; +use testkit::sdk::raw::proto; +use testkit::{ + Context, Error, GrpcCode, PYTHON_BINARIES, Poll, Result, ResultExt as _, Sandbox, YamlFile, +}; + +const TEST_HOST: &str = "host.openshell.internal"; +const TOKEN_ENV: &str = "E2E_GATING_TOKEN"; +const TEST_SECRET: &str = "e2e-gating-secret-value"; +const PLACEHOLDER_PREFIX: &str = "openshell:resolve:env:"; + +/// POSTs a body containing the provider's token to `TEST_HOST` on the port +/// in `argv[1]`, directly rather than through a proxy. +const BODY_CLIENT: &str = r#" +import os, sys, urllib.request +body = ("prefix-" + os.environ["E2E_GATING_TOKEN"] + "-suffix").encode() +request = urllib.request.Request(f"http://host.openshell.internal:{sys.argv[1]}/token", data=body, headers={"Content-Type": "text/plain"}) +urllib.request.build_opener(urllib.request.ProxyHandler({})).open(request, timeout=10).read() +"#; + +/// Upgrades to WebSocket at `TEST_HOST` on the port in `argv[1]`, sends one +/// binary frame, and waits briefly for the connection to be closed. +const WEBSOCKET_CLIENT: &str = r#" +import base64, os, socket, sys +host, port = "host.openshell.internal", int(sys.argv[1]) +with socket.create_connection((host, port), timeout=10) as sock: + key = base64.b64encode(os.urandom(16)).decode() + sock.sendall(f"GET /ws HTTP/1.1\r\nHost: {host}:{port}\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n".encode()) + response = b"" + while b"\r\n\r\n" not in response: + chunk = sock.recv(4096) + if not chunk: + sys.exit(f"closed during the upgrade: {response!r}") + response += chunk + if not response.startswith(b"HTTP/1.1 101"): + sys.exit(f"upgrade failed: {response!r}") + payload, mask = b"binary-credential-channel", os.urandom(4) + sock.sendall(bytes([0x82, 0x80 | len(payload)]) + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload))) + sock.settimeout(3) + try: + sock.recv(4096) + except (TimeoutError, ConnectionError): + pass +"#; + +/// Prints `TOKEN_ABSENT`, `TOKEN_PLACEHOLDER`, or `TOKEN_UNSAFE` for the +/// provider's environment variable, without printing its value. +const TOKEN_PRESENCE: &str = r#"if [ "${E2E_GATING_TOKEN+x}" != x ]; then + echo TOKEN_ABSENT +else + case "$E2E_GATING_TOKEN" in + openshell:resolve:env:*) echo TOKEN_PLACEHOLDER ;; + *) echo TOKEN_UNSAFE ;; + esac +fi"#; + +/// A provider whose profile declares the REST and WebSocket servers as its +/// endpoints keeps the credential placeholder in a REST body. +#[testkit::test] +async fn profile_rest_endpoint_keeps_body_placeholder(cx: &mut Context) -> testkit::Result { + let server = HttpRecorder::start("ok").await?; + let websocket = WebSocketRecorder::start().await?; + let provider = install_gating_provider(cx, Some((server.port, websocket.port))).await?; + + let mut policy = policy::base(); + allow(&mut policy, server.port) + .rest(Access::Full) + .request_body_credential_rewrite(false); + let sandbox = create_sandbox(cx, "rest", policy, &[&provider]).await?; + run_python(&sandbox, BODY_CLIENT, server.port).await?; + expect_placeholder_body(&server).await +} + +/// A provider's WebSocket endpoint completes the handshake but doesn't +/// forward a binary frame, which the supervisor can't inspect for the +/// credential. The same client in a sandbox without the provider gets its +/// binary frame through, which shows the denial comes from the credential. +#[testkit::test] +async fn profile_websocket_endpoint_denies_binary_frames(cx: &mut Context) -> testkit::Result { + let server = HttpRecorder::start("ok").await?; + let websocket = WebSocketRecorder::start().await?; + let provider = install_gating_provider(cx, Some((server.port, websocket.port))).await?; + + let control = WebSocketRecorder::start().await?; + let binary_seen = send_binary_frame(cx, "wsctl", &control, &[]).await?; + testkit::ensure!( + binary_seen, + "the binary frame didn't reach upstream from a sandbox without the provider" + ); + + let binary_seen = send_binary_frame(cx, "ws", &websocket, &[&provider]).await?; + testkit::ensure!( + !binary_seen, + "credentialed WebSocket binary frame reached upstream" + ); + Ok(()) +} + +/// The gateway rejects a policy that binds a provider to an L4-only +/// endpoint or to an endpoint with `tls: skip`, and accepts an L4 endpoint +/// that opts in to uninspected credentials. +#[testkit::test] +async fn policy_binding_requires_inspection_or_opt_in(cx: &mut Context) -> testkit::Result { + let server = HttpRecorder::start("ok").await?; + let provider = install_gating_provider(cx, None).await?; + + let mut l4 = policy::base(); + allow(&mut l4, server.port).credential_binding(&provider); + // The wording checks follow the #5264 tests. + let error = expect_rejected(cx, "l4", l4, &provider).await?; + testkit::ensure!(error.contains("credentialed endpoint"), "{error}"); + testkit::ensure!(error.contains("L4-only"), "{error}"); + + let mut tls_skip = policy::base(); + allow(&mut tls_skip, server.port) + .rest(Access::Full) + .tls_skip() + .credential_binding(&provider); + let error = expect_rejected(cx, "tls", tls_skip, &provider).await?; + testkit::ensure!(error.contains("credentialed endpoint"), "{error}"); + testkit::ensure!(error.contains("tls:") && error.contains("skip"), "{error}"); + + let mut opt_in = policy::base(); + allow(&mut opt_in, server.port) + .allow_uninspected_credentials() + .credential_binding(&provider); + let sandbox = create_sandbox(cx, "optin", opt_in, &[&provider]).await?; + sandbox.exec(["true"]).await?.success()?; + Ok(()) +} + +/// A policy binding to a REST endpoint without body rewriting sends the +/// credential placeholder upstream, not the secret. +#[testkit::test] +async fn policy_binding_rest_body_keeps_placeholder_without_rewrite( + cx: &mut Context, +) -> testkit::Result { + let server = HttpRecorder::start("ok").await?; + let provider = install_gating_provider(cx, None).await?; + + let mut policy = policy::base(); + allow(&mut policy, server.port) + .rest(Access::Full) + .request_body_credential_rewrite(false) + .credential_binding(&provider); + let sandbox = create_sandbox(cx, "body", policy, &[&provider]).await?; + run_python(&sandbox, BODY_CLIENT, server.port).await?; + expect_placeholder_body(&server).await +} + +/// A policy binding with `request_body_credential_rewrite: true` replaces +/// the placeholder in the REST body with the secret upstream. +#[testkit::test] +async fn policy_binding_rest_body_rewrite_sends_secret_upstream( + cx: &mut Context, +) -> testkit::Result { + let server = HttpRecorder::start("ok").await?; + let provider = install_gating_provider(cx, None).await?; + + let mut policy = policy::base(); + allow(&mut policy, server.port) + .rest(Access::Full) + .request_body_credential_rewrite(true) + .credential_binding(&provider); + let sandbox = create_sandbox(cx, "rw", policy, &[&provider]).await?; + run_python(&sandbox, BODY_CLIENT, server.port).await?; + let request = one_request(&server).await?; + testkit::ensure!( + request.body == format!("prefix-{TEST_SECRET}-suffix"), + "expected the secret in the body upstream; got {}", + redact(&request) + ); + Ok(()) +} + +/// Binding and unbinding a provider with `policy set --wait` adds and +/// removes its environment variable in a running sandbox. +#[testkit::test] +async fn policy_binding_updates_provider_environment_live(cx: &mut Context) -> testkit::Result { + let server = HttpRecorder::start("ok").await?; + let provider = install_gating_provider(cx, None).await?; + + let policy_binding = |bind: bool| { + let mut policy = policy::base(); + let endpoint = allow(&mut policy, server.port) + .rest(Access::Full) + .request_body_credential_rewrite(false); + if bind { + endpoint.credential_binding(&provider); + } + policy + }; + let (unbound, bound) = (policy_binding(false), policy_binding(true)); + + let sandbox = create_sandbox(cx, "live", unbound.clone(), &[&provider]).await?; + wait_for_token(&sandbox, "TOKEN_ABSENT").await?; + + set_policy(cx, &sandbox, &bound) + .await + .context("bind the provider")?; + wait_for_token(&sandbox, "TOKEN_PLACEHOLDER").await?; + + set_policy(cx, &sandbox, &unbound) + .await + .context("unbind the provider")?; + wait_for_token(&sandbox, "TOKEN_ABSENT").await +} + +/// Run [`WEBSOCKET_CLIENT`] against `server` in a sandbox with `providers`, +/// and return whether its binary frame reached `server`. +async fn send_binary_frame( + cx: &Context, + suffix: &str, + server: &WebSocketRecorder, + providers: &[&str], +) -> Result { + let mut policy = policy::base(); + allow(&mut policy, server.port).websocket(Access::ReadWrite); + let sandbox = create_sandbox(cx, suffix, policy, providers).await?; + run_python(&sandbox, WEBSOCKET_CLIENT, server.port).await?; + server + .wait_for_handshake() + .await + .context("upstream should receive the WebSocket handshake")?; + server.wait_until_finished().await +} + +/// Let the image's Python reach `TEST_HOST` on `port`. +fn allow(policy: &mut proto::SandboxPolicy, port: u16) -> &mut proto::NetworkEndpoint { + policy::allow( + policy, + "credential_gating", + TEST_HOST, + port, + PYTHON_BINARIES, + ) +} + +/// Install a run-unique provider holding `TEST_SECRET` in `TOKEN_ENV`. With +/// `endpoints`, its profile declares the REST and WebSocket ports as its +/// endpoints; without, a policy must bind the provider explicitly. +async fn install_gating_provider(cx: &Context, endpoints: Option<(u16, u16)>) -> Result { + let name = cx.name("cg"); + let endpoints = endpoints.map_or_else(String::new, |(rest, websocket)| { + format!( + "endpoints: + - host: {TEST_HOST} + port: {rest} + protocol: rest + access: full + - host: {TEST_HOST} + port: {websocket} + protocol: websocket + access: read-write +" + ) + }); + let binaries: String = PYTHON_BINARIES + .iter() + .map(|path| format!(" - path: {path}\n")) + .collect(); + let profile = format!( + "id: {name} +display_name: E2E Credential Gating +category: other +credentials: + - name: token + env_vars: [{TOKEN_ENV}] + required: true + auth_style: bearer + header_name: authorization +{endpoints}binaries: +{binaries}" + ); + cx.install_provider(&name, &profile, &[(TOKEN_ENV, TEST_SECRET)]) + .await?; + Ok(name) +} + +fn sandbox_with( + cx: &Context, + suffix: &str, + policy: proto::SandboxPolicy, + providers: &[&str], +) -> testkit::SandboxBuilder { + cx.sandbox(suffix) + .policy(policy) + .spec(|spec| { + spec.image = Some(testkit::python_image()); + spec.providers = providers.iter().map(ToString::to_string).collect(); + }) + .command(["sleep", "infinity"]) +} + +/// Create a long-running sandbox from the Python image with `policy` and +/// `providers`, and wait until it is Ready. +async fn create_sandbox( + cx: &Context, + suffix: &str, + policy: proto::SandboxPolicy, + providers: &[&str], +) -> Result { + let sandbox = sandbox_with(cx, suffix, policy, providers).create().await?; + sandbox.wait_ready().await?; + Ok(sandbox) +} + +/// Fail unless the gateway rejects the sandbox as a failed precondition, and +/// return the rejection. +async fn expect_rejected( + cx: &Context, + suffix: &str, + policy: proto::SandboxPolicy, + provider: &str, +) -> Result { + match sandbox_with(cx, suffix, policy, &[provider]).create().await { + Ok(sandbox) => Err(Error::msg(format!( + "the gateway accepted sandbox {}", + sandbox.name() + ))), + Err(error) if error.grpc_code() == Some(GrpcCode::FailedPrecondition) => { + Ok(error.to_string()) + } + Err(error) => Err(error.context("expected a failed precondition")), + } +} + +/// Run `script` with the image's Python and `port` as its argument. +async fn run_python(sandbox: &Sandbox, script: &str, port: u16) -> Result { + let port = port.to_string(); + sandbox + .exec(["python3", "-c", script, &port]) + .await? + .success() + .context("run the client in the sandbox")?; + Ok(()) +} + +/// The one request `server` saw. +async fn one_request(server: &HttpRecorder) -> Result { + let mut requests = server.wait_for_requests(1).await?; + testkit::ensure!( + requests.len() == 1, + "expected one request upstream; got {}", + requests.len() + ); + Ok(requests.remove(0)) +} + +/// Fail unless `server` saw one request whose body has the credential +/// placeholder in place of the token, and the secret appears nowhere in it. +async fn expect_placeholder_body(server: &HttpRecorder) -> Result { + let request = one_request(server).await?; + testkit::ensure!( + request + .body + .starts_with(&format!("prefix-{PLACEHOLDER_PREFIX}")) + && request.body.ends_with("-suffix") + && !request.head.contains(TEST_SECRET) + && !request.body.contains(TEST_SECRET), + "expected the placeholder in the body upstream and no secret; got {}", + redact(&request) + ); + Ok(()) +} + +/// `request` for a failure message, with the secret replaced. +fn redact(request: &Request) -> String { + format!("{}\n\n{}", request.head, request.body).replace(TEST_SECRET, "") +} + +/// Wait until new exec sessions report `expected` from [`TOKEN_PRESENCE`]. +async fn wait_for_token(sandbox: &Sandbox, expected: &str) -> Result { + testkit::poll_until( + &format!("the provider environment to report {expected}"), + Duration::from_secs(30), + Duration::from_millis(500), + async || { + let output = sandbox.exec(["sh", "-lc", TOKEN_PRESENCE]).await?; + Ok(if output.stdout.contains(expected) { + Poll::Ready(()) + } else { + Poll::Pending(format!( + "exit {}; stdout: {}; stderr: {}", + output.exit_code, + output.stdout.trim(), + output.stderr.trim() + )) + }) + }, + ) + .await +} + +/// Replace `sandbox`'s policy with `policy` and wait until it is applied. +async fn set_policy(cx: &Context, sandbox: &Sandbox, policy: &proto::SandboxPolicy) -> Result { + let file = YamlFile::new(&policy::to_yaml(policy)?)?; + cx.cli([ + "policy", + "set", + sandbox.name(), + "--policy", + file.path()?, + "--wait", + "--timeout", + "120", + ]) + .await? + .success() + .context("policy set")?; + Ok(()) +} diff --git a/tests/suites/conformance/src/lib.rs b/tests/suites/conformance/src/lib.rs index 00a4d45027..98b916092f 100644 --- a/tests/suites/conformance/src/lib.rs +++ b/tests/suites/conformance/src/lib.rs @@ -9,6 +9,7 @@ //! program in `src/main.rs` runs them by linking this library. mod cli; +mod credential_gating; mod file_transfer; mod lifecycle; mod persistent_workspace; diff --git a/tests/suites/drivers/podman/Cargo.lock b/tests/suites/drivers/podman/Cargo.lock index 6633a3e703..6657ca1da6 100644 --- a/tests/suites/drivers/podman/Cargo.lock +++ b/tests/suites/drivers/podman/Cargo.lock @@ -493,6 +493,16 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + [[package]] name = "concurrent-queue" version = "2.5.0" @@ -560,6 +570,12 @@ version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + [[package]] name = "crossbeam-queue" version = "0.3.12" @@ -1137,6 +1153,25 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hickory-proto" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bab31817bfb44672a252e97fe81cd0c18d1b2cf892108922f6818820df8c643" +dependencies = [ + "data-encoding", + "idna", + "ipnet", + "jni", + "once_cell", + "rand 0.10.2", + "ring", + "thiserror 2.0.20", + "tinyvec", + "tracing", + "url", +] + [[package]] name = "hkdf" version = "0.13.0" @@ -1517,6 +1552,55 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror 2.0.20", + "walkdir", + "windows-link", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -2020,6 +2104,10 @@ name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "critical-section", + "portable-atomic", +] [[package]] name = "once_cell_polyfill" @@ -2108,6 +2196,19 @@ dependencies = [ "tower", ] +[[package]] +name = "openshell-policy" +version = "0.0.0" +dependencies = [ + "hickory-proto", + "miette", + "openshell-core", + "openshell-policy-schema", + "prost-types", + "serde", + "serde_json", +] + [[package]] name = "openshell-policy-schema" version = "0.0.0" @@ -2154,15 +2255,19 @@ dependencies = [ "libc", "libtest-mimic", "linkme", + "noyalib", "openshell-bootstrap", "openshell-core", + "openshell-policy", "openshell-sdk", "openshell-testkit-macros", "quick-junit", "rand 0.9.4", "serde", "serde_json", + "tempfile", "tokio", + "tonic", ] [[package]] @@ -2353,6 +2458,12 @@ dependencies = [ "openshell-testkit", ] +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + [[package]] name = "potential_utf" version = "0.1.5" @@ -2771,6 +2882,15 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rusticata-macros" version = "4.1.0" @@ -2853,6 +2973,15 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + [[package]] name = "schannel" version = "0.1.29" @@ -3099,6 +3228,22 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version", + "simdutf8", +] + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + [[package]] name = "simple_asn1" version = "0.6.4" @@ -3967,6 +4112,16 @@ dependencies = [ "memchr", ] +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + [[package]] name = "want" version = "0.3.1" @@ -4114,6 +4269,15 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "626c4bac6755d76ffc12cb01b2eac751db1996b9e0041de9aa02c8c211ddc82c" +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "windows-core" version = "0.62.2" diff --git a/tests/suites/features/Cargo.lock b/tests/suites/features/Cargo.lock index bb1a1427ee..bcc428d57d 100644 --- a/tests/suites/features/Cargo.lock +++ b/tests/suites/features/Cargo.lock @@ -493,6 +493,16 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + [[package]] name = "concurrent-queue" version = "2.5.0" @@ -560,6 +570,12 @@ version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + [[package]] name = "crossbeam-queue" version = "0.3.12" @@ -1137,6 +1153,25 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hickory-proto" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bab31817bfb44672a252e97fe81cd0c18d1b2cf892108922f6818820df8c643" +dependencies = [ + "data-encoding", + "idna", + "ipnet", + "jni", + "once_cell", + "rand 0.10.2", + "ring", + "thiserror 2.0.20", + "tinyvec", + "tracing", + "url", +] + [[package]] name = "hkdf" version = "0.13.0" @@ -1167,10 +1202,18 @@ dependencies = [ [[package]] name = "host-gateway" version = "0.0.0" +dependencies = [ + "host-server-support", + "openshell-testkit", +] + +[[package]] +name = "host-server-support" +version = "0.0.0" dependencies = [ "openshell-testkit", - "tempfile", "tokio", + "tungstenite", ] [[package]] @@ -1526,6 +1569,55 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror 2.0.20", + "walkdir", + "windows-link", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -2057,6 +2149,10 @@ name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "critical-section", + "portable-atomic", +] [[package]] name = "once_cell_polyfill" @@ -2145,6 +2241,19 @@ dependencies = [ "tower", ] +[[package]] +name = "openshell-policy" +version = "0.0.0" +dependencies = [ + "hickory-proto", + "miette", + "openshell-core", + "openshell-policy-schema", + "prost-types", + "serde", + "serde_json", +] + [[package]] name = "openshell-policy-schema" version = "0.0.0" @@ -2203,15 +2312,19 @@ dependencies = [ "libc", "libtest-mimic", "linkme", + "noyalib", "openshell-bootstrap", "openshell-core", + "openshell-policy", "openshell-sdk", "openshell-testkit-macros", "quick-junit", "rand 0.9.4", "serde", "serde_json", + "tempfile", "tokio", + "tonic", ] [[package]] @@ -2385,6 +2498,12 @@ version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + [[package]] name = "potential_utf" version = "0.1.5" @@ -2803,6 +2922,15 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rusticata-macros" version = "4.1.0" @@ -2885,6 +3013,15 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + [[package]] name = "schannel" version = "0.1.29" @@ -3131,6 +3268,22 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version", + "simdutf8", +] + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + [[package]] name = "simple_asn1" version = "0.6.4" @@ -3999,6 +4152,16 @@ dependencies = [ "memchr", ] +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + [[package]] name = "want" version = "0.3.1" @@ -4146,6 +4309,15 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "626c4bac6755d76ffc12cb01b2eac751db1996b9e0041de9aa02c8c211ddc82c" +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "windows-core" version = "0.62.2" diff --git a/tests/suites/features/host-gateway/Cargo.toml b/tests/suites/features/host-gateway/Cargo.toml index d81b6cafb2..0dfef50bc5 100644 --- a/tests/suites/features/host-gateway/Cargo.toml +++ b/tests/suites/features/host-gateway/Cargo.toml @@ -12,6 +12,5 @@ repository = "https://github.com/NVIDIA/OpenShell" publish = false [dependencies] +host-server-support = { path = "../../../support/host-server" } openshell-testkit = { path = "../../../testkit/testkit" } -tempfile = "3" -tokio = { version = "1.43", features = ["io-util", "net", "rt"] } diff --git a/tests/suites/features/host-gateway/tests/host_gateway.rs b/tests/suites/features/host-gateway/tests/host_gateway.rs index 024b1cd84b..472197dfd1 100644 --- a/tests/suites/features/host-gateway/tests/host_gateway.rs +++ b/tests/suites/features/host-gateway/tests/host_gateway.rs @@ -9,34 +9,24 @@ //! Python test image ([`testkit::python_image`]) that requests a page from that //! server through the name. -use testkit::{Context, Error, Result, ResultExt as _}; -use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; -use tokio::net::{TcpListener, TcpStream}; -use tokio::task::JoinHandle; +use host_server_support::HttpRecorder; +use testkit::policy; +use testkit::{Context, PYTHON_BINARIES, Result, ResultExt as _, YamlFile}; -/// The image's system Python, which needs nothing beyond its standard library. -/// The policy names its real path, `/usr/bin/python3.*`, because a rule matches -/// the executable behind a symlink. -const PYTHON: &str = "/usr/bin/python3"; +/// The image's Python, which needs nothing beyond its standard library. The +/// policy allows [`PYTHON_BINARIES`], the real paths behind `python3` in +/// `testkit::python_image()` images, because a rule matches the executable +/// behind a symlink. +const PYTHON: &str = "python3"; /// Requests `/` from `host.openshell.internal` on the port in `argv[1]` and -/// prints the response. A failed connection is retried for 20 seconds, so a -/// slow start doesn't fail the test, and reported on stderr. +/// prints the response. The server listens before the sandbox is created, +/// and a Ready sandbox's network is up, so the client doesn't retry. const CLIENT: &str = r#" -import socket, sys, time +import socket, sys host, port = "host.openshell.internal", int(sys.argv[1]) -deadline = time.monotonic() + 20 -while True: - try: - conn = socket.create_connection((host, port), timeout=10) - break - except OSError as error: - print(f"connect {host}:{port}: {error}", file=sys.stderr, flush=True) - if time.monotonic() > deadline: - sys.exit(1) - time.sleep(1) -with conn: +with socket.create_connection((host, port), timeout=10) as conn: conn.sendall(f"GET / HTTP/1.1\r\nHost: {host}:{port}\r\nConnection: close\r\n\r\n".encode()) response = b"" while chunk := conn.recv(4096): @@ -51,8 +41,8 @@ print(response.decode(errors="replace")) #[testkit::test] async fn python_sandbox_reaches_host_server(cx: &mut Context) -> testkit::Result { let marker = format!("host-gateway-{}", cx.run_id()); - let server = HostServer::start(marker.clone()).await?; - let policy = PolicyFile::new(&policy(server.port))?; + let server = HttpRecorder::start(marker.clone()).await?; + let policy = YamlFile::new(&policy(server.port)?)?; let port = server.port.to_string(); let image = testkit::python_image(); @@ -100,110 +90,18 @@ async fn python_sandbox_reaches_host_server(cx: &mut Context) -> testkit::Result Ok(()) } -/// A sandbox policy that allows the image's system Python ([`PYTHON`]) to +/// A sandbox policy that allows the image's Python ([`PYTHON`]) to /// connect to `host.openshell.internal` on `port`. The endpoint names only the /// host and port, as the policy docs describe, so the driver's address for the /// name is what makes it reachable, not an `allowed_ips` range. -fn policy(port: u16) -> String { - format!( - r#"version: 1 - -filesystem_policy: - include_workdir: true - read_only: [/usr, /lib, /lib64, /proc, /dev/urandom, /etc] - read_write: [/sandbox, /tmp, /dev/null] -landlock: - compatibility: best_effort - -process: - run_as_user: sandbox - run_as_group: sandbox - -network_policies: - host_server: - name: host_server - endpoints: - - host: host.openshell.internal - port: {port} - binaries: - - path: /usr/bin/python3.* -"# - ) -} - -/// An HTTP server on the host that answers every request with `body`. It -/// stops when dropped. -struct HostServer { - port: u16, - task: JoinHandle<()>, -} - -impl HostServer { - /// Listen on every host address, since drivers map - /// `host.openshell.internal` to different ones. - async fn start(body: String) -> Result { - let listener = TcpListener::bind(("0.0.0.0", 0)) - .await - .map_err(|error| Error::from_error(&error)) - .context("bind the host server")?; - let port = listener - .local_addr() - .map_err(|error| Error::from_error(&error)) - .context("read the host server's address")? - .port(); - let task = tokio::spawn(async move { - while let Ok((stream, _)) = listener.accept().await { - tokio::spawn(respond(stream, body.clone())); - } - }); - Ok(Self { port, task }) - } -} - -impl Drop for HostServer { - fn drop(&mut self) { - self.task.abort(); - } -} - -/// Read one request's headers from `stream` and answer with `body`. -async fn respond(mut stream: TcpStream, body: String) { - let _ = stream.set_nodelay(true); - let mut request = Vec::new(); - let mut buf = [0_u8; 1024]; - while !request.windows(4).any(|window| window == b"\r\n\r\n") { - match stream.read(&mut buf).await { - Ok(read) if read > 0 && request.len() < 64 * 1024 => { - request.extend_from_slice(&buf[..read]); - } - _ => return, - } - } - let response = format!( - "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", - body.len() +fn policy(port: u16) -> Result { + let mut policy = policy::base(); + policy::allow( + &mut policy, + "host_server", + "host.openshell.internal", + port, + PYTHON_BINARIES, ); - let _ = stream.write_all(response.as_bytes()).await; - let _ = stream.shutdown().await; -} - -/// A sandbox policy written to a temporary file for `sandbox create --policy`. -struct PolicyFile(tempfile::NamedTempFile); - -impl PolicyFile { - fn new(contents: &str) -> Result { - let file = tempfile::NamedTempFile::new() - .map_err(|error| Error::from_error(&error)) - .context("create policy file")?; - std::fs::write(file.path(), contents) - .map_err(|error| Error::from_error(&error)) - .context(format!("write {}", file.path().display()))?; - Ok(Self(file)) - } - - fn path(&self) -> Result<&str> { - let path = self.0.path(); - path.to_str() - .ok_or_else(|| Error::msg(format!("{} is not UTF-8", path.display()))) - } + policy::to_yaml(&policy) } diff --git a/tests/support/Cargo.lock b/tests/support/Cargo.lock index f8b591438f..94cd21327d 100644 --- a/tests/support/Cargo.lock +++ b/tests/support/Cargo.lock @@ -493,6 +493,16 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + [[package]] name = "concurrent-queue" version = "2.5.0" @@ -560,6 +570,12 @@ version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + [[package]] name = "crossbeam-queue" version = "0.3.12" @@ -1137,6 +1153,25 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hickory-proto" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bab31817bfb44672a252e97fe81cd0c18d1b2cf892108922f6818820df8c643" +dependencies = [ + "data-encoding", + "idna", + "ipnet", + "jni", + "once_cell", + "rand 0.10.2", + "ring", + "thiserror 2.0.20", + "tinyvec", + "tracing", + "url", +] + [[package]] name = "hkdf" version = "0.13.0" @@ -1164,6 +1199,15 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "host-server-support" +version = "0.0.0" +dependencies = [ + "openshell-testkit", + "tokio", + "tungstenite", +] + [[package]] name = "hostname" version = "0.4.2" @@ -1517,6 +1561,55 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror 2.0.20", + "walkdir", + "windows-link", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -2029,6 +2122,10 @@ name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "critical-section", + "portable-atomic", +] [[package]] name = "once_cell_polyfill" @@ -2117,6 +2214,19 @@ dependencies = [ "tower", ] +[[package]] +name = "openshell-policy" +version = "0.0.0" +dependencies = [ + "hickory-proto", + "miette", + "openshell-core", + "openshell-policy-schema", + "prost-types", + "serde", + "serde_json", +] + [[package]] name = "openshell-policy-schema" version = "0.0.0" @@ -2163,15 +2273,19 @@ dependencies = [ "libc", "libtest-mimic", "linkme", + "noyalib", "openshell-bootstrap", "openshell-core", + "openshell-policy", "openshell-sdk", "openshell-testkit-macros", "quick-junit", "rand 0.9.4", "serde", "serde_json", + "tempfile", "tokio", + "tonic", ] [[package]] @@ -2353,6 +2467,12 @@ dependencies = [ "openshell-testkit", ] +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + [[package]] name = "potential_utf" version = "0.1.5" @@ -2771,6 +2891,15 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rusticata-macros" version = "4.1.0" @@ -2853,6 +2982,15 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + [[package]] name = "schannel" version = "0.1.29" @@ -3099,6 +3237,22 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version", + "simdutf8", +] + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + [[package]] name = "simple_asn1" version = "0.6.4" @@ -3967,6 +4121,16 @@ dependencies = [ "memchr", ] +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + [[package]] name = "want" version = "0.3.1" @@ -4114,6 +4278,15 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "626c4bac6755d76ffc12cb01b2eac751db1996b9e0041de9aa02c8c211ddc82c" +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "windows-core" version = "0.62.2" diff --git a/tests/support/Cargo.toml b/tests/support/Cargo.toml index f56ee275f5..4c7c495f97 100644 --- a/tests/support/Cargo.toml +++ b/tests/support/Cargo.toml @@ -5,4 +5,4 @@ # environment runs. Suites depend on them by path. [workspace] resolver = "2" -members = ["container-engine", "keycloak", "podman"] +members = ["container-engine", "host-server", "keycloak", "podman"] diff --git a/tests/support/host-server/Cargo.toml b/tests/support/host-server/Cargo.toml new file mode 100644 index 0000000000..e7db63f53d --- /dev/null +++ b/tests/support/host-server/Cargo.toml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "host-server-support" +description = "Servers on the test host that sandboxes reach through host.openshell.internal" +version = "0.0.0" +edition = "2024" +rust-version = "1.94" +license = "Apache-2.0" +repository = "https://github.com/NVIDIA/OpenShell" +publish = false + +[dependencies] +openshell-testkit = { path = "../../testkit/testkit" } +tokio = { version = "1.43", features = ["io-util", "net", "rt", "time"] } +tungstenite = "0.26" + +[dev-dependencies] +tokio = { version = "1.43", features = ["macros", "rt"] } diff --git a/tests/support/host-server/src/http.rs b/tests/support/host-server/src/http.rs new file mode 100644 index 0000000000..4d8ff8c812 --- /dev/null +++ b/tests/support/host-server/src/http.rs @@ -0,0 +1,125 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::sync::{Arc, Mutex, PoisonError}; + +use testkit::{Poll, Result}; +use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; +use tokio::net::TcpStream; +use tokio::task::JoinHandle; + +use crate::{MAX_REQUEST, READ_TIMEOUT, WAIT_INTERVAL, WAIT_TIMEOUT, bind, find, header}; + +/// A request an [`HttpRecorder`] read. +#[derive(Clone, Debug)] +pub struct Request { + /// The request line and headers. + pub head: String, + pub body: String, +} + +/// An HTTP server that answers every request with `200` and a fixed body, +/// and records each request it read. It reads a body by `Content-Length` +/// only, not a chunked one. +pub struct HttpRecorder { + pub port: u16, + requests: Arc>>, + task: JoinHandle<()>, +} + +impl HttpRecorder { + pub async fn start(body: impl Into) -> Result { + let (listener, port) = bind().await?; + let body: Arc = body.into().into(); + let requests = Arc::new(Mutex::new(Vec::new())); + let task_requests = Arc::clone(&requests); + let task = tokio::spawn(async move { + while let Ok((stream, _)) = listener.accept().await { + tokio::spawn(respond( + stream, + Arc::clone(&body), + Arc::clone(&task_requests), + )); + } + }); + Ok(Self { + port, + requests, + task, + }) + } + + /// The requests the server has read, once there are at least `count`. + pub async fn wait_for_requests(&self, count: usize) -> Result> { + testkit::poll_until( + &format!( + "{count} request(s) at the host server on port {}", + self.port + ), + WAIT_TIMEOUT, + WAIT_INTERVAL, + async || { + let requests = self + .requests + .lock() + .unwrap_or_else(PoisonError::into_inner) + .clone(); + Ok(if requests.len() >= count { + Poll::Ready(requests) + } else { + Poll::Pending(format!("{} request(s)", requests.len())) + }) + }, + ) + .await + } +} + +impl Drop for HttpRecorder { + fn drop(&mut self) { + self.task.abort(); + } +} + +/// Read one request, headers and `Content-Length` body, record it, and +/// answer with `body`. +async fn respond(mut stream: TcpStream, body: Arc, requests: Arc>>) { + let _ = stream.set_nodelay(true); + let Some(request) = read_request(&mut stream).await else { + return; + }; + requests + .lock() + .unwrap_or_else(PoisonError::into_inner) + .push(request); + let response = format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + let _ = stream.write_all(response.as_bytes()).await; + let _ = stream.shutdown().await; +} + +pub(crate) async fn read_request(stream: &mut TcpStream) -> Option { + let mut request = Vec::new(); + let mut buffer = [0_u8; 4096]; + loop { + if let Some(end) = find(&request, b"\r\n\r\n") { + let head = String::from_utf8_lossy(&request[..end]).into_owned(); + let length = header(&head, "content-length") + .and_then(|value| value.parse::().ok()) + .unwrap_or(0); + if request.len() >= end + 4 + length { + let body = + String::from_utf8_lossy(&request[end + 4..end + 4 + length]).into_owned(); + return Some(Request { head, body }); + } + } + match tokio::time::timeout(READ_TIMEOUT, stream.read(&mut buffer)).await { + Ok(Ok(read)) if read > 0 && request.len() < MAX_REQUEST => { + request.extend_from_slice(&buffer[..read]); + } + _ => return None, + } + } +} diff --git a/tests/support/host-server/src/lib.rs b/tests/support/host-server/src/lib.rs new file mode 100644 index 0000000000..0c11e4ceca --- /dev/null +++ b/tests/support/host-server/src/lib.rs @@ -0,0 +1,56 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Servers on the test host that sandboxes reach through +//! `host.openshell.internal`, and that record what arrived, so a test can +//! check what the supervisor forwarded upstream. +//! +//! The servers listen on every host address, since drivers map +//! `host.openshell.internal` to different ones, and stop when dropped. They +//! run where the tests run, so a gateway whose sandboxes can't reach the test +//! host, such as one on a remote cluster, can't run tests that use them. + +mod http; +mod websocket; + +use std::time::Duration; + +use testkit::{Error, Result, ResultExt as _}; +use tokio::net::TcpListener; + +pub use http::{HttpRecorder, Request}; +pub use websocket::WebSocketRecorder; + +const WAIT_TIMEOUT: Duration = Duration::from_secs(10); +const WAIT_INTERVAL: Duration = Duration::from_millis(100); +const READ_TIMEOUT: Duration = Duration::from_secs(10); +const MAX_REQUEST: usize = 1024 * 1024; + +async fn bind() -> Result<(TcpListener, u16)> { + let listener = TcpListener::bind(("0.0.0.0", 0)) + .await + .map_err(|error| Error::from_error(&error)) + .context("bind a host server")?; + let port = listener + .local_addr() + .map_err(|error| Error::from_error(&error)) + .context("read the host server's address")? + .port(); + Ok((listener, port)) +} + +fn find(haystack: &[u8], needle: &[u8]) -> Option { + haystack + .windows(needle.len()) + .position(|window| window == needle) +} + +fn header(head: &str, name: &str) -> Option { + head.lines().find_map(|line| { + let (header, value) = line.split_once(':')?; + header + .trim() + .eq_ignore_ascii_case(name) + .then(|| value.trim().to_string()) + }) +} diff --git a/tests/support/host-server/src/websocket.rs b/tests/support/host-server/src/websocket.rs new file mode 100644 index 0000000000..264dd4b5d3 --- /dev/null +++ b/tests/support/host-server/src/websocket.rs @@ -0,0 +1,223 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; + +use testkit::{Poll, Result}; +use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; +use tokio::net::TcpStream; +use tokio::task::JoinHandle; + +use crate::http::read_request; +use crate::{READ_TIMEOUT, WAIT_INTERVAL, WAIT_TIMEOUT, bind, header}; + +/// A WebSocket server that completes the handshake, then reads frames until +/// the connection closes, and records whether any was a binary frame. +pub struct WebSocketRecorder { + pub port: u16, + state: Arc, + task: JoinHandle<()>, +} + +#[derive(Default)] +struct WebSocketState { + handshake_seen: AtomicBool, + binary_seen: AtomicBool, + finished: AtomicBool, +} + +impl WebSocketRecorder { + pub async fn start() -> Result { + let (listener, port) = bind().await?; + let state = Arc::new(WebSocketState::default()); + let task_state = Arc::clone(&state); + let task = tokio::spawn(async move { + while let Ok((stream, _)) = listener.accept().await { + tokio::spawn(accept_websocket(stream, Arc::clone(&task_state))); + } + }); + Ok(Self { port, state, task }) + } + + /// Wait until a client completed the handshake. + pub async fn wait_for_handshake(&self) -> Result { + testkit::poll_until( + &format!( + "a WebSocket handshake at the host server on port {}", + self.port + ), + WAIT_TIMEOUT, + WAIT_INTERVAL, + async || { + Ok(if self.state.handshake_seen.load(Ordering::Acquire) { + Poll::Ready(()) + } else { + Poll::Pending("no handshake".to_string()) + }) + }, + ) + .await + } + + /// Wait until a connection that completed the handshake has ended, by a + /// close frame, the client closing it, or [`READ_TIMEOUT`] passing, and + /// return whether a binary frame arrived on it. + pub async fn wait_until_finished(&self) -> Result { + testkit::poll_until( + &format!( + "the WebSocket connection at the host server on port {} to end", + self.port + ), + READ_TIMEOUT + WAIT_TIMEOUT, + WAIT_INTERVAL, + async || { + Ok(if self.state.finished.load(Ordering::Acquire) { + Poll::Ready(self.state.binary_seen.load(Ordering::Acquire)) + } else { + Poll::Pending("the connection is open".to_string()) + }) + }, + ) + .await + } +} + +impl Drop for WebSocketRecorder { + fn drop(&mut self) { + self.task.abort(); + } +} + +async fn accept_websocket(mut stream: TcpStream, state: Arc) { + let _ = stream.set_nodelay(true); + let Some(request) = read_request(&mut stream).await else { + return; + }; + let Some(key) = header(&request.head, "sec-websocket-key") else { + return; + }; + // The supervisor checks Sec-WebSocket-Accept, so compute it properly. + let accept = tungstenite::handshake::derive_accept_key(key.as_bytes()); + let response = format!( + "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: {accept}\r\n\r\n" + ); + if stream.write_all(response.as_bytes()).await.is_err() { + return; + } + state.handshake_seen.store(true, Ordering::Release); + + let _ = tokio::time::timeout(READ_TIMEOUT, read_frames(&mut stream, &state)).await; + state.finished.store(true, Ordering::Release); +} + +/// Read frames until a close frame or the end of the stream, and record a +/// binary frame. +async fn read_frames(stream: &mut TcpStream, state: &WebSocketState) -> std::io::Result<()> { + loop { + let mut frame_header = [0_u8; 2]; + stream.read_exact(&mut frame_header).await?; + let opcode = frame_header[0] & 0x0f; + let length = match frame_header[1] & 0x7f { + 126 => u64::from(stream.read_u16().await?), + 127 => stream.read_u64().await?, + length => u64::from(length), + }; + if opcode == 0x02 { + state.binary_seen.store(true, Ordering::Release); + } + if opcode == 0x08 { + return Ok(()); + } + let mask = if frame_header[1] & 0x80 == 0 { 0 } else { 4 }; + let skipped = tokio::io::copy( + &mut (&mut *stream).take(mask + length), + &mut tokio::io::sink(), + ) + .await?; + if skipped < mask + length { + return Ok(()); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::find; + + /// Complete a handshake with `recorder`, send `frames`, and close. + async fn send_frames(recorder: &WebSocketRecorder, frames: &[(u8, Vec)]) { + #[allow( + clippy::disallowed_methods, + reason = "standalone test support has no openshell-core dependency and enables TCP_NODELAY below" + )] + let mut stream = TcpStream::connect(("127.0.0.1", recorder.port)) + .await + .unwrap(); + stream.set_nodelay(true).unwrap(); + stream + .write_all( + b"GET / HTTP/1.1\r\nHost: test\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\ + Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\n\r\n", + ) + .await + .unwrap(); + let mut response = Vec::new(); + while find(&response, b"\r\n\r\n").is_none() { + let mut buffer = [0_u8; 1024]; + let read = stream.read(&mut buffer).await.unwrap(); + assert!(read > 0, "closed during the handshake"); + response.extend_from_slice(&buffer[..read]); + } + let response = String::from_utf8_lossy(&response); + assert!(response.starts_with("HTTP/1.1 101"), "{response}"); + // The RFC 6455 sample key's accept value. + assert!( + response.contains("Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo="), + "{response}" + ); + for (opcode, payload) in frames { + stream.write_all(&frame(*opcode, payload)).await.unwrap(); + } + stream.write_all(&frame(0x08, &[])).await.unwrap(); + } + + /// A final, masked client frame. + fn frame(opcode: u8, payload: &[u8]) -> Vec { + let mask = [1_u8, 2, 3, 4]; + let mut frame = vec![0x80 | opcode]; + match payload.len() { + length @ 0..=125 => frame.push(0x80 | u8::try_from(length).unwrap()), + length => { + frame.push(0x80 | 126); + frame.extend_from_slice(&u16::try_from(length).unwrap().to_be_bytes()); + } + } + frame.extend_from_slice(&mask); + frame.extend(payload.iter().enumerate().map(|(i, b)| b ^ mask[i % 4])); + frame + } + + #[tokio::test] + async fn websocket_recorder_sees_a_binary_frame_after_a_long_text_frame() { + let recorder = WebSocketRecorder::start().await.unwrap(); + send_frames( + &recorder, + &[(0x01, vec![b'x'; 300]), (0x02, b"binary".to_vec())], + ) + .await; + assert!(recorder.wait_until_finished().await.unwrap()); + } + + #[tokio::test] + async fn websocket_recorder_sees_no_binary_frame_in_text_frames() { + let recorder = WebSocketRecorder::start().await.unwrap(); + send_frames( + &recorder, + &[(0x01, vec![b'x'; 300]), (0x01, b"text".to_vec())], + ) + .await; + assert!(!recorder.wait_until_finished().await.unwrap()); + } +} diff --git a/tests/testkit/Cargo.lock b/tests/testkit/Cargo.lock index db2e79b6ca..a02c314451 100644 --- a/tests/testkit/Cargo.lock +++ b/tests/testkit/Cargo.lock @@ -493,6 +493,16 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + [[package]] name = "concurrent-queue" version = "2.5.0" @@ -551,6 +561,12 @@ version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" +[[package]] +name = "critical-section" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b" + [[package]] name = "crossbeam-queue" version = "0.3.12" @@ -1128,6 +1144,25 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hickory-proto" +version = "0.26.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bab31817bfb44672a252e97fe81cd0c18d1b2cf892108922f6818820df8c643" +dependencies = [ + "data-encoding", + "idna", + "ipnet", + "jni", + "once_cell", + "rand 0.10.2", + "ring", + "thiserror 2.0.20", + "tinyvec", + "tracing", + "url", +] + [[package]] name = "hkdf" version = "0.13.0" @@ -1508,6 +1543,55 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "jni-macros", + "jni-sys", + "log", + "simd_cesu8", + "thiserror 2.0.20", + "walkdir", + "windows-link", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -2011,6 +2095,10 @@ name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +dependencies = [ + "critical-section", + "portable-atomic", +] [[package]] name = "once_cell_polyfill" @@ -2099,6 +2187,19 @@ dependencies = [ "tower", ] +[[package]] +name = "openshell-policy" +version = "0.0.0" +dependencies = [ + "hickory-proto", + "miette", + "openshell-core", + "openshell-policy-schema", + "prost-types", + "serde", + "serde_json", +] + [[package]] name = "openshell-policy-schema" version = "0.0.0" @@ -2145,8 +2246,10 @@ dependencies = [ "libc", "libtest-mimic", "linkme", + "noyalib", "openshell-bootstrap", "openshell-core", + "openshell-policy", "openshell-sdk", "openshell-testkit-macros", "quick-junit", @@ -2329,6 +2432,12 @@ version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + [[package]] name = "potential_utf" version = "0.1.5" @@ -2747,6 +2856,15 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + [[package]] name = "rusticata-macros" version = "4.1.0" @@ -2829,6 +2947,15 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + [[package]] name = "schannel" version = "0.1.29" @@ -3075,6 +3202,22 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "simd_cesu8" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520" +dependencies = [ + "rustc_version", + "simdutf8", +] + +[[package]] +name = "simdutf8" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" + [[package]] name = "simple_asn1" version = "0.6.4" @@ -3943,6 +4086,16 @@ dependencies = [ "memchr", ] +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + [[package]] name = "want" version = "0.3.1" @@ -4090,6 +4243,15 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "626c4bac6755d76ffc12cb01b2eac751db1996b9e0041de9aa02c8c211ddc82c" +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "windows-core" version = "0.62.2" diff --git a/tests/testkit/README.md b/tests/testkit/README.md index c02ad01c25..09fcd05052 100644 --- a/tests/testkit/README.md +++ b/tests/testkit/README.md @@ -65,7 +65,8 @@ fails. A killed test process skips cleanup, so names are unique to the run. - `cx.sandbox(suffix)` to build and create a sandbox with a run-unique name. The returned `Sandbox` can wait for ready or finished, stop, start, exec, - read its config and draft policy, set a setting, and delete. + read its config and draft policy, set a setting, and delete. `exec` output + has `success()`. - `cx.track(name)` to register a sandbox created some other way, such as through the CLI, for cleanup. Call it before the create, so a create that fails after the gateway accepted it is still cleaned @@ -78,12 +79,28 @@ fails. A killed test process skips cleanup, so names are unique to the run. - `testkit::python_image()` for sandboxes that run `python3`, curl, git, or openssl: `OPENSHELL_TEST_PYTHON_IMAGE`, or `openshell/e2e-python:dev` (`e2e/python/Dockerfile.workload`). The tmachine installers load it into - every environment; `mise run e2e:workload:build` builds it locally. + every environment; `mise run e2e:workload:build` builds it locally. An + override image must run `python3` from one of `testkit::PYTHON_BINARIES`. - `cx.cli([...])` to run the installed CLI against the same gateway, with `success()`, `failure()`, and `json()` on the output. `cx.cli_handle()` returns that CLI for code that can't borrow `cx`, such as a cleanup hook. - `cx.on_cleanup(name, hook)` to remove something when the test ends. +- `testkit::policy` for sandbox policies: `policy::base()` reads + `testkit/src/base-policy.yaml`, which allows no network access, and + `policy::allow(&mut policy, name, host, port, binaries)` adds an endpoint + (binaries apply to every endpoint under `name`) that `policy::Endpoint` + configures (`.rest(Access::Full)`, `.credential_binding(provider)`, ...). + Pass the result to `cx.sandbox(..).policy(policy)`, or `policy::to_yaml` + for the CLI. + `testkit::PYTHON_BINARIES` names `python3` in `python_image()` images. +- `cx.import_profile(profile)` to import a provider profile and return its + `id`, and `cx.create_provider(name, type, credentials)` to create a + provider, each with cleanup. `cx.install_provider(name, profile, + credentials)` does both. +- `testkit::YamlFile` for YAML a CLI command reads from a file. - `cx.client()` for SDK calls testkit does not wrap. +- `error.grpc_code()` for the gRPC status code (`testkit::GrpcCode`) of a + request the gateway refused, such as a sandbox create. - `poll_until` for waiting on other state. ## Running against a gateway diff --git a/tests/testkit/testkit/Cargo.toml b/tests/testkit/testkit/Cargo.toml index 47ff56c13c..abfa8cf97b 100644 --- a/tests/testkit/testkit/Cargo.toml +++ b/tests/testkit/testkit/Cargo.toml @@ -17,6 +17,7 @@ name = "testkit" [dependencies] openshell-bootstrap = { path = "../../../crates/openshell-bootstrap" } openshell-core = { path = "../../../crates/openshell-core", default-features = false } +openshell-policy = { path = "../../../crates/openshell-policy" } openshell-sdk = { path = "../../../crates/openshell-sdk" } openshell-testkit-macros = { path = "../testkit-macros" } clap = { version = "4", features = ["derive"] } @@ -25,17 +26,16 @@ libtest-mimic = "0.8" linkme = "0.3" quick-junit = "0.8" rand = "0.9" -serde = "1" +serde = { version = "1", features = ["derive"] } serde_json = "1" +serde_yml = { package = "noyalib", version = "0.0.53", default-features = false, features = ["std", "compat-serde-yaml"] } +tempfile = "3" tokio = { version = "1.43", features = ["macros", "process", "rt", "time"] } +tonic = "0.14" [target.'cfg(unix)'.dependencies] libc = "0.2" -[dev-dependencies] -tempfile = "3" -tonic = "0.14" - # A test program, run by `cargo test` and nextest through `testkit::main`'s # libtest interface, that checks the program's own subcommands. [[test]] diff --git a/tests/testkit/testkit/src/base-policy.yaml b/tests/testkit/testkit/src/base-policy.yaml new file mode 100644 index 0000000000..0935dbaa9a --- /dev/null +++ b/tests/testkit/testkit/src/base-policy.yaml @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# The sandbox policy tests start from (testkit::policy::base()): the image's +# system directories read-only, the sandbox's home and /tmp writable, the +# sandbox user, and no network access. Tests add the network access they need +# with testkit::policy::allow(). +version: 1 +filesystem_policy: + include_workdir: true + read_only: [/usr, /lib, /lib64, /proc, /dev/urandom, /app, /etc, /var/log] + read_write: [/sandbox, /tmp, /dev/null] +landlock: + compatibility: best_effort +process: + run_as_user: sandbox + run_as_group: sandbox diff --git a/tests/testkit/testkit/src/cleanup.rs b/tests/testkit/testkit/src/cleanup.rs index f1a1e0228e..5ae791b7c4 100644 --- a/tests/testkit/testkit/src/cleanup.rs +++ b/tests/testkit/testkit/src/cleanup.rs @@ -18,26 +18,40 @@ pub const CLEANUP_TIMEOUT: Duration = Duration::from_mins(2); type HookFn = Box BoxFuture<'static, Result> + Send>; +/// What a hook is registered for, so it can be registered once and cancelled. +/// The variants are separate namespaces, so keys of different kinds never +/// collide. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Key { + /// The hook that deletes the sandbox with this name. + Sandbox(String), +} + struct Hook { /// What the hook does, for reporting a failure. name: String, /// Registers the hook at most once; see [`Cleanup::push_keyed`]. - key: Option, + key: Option, run: HookFn, } impl Hook { - fn new(name: String, key: Option, hook: F) -> Self + fn new(name: String, hook: F) -> Self where F: FnOnce() -> Fut + Send + 'static, Fut: Future + Send + 'static, { Self { name, - key, + key: None, run: Box::new(move || hook().boxed()), } } + + fn keyed(mut self, key: Key) -> Self { + self.key = Some(key); + self + } } /// An ordered list of cleanup hooks, shared by a test's context and the @@ -54,25 +68,25 @@ impl Cleanup { F: FnOnce() -> Fut + Send + 'static, Fut: Future + Send + 'static, { - lock(&self.hooks).push(Hook::new(name.into(), None, hook)); + lock(&self.hooks).push(Hook::new(name.into(), hook)); } /// Add a hook unless one with `key` is registered, in which case the /// existing hook keeps its place in the order. - pub(crate) fn push_keyed(&self, key: &str, name: impl Into, hook: F) + pub(crate) fn push_keyed(&self, key: Key, name: impl Into, hook: F) where F: FnOnce() -> Fut + Send + 'static, Fut: Future + Send + 'static, { let mut hooks = lock(&self.hooks); - if !hooks.iter().any(|hook| hook.key.as_deref() == Some(key)) { - hooks.push(Hook::new(name.into(), Some(key.to_string()), hook)); + if !hooks.iter().any(|hook| hook.key.as_ref() == Some(&key)) { + hooks.push(Hook::new(name.into(), hook).keyed(key)); } } /// Drop the hook registered with `key`, if any, without running it. - pub(crate) fn remove(&self, key: &str) { - lock(&self.hooks).retain(|hook| hook.key.as_deref() != Some(key)); + pub(crate) fn remove(&self, key: &Key) { + lock(&self.hooks).retain(|hook| hook.key.as_ref() != Some(key)); } /// Run every hook, last registered first, each limited to `timeout`. A @@ -121,7 +135,11 @@ mod tests { use std::sync::{Arc, Mutex}; use std::time::Duration; - use super::Cleanup; + use super::{Cleanup, Key}; + + fn sandbox(name: &str) -> Key { + Key::Sandbox(name.to_string()) + } use crate::error::Error; const TIMEOUT: Duration = Duration::from_secs(5); @@ -130,7 +148,7 @@ mod tests { /// Register a hook, keyed if `key` is set, that records `name` in `log` /// when it runs. - fn record(cleanup: &Cleanup, key: Option<&str>, log: &Log, name: &'static str) { + fn record(cleanup: &Cleanup, key: Option, log: &Log, name: &'static str) { let log = Arc::clone(log); let hook = async move || { log.lock().unwrap().push(name); @@ -160,12 +178,12 @@ mod tests { async fn keyed_hook_is_registered_once_and_can_be_removed() { let log = Arc::new(Mutex::new(Vec::new())); let cleanup = Cleanup::default(); - record(&cleanup, Some("a"), &log, "a"); + record(&cleanup, Some(sandbox("a")), &log, "a"); record(&cleanup, None, &log, "b"); // Already registered: keeps its place before "b". - record(&cleanup, Some("a"), &log, "a again"); - record(&cleanup, Some("c"), &log, "c"); - cleanup.remove("c"); + record(&cleanup, Some(sandbox("a")), &log, "a again"); + record(&cleanup, Some(sandbox("c")), &log, "c"); + cleanup.remove(&sandbox("c")); assert!(cleanup.run(TIMEOUT).await.is_empty()); assert_eq!(*log.lock().unwrap(), ["b", "a"]); } @@ -174,10 +192,10 @@ mod tests { async fn a_running_hook_can_change_the_remaining_hooks() { let log = Arc::new(Mutex::new(Vec::new())); let cleanup = Cleanup::default(); - record(&cleanup, Some("removed"), &log, "removed"); + record(&cleanup, Some(sandbox("removed")), &log, "removed"); let handle = cleanup.clone(); cleanup.push("remove", async move || { - handle.remove("removed"); + handle.remove(&sandbox("removed")); Ok(()) }); assert!(cleanup.run(TIMEOUT).await.is_empty()); diff --git a/tests/testkit/testkit/src/context.rs b/tests/testkit/testkit/src/context.rs index ea20878837..7f09ad7397 100644 --- a/tests/testkit/testkit/src/context.rs +++ b/tests/testkit/testkit/src/context.rs @@ -12,7 +12,7 @@ use openshell_sdk::{ }; use rand::Rng; -use crate::cleanup::{CLEANUP_TIMEOUT, Cleanup}; +use crate::cleanup::{CLEANUP_TIMEOUT, Cleanup, Key}; use crate::cli::{Cli, CliOutput}; use crate::error::{Error, Result, ResultExt}; use crate::poll::{Poll, poll_until}; @@ -269,18 +269,22 @@ fn track_sandbox(cleanup: &Cleanup, client: &WorkspaceScopedClient, name: &str) let owned = name.to_string(); // The timeout wraps the whole hook, not just the wait, so a gateway that // never answers the delete cannot hang cleanup. - cleanup.push_keyed(name, format!("delete sandbox {name}"), async move || { - client - .delete_sandbox( - &owned, - DeleteOptions { - allow_missing: true, - }, - ) - .await?; - client.wait_deleted(&owned, CLEANUP_TIMEOUT, None).await?; - Ok(()) - }); + cleanup.push_keyed( + Key::Sandbox(name.to_string()), + format!("delete sandbox {name}"), + async move || { + client + .delete_sandbox( + &owned, + DeleteOptions { + allow_missing: true, + }, + ) + .await?; + client.wait_deleted(&owned, CLEANUP_TIMEOUT, None).await?; + Ok(()) + }, + ); } /// The generated gRPC client, for public API the SDK does not wrap yet. @@ -358,32 +362,16 @@ impl SandboxBuilder { // The SDK's spec has no policy field, so create through gRPC. Some(policy) => { crate::ensure!( - self.spec.image.is_none() - && self.spec.environment.is_empty() - && self.spec.providers.is_empty() - && !self.spec.gpu - && self.spec.service_exposures.is_empty(), - "testkit creates a sandbox with a policy from its name, labels, command, restart policy, and policy only" + !self.spec.gpu && self.spec.service_exposures.is_empty(), + "testkit doesn't create a sandbox with a policy and a GPU or service exposures" ); - let request = proto::CreateSandboxRequest { - name: name.clone(), - labels: self.spec.labels, - workspace_scope: self.grpc.scope(), - spec: Some(proto::SandboxSpec { - command: self.spec.command, - tty: self.spec.tty, - policy: Some(policy), - restart_policy: restart_policy(self.spec.restart_policy) as i32, - ..proto::SandboxSpec::default() - }), - ..proto::CreateSandboxRequest::default() - }; + let request = policy_create_request(self.spec, policy, self.grpc.scope()); self.grpc .client() .await? .create_sandbox(request) .await - .map_err(|status| Error::from_error(&status)) + .map_err(|status| Error::from_status(&status)) .context(format!("create sandbox {name}"))?; self.client .get_sandbox(&name) @@ -402,11 +390,47 @@ impl SandboxBuilder { } } -const fn restart_policy(policy: SandboxRestartPolicy) -> proto::SandboxRestartPolicy { - match policy { - SandboxRestartPolicy::Never => proto::SandboxRestartPolicy::Never, - SandboxRestartPolicy::OnFailure => proto::SandboxRestartPolicy::OnFailure, - SandboxRestartPolicy::Always => proto::SandboxRestartPolicy::Always, +/// The gRPC request for a sandbox created from `spec` with `policy`, built +/// as the SDK builds one from `spec` alone. The spec is destructured without +/// `..`, so a field added to the SDK's spec fails to compile here instead of +/// being dropped. [`SandboxBuilder::create`] rejects a GPU or service +/// exposures before calling this. A policy field on the SDK's spec would +/// replace this (#5410). +fn policy_create_request( + spec: SandboxSpec, + policy: proto::SandboxPolicy, + workspace_scope: Option, +) -> proto::CreateSandboxRequest { + let SandboxSpec { + name, + image, + labels, + environment, + providers, + gpu: _, + command, + tty, + service_exposures: _, + restart_policy, + } = spec; + proto::CreateSandboxRequest { + name: name.unwrap_or_default(), + labels, + workspace_scope, + spec: Some(proto::SandboxSpec { + environment, + template: image.map(|image| proto::SandboxTemplate { + image, + ..proto::SandboxTemplate::default() + }), + providers, + command, + tty, + policy: Some(policy), + restart_policy: proto::SandboxRestartPolicy::from(restart_policy) as i32, + ..proto::SandboxSpec::default() + }), + ..proto::CreateSandboxRequest::default() } } @@ -427,6 +451,20 @@ pub struct ExecOutput { pub stderr: String, } +impl ExecOutput { + /// Fail unless the command exited 0. + pub fn success(self) -> Result { + crate::ensure!( + self.exit_code == 0, + "exec exited with {}; stdout: {}; stderr: {}", + self.exit_code, + self.stdout.trim(), + self.stderr.trim() + ); + Ok(self) + } +} + impl Sandbox { pub fn name(&self) -> &str { &self.name @@ -603,7 +641,7 @@ impl Sandbox { .wait_deleted(&self.name, DELETE_TIMEOUT, Some(&self.id)) .await .context(format!("wait for sandbox {} to be deleted", self.name))?; - self.cleanup.remove(&self.name); + self.cleanup.remove(&Key::Sandbox(self.name.clone())); Ok(()) } } @@ -631,3 +669,26 @@ fn generate_run_id() -> String { .map(|_| char::from(ALPHABET[rng.random_range(0..ALPHABET.len())])) .collect() } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn policy_create_request_keeps_the_spec_and_adds_the_policy() { + let spec = SandboxSpec { + image: Some("python:3".to_string()), + providers: vec!["provider".to_string()], + ..SandboxSpec::default() + }; + let policy = crate::policy::base(); + + let spec = policy_create_request(spec, policy.clone(), None) + .spec + .expect("spec"); + + assert_eq!(spec.template.expect("template").image, "python:3"); + assert_eq!(spec.providers, ["provider"]); + assert_eq!(spec.policy, Some(policy)); + } +} diff --git a/tests/testkit/testkit/src/error.rs b/tests/testkit/testkit/src/error.rs index 70a702ddc1..51d2fc3904 100644 --- a/tests/testkit/testkit/src/error.rs +++ b/tests/testkit/testkit/src/error.rs @@ -7,10 +7,12 @@ use std::fmt; use openshell_sdk::SdkError; -/// A test failure: a message with the context it happened in. +/// A test failure: a message with the context it happened in, and the gRPC +/// status code when the gateway refused a request. #[derive(Debug)] pub struct Error { message: String, + code: Option, } /// The result of a test or test step. @@ -20,9 +22,23 @@ impl Error { pub fn msg(message: impl Into) -> Self { Self { message: message.into(), + code: None, } } + /// A gateway's refusal, keeping its status code for [`Error::grpc_code`]. + pub fn from_status(status: &tonic::Status) -> Self { + Self { + code: Some(status.code()), + ..Self::from_error(status) + } + } + + /// The gRPC status code, when the gateway refused a request. + pub fn grpc_code(&self) -> Option { + self.code + } + pub fn from_display(error: impl fmt::Display) -> Self { Self::msg(error.to_string()) } @@ -45,7 +61,10 @@ impl Error { #[must_use] pub fn context(self, context: impl fmt::Display) -> Self { - Self::msg(format!("{context}: {}", self.message)) + Self { + message: format!("{context}: {}", self.message), + code: self.code, + } } } @@ -59,7 +78,10 @@ impl std::error::Error for Error {} impl From for Error { fn from(error: SdkError) -> Self { - Self::from_error(&error) + Self { + code: error.grpc_status().map(tonic::Status::code), + ..Self::from_error(&error) + } } } diff --git a/tests/testkit/testkit/src/image.rs b/tests/testkit/testkit/src/image.rs index 26bb2278fe..07a6e2d937 100644 --- a/tests/testkit/testkit/src/image.rs +++ b/tests/testkit/testkit/src/image.rs @@ -9,7 +9,9 @@ const BASE_IMAGE_ENV: &str = "OPENSHELL_TEST_BASE_IMAGE"; /// nvcr.io. const DEFAULT_BASE_IMAGE: &str = "nvcr.io/nvidia/base/ubuntu:24.04"; -/// Overrides [`DEFAULT_PYTHON_IMAGE`]. +/// Overrides [`DEFAULT_PYTHON_IMAGE`]. The image must run `python3` from one +/// of [`PYTHON_BINARIES`]; otherwise policies that allow those paths don't +/// match it, and its connections are denied. const PYTHON_IMAGE_ENV: &str = "OPENSHELL_TEST_PYTHON_IMAGE"; /// The workload image for tests that run `python3`, curl, git or openssl in a /// sandbox, built from `e2e/python/Dockerfile.workload`. It isn't published: @@ -17,6 +19,16 @@ const PYTHON_IMAGE_ENV: &str = "OPENSHELL_TEST_PYTHON_IMAGE"; /// `mise run e2e:workload:build` builds it locally. const DEFAULT_PYTHON_IMAGE: &str = "openshell/e2e-python:dev"; +/// The paths `python3` runs from in [`python_image`] images, as network +/// policy binaries: the system Python, the pullable uv image's Python, and the +/// workload image's uv-installed Python behind its virtualenv. Add a path +/// here for an image that installs Python elsewhere. +pub const PYTHON_BINARIES: &[&str] = &[ + "/usr/bin/python*", + "/usr/local/bin/python*", + "/sandbox/.uv/python/*/bin/python*", +]; + /// The image named by [`BASE_IMAGE_ENV`], or [`DEFAULT_BASE_IMAGE`]. pub fn base_image() -> String { image_from_env(BASE_IMAGE_ENV, DEFAULT_BASE_IMAGE) diff --git a/tests/testkit/testkit/src/lib.rs b/tests/testkit/testkit/src/lib.rs index 834f28c1e7..8520fb80b3 100644 --- a/tests/testkit/testkit/src/lib.rs +++ b/tests/testkit/testkit/src/lib.rs @@ -32,11 +32,14 @@ mod context; mod error; pub mod gateway; mod image; +pub mod policy; mod poll; pub mod program; +mod provider; mod ready; mod registry; mod run; +mod yaml_file; pub use cleanup::CLEANUP_TIMEOUT; pub use cli::{CLI_TIMEOUT, Cli, CliOutput}; @@ -46,7 +49,7 @@ pub use context::{ }; pub use error::{Error, Result, ResultExt}; pub use gateway::Gateway; -pub use image::{base_image, python_image}; +pub use image::{PYTHON_BINARIES, base_image, python_image}; pub use openshell_sdk as sdk; pub use poll::{Poll, poll_until}; pub use program::main; @@ -54,6 +57,8 @@ pub use ready::check as check_ready; pub use registry::Test; pub use run::{TestInfo, run}; pub use testkit_macros::test; +pub use tonic::Code as GrpcCode; +pub use yaml_file::YamlFile; #[doc(hidden)] pub mod __private { diff --git a/tests/testkit/testkit/src/policy.rs b/tests/testkit/testkit/src/policy.rs new file mode 100644 index 0000000000..ab7be7cae3 --- /dev/null +++ b/tests/testkit/testkit/src/policy.rs @@ -0,0 +1,188 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Sandbox policies for tests: [`base`], read from `base-policy.yaml`, plus +//! [`allow`] and the [`Endpoint`] settings for the network access a test +//! needs. +//! +//! ```ignore +//! let mut policy = testkit::policy::base(); +//! testkit::policy::allow(&mut policy, "upstream", "host.openshell.internal", port, PYTHON_BINARIES) +//! .rest(Access::Full) +//! .credential_binding(&provider); +//! ``` + +use openshell_sdk::raw::proto; + +use crate::error::{Error, Result, ResultExt as _}; + +pub use proto::NetworkAccessPreset as Access; + +const BASE: &str = include_str!("base-policy.yaml"); + +/// The policy in `base-policy.yaml`. It allows no network access. +pub fn base() -> proto::SandboxPolicy { + openshell_policy::parse_sandbox_policy(BASE).expect("base-policy.yaml is a valid policy") +} + +/// Let `binaries` connect to `host:port` under the network policy `name`, +/// adding the policy if it doesn't exist. Returns the new endpoint, which +/// [`Endpoint`] configures further. +/// +/// A policy's binaries apply to all its endpoints, so calling this again with +/// the same `name` adds `binaries` to the existing ones, and every endpoint +/// under `name` allows all of them. Use a different `name` to keep them apart. +pub fn allow<'a>( + policy: &'a mut proto::SandboxPolicy, + name: &str, + host: &str, + port: u16, + binaries: &[&str], +) -> &'a mut proto::NetworkEndpoint { + let rule = policy + .network_policies + .entry(name.to_string()) + .or_insert_with(|| proto::NetworkPolicyRule { + name: name.to_string(), + ..proto::NetworkPolicyRule::default() + }); + for binary in binaries { + if !rule + .binaries + .iter() + .any(|existing| existing.path == *binary) + { + rule.binaries.push(proto::NetworkBinary { + path: (*binary).to_string(), + }); + } + } + rule.endpoints.push(proto::NetworkEndpoint { + host: host.to_string(), + port: port.into(), + ports: vec![port.into()], + ..proto::NetworkEndpoint::default() + }); + rule.endpoints + .last_mut() + .expect("an endpoint was just added") +} + +/// The policy as YAML, for CLI commands such as `policy set`. +pub fn to_yaml(policy: &proto::SandboxPolicy) -> Result { + openshell_policy::serialize_sandbox_policy(policy) + .map_err(|error| Error::msg(format!("{error:?}"))) + .context("serialize the policy") +} + +/// Settings for an endpoint from [`allow`], named as in policy YAML. +pub trait Endpoint { + /// `protocol: rest` with `access`. + fn rest(&mut self, access: Access) -> &mut Self; + /// `protocol: websocket` with `access`. + fn websocket(&mut self, access: Access) -> &mut Self; + /// `tls: skip`. + fn tls_skip(&mut self) -> &mut Self; + /// `request_body_credential_rewrite`. + fn request_body_credential_rewrite(&mut self, rewrite: bool) -> &mut Self; + /// `credential_binding: {provider: }`. + fn credential_binding(&mut self, provider: &str) -> &mut Self; + /// `allow_uninspected_credentials: true`. + fn allow_uninspected_credentials(&mut self) -> &mut Self; +} + +impl Endpoint for proto::NetworkEndpoint { + fn rest(&mut self, access: Access) -> &mut Self { + self.protocol = "rest".to_string(); + self.access = access as i32; + self + } + + fn websocket(&mut self, access: Access) -> &mut Self { + self.protocol = "websocket".to_string(); + self.access = access as i32; + self + } + + fn tls_skip(&mut self) -> &mut Self { + self.tls = proto::NetworkTlsMode::Skip as i32; + self + } + + fn request_body_credential_rewrite(&mut self, rewrite: bool) -> &mut Self { + self.request_body_credential_rewrite = rewrite; + self + } + + fn credential_binding(&mut self, provider: &str) -> &mut Self { + self.credential_binding = Some(proto::NetworkCredentialBinding { + provider: provider.to_string(), + }); + self + } + + fn allow_uninspected_credentials(&mut self) -> &mut Self { + self.allow_uninspected_credentials = true; + self + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Each helper builds what the YAML it is named after parses to, so the + /// gateway can't tell the two apart. + #[test] + fn helpers_match_the_yaml_they_name() { + let mut policy = base(); + allow( + &mut policy, + "upstream", + "example.test", + 8080, + &["/usr/bin/python*"], + ) + .rest(Access::Full) + .tls_skip() + .request_body_credential_rewrite(true) + .credential_binding("provider"); + allow( + &mut policy, + "upstream", + "example.test", + 8081, + &["/usr/bin/python*"], + ) + .websocket(Access::ReadWrite) + .allow_uninspected_credentials(); + + let yaml = format!( + "{BASE}network_policies: + upstream: + name: upstream + endpoints: + - host: example.test + port: 8080 + protocol: rest + access: full + tls: skip + request_body_credential_rewrite: true + credential_binding: {{provider: provider}} + - host: example.test + port: 8081 + protocol: websocket + access: read-write + allow_uninspected_credentials: true + binaries: + - path: /usr/bin/python* +" + ); + let expected = openshell_policy::parse_sandbox_policy(&yaml).expect("the YAML parses"); + assert_eq!(policy, expected); + let round_trip = + openshell_policy::parse_sandbox_policy(&to_yaml(&policy).expect("serializes")) + .expect("the serialized policy parses"); + assert_eq!(round_trip, expected); + } +} diff --git a/tests/testkit/testkit/src/provider.rs b/tests/testkit/testkit/src/provider.rs new file mode 100644 index 0000000000..c82901c74a --- /dev/null +++ b/tests/testkit/testkit/src/provider.rs @@ -0,0 +1,88 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use serde::Deserialize; + +use crate::context::Context; +use crate::error::{Error, Result, ResultExt as _}; +use crate::yaml_file::YamlFile; + +impl Context { + /// Import `profile`, a provider profile in YAML, and return its `id`, the + /// provider type it defines. Cleanup deletes the profile after the test's + /// sandboxes and the providers created after it. + pub async fn import_profile(&self, profile: &str) -> Result { + #[derive(Deserialize)] + struct Profile { + id: String, + } + let id = serde_yml::from_str::(profile) + .map_err(|error| Error::from_error(&error)) + .context("read the provider profile's id")? + .id; + // Deleting a missing profile succeeds, so register the delete before + // the import. + self.on_cleanup_delete("profile", &id); + let file = YamlFile::new(profile)?; + self.cli(["profile", "import", "--file", file.path()?]) + .await? + .success() + .context(format!("import profile {id}"))?; + Ok(id) + } + + /// Create a provider called `name` of `provider_type` with `credentials`, + /// as pairs of environment variable and value. Cleanup deletes the + /// provider after the test's sandboxes. + /// + /// The CLI reads the credential values from its environment, so they + /// aren't in the reported command. + pub async fn create_provider( + &self, + name: &str, + provider_type: &str, + credentials: &[(&str, &str)], + ) -> Result { + // Deleting a missing provider succeeds, so register the delete before + // the create. + self.on_cleanup_delete("provider", name); + let mut args = vec![ + "provider", + "create", + "--name", + name, + "--type", + provider_type, + ]; + for (key, _) in credentials { + args.extend(["--credential", key]); + } + self.cli_with_env(credentials, args) + .await? + .success() + .context(format!("create provider {name}"))?; + Ok(()) + } + + /// Import `profile` with [`Context::import_profile`] and create a provider + /// called `name` of its type with [`Context::create_provider`]. + pub async fn install_provider( + &self, + name: &str, + profile: &str, + credentials: &[(&str, &str)], + ) -> Result { + let provider_type = self.import_profile(profile).await?; + self.create_provider(name, &provider_type, credentials) + .await + } + + fn on_cleanup_delete(&self, kind: &'static str, name: &str) { + let cli = self.cli_handle(); + let owned = name.to_string(); + self.on_cleanup(format!("delete {kind} {name}"), async move || { + cli.run([kind, "delete", &owned]).await?.success()?; + Ok(()) + }); + } +} diff --git a/tests/testkit/testkit/src/yaml_file.rs b/tests/testkit/testkit/src/yaml_file.rs new file mode 100644 index 0000000000..04ca3d815d --- /dev/null +++ b/tests/testkit/testkit/src/yaml_file.rs @@ -0,0 +1,30 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use crate::error::{Error, Result, ResultExt as _}; + +/// YAML in a temporary file, for CLI commands that read one, such as +/// `sandbox create --policy` or `profile import --file`. The file is removed +/// when dropped. +pub struct YamlFile(tempfile::NamedTempFile); + +impl YamlFile { + pub fn new(contents: &str) -> Result { + let file = tempfile::Builder::new() + .suffix(".yaml") + .tempfile() + .map_err(|error| Error::from_error(&error)) + .context("create a temporary YAML file")?; + std::fs::write(file.path(), contents) + .map_err(|error| Error::from_error(&error)) + .context(format!("write {}", file.path().display()))?; + Ok(Self(file)) + } + + /// The file's path, as a CLI argument. + pub fn path(&self) -> Result<&str> { + let path = self.0.path(); + path.to_str() + .ok_or_else(|| Error::msg(format!("{} is not UTF-8", path.display()))) + } +} diff --git a/tests/tmachine/ansible/playbooks/drivers/podman/e2e.yaml b/tests/tmachine/ansible/playbooks/drivers/podman/e2e.yaml index e839a2cfff..a9270a0922 100644 --- a/tests/tmachine/ansible/playbooks/drivers/podman/e2e.yaml +++ b/tests/tmachine/ansible/playbooks/drivers/podman/e2e.yaml @@ -48,13 +48,30 @@ HOME: /home/tmachine XDG_RUNTIME_DIR: /run/user/1000 + # The binaries installer runs a plain-HTTP gateway named tmachine from a + # system service and puts the CLI in /usr/local/bin. The package installers + # run the packaged user service, install the CLI in /usr/bin and register + # the gateway as openshell, using the package default network and namespace. + - name: Detect the installer + ansible.builtin.stat: + path: /usr/local/bin/openshell + register: podman_e2e_binaries_cli + + - name: Select the gateway under test + ansible.builtin.set_fact: + podman_e2e_binaries: "{{ podman_e2e_binaries_cli.stat.exists }}" + podman_e2e_cli: "{{ '/usr/local/bin/openshell' if podman_e2e_binaries_cli.stat.exists else '/usr/bin/openshell' }}" + podman_e2e_gateway: "{{ 'tmachine' if podman_e2e_binaries_cli.stat.exists else 'openshell' }}" + - name: Remove any previous OpenShell gateway registration + when: podman_e2e_binaries ansible.builtin.command: argv: [/usr/local/bin/openshell, gateway, remove, tmachine] changed_when: false failed_when: false - name: Register the configured OpenShell gateway + when: podman_e2e_binaries ansible.builtin.command: argv: [/usr/local/bin/openshell, gateway, add, http://127.0.0.1:17670, --local, --name, tmachine] @@ -75,12 +92,12 @@ CONTAINER_ENGINE: podman CONTAINER_HOST: unix:///run/user/1000/podman/podman.sock HOME: /home/tmachine - OPENSHELL_BIN: /usr/local/bin/openshell + OPENSHELL_BIN: "{{ podman_e2e_cli }}" OPENSHELL_E2E_CONTAINER_ENGINE_UNSET_XDG_CONFIG_HOME: "1" OPENSHELL_E2E_DRIVER: podman - OPENSHELL_E2E_NETWORK_NAME: tmachine - OPENSHELL_E2E_SANDBOX_NAMESPACE: tmachine - OPENSHELL_GATEWAY: tmachine + OPENSHELL_E2E_NETWORK_NAME: "{{ podman_e2e_gateway }}" + OPENSHELL_E2E_SANDBOX_NAMESPACE: "{{ podman_e2e_gateway }}" + OPENSHELL_GATEWAY: "{{ podman_e2e_gateway }}" OPENSHELL_PODMAN_SOCKET: /run/user/1000/podman/podman.sock XDG_RUNTIME_DIR: /run/user/1000 register: podman_e2e_test_result @@ -96,9 +113,12 @@ var: podman_e2e_test_result.stdout_lines - name: Capture OpenShell gateway journal after Podman E2E test failure - become: true + become: "{{ podman_e2e_binaries }}" ansible.builtin.command: - argv: [journalctl, --unit, openshell-gateway.service, --no-pager, --lines, "200"] + argv: >- + {{ ['journalctl', '--unit', 'openshell-gateway.service', '--no-pager', '--lines', '200'] + if podman_e2e_binaries else + ['journalctl', '--user-unit', 'openshell-gateway.service', '--no-pager', '--lines', '200'] }} changed_when: false failed_when: false when: podman_e2e_test_result.rc != 0 diff --git a/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml b/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml index 89022bd769..202270100f 100644 --- a/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml +++ b/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml @@ -5,8 +5,9 @@ # # `podman_userns_mode` is default, auto, keep-id or private. Every mode except # default writes its `[openshell.drivers.podman]` settings to -# `podman_userns_driver_config`, which the openshell_gateway role (the -# `binaries` installer) appends to gateway.toml; default removes the file. The +# `/etc/openshell-test/gateway-overrides.toml`, which every installer applies to the active +# gateway-overrides.toml (the openshell_gateway role for `binaries`, the +# openshell_packaged_gateway role for the packages); default removes the file. The # Podman support crate (tests/support/podman) reads the published mode and # runs Podman's own reference container in it. --- @@ -14,14 +15,17 @@ hosts: all gather_facts: false vars: - podman_userns_driver_config: /etc/openshell-test/podman-driver.toml + podman_userns_driver_config: /etc/openshell-test/gateway-overrides.toml podman_userns_settings: default: "" auto: | + [openshell.drivers.podman] userns = "auto" keep-id: | + [openshell.drivers.podman] userns = "keep-id" private: | + [openshell.drivers.podman] userns = "private" uidmap = ["0:0:1", "1:1:65535"] gidmap = ["0:0:1", "1:1:65535"] @@ -59,7 +63,7 @@ when: podman_userns_mode != "default" # The fragment is the only source of userns, uidmap and gidmap settings in - # the rendered gateway.toml, so default mode leaves them unset. + # the gateway's configuration, so default mode leaves them unset. - name: Remove Podman driver user-namespace settings for default mode become: true ansible.builtin.file: diff --git a/tests/tmachine/ansible/roles/openshell_gateway/files/merge-gateway-toml.py b/tests/tmachine/ansible/roles/openshell_gateway/files/merge-gateway-toml.py new file mode 100755 index 0000000000..02ef6e874c --- /dev/null +++ b/tests/tmachine/ansible/roles/openshell_gateway/files/merge-gateway-toml.py @@ -0,0 +1,165 @@ +#!/usr/bin/python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Apply an environment's gateway overrides to a gateway.toml. + +usage: merge-gateway-toml.py BASE OVERRIDES [--default DEFAULT] + +OVERRIDES holds whole tables, such as `[openshell.gateway]` or +`[openshell.drivers.podman]`. Each key in it replaces the same key in BASE's +table, or is added to the table, and a table BASE lacks is appended. Everything +else in BASE stays, including its comments, so a package's own configuration is +never replaced. Running it again with the same overrides changes nothing. + +If BASE does not exist it is created first: from DEFAULT when that file exists +(the rpm's packaged default), else as a minimal `version = 2` file. The merged +text is parsed as TOML before it is written, and the script fails, leaving BASE +as it was, if it is not valid. + +The overrides must be valid TOML, and the script fails if any of their values is +missing from the merged result, so a statement it cannot rewrite is an error +rather than a silent no-op. + +Limits: a key's value in BASE is replaced as written lines, up to the next key, +header, comment or blank line. A key can't be removed. +""" + +import argparse +import os +import re +import sys +import tomllib + +HEADER = re.compile(r"^\s*\[([^\[\]]+)\]\s*(#.*)?$") +ARRAY_HEADER = re.compile(r"^\s*\[\[[^\[\]]+\]\]\s*(#.*)?$") +KEY = re.compile(r"^\s*([A-Za-z0-9_-]+|\"[^\"]+\"|'[^']+')\s*=") +MINIMAL = "[openshell]\nversion = 2\n" + + +def table_name(line): + return re.sub(r"\s+", "", HEADER.match(line).group(1)) + + +def is_boundary(line): + """A table or array-of-tables header ends the current table.""" + return bool(HEADER.match(line) or ARRAY_HEADER.match(line)) + + +def key_name(raw): + return raw[1:-1] if raw[0] in "\"'" else raw + + +def is_statement_start(line): + return bool(is_boundary(line) or KEY.match(line)) + + +def is_break(line): + stripped = line.strip() + return not stripped or stripped.startswith("#") + + +def statements(lines): + """The (key, lines) pairs of a table body, in order.""" + found = [] + for line in lines: + key = KEY.match(line) + if key: + found.append((key_name(key.group(1)), [line])) + elif found and not is_break(line): + found[-1][1].append(line) + return found + + +def override_tables(text): + if any(line.lstrip().startswith("[[") for line in text.splitlines()): + sys.exit("array-of-tables headers are not supported in the overrides") + tables, current = {}, None + for line in text.splitlines(): + if HEADER.match(line): + current = table_name(line) + tables.setdefault(current, []) + elif current is not None: + tables[current].append(line) + elif not is_break(line): + sys.exit(f"override key outside a table: {line!r}") + return {name: statements(body) for name, body in tables.items()} + + +def apply(lines, table, new): + start = next( + (i for i, l in enumerate(lines) if HEADER.match(l) and table_name(l) == table), + None, + ) + if start is None: + return lines + ["", f"[{table}]", *(l for _, stmt in new for l in stmt)] + for key, stmt in new: + end = next( + (i for i in range(start + 1, len(lines)) if is_boundary(lines[i])), + len(lines), + ) + existing = next( + ( + i + for i in range(start + 1, end) + if (m := KEY.match(lines[i])) and key_name(m.group(1)) == key + ), + None, + ) + if existing is None: + lines[start + 1 : start + 1] = stmt + continue + stop = existing + 1 + while stop < end and not is_statement_start(lines[stop]) and not is_break(lines[stop]): + stop += 1 + lines[existing:stop] = stmt + return lines + + +def contains(merged, expected): + """Whether every table and value in EXPECTED is present in MERGED.""" + if isinstance(expected, dict): + return isinstance(merged, dict) and all( + k in merged and contains(merged[k], v) for k, v in expected.items() + ) + return merged == expected + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("base") + parser.add_argument("overrides") + parser.add_argument("--default") + args = parser.parse_args() + + created = not os.path.exists(args.base) + if not created: + text = open(args.base).read() + elif args.default and os.path.exists(args.default): + text = open(args.default).read() + else: + text = MINIMAL + lines = text.splitlines() + override_text = open(args.overrides).read() + try: + wanted = tomllib.loads(override_text) + except tomllib.TOMLDecodeError as error: + sys.exit(f"{args.overrides} is not valid TOML: {error}") + for table, new in override_tables(override_text).items(): + lines = apply(lines, table, new) + merged = "\n".join(lines) + "\n" + try: + result = tomllib.loads(merged) + except tomllib.TOMLDecodeError as error: + sys.exit(f"merged {args.base} is not valid TOML: {error}") + # The rewriter works on lines, so confirm every override really landed + # rather than trusting it to have handled each statement. + if not contains(result, wanted): + sys.exit(f"the overrides in {args.overrides} were not all applied to {args.base}") + with open(args.base, "w") as out: + out.write(merged) + if created: + os.chmod(args.base, 0o644) + + +main() diff --git a/tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py b/tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py new file mode 100644 index 0000000000..ba6926169a --- /dev/null +++ b/tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import subprocess +import sys +import tomllib +from pathlib import Path + +import pytest + +SCRIPT = Path(__file__).with_name("merge-gateway-toml.py") +BASE = '[openshell]\nversion = 2\n\n[openshell.drivers.podman]\n# keep\nhealth_check_interval_secs = 10\n' + + +def merge(tmp_path, base, overrides, default=None): + base_file, override_file = tmp_path / "gateway.toml", tmp_path / "overrides.toml" + if base is not None: + base_file.write_text(base) + override_file.write_text(overrides) + args = [sys.executable, str(SCRIPT), str(base_file), str(override_file)] + if default is not None: + default_file = tmp_path / "default.toml" + default_file.write_text(default) + args += ["--default", str(default_file)] + run = subprocess.run(args, capture_output=True, text=True) + return run, base_file + + +def test_replaces_and_preserves_comments(tmp_path): + run, base = merge(tmp_path, BASE, '[openshell.drivers.podman]\nhealth_check_interval_secs = 15\nuserns = "keep-id"\n') + assert run.returncode == 0, run.stderr + text = base.read_text() + assert "# keep" in text + assert tomllib.loads(text)["openshell"]["drivers"]["podman"] == { + "health_check_interval_secs": 15, + "userns": "keep-id", + } + + +def test_idempotent(tmp_path): + overrides = '[openshell.drivers.podman]\nuserns = "private"\nuidmap = ["0:0:1",\n "1:1:65535"]\n' + merge(tmp_path, BASE, overrides) + first = (tmp_path / "gateway.toml").read_text() + run, base = merge(tmp_path, first, overrides) + assert run.returncode == 0 and base.read_text() == first + + +def test_array_of_tables_header_is_a_boundary(tmp_path): + base = ( + "[openshell.drivers.podman]\nhealth_check_interval_secs = 10\n" + '[[openshell.supervisor.middleware]]\nname = "local-guard"\n' + 'grpc_endpoint = "https://127.0.0.1:50051"\n' + ) + run, out = merge(tmp_path, base, "[openshell.drivers.podman]\nhealth_check_interval_secs = 15\n") + assert run.returncode == 0, run.stderr + parsed = tomllib.loads(out.read_text())["openshell"] + assert parsed["drivers"]["podman"] == {"health_check_interval_secs": 15} + assert parsed["supervisor"]["middleware"] == [ + {"name": "local-guard", "grpc_endpoint": "https://127.0.0.1:50051"} + ] + + +@pytest.mark.parametrize("quote", ['"', "'"]) +def test_quoted_key_is_applied(tmp_path, quote): + run, base = merge(tmp_path, BASE, f'[openshell.drivers.podman]\n{quote}userns{quote} = "keep-id"\n') + assert run.returncode == 0, run.stderr + assert tomllib.loads(base.read_text())["openshell"]["drivers"]["podman"]["userns"] == "keep-id" + + +@pytest.mark.parametrize( + "overrides", + [ + '[openshell.drivers.podman]\nuserns "keep-id"\n', + '[openshell.drivers.podman]\nuserns = "keep-id\n', + '[[openshell.supervisor.middleware]]\nname = "x"\n', + 'userns = "keep-id"\n', + ], +) +def test_rejects_unusable_overrides_and_leaves_base(tmp_path, overrides): + run, base = merge(tmp_path, BASE, overrides) + assert run.returncode != 0 + assert base.read_text() == BASE + + +def test_dotted_key_not_silently_dropped(tmp_path): + run, base = merge(tmp_path, BASE, "[openshell.drivers]\npodman.userns = \"auto\"\n") + if run.returncode == 0: + assert tomllib.loads(base.read_text())["openshell"]["drivers"]["podman"]["userns"] == "auto" + else: + assert base.read_text() == BASE + + +def test_creates_from_default_else_minimal(tmp_path): + run, base = merge(tmp_path, None, '[openshell.drivers.podman]\nuserns = "auto"\n', default=BASE) + assert run.returncode == 0 and "health_check_interval_secs = 10" in base.read_text() + other = tmp_path / "other" + other.mkdir() + run, base = merge(other, None, '[openshell.drivers.podman]\nuserns = "auto"\n') + parsed = tomllib.loads(base.read_text()) + assert parsed["openshell"]["version"] == 2 diff --git a/tests/tmachine/ansible/roles/openshell_gateway/tasks/main.yaml b/tests/tmachine/ansible/roles/openshell_gateway/tasks/main.yaml index c8089f5ae0..2d23c352c3 100644 --- a/tests/tmachine/ansible/roles/openshell_gateway/tasks/main.yaml +++ b/tests/tmachine/ansible/roles/openshell_gateway/tasks/main.yaml @@ -58,6 +58,11 @@ dest: /etc/openshell/gateway.toml mode: "0644" +- name: Merge environment gateway settings + ansible.builtin.include_tasks: merge-settings.yaml + vars: + openshell_gateway_merge_target: /etc/openshell/gateway.toml + - name: Install OpenShell gateway service become: true ansible.builtin.template: diff --git a/tests/tmachine/ansible/roles/openshell_gateway/tasks/merge-settings.yaml b/tests/tmachine/ansible/roles/openshell_gateway/tasks/merge-settings.yaml new file mode 100644 index 0000000000..6d167a3ec8 --- /dev/null +++ b/tests/tmachine/ansible/roles/openshell_gateway/tasks/merge-settings.yaml @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# An environment overrides gateway settings by writing the tables to change, +# such as `[openshell.drivers.podman]`, to +# /etc/openshell-test/gateway-overrides.toml before the installer runs (for +# example playbooks/drivers/podman/userns.yaml). Apply them to +# `openshell_gateway_merge_target` without replacing the file; see +# files/merge-gateway-toml.py. +# +# `openshell_gateway_merge_default` is a default configuration to start from +# when the target doesn't exist yet, and `openshell_gateway_merge_user` is who +# owns the target. +--- +- name: Check for environment gateway overrides + ansible.builtin.stat: + path: /etc/openshell-test/gateway-overrides.toml + register: openshell_gateway_overrides + +- name: Apply environment gateway overrides + become: true + become_user: "{{ openshell_gateway_merge_user | default('root') }}" + ansible.builtin.script: + cmd: >- + merge-gateway-toml.py {{ openshell_gateway_merge_target }} + /etc/openshell-test/gateway-overrides.toml + {{ '--default ' ~ openshell_gateway_merge_default + if openshell_gateway_merge_default | default('') else '' }} + executable: /usr/bin/python3 + when: openshell_gateway_overrides.stat.exists diff --git a/tests/tmachine/ansible/roles/openshell_gateway/tasks/podman.yaml b/tests/tmachine/ansible/roles/openshell_gateway/tasks/podman.yaml index b85361b769..b47c7a65dd 100644 --- a/tests/tmachine/ansible/roles/openshell_gateway/tasks/podman.yaml +++ b/tests/tmachine/ansible/roles/openshell_gateway/tasks/podman.yaml @@ -36,22 +36,3 @@ environment: HOME: "{{ openshell_gateway_home }}" XDG_RUNTIME_DIR: "/run/user/{{ openshell_gateway_uid }}" - -# An environment can add [openshell.drivers.podman] settings, such as the -# user-namespace mode (playbooks/drivers/podman/userns.yaml). -- name: Check for environment Podman driver settings - ansible.builtin.stat: - path: /etc/openshell-test/podman-driver.toml - register: openshell_gateway_podman_driver_file - -- name: Read environment Podman driver settings - ansible.builtin.slurp: - src: /etc/openshell-test/podman-driver.toml - register: openshell_gateway_podman_driver_slurp - when: openshell_gateway_podman_driver_file.stat.exists - -- name: Select environment Podman driver settings - ansible.builtin.set_fact: - openshell_gateway_podman_driver_settings: >- - {{ openshell_gateway_podman_driver_slurp.content | b64decode | trim - if openshell_gateway_podman_driver_file.stat.exists else '' }} diff --git a/tests/tmachine/ansible/roles/openshell_gateway/templates/gateway-podman.toml.j2 b/tests/tmachine/ansible/roles/openshell_gateway/templates/gateway-podman.toml.j2 index 0ee292ea29..e59314f632 100644 --- a/tests/tmachine/ansible/roles/openshell_gateway/templates/gateway-podman.toml.j2 +++ b/tests/tmachine/ansible/roles/openshell_gateway/templates/gateway-podman.toml.j2 @@ -8,7 +8,4 @@ image_pull_policy = "if_not_present" network_name = "tmachine" health_check_interval_secs = 10 stop_timeout_secs = 15 -{% if openshell_gateway_podman_driver_settings %} -{{ openshell_gateway_podman_driver_settings }} -{% endif %} {% endblock %} diff --git a/tests/tmachine/ansible/roles/openshell_packaged_gateway/tasks/main.yaml b/tests/tmachine/ansible/roles/openshell_packaged_gateway/tasks/main.yaml index 155eaa44c8..43db126598 100644 --- a/tests/tmachine/ansible/roles/openshell_packaged_gateway/tasks/main.yaml +++ b/tests/tmachine/ansible/roles/openshell_packaged_gateway/tasks/main.yaml @@ -21,17 +21,6 @@ name: openshell_gateway tasks_from: "{{ openshell_gateway_driver }}.yaml" -# The packaged gateway's configuration comes from its environment file, so it -# can't apply an environment's Podman driver settings. Fail rather than test a -# gateway without them. -- name: Require no environment Podman driver settings - ansible.builtin.assert: - that: - - not (openshell_gateway_podman_driver_settings | default('')) - fail_msg: >- - This environment configures Podman driver settings, which only the - binaries installer applies. - - name: Create packaged gateway environment directory become: true ansible.builtin.file: @@ -56,6 +45,21 @@ OPENSHELL_SANDBOX_RUNTIME_IMAGE=docker.io/openshell/sandbox:tmachine OPENSHELL_SUPERVISOR_IMAGE=docker.io/openshell/supervisor:tmachine +# Override settings in the package's own configuration. The rpm seeds +# ~/.config/openshell/gateway.toml from its default on first start, which comes +# after this, so start from that default here and the package's seeding then +# leaves the file alone. A package with no default, such as the deb, gets a +# minimal file; its driver comes from gateway.env. An environment with no +# overrides leaves the packaged configuration untouched. +- name: Apply environment gateway overrides to the packaged gateway + ansible.builtin.include_role: + name: openshell_gateway + tasks_from: merge-settings.yaml + vars: + openshell_gateway_merge_target: "{{ openshell_gateway_home }}/.config/openshell/gateway.toml" + openshell_gateway_merge_default: /usr/share/openshell-gateway/gateway.toml.default + openshell_gateway_merge_user: "{{ openshell_gateway_user }}" + - name: Start gateway user manager ansible.builtin.include_role: name: tmachine_user_manager diff --git a/tests/tmachine/config.nix b/tests/tmachine/config.nix index 24206097f9..8a8a798176 100644 --- a/tests/tmachine/config.nix +++ b/tests/tmachine/config.nix @@ -71,7 +71,8 @@ let ) podmanModes; # fedora-podman-{rootful,rootless}-userns-{default,auto,keep-id,private} # configure the Podman gateway's user-namespace mode, for the Podman driver - # suite. Only the binaries installer applies the mode. + # suite. Every installer applies the mode through + # /etc/openshell-test/gateway-overrides.toml. podmanUsernsEnvironments = pkgs.lib.concatMap ( podman: map ( @@ -274,7 +275,7 @@ let }; # Runs a test family in a tmachine environment with tests/runner - # (`nix run .#openshell-test-runner -- --environment ubuntu-docker-rootful + # (`nix run .#openshell-test-runner -- run --environment ubuntu-docker-rootful # --installer deb --family conformance`). It cross-builds guest test bundles # and drives tmachine sessions, so it needs the toolchains and the VM tools. testRunner = pkgs.writeShellApplication {