test(testkit): add the testkit library for end-to-end suites (#4724)

* test(testkit): add the testkit workspace for end-to-end suites

Add tests/testkit as its own Cargo workspace: testkit, testkit-macros,
and testkit-spec. testkit reads a target file, connects to the gateway
through openshell-sdk, creates and cleans up sandboxes, and runs the
installed CLI. #[testkit::test] requires a stable ID and a doc comment and
declares capability requirements. testkit-ready waits for a target's
gateway and CLI before tests run.

Nothing uses testkit yet. The lockfile is seeded from the root lockfile
so shared crates resolve to the same versions.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): format, lint and test testkit in Branch Checks

Nothing else builds testkit, so check it in Branch Checks and the
matching mise tasks to catch openshell-sdk changes that break it.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(tests): describe how to write and run testkit tests

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): give each sandbox its own cleanup timeout

Cleanup shared one two-minute budget across all of a test's sandboxes, so
a slow delete left the rest with no time and could leak them on a shared
gateway. Delete them concurrently, each with the full timeout.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testkit): show how to exclude tests by capability

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs: list tests/testkit in the project structure tables

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(testkit): drop test IDs and capabilities for now

Nothing reads test IDs or capability requirements until a runner, waivers,
or result history exist, so remove the id and requires arguments and the
target file's capabilities section. Both can return as optional attribute
arguments without changing existing tests.

With them gone, testkit-spec only checked the function shape and read the
doc comment, so fold it into testkit-macros.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testkit): describe what testkit provides today

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(testkit): find the gateway the way the CLI does

Replace the target file (OPENSHELL_TEST_TARGET) with the CLI's own gateway
selection: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the
active gateway. A registered gateway's endpoint, auth mode, and mTLS
material come from its CLI config directory. cx.cli runs OPENSHELL_BIN
against the same gateway and workspace as the SDK.

Suites now run unchanged under e2e/with-docker-gateway.sh and
e2e/with-podman-gateway.sh, and environments need no extra file.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): keep labels when creating a sandbox with a policy

The policy path creates the sandbox through gRPC and dropped the spec's
labels without an error. Pass them in the create request.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): use the registered gateway's credentials for an endpoint

When OPENSHELL_GATEWAY_ENDPOINT is set, the CLI takes credentials from the
gateway named by OPENSHELL_GATEWAY, or else the registered gateway with
that endpoint. testkit connected without a client certificate instead, so
an mTLS endpoint such as the one e2e/rust/e2e-vm.sh exports worked for
cx.cli but not for the SDK. Resolve it the way the CLI does, and pass
both the name and the endpoint to cx.cli.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(tests): list testkit's gateway variables in precedence order

OPENSHELL_GATEWAY_ENDPOINT takes precedence over OPENSHELL_GATEWAY. Also
mention OPENSHELL_BIN.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): apply other attributes to the generated test

#[testkit::test] left attributes such as #[ignore] and #[cfg] on the
nested body function, where they have no effect on the test. Move them
to the generated test, and reject a second test attribute such as

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
#[test] or #[tokio::test].

* fix(testkit): show usage for testkit-ready --help

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(testkit): cover cleanup after pass, failure, and panic

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): reject #[should_panic] on testkit tests

A testkit test panics on any failure, including an unreachable gateway or
a failed cleanup, so #[should_panic] would let such a failure pass. Reject
it and point to checking the expected failure in the body.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(testkit): keep only the macro's misuse checks as unit tests

Suites built on testkit exercise gateway resolution, the CLI wrapper,
polling, errors, and cleanup against a real gateway. Keep one unit test
for the macro's compile errors, which a passing suite never reaches, and
drop the test-only seams the other tests needed.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(testkit): connect through the SDK's mTLS support

The SDK's ClientConfig now takes a client identity, so testkit no longer builds
its own channel. It still resolves the registered gateway and reads its
credential files, then hands the CA and client certificate to
OpenShellClient::connect. That keeps TLS behavior, connect timeouts and
keep-alive settings in the SDK, and drops testkit's direct tonic dependency.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(testkit): bound readiness and cleanup with timeouts

wait_ready checked elapsed time only after a health call returned, and cleanup
started its timeout only after the delete returned, so a gateway that accepts a
connection but never answers could hang either one. Wrap each whole operation in
tokio::time::timeout: the readiness wait, including every connect and health
call, and each sandbox's delete plus wait for deletion. A test covers a peer
that accepts connections and never replies.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Author: matthewgrossman

.github/workflows/branch-checks.yml

@@ -221,6 +221,7 @@ jobs:
           cargo fmt --manifest-path examples/governance-interceptor/Cargo.toml --all -- --check
           cargo fmt --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all -- --check
           cargo fmt --manifest-path examples/isolation-backend-v8/Cargo.toml --all -- --check
+          cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
 
   cargo-lockfiles:
     name: Cargo lockfiles
@@ -277,6 +278,7 @@ jobs:
           cargo clippy --locked --manifest-path examples/governance-interceptor/Cargo.toml --all-targets -- -D warnings
           cargo clippy --locked --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all-targets -- -D warnings
           cargo clippy --locked --manifest-path examples/isolation-backend-v8/Cargo.toml --all-targets -- -D warnings
+          cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
 
   rust-test:
     name: Rust test (${{ matrix.system }})
@@ -325,6 +327,7 @@ jobs:
         run: |
           cargo nextest run --locked --profile ci --workspace --features openshell-server/test-support
           cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml
+          cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path tests/testkit/Cargo.toml
 
   rust-postgres-test:
     name: Rust PostgreSQL tests

AGENTS.md

@@ -61,6 +61,7 @@ Do not rely on this file for a full inventory. The detailed public and contribut
 | `python/openshell/` | Python SDK | Python bindings and CLI packaging |
 | `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |
 | `proto/` | Protobuf definitions | gRPC service contracts |
+| `tests/testkit/` | E2E testkit | Support library and `#[testkit::test]` attribute for SDK-driven E2E suites |
 | `deploy/` | Docker, Helm, K8s | Dockerfiles, Helm chart, manifests |
 | `docs/` | Published docs | MDX pages, navigation, and content assets |
 | `fern/` | Docs site config | Fern site config, components, and theme assets |

CONTRIBUTING.md

@@ -344,6 +344,7 @@ These are the primary `mise` tasks for day-to-day development:
 | `sdk/typescript/` | TypeScript SDK (Connect client and generated protobuf bindings) |
 | `proto/`        | Protocol buffer definitions and [public API conventions](proto/README.md) |
 | `tasks/`        | `mise` task definitions and build scripts     |
+| `tests/testkit/` | Support library for SDK-driven end-to-end suites |
 | `deploy/`       | Dockerfiles, Helm chart, Kubernetes manifests |
 | `docs/`         | Published Fern docs source, navigation, and content assets |
 | `fern/`         | Fern site config, components, and theme assets |

TESTING.md

@@ -16,6 +16,7 @@ crates/*/tests/        # Rust integration tests
 python/openshell/      # Python unit tests (*_test.py suffix)
 e2e/python/            # Python E2E tests (test_*.py prefix)
 e2e/rust/              # Rust CLI E2E tests
+tests/testkit/         # Support library for SDK-driven E2E suites
 ```
 
 ## Rust Tests
@@ -129,6 +130,16 @@ requires mTLS.
 runtime. It covers the current schema and the Podman in-tree versus external
 driver configuration boundary.
 
+### testkit (`tests/testkit/`)
+
+`testkit` is the support library for SDK-driven end-to-end suites. Tests are
+`#[testkit::test]` functions with a required doc comment. They find the
+gateway the same way the CLI does, through `OPENSHELL_GATEWAY_ENDPOINT`,
+`OPENSHELL_GATEWAY`, or the active gateway, and `cx.cli` runs the CLI named
+by `OPENSHELL_BIN`. See
+[tests/testkit/README.md](tests/testkit/README.md) for how to write and run
+tests.
+
 ### Python E2E (`e2e/python/`)
 
 `mise run e2e:python` builds `openshell/e2e-python:dev` from

tasks/rust.toml

@@ -33,6 +33,7 @@ run = [
   "cargo clippy --manifest-path examples/governance-interceptor/Cargo.toml --all-targets -- -D warnings",
   "cargo clippy --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all-targets -- -D warnings",
   "cargo clippy --manifest-path examples/isolation-backend-v8/Cargo.toml --all-targets -- -D warnings",
+  "cargo clippy --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings",
 ]
 run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 lint native"
 hide = true
@@ -45,6 +46,7 @@ run = [
   "cargo fmt --manifest-path examples/governance-interceptor/Cargo.toml --all",
   "cargo fmt --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all",
   "cargo fmt --manifest-path examples/isolation-backend-v8/Cargo.toml --all",
+  "cargo fmt --manifest-path tests/testkit/Cargo.toml --all",
 ]
 hide = true
 
@@ -56,6 +58,7 @@ run = [
   "cargo fmt --manifest-path examples/governance-interceptor/Cargo.toml --all -- --check",
   "cargo fmt --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all -- --check",
   "cargo fmt --manifest-path examples/isolation-backend-v8/Cargo.toml --all -- --check",
+  "cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check",
 ]
 hide = true
 

tasks/test.toml

@@ -119,6 +119,7 @@ run = [
   "cargo test --workspace --exclude openshell-server",
   "cargo test -p openshell-server --features test-support",
   "cargo nextest run --config-file .config/nextest.toml --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml",
+  "cargo nextest run --config-file .config/nextest.toml --manifest-path tests/testkit/Cargo.toml",
 ]
 run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-precommit native"
 hide = true