docs(tests): document the runner and the conformance suite

Describe how the runner runs a family in a tmachine environment, locally and
on CI, in TESTING.md, add a README for the sandbox suite, point the testkit
README at the runner, and add rows for tests/suites, tests/tmachine and
tests/runner to the project-structure tables in AGENTS.md and CONTRIBUTING.md.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Author: matthewgrossman

AGENTS.md

@@ -63,6 +63,9 @@ Do not rely on this file for a full inventory. The detailed public and contribut
 | `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |
 | `proto/` | Protobuf definitions | gRPC service contracts |
 | `tests/testkit/` | E2E testkit | Support library and `#[testkit::test]` attribute for SDK-driven E2E suites |
+| `tests/suites/` | E2E suites | One Cargo workspace per test family; `conformance/sandbox` is the first testkit suite |
+| `tests/tmachine/` | Test machines | QEMU machines provisioned with Ansible, and the `Session` API that drives them |
+| `tests/runner/` | Test runner | `openshell-test-runner`: builds a family's tests, runs them in one tmachine environment, checks the JUnit results |
 | `deploy/` | Docker, Helm, K8s | Dockerfiles, Helm chart, manifests |
 | `docs/` | Published docs | MDX pages, navigation, and content assets |
 | `fern/` | Docs site config | Fern site config, components, and theme assets |

CONTRIBUTING.md

@@ -345,6 +345,9 @@ These are the primary `mise` tasks for day-to-day development:
 | `proto/`        | Protocol buffer definitions and [public API conventions](proto/README.md) |
 | `tasks/`        | `mise` task definitions and build scripts     |
 | `tests/testkit/` | Support library for SDK-driven end-to-end suites |
+| `tests/suites/` | End-to-end suites, one Cargo workspace per family |
+| `tests/tmachine/` | QEMU test machines provisioned with Ansible, and the `Session` API that drives them |
+| `tests/runner/` | `openshell-test-runner`: runs a test family in a tmachine environment and checks the results |
 | `deploy/`       | Dockerfiles, Helm chart, Kubernetes manifests |
 | `docs/`         | Published Fern docs source, navigation, and content assets |
 | `fern/`         | Fern site config, components, and theme assets |

TESTING.md

@@ -17,6 +17,9 @@ python/openshell/      # Python unit tests (*_test.py suffix)
 e2e/python/            # Python E2E tests (test_*.py prefix)
 e2e/rust/              # Rust CLI E2E tests
 tests/testkit/         # Support library for SDK-driven E2E suites
+tests/suites/conformance/  # Testkit suites (the `sandbox` crate) beside the older CLI conformance crate
+tests/tmachine/        # QEMU test machines, Ansible playbooks, and the Session API
+tests/runner/          # openshell-test-runner: runs a test family in a tmachine environment
 ```
 
 ## Rust Tests
@@ -140,6 +143,71 @@ by `OPENSHELL_BIN`. See
 [tests/testkit/README.md](tests/testkit/README.md) for how to write and run
 tests.
 
+Suites live under `tests/suites/<family>/`. Each family is a Cargo workspace
+with its own lockfile. `tests/suites/conformance/sandbox` is the first testkit
+suite; it has one smoke test. Run it against a local Docker gateway with:
+
+```shell
+mise run e2e:testkit
+```
+
+That task runs `e2e/with-docker-gateway.sh`, which starts a gateway and exports
+`OPENSHELL_GATEWAY` and `OPENSHELL_BIN`, so the suite runs the same way the
+runner runs it in a guest.
+
+Workspaces that build root crates through path dependencies must lock shared
+crates at the versions in the root `Cargo.lock`; otherwise CI can build against
+dependencies the Nix toolchain doesn't support. `mise run rust:lockfiles:check`
+verifies this, and `uv run tasks/scripts/check_cargo_lockfile_alignment.py --fix`
+re-locks a workspace from the root lockfile. Add new testkit workspaces to
+`ALIGNED_WORKSPACES` in that script.
+
+### Running suites in a tmachine environment (`tests/runner/`)
+
+`openshell-test-runner` runs a test family inside a tmachine VM, against the
+OpenShell that the VM's installer set up. It is adapted from the runner in
+NVIDIA/OpenShell#4629 by Evan Lezar.
+
+```shell
+nix run .#openshell-test-runner -- \
+  --environment ubuntu-docker-rootful --installer deb \
+  --family conformance --package sandbox
+```
+
+It:
+
+1. builds the family's packages on the host as a nextest archive for the
+   guest's musl target, with `testkit-ready`;
+2. boots the tmachine environment with the installer (this needs KVM, and
+   tmachine only boots guests of the host's architecture);
+3. copies the archive in, waits for the gateway with `testkit-ready`, and runs
+   the tests with the guest's `cargo-nextest`, pointing them at the installed
+   CLI (`OPENSHELL_BIN`) and the gateway the installer registered
+   (`OPENSHELL_GATEWAY`, `--gateway`, default `openshell`);
+4. writes `junit.xml`, `nextest.log` and, after a failure, the guest's
+   diagnostics to `artifacts/test-results/` (`--results`);
+5. shuts the VM down, also on failure or SIGTERM.
+
+A run passes if nextest succeeds and its JUnit report shows at least one test
+and no failures. `--filter <filterset>` narrows the tests with a nextest
+filterset, and `--keep` leaves the VM running until you press Enter. Every SSH
+and nextest step has a timeout. Two runs on one host need different
+`TMACHINE_SSH_PORT` values (default 2222). The installer inputs, such as the
+Debian package and the images, come from `artifacts/`, built by
+`nix run .#build-artifacts` or downloaded by CI.
+
+A flaky or broken test is marked `#[ignore = "<issue link>"]`, which takes it
+out of every environment until it's fixed.
+
+**CI.** `.github/workflows/e2e-suites.yml` lists the environments and suites as
+an explicit matrix and runs the runner once per row. Branch E2E calls it when
+the `test:e2e` label is set. It is advisory for now. To run a suite in another
+environment on CI, add a row.
+
+The older conformance, feature and driver lanes in `.github/workflows/` still
+run through the tmachine testsuites in `tests/tmachine/config.nix`. They move
+to the runner as their suites move to testkit.
+
 ### Python E2E (`e2e/python/`)
 
 `mise run e2e:python` builds `openshell/e2e-python:dev` from

tests/suites/conformance/sandbox/README.md

@@ -0,0 +1,60 @@
+# Sandbox conformance suite
+
+The `sandbox` crate holds testkit suites for sandbox behavior that every compute
+driver must provide. It is a member of the `conformance` workspace. Each file in
+`tests/` is one suite.
+
+| Suite | Covers |
+|---|---|
+| `smoke` | Create a sandbox, run `echo` in it, delete it |
+
+The suite is small on purpose. It proves that the runner can run a real test in
+a tmachine environment on CI. The older suites move to testkit by adding suites
+here, or a new family under `tests/suites/`.
+
+## Running it
+
+Against a local Docker gateway:
+
+```shell
+mise run e2e:testkit
+```
+
+Against any gateway the CLI can reach, with the testkit environment variables
+described in [tests/testkit/README.md](../../../testkit/README.md):
+
+```shell
+OPENSHELL_GATEWAY=<name> cargo nextest run --locked \
+  --manifest-path tests/suites/conformance/Cargo.toml -p sandbox --test-threads 1
+```
+
+In a tmachine VM, as CI runs it (see [TESTING.md](../../../../TESTING.md)):
+
+```shell
+nix run .#openshell-test-runner -- \
+  --environment ubuntu-docker-rootful --installer deb \
+  --family conformance --package sandbox
+```
+
+## Adding a test
+
+Add a `#[testkit::test]` function to a suite, or a new file in `tests/` for a new
+area. No playbook, test archive or workflow entry is needed: the runner builds
+and runs every test in the package.
+
+## What a test may assert
+
+These tests run on every compute driver, so they check the public contract and
+leave driver behavior alone.
+
+- Assert only what the public contract promises: the proto comments and the
+  published docs.
+- Assert `phase` and `exit_code`. Don't assert condition reasons.
+- Check that something did not happen by reading state, not by sleeping.
+- Every environment must pass every test here. If a behavior is left to the
+  driver, the test belongs in a `features` or `drivers` suite instead.
+- Use the CLI for upload, download, connect and forward. Rely on its exit code
+  and `--output json`.
+- Add a testkit helper instead of calling raw gRPC from a test.
+
+Guidance adapted from NVIDIA/OpenShell#4629 by Evan Lezar.

tests/testkit/README.md

@@ -89,6 +89,12 @@ time or the CLI does not report it connected.
 ephemeral gateway and export `OPENSHELL_GATEWAY` and `OPENSHELL_BIN`, so a
 suite runs under them unchanged.
 
+`mise run e2e:testkit` does this for the suites in `tests/suites/conformance`.
+In a tmachine VM, the test runner (`tests/runner/`) does the same: it sets
+`OPENSHELL_GATEWAY` to the gateway the installer registered and
+`OPENSHELL_BIN` to the installed CLI, then runs the suites with nextest. See
+[TESTING.md](../../TESTING.md).
+
 ## Checks
 
 ```shell