Skip to content

Commit 6b72502

Browse files
feat(runner): add the test runner, environments and the cpu-tests plan
`openshell-test-runner` expands a plan (tests/plans/*.yaml) into one job per environment, builds the family's nextest archive for the guest, boots the environment through the tmachine Session API, runs nextest in it and brings back the JUnit report and logs. No per-suite Ansible playbook is involved. The runner points the tests at the system under test with environment variables: OPENSHELL_GATEWAY (the gateway the installer registered, from the environment's `gateway` field) and OPENSHELL_BIN (the installed CLI). Nextest runs with `--test-threads 1` and a slow-timeout that terminates a hung test. The JUnit report is checked against the catalog, so a missing or skipped expected test fails the job; `#[ignore]` tests are not expected and nextest's skipped case for them is accepted. `--strict` fails on an unsupported environment, a family with no tests, or an empty selection. `--emit-github-matrix` prints the GitHub Actions matrix for a plan, so the plan is the only list of what runs where. One environment is defined, amd64 Ubuntu with rootful Docker and the Debian package, which is the one CI can provision today; tests/plans/cpu-tests.yaml runs the conformance family on it. Further environments (Podman, k3s, arm64) are added with the suites that need them. There are no waivers or per-test requirements: the plan picks suites, and a broken test is `#[ignore]`d. Most of this comes from #4629, adapted to testkit finding the gateway the way the CLI does instead of reading a target file. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent 55cc230 commit 6b72502

16 files changed

Lines changed: 3895 additions & 0 deletions

File tree

‎tests/environments/docker.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Docker-backed environments, by Evan Lezar (NVIDIA/OpenShell#4629).
5+
#
6+
# The ID must equal <backend>.<os><version without dots>-<arch>.<runtime>.<installer>.
7+
# `tmachine` names the environment and installer in tests/tmachine/config.nix
8+
# that provision it. `gateway` is the name the installer registers the gateway
9+
# under with the CLI (`openshell gateway add --name`); the packaged installers
10+
# register `openshell`.
11+
#
12+
# Only amd64 is defined: CI has no arm64 runner with KVM for tmachine yet. To add
13+
# arm64, copy the entry with `arch: arm64` and a `ubuntu2404-arm64` ID, then list
14+
# it in a plan. The CI matrix picks the `ubuntu-24.04-arm` runner for it (see
15+
# runner_label in tests/runner/src/github.rs); the calling workflow must also
16+
# build arm64 inputs.
17+
environments:
18+
- id: tmachine.ubuntu2404-amd64.docker-rootful.deb
19+
backend: tmachine
20+
machine: { os: ubuntu, version: "24.04", arch: amd64 }
21+
runtime: docker-rootful
22+
installer: deb
23+
tmachine: { environment: ubuntu-docker-rootful, installer: deb }
24+
gateway: openshell

‎tests/plans/cpu-tests.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Tests that need no GPU. `type` names a family under tests/suites/ (a Cargo
5+
# workspace of testkit suites). Omit `suites` to run every suite in the family,
6+
# or list suite names (a suite is a file under <crate>/tests/) to run only
7+
# those. `environments` are IDs from tests/environments/.
8+
#
9+
# CI runs this plan from .github/workflows/e2e-suites.yml, one job per
10+
# environment listed here. To add coverage, add an environment to an entry or an
11+
# entry for another family; no workflow change is needed.
12+
#
13+
# Only amd64 is listed: Branch E2E has no arm64 runner with KVM for tmachine
14+
# yet. See tests/environments/docker.yaml for how to add arm64.
15+
matrix:
16+
- type: conformance
17+
environments:
18+
- tmachine.ubuntu2404-amd64.docker-rootful.deb

0 commit comments

Comments
 (0)