Repository navigation
Commit 6b72502
feat(runner): add the test runner, environments and the cpu-tests plan
`openshell-test-runner` expands a plan (tests/plans/*.yaml) into one job per
environment, builds the family's nextest archive for the guest, boots the
environment through the tmachine Session API, runs nextest in it and brings back
the JUnit report and logs. No per-suite Ansible playbook is involved.
The runner points the tests at the system under test with environment variables:
OPENSHELL_GATEWAY (the gateway the installer registered, from the environment's
`gateway` field) and OPENSHELL_BIN (the installed CLI). Nextest runs with
`--test-threads 1` and a slow-timeout that terminates a hung test. The JUnit
report is checked against the catalog, so a missing or skipped expected test
fails the job; `#[ignore]` tests are not expected and nextest's skipped case for
them is accepted. `--strict` fails on an unsupported environment, a family
with no tests, or an empty selection.
`--emit-github-matrix` prints the GitHub Actions matrix for a plan, so the plan
is the only list of what runs where. One environment is defined, amd64
Ubuntu with rootful Docker and the Debian package, which is the one CI can
provision today; tests/plans/cpu-tests.yaml runs the conformance family on it.
Further environments (Podman, k3s, arm64) are added with the suites that need them.
There are no waivers or per-test requirements: the plan picks suites, and a
broken test is `#[ignore]`d.
Most of this comes from #4629, adapted to testkit finding the gateway the way the
CLI does instead of reading a target file.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>1 parent 55cc230 commit 6b72502
16 files changed
Lines changed: 3895 additions & 0 deletions
File tree
- tests
- environments
- plans
- runner
- src
- testdata
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
0 commit comments