Skip to content

Commit 5d7cb2b

Browse files
test(plans): run the ported conformance suites in the cpu-tests plan
Run the sandbox suites on rootful Docker and rootful and rootless Podman, and the OCI image suite on rootful Docker. Document the suites, the plan and what the Keycloak and Podman userns suites still need before they can join it. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent 6f1d95e commit 5d7cb2b

4 files changed

Lines changed: 61 additions & 2 deletions

File tree

‎TESTING.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,27 @@ The older conformance, feature and driver lanes in `.github/workflows/` are stil
229229
written by hand and run through the tmachine testsuites in
230230
`tests/tmachine/config.nix`. They move to plans as their suites move to testkit.
231231

232+
The old suites now also exist as testkit suites, and the plan runs the ones the
233+
runner can set up: the `sandbox` and `sandbox-image` crates of the `conformance`
234+
family. The old crates and playbooks stay until the release lanes switch to
235+
plans. Two ports are not in a plan yet:
236+
237+
- `features/providers` (provider OAuth refresh against Keycloak) needs the
238+
`keycloak` Ansible role to run a Keycloak service on the guest with the realm in
239+
`scripts/keycloak-realm.json`, and `OPENSHELL_E2E_OIDC_ISSUER`,
240+
`OPENSHELL_E2E_OIDC_USERNAME` and `OPENSHELL_E2E_OIDC_PASSWORD` set for the
241+
tests. The runner provisions only an environment and an installer, so it has no
242+
place for suite-specific guest setup.
243+
- `features/podman` (user namespaces) needs the gateway reconfigured for each
244+
`userns` mode with the fixtures in `tests/suites/features/podman/fixtures`, and
245+
a reference UID map from native Podman in `OPENSHELL_TEST_INPUT_DIR`, as the old
246+
`driver-podman` playbooks do. The runner starts one gateway configuration per
247+
environment and runs a suite once.
248+
249+
Both crates are compiled and linted by Branch Checks. The Keycloak one runs
250+
locally with `e2e/with-keycloak.sh e2e/with-docker-gateway.sh cargo test
251+
--manifest-path tests/suites/features/Cargo.toml -p providers`.
252+
232253
### Python E2E (`e2e/python/`)
233254

234255
`mise run e2e:python` builds `openshell/e2e-python:dev` from

‎tests/plans/README.md‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,14 @@ environment does not provide is left out of that environment.
1919

2020
| Plan | Runs |
2121
|---|---|
22-
| `cpu-tests.yaml` | The `conformance` family on Ubuntu 24.04 amd64 with rootful Docker and the Debian package |
22+
| `cpu-tests.yaml` | The `conformance` sandbox suites (`smoke`, `lifecycle`, `cli`, `persistent-workspace`, `file-transfer`, `policy-advisor`) on rootful Docker (Debian package) and rootful and rootless Podman (RPM), and the `oci-image` suite on rootful Docker, all Ubuntu 24.04 or Fedora 44 amd64 |
23+
24+
The Kubernetes (k3s) environments are defined but not in a plan: the old lane
25+
verified that the Agent Sandbox controller reconciles before it ran tests, and
26+
the runner only waits for the gateway. The `features` family (`providers`
27+
against Keycloak, `podman` user namespaces) is not in a plan either, because both
28+
need guest setup the runner does not do; see
29+
[TESTING.md](../../TESTING.md#test-plans-and-the-runner).
2330

2431
## How a plan runs on CI
2532

@@ -40,6 +47,10 @@ Branch E2E calls `.github/workflows/e2e-suites.yml` with the plan name when the
4047
4. The job uploads the JUnit reports and logs, and an aggregator job reports the
4148
result.
4249

50+
An environment can add environment variables for the test processes with `env`,
51+
for what a suite reads that is not a capability. `docker-rootful` sets
52+
`OPENSHELL_TEST_CONTAINER_ENGINE=docker` for the `oci-image` suite.
53+
4354
Adding an environment or a suite to a plan changes what CI runs with no workflow
4455
change. Arm64 environments are defined in `tests/environments/` but are not in a
4556
plan until a runner that can boot them with KVM exists.

‎tests/plans/cpu-tests.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,28 @@
1414
# for tmachine yet. The arm64 environments in tests/environments/ are defined
1515
# so a plan can use them once one exists.
1616
matrix:
17+
# Sandbox behavior through the SDK and the CLI. These tests are
18+
# driver-agnostic, so every environment that has a Docker or Podman gateway
19+
# the runner can wait for runs them. The Kubernetes (k3s) environments are
20+
# left out: before tests, the old lane verified that the Agent Sandbox
21+
# controller reconciles (tests/tmachine/ansible/playbooks/conformance/
22+
# k3s-ready.yaml), and the runner only waits for the gateway.
1723
- type: conformance
24+
suites:
25+
- smoke
26+
- lifecycle
27+
- cli
28+
- persistent-workspace
29+
- file-transfer
30+
- policy-advisor
31+
environments:
32+
- tmachine.ubuntu2404-amd64.docker-rootful.deb
33+
- tmachine.fedora44-amd64.podman-rootful.rpm
34+
- tmachine.fedora44-amd64.podman-rootless.rpm
35+
# OCI image behavior. The suite builds images with the gateway's container
36+
# engine, which the environment names in `env`. Only Docker names it so far.
37+
- type: conformance
38+
suites:
39+
- oci-image
1840
environments:
1941
- tmachine.ubuntu2404-amd64.docker-rootful.deb

‎tests/testkit/README.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,12 @@ test by hand needs the variable:
106106
OPENSHELL_TEST_CAPABILITIES=persistent_workspace cargo nextest run ...
107107
```
108108

109-
Tests without `requires`, which is every test today, never read it. To run only
109+
The `mise run e2e:testkit` task sets it for a Docker gateway. Other settings a
110+
suite reads, such as `OPENSHELL_TEST_CONTAINER_ENGINE` for the `oci-image`
111+
suite, are plain environment variables; an environment definition supplies them
112+
with its `env` field.
113+
114+
Tests without `requires` never read it. To run only
110115
the tests an environment supports by hand, exclude the others by name with a
111116
nextest filter, for example `-E 'not test(=stopped_sandbox_keeps_workspace)'`.
112117

0 commit comments

Comments
 (0)