Skip to content

Commit a509ab3

Browse files
docs(testing): describe how to add a feature suite
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent e6e3d16 commit a509ab3

1 file changed

Lines changed: 41 additions & 10 deletions

File tree

‎TESTING.md‎

Lines changed: 41 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -190,17 +190,48 @@ an explicit matrix and runs the runner once per row. Branch E2E calls it when
190190
the `test:e2e` label is set. It is advisory, so a failure doesn't block
191191
merging. To run a suite in another environment on CI, add a row.
192192

193-
**Environments with services.** A suite that needs something besides the
193+
### Adding a feature suite
194+
195+
A feature suite tests one OpenShell feature and lives in the
196+
`tests/suites/features` workspace. `tests/suites/features/keycloak` is the
197+
example to copy.
198+
199+
1. Add a crate under `tests/suites/features/<name>` that depends on
200+
`openshell-testkit`, and add it to `members` in
201+
`tests/suites/features/Cargo.toml`.
202+
2. Write the tests in `tests/*.rs` as `#[testkit::test]` functions.
203+
3. Re-lock the workspace: `cargo metadata --manifest-path
204+
tests/suites/features/Cargo.toml`, then
205+
`uv run tasks/scripts/check_cargo_lockfile_alignment.py --fix`.
206+
4. Add a row for the suite to the matrix in `.github/workflows/e2e-suites.yml`.
207+
208+
If the feature only needs the gateway, run the suite in an existing
209+
environment such as `ubuntu-docker-rootful`, and skip the rest of this
210+
section.
211+
212+
**Suites that need a service.** A feature that needs something besides the
194213
gateway, such as an OIDC issuer, runs in a tmachine environment that provides
195-
it. The environment's setup starts the service (with a restart policy, since
196-
setup is cached as a disk) and writes how to reach it as `OPENSHELL_TEST_*`
197-
lines in `/etc/openshell-test/env.d/<service>.env`. The runner passes those
198-
variables to the tests and rejects names without that prefix. A suite-local
199-
fixture reads them with plain environment variables, so you can run the suite
200-
against your own deployment by setting them yourself. See the
201-
`ubuntu-docker-rootful-keycloak` environment in `tests/tmachine/config.nix`,
202-
`tests/tmachine/ansible/playbooks/keycloak.yaml` and
203-
`tests/suites/features/keycloak`:
214+
it. The environment and the suite share a contract: a set of
215+
`OPENSHELL_TEST_<SERVICE>_*` environment variables that say how to reach the
216+
service.
217+
218+
- **Environment.** Add an environment to `tests/tmachine/config.nix` whose
219+
setup playbook starts the service (with a restart policy, since setup is
220+
cached as a disk) and writes the variables as `KEY=VALUE` lines to
221+
`/etc/openshell-test/env.d/<service>.env`. Pass files the playbook needs
222+
through the environment's setup `inputs`, and add them to the tmachine disk
223+
cache key in `e2e-suites.yml`. The runner passes the variables to the tests
224+
and rejects names without the `OPENSHELL_TEST_` prefix.
225+
- **Fixture.** In the suite's `src/lib.rs`, define the variable names as
226+
constants and a `<Service>Context::new(cx)` that reads them, waits for the
227+
service, and wraps the testkit `Context`. Each test calls it on its first
228+
line. Nextest runs each test in its own process, so the fixture builds its
229+
clients there from the variables. A missing variable fails the test; it
230+
never skips it.
231+
- **Documentation.** Document the variables in the fixture's module docs, so
232+
anyone can run the suite against their own deployment by setting them.
233+
234+
Run the Keycloak suite with:
204235

205236
```shell
206237
nix run .#openshell-test-runner -- \

0 commit comments

Comments
 (0)