You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a2afbcd
Browse filesBrowse the repository at this point in the historyBrowse files
test(testkit): add the openshell-conformance test program (#5249)
* build(testkit): add dependencies for the test registry and program
Add linkme for the test registry, libtest-mimic and clap for the test
program's command line, quick-junit for its report, and libc for process
control on Unix. Refresh the lockfiles of the workspaces that depend on
testkit.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(testkit): register tests and add a test program with list
`#[testkit::test]` now also registers each test, with its family, name,
description and `#[ignore]` reason, in a link-time registry. The hidden
function gets only `cfg` and lint attributes; `#[should_panic]`,
`#[cfg_attr]` and `#[expect]` are rejected with an error that says why.
`testkit::main()` turns a binary into a test program. It takes libtest's
arguments, so it also works as a `harness = false` target under cargo
test and nextest, and it has a `list` subcommand that prints each test's
name and description. A program that links no tests exits 2, since Rust
does not link a suite crate that is never named.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(testkit): add a ready subcommand to the test program
`ready [--timeout SECS]` waits for the gateway the program will test and
exits 2 if it is not ready in time. It shares its check with the
`testkit-ready` binary, which now calls it instead of keeping its own
copy.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(testkit): add a run subcommand to the test program
`run [FILTER]... [--include-ignored] [--timeout SECS] [--grace SECS]`
runs the selected tests one at a time. Each test is a new process, the
program started as `<name> --exact --nocapture`, in its own process
group, so nothing a test starts outlives it. A test that runs past its
timeout (900s by default) gets SIGTERM, then SIGKILL after the grace
period (10s by default).
Interrupting `run` stops the current test and runs no more; a second
interrupt skips the grace period. `run` exits 1 when a test failed and 2
when tests could not run or the run was interrupted. It is Unix only; on
other platforms it says to use cargo test or nextest.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(testkit): write a JUnit report from the test program's run
`run --junit PATH` writes a JUnit report with one test suite per family
and one test case per test, including skipped, failed and timed-out
tests. Each test's output is shown as it comes and the last 4 MiB of each
stream is kept for the report. The supervisor waits for a test's output
to close before it moves on, so a signal that arrives while it waits
still ends the run.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* docs(testkit): document the test program and test registration
Describe how `#[testkit::test]` registers tests, the `list`, `ready` and
`run` subcommands, their exit codes, and the limits of `run`: Unix only,
and a test stopped on a timeout does not run its cleanup.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(conformance): move the conformance suites into a library
Cargo builds each file in tests/ as its own test executable, which no other
program can link. Move the conformance suites into src/ as modules of a
library, so a test program can link them. Test code is unchanged; cargo test
and nextest run the same 15 tests as the library's unit tests.
Nextest now names them as binary `conformance`, test `<suite>::<fn>`, so
filters that selected a suite by binary select it by test name instead.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(conformance): add the openshell-conformance program
The conformance crate now also builds openshell-conformance, a binary whose
main calls testkit::main() and links the crate's library. It lists, waits
for the gateway, and runs the conformance tests without cargo or nextest.
It runs only the conformance suite and promises only the conformance
contract; feature and driver suites are not part of it.
The binary has test = false, so cargo test and nextest still see one
conformance test binary, the library's unit tests, and the e2e lanes and
the runner's filters are unchanged. The library is linked without
cfg(test), so the program gets the registrations and none of the tokio
test wrappers. tasks/scripts/check-conformance-program.sh checks that
openshell-conformance list names the same tests as nextest; Branch Checks
and mise test:rust run it.
A dependency that is never named is not linked, so a suite the program
forgets to name would silently go missing. testkit::main() now fails with
exit 2 when a program links no tests, under every command.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(e2e): run the conformance host lanes with openshell-conformance
The Docker, Podman, VM and Kubernetes host lanes ran the conformance suite
with cargo nextest. They now run it with openshell-conformance: ready, then
run --junit, so CI exercises the program that gateway owners use.
The script uses OPENSHELL_CONFORMANCE_BIN when set and otherwise builds the
program from the checkout. The JUnit report goes to
OPENSHELL_CONFORMANCE_JUNIT or results/conformance-<gateway>.xml; the
Kubernetes runner names it after its report so credential-driver runs keep
both.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(e2e): build openshell-conformance once per architecture in Branch E2E
Each Docker, Podman, VM and Kubernetes host lane built the conformance
suite itself. Branch E2E now builds openshell-conformance once per
architecture as a static musl binary, checks that it is static and lists
its tests, and uploads it. setup-e2e-cli downloads the one that matches the
runner when a lane sets prebuilt-conformance, and the lane runs it through
OPENSHELL_CONFORMANCE_BIN.
The job is separate from build-binaries.yml, which the release workflows
also call: the program has its own workspace and is not released. Lanes
called without prebuilt-conformance, such as the release qualification
lanes, still build the program from the checkout.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(e2e): harden the conformance program helpers
Take the openshell-conformance path from cargo's build messages instead of
assuming <target>/debug, so CARGO_BUILD_TARGET and other profiles work. Make
the drift check fail clearly when it reads no test names from nextest.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Matthew Grossman <mgrossman@nvidia.com>
[{"suite":"ci","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:ci"}]
[{"suite":"additional-ca","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"OPENSHELL_E2E_PODMAN_TEST=additional_ca mise run --no-deps --skip-deps e2e:podman"}]
[{"suite":"external-driver","cmd":"mise run --no-deps --skip-deps e2e:docker:external-driver","apt_packages":"openssh-client","python_proto":false,"mcp":false}]
[{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver"}]
0 commit comments