Skip to content

Commit 4e6a9c4

Browse files
ci(tests): run conformance-integration through the runner
Switch the required conformance-integration job in Branch E2E, Release Dev and Release Qualification from the tmachine conformance testsuite to the E2E Suites workflow, selecting the conformance family. The job ID is unchanged. Delete the tmachine conformance and policy-advisor testsuites, their playbooks and the old conformance test archive: the sandbox package covers the same tests, including the ported policy-advisor ones. The Branch E2E E2E Suites job now runs the features family only, so conformance does not run twice. Point the integration runner defaults and the docs at what remains. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
1 parent 4905404 commit 4e6a9c4

13 files changed

Lines changed: 48 additions & 262 deletions

File tree

‎.agents/skills/watch-github-actions/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId
125125

126126
## View Job Logs
127127

128-
For tmachine Kubernetes conformance failures, inspect the `tmachine-diagnostics-*` artifact. The K3s report includes Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The conformance playbook checks reconciliation after guest boot with a temporary scheduling-gated Sandbox probe before running tests. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
128+
For tmachine Kubernetes conformance failures, inspect the `e2e-results-conformance-sandbox-ubuntu-k3s-*` artifact. Its `diagnostics/` holds the K3s report with Agent Sandbox controller state, leader leases, events, and current and previous controller logs. The k3s installers check reconciliation after guest boot with a temporary scheduling-gated Sandbox probe, so a failed probe fails provisioning before any test runs. Distinguish a failed controller probe from a failed conformance scenario; gateway connectivity and controller Deployment availability do not prove reconciliation works.
129129

130130
For the `e2e-suites` lane (`.github/workflows/e2e-suites.yml`), the last lines of the job log give the runner's result: PASS, FAIL (exit 1) after the end of the nextest log (which names the failed tests), or ERROR (exit 2) with the step that broke. FAIL and ERROR also appear as annotations on the run's summary page. Then inspect the `e2e-results-<family>-<package>-<environment>-<installer>` artifact, which holds `junit.xml`, `nextest.log` and, after a failure, the guest's `diagnostics/`.
131131

‎.github/workflows/branch-e2e.yml‎

Lines changed: 4 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -247,19 +247,13 @@ jobs:
247247
actions: read
248248
contents: read
249249
packages: read
250-
uses: ./.github/workflows/integration-runner.yml
250+
uses: ./.github/workflows/e2e-suites.yml
251251
with:
252-
category: conformance
252+
family: conformance
253253
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
254254
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
255-
test-matrix: >-
256-
[
257-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
258-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
259-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
260-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
261-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
262-
]
255+
secrets:
256+
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
263257

264258
# Run feature-specific integration tests:
265259
feature-specific-integration:
@@ -303,22 +297,6 @@ jobs:
303297
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"}
304298
]
305299
306-
# Run the testkit suites in tests/suites/ on tmachine. The environments and
307-
# suites are listed in e2e-suites.yml.
308-
e2e-suites:
309-
needs: [pr_metadata, prepare-integration]
310-
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
311-
permissions:
312-
actions: read
313-
contents: read
314-
packages: read
315-
uses: ./.github/workflows/e2e-suites.yml
316-
with:
317-
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
318-
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
319-
secrets:
320-
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
321-
322300
docker-e2e:
323301
needs: [pr_metadata, build-binaries, build-images]
324302
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
@@ -540,7 +518,6 @@ jobs:
540518
needs:
541519
- pr_metadata
542520
- conformance-integration
543-
- e2e-suites
544521
- feature-specific-integration
545522
- driver-specific-integration
546523
- docker-e2e

‎.github/workflows/integration-runner.yml‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,8 @@ on:
2424
type: string
2525
default: >-
2626
[
27-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
28-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
29-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
30-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
31-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
27+
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
28+
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
3229
]
3330
3431
permissions:

‎.github/workflows/integration-test.yml‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,15 @@ on:
2020
description: Integration-test category, used for the job name and concurrency key
2121
required: true
2222
type: string
23-
default: conformance
23+
default: feature-specific
2424
test-matrix:
2525
description: JSON array of environment, installer, and testsuite combinations
2626
required: true
2727
type: string
2828
default: >-
2929
[
30-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
31-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
32-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
33-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
34-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
30+
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
31+
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"}
3532
]
3633
3734
permissions:

‎.github/workflows/release-dev.yml‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -118,19 +118,13 @@ jobs:
118118
actions: read
119119
contents: read
120120
packages: read
121-
uses: ./.github/workflows/integration-runner.yml
121+
uses: ./.github/workflows/e2e-suites.yml
122122
with:
123-
category: conformance
123+
family: conformance
124124
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
125125
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
126-
test-matrix: >-
127-
[
128-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
129-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
130-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
131-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
132-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
133-
]
126+
secrets:
127+
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
134128

135129
feature-specific-integration:
136130
needs: prepare-integration

‎.github/workflows/release-qualification.yml‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -75,19 +75,13 @@ jobs:
7575
actions: read
7676
contents: read
7777
packages: read
78-
uses: ./.github/workflows/integration-runner.yml
78+
uses: ./.github/workflows/e2e-suites.yml
7979
with:
80-
category: conformance
80+
family: conformance
8181
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
8282
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
83-
test-matrix: >-
84-
[
85-
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"conformance"},
86-
{"environment":"ubuntu-k3s","installer":"k3s","testsuite":"conformance"},
87-
{"environment":"ubuntu-k3s","installer":"k3s-ha","testsuite":"conformance"},
88-
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"conformance"},
89-
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"conformance"}
90-
]
83+
secrets:
84+
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
9185

9286
feature-specific-integration:
9387
needs: prepare-integration

‎CI.md‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,7 @@ push-triggered workflow jobs themselves.
213213

214214
### K3s conformance version baseline
215215

216-
The tmachine `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the
216+
The `ubuntu-k3s` conformance lanes pin Agent Sandbox v0.5.0 as the
217217
compatibility baseline for the v1beta1 Sandbox API. They do not track the local
218218
K3s development default, currently v1.0.3. OpenShell also supports v0.4.6 through
219219
its v1alpha1 fallback, so v0.5.0 is not the overall minimum supported version.
@@ -223,26 +223,26 @@ recreating the Service requires reprovisioning the fixture. The `k3s` lane runs
223223
one plaintext gateway. The `k3s-ha` lane runs the supported HA configuration:
224224
three gateway replicas sharing PostgreSQL, serving TLS, with the CLI connecting
225225
over mTLS. Before conformance, every installer waits up to five minutes for a
226-
connected gateway; interactive shells skip the wait. K3s failures upload
227-
diagnostics as `tmachine-diagnostics-*` artifacts.
226+
connected gateway; interactive shells skip the wait. Each lane uploads its
227+
JUnit report, nextest log and, after a failure, the guest's diagnostics as an
228+
`e2e-results-*` artifact.
228229

229-
### Run only the policy advisor conformance tests
230+
### Run a subset of the conformance tests
230231

231-
Manually dispatch `Integration Tests` on the candidate branch with an
232-
`artifact-run-id` from a build of the same commit. When a `Release` or `Pre-release` run was
233-
dispatched from a different commit, also set `source-sha` to the release tag's
234-
resolved commit so runtime images and test inputs match the candidate binaries.
235-
Otherwise, `source-sha` defaults to the artifact run's head SHA. Set `category`
236-
to `policy-advisor` and `test-matrix` to:
232+
The conformance lanes run the `sandbox` package of `tests/suites/conformance`
233+
through `.github/workflows/e2e-suites.yml`, which keeps the list of
234+
environments. To run one suite in one environment from a checkout, build the
235+
candidate inputs (see [TESTING.md](TESTING.md)) and narrow the tests with a
236+
nextest filterset:
237237

238-
```json
239-
[{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]
238+
```shell
239+
nix run .#openshell-test-runner -- \
240+
--environment ubuntu-docker-rootful --installer deb \
241+
--family conformance --package sandbox \
242+
--filter 'binary_id(=sandbox::policy_advisor)'
240243
```
241244

242-
This runs the `mechanistic-proposal`, `new-hostname-proposal`, and
243-
`policy-local` conformance tests in the installed-artifact suite. The artifact run must contain the candidate CLI and
244-
gateway binaries and runtime images. This manual run does not replace the
245-
required PR E2E gate.
245+
A local run does not replace the required PR E2E gate.
246246

247247
## Informational security reports
248248

‎TESTING.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -358,9 +358,8 @@ Suites:
358358
- CLI conformance (`tests/suites/conformance`) - portable Cargo tests for
359359
lifecycle, mechanistic drafts, file transfer, and the sandbox-local API,
360360
including agent-authored permission requests. Driver E2E runs the complete
361-
Cargo test package. The installed-artifact conformance suite runs the same
362-
tests from a nextest archive and offers a focused `policy-advisor` testsuite
363-
for manual integration runs.
361+
Cargo test package. The installed-artifact conformance lanes run the `sandbox`
362+
package through `openshell-test-runner` (see below).
364363
- Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
365364
`e2e-vm`) - CLI conformance plus the common and driver-specific coverage for
366365
the selected deployment.
@@ -470,7 +469,9 @@ rebuilds only the RPMs. Run conformance against the installed RPMs:
470469

471470
```shell
472471
nix run .#build-artifacts
473-
nix run .#tmachine -- test fedora-podman-rootful rpm conformance
472+
nix run .#openshell-test-runner -- \
473+
--environment fedora-podman-rootful --installer rpm \
474+
--family conformance --package sandbox
474475
```
475476

476477
Run the portable subset in a disposable rootless Podman guest:

‎e2e/policy-advisor/README.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -55,19 +55,21 @@ contents write on the repository. The test auto-resolves the token from
5555

5656
## Conformance coverage
5757

58-
The portable CLI conformance suite may include policy-related scenarios. The
58+
The portable conformance suite may include policy-related scenarios. The
5959
current coverage checks draft generation for denied IP addresses and hostnames
6060
absent from policy, plus sandbox-local policy inspection and permission
6161
requests through `policy.local`. See [TESTING.md](../../TESTING.md#rust-cli-e2e)
6262
for the supported source and installed-artifact test paths.
6363

64-
A manual `Integration Tests` workflow run can select the `policy-advisor`
65-
testsuite to run the policy-related subset against an installed candidate. Set
66-
`artifact-run-id` to the candidate build's workflow run ID and `test-matrix`
67-
to:
64+
To run only the policy-related conformance tests against an installed
65+
candidate, run the `policy-advisor` suite of the `sandbox` package through the
66+
test runner:
6867

69-
```json
70-
[{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]
68+
```bash
69+
nix run .#openshell-test-runner -- \
70+
--environment ubuntu-docker-rootful --installer deb \
71+
--family conformance --package sandbox \
72+
--filter 'binary_id(=sandbox::policy_advisor)'
7173
```
7274

7375
The GitHub write test above and the regressions below still exercise distinct

‎tests/tmachine/ansible/playbooks/conformance/cli.yaml‎

Lines changed: 0 additions & 144 deletions
This file was deleted.

0 commit comments

Comments
 (0)