[Bugs5382]

Rebased onto main (2026-10-01 ET) at 148b4a3, after #318, #321, #322, #324, #325, #326 and #328 landed. Head 5bf4df4. The diff is this PR's own work. Merge any time; it has no open dependency.

Red suite job fixed again (the suite trusted the pre-#318 management certificate). Run 36790605504 failed the first step: after the Root's ceremony, cryptosctl got x509: certificate is not valid for any names, but wanted to match localhost. Since #318 a node with a CA presents a management leaf signed by that CA for the network.address IP and the pki.est.hostnames entries (no localhost), followed by its CA chain. The suite pinned the self-signed leaf and checked the name localhost. The last commit follows what #318 did in test/integration: cryptosctl checks the node by its address (--server-name 10.0.0.10), the suite trusts the last certificate the listener presents (the self-signed leaf before a CA, the root after), it switches to the root once the ceremony, the subordinate submit or the escrow import commits a CA, and it tells boots apart by the leaf rather than the trust file. The node's certificate is unchanged.

Local QEMU for this fix, only as far as needed to prove it: a TCG run of the Root, the Intermediate, both protocol switch steps and both escrow steps (the import is the third CA commit) on the rebased tree: 6 passed, 0 failed, 0 skipped. The full image check is this PR's "Boot the image and run the suite" job.

Upgrade step on the TPM Intermediate fixed (boot detection in the suite, not the node). Run 36814870750 passed every step but "image upgrade in place on the TPM Intermediate", which read the image status 0 s after image activate and found the old boot. The suite took a server certificate other than the pinned one as proof of a new boot, but a CA-signed management certificate is re-minted without a reboot when the CA is re-certified, and the Intermediate is re-certified earlier in the run (the Root never is, so its upgrade passed). After a reboot or an activation the suite now waits for the status to report a higher boot count. Local TCG runs limited to the Root, the Intermediate, re-certify and both upgrade steps: on 9e45bc6 the TPM upgrade step fails the same way (old boot answering 2 s after the request); with the fix all 5 pass, the Intermediate back on the successor 30 s after the request and still issuing. A second run of the two protocol switch steps, which share the reboot helper: 4 passed.

Earlier red suite job fixed (stale test expectation, not a node regression). Run 36748739042 failed "protocol switch: ACME and EST on, reboot pending, reboot, running". The switch itself worked: after the reboot the status read Protocols: ACME on, EST on, SCEP off with no pending reboot. Since #288 the Protocols line lists SCEP, and the step matched ACME on, EST on up to the newline. The last commit expects the SCEP entry in both protocol switch steps. That run's merge base with main was e5a15f6, before #312 and #317, so neither is involved. The failure reproduced locally on the tree then rebased onto 03f2859, and with the fix both protocol switch steps passed there.

Local QEMU was used only for root cause: a TCG run (no /dev/kvm on this box) limited to the Root, the Intermediate and the two protocol switch steps. Before the fix: the "on" step fails with the same message as CI. After: 4 passed, 0 failed, 0 skipped. The full image check is this PR's "Boot the image and run the suite" job.

Pins: this PR does not touch go.mod; github.com/CryptOS-PKI/api stays at whatever main pins.

Headers: the two new Go files (cmd/init/coverflush_e2ecover_linux.go, test/integration/image_suite_test.go) carry "Copyright The CryptOS Authors." (golic v1.2.0 replace), and task license exits 0.

Root upgrade step re-enabled. #305 made later boots read the state-key mode from the LUKS2 header, so the software-key Root (state_key.mode: nodeid on a tpm-default image) now comes back from a reboot, and the step "image upgrade in place on the software-key Root, then it still signs" runs again.

Local gate (Linux box): task ci passes on head 5bf4df4.

What and why

The kind test from #293 proves cert-manager can enrol over ACME, but its node runs in-process. Nothing in CI booted the image operators install. This adds a second, slower test that does. It boots the real CryptOS image in QEMU (swtpm, OVMF) on a GitHub-hosted runner, as a software-key Root and a TPM-key Intermediate. A suite of named steps then drives both over the mTLS API with cryptosctl. Each step reports pass, fail or skip (with the reason) in the job summary.

• The steps (test/integration/image_suite_test.go, TestImageSuite):
  • Root: the Root's ceremony.
  • Intermediate: ca get-subordinate-csr, ca sign-subordinate on the Root, ca submit-subordinate-cert, and the chain verifies.
  • Leaves: ca issue-leaf from CSRs. ECDSA P-384 and RSA 3072 issue; RSA 2048 is refused.
  • nginx (pinned by digest) serves a leaf with OCSP stapling. curl --cacert, openssl s_client -verify_return_error and curl --cert-status all pass.
  • Revocation: ca revoke, then OCSP says revoked (POST, GET and openssl ocsp) and the CRL lists the serial. nginx stops stapling good after a reload; it logs the revoked status and staples nothing, and curl --cert-status refuses.
  • Protocol switch: config apply with pki.acme and pki.est shows ACME on (not running, reboot pending) and Reboot: pending. After a reboot, both are running and nothing is pending. Switching ACME off later reads still running, reboot pending until the reboot, then off.
  • ACME: cert-manager in kind (the pins from test(e2e): prove cert-manager gets a CryptOS certificate over ACME in a kind cluster #293) gets a Certificate from the booted Intermediate over http-01. The chain verifies to the Root, and ca list-issued shows the serial. A forced renewal gives a new serial, also listed.
  • EST: simpleenroll (HTTP Basic, with a wrong password refused with 401) and simplereenroll (TLS client certificate), both with curl, with openssl reading the PKCS#7.
  • Re-certify: the Intermediate is re-certified on the same key, and a leaf issued afterwards chains through the new certificate.
  • Escrow: export is refused on the TPM node, and a 17-byte passphrase is refused. The Root exports, and a fresh node imports the backup (a wrong passphrase is refused first) and holds the Root certificate.
  • Image upgrade on the TPM Intermediate: image stage of a signed successor, image activate, back on the new version with the same identity, and still issuing.
  • Console reset on the throwaway restored node: Ctrl-R, the CN and Enter over the serial line, then back in re-provision maintenance.
• Skipped, with the reason in the summary:
  • Audit chain: no RPC or CLI verifies it.
  • SNTP: the SNTP client is on main (feat(init): keep the node clock in sync with a client-only SNTP #271), but the suite runs no NTP server for the node to sync against yet.
  • SCEP: the SCEP server is on main (feat(scep): serve SCEP so network devices can enrol a trustpoint #288), but the suite runs no SCEP client against it yet.
• Networking: each VM gets its own QEMU user-mode network (node 10.0.0.10, host 10.0.0.1, QEMU's DNS proxy 10.0.0.3) with host-forwards for the ports the steps dial. This needs no root and no bridge on the runner, and each VM is isolated from the others. The node's http-01 fetch resolves the test name through QEMU's DNS proxy and the runner's resolver. run.sh maps the name to 10.0.0.1 in /etc/hosts, the runner's loopback as the VM sees it, and kind maps the ingress there. The cluster reaches the ACME origin through a TLS front on the kind bridge, whose certificate the booted Intermediate issued. There is no validator bypass.
• KVM: hosted ubuntu-latest runners have /dev/kvm. The workflow opens it with the udev rule GitHub documents, and the job summary says whether KVM was there. TCG is the fallback, with longer boot timeouts.
• Coverage: test/image/build.sh builds a variant used only by this test. It has the same kernel, static tools and rootfs; /init is built with -cover -covermode=atomic and the e2ecover tag, and the kernel command line is qemu-dev, so the boot log and console are on serial. The image carries a throwaway upgrade anchor, discarded with the build, plus a signed successor image. PID 1 never exits, so cmd/init/coverflush_e2ecover_linux.go, compiled only with that tag, rewrites the counters every 2 s to a small FAT disk the harness reads back after the VM stops. cryptosctl on the runner is instrumented too. test/image/coverage.sh merges the counters into a per-package table in the summary and an HTML report in the artifacts. No release or CI image is built with -cover or the tag.
• Workflow (ci-e2e-image.yml):
  • Triggers: nightly, pull requests that touch the image build, the boot, or the ACME, EST, config or revocation code, and manual dispatch. Drafts are skipped, and it is not a required check. The in-process kind test stays the pull-request check.
  • Jobs: an image job and a suite job, 40 minutes each.
  • Image reuse: the image is cached by the hash of everything that goes into it, so an unchanged tree reuses the last build. The kernel shares ci-image.yml's cache.
  • Pins: kind, kubectl and the manifests keep their checksums from test/kind/versions.env; nginx is pinned by digest in test/image/versions.env. QEMU, swtpm and OVMF come from the runner's distro packages, like ci-image.yml's toolchain.
• Running it locally: task e2e:image on Linux. It skips with a message on a non-Linux host or without docker, qemu, swtpm or OVMF.

Found by booting the image:

• Fixed here: OCSP over GET answered 404 (fix(revocation) commit). RFC 6960 appendix A.1 lets a client append the URL-encoded base64 request to the responder URL, and nginx staples that way. The mux only routed the exact path /ocsp. It now routes /ocsp/ and decodes from the escaped path, so a percent-encoded / survives. Test: TestOCSPEndpointGETServesResponse, which failed with 404 before the fix.
• Fixed in fix(init): read the state-key mode from the volume after the first boot #305, found here: a node whose state_key.mode came from the machine config (nodeid on a tpm-default image) could not come back from any reboot, because later boots used the image's build-time mode. fix(init): read the state-key mode from the volume after the first boot #305 reads the mode from the LUKS2 header instead, and the Root upgrade step, skipped until then, runs and passes again.

Design calls:

• Image reuse: the image job reuses its own cached build rather than an artifact from ci-image.yml. That workflow uploads nothing on main (its per-run key is discarded), and its release assets carry neither counters nor a serial console. A cache keyed on the build inputs gives the same reuse without either problem.
• No serial log for boot detection: readiness and reboots are detected from the management certificate, which is new on every boot, and the boot count, not from serial log lines. init logs through /dev/kmsg, which the kernel rate-limits, and lines were lost in an early run.

Closes #295

Related: #293, #284, #286, #305. Docs: CryptOS-PKI/docs#31 (docs/25-use-cases-pages, 255d7af) extends the Kubernetes page's "Tested in CI" note to cover this suite; it depends on this PR. The README gains the layout lines, task e2e:image and the workflow, and docs/image-upgrade.md says what the suite now proves about the upgrade path on both a TPM node and a nodeid node.

Verification

• Lint clean (golangci-lint v2.12.2, also with --build-tags integration,e2ecover for Linux; go vet; gofmt; actionlint; shellcheck)
• Tests pass (go test ./...)
• Build succeeds
• Documentation updated

How this was verified

• go test ./... locally (CGO on, for the TPM simulator tests), and the new OCSP GET test failed before the fix.
• The suite was brought up on hosted runners from this branch. Runs:
  • 36702744115: KVM boots in 16 s. It found the /dev/kmsg rate limit and the set-mode coverage error.
  • 36704068511: 13 pass, 0 fail, 4 skip, 46.0% coverage, about 6 minutes for the run.
  • 36704870517: added the upgrade steps and found the state_key.mode reboot bug.
• An earlier run was this PR's ci-e2e-image check, 36706496825 on da2a14a: 14 pass, 0 fail, 4 skip.
  • Durations: the image job took 3m12s and the suite job 4m53s, about 8 minutes in all.
  • Boots under KVM: 16-17 s for each first boot, 12 s for each reboot and for the upgrade.
  • Coverage: 50.7% of the statements in the packages the binaries link. The lowest are internal/install and internal/kms (0%; no step installs from the ISO or runs a KMS-mode node), internal/cms (6%; SCEP is not on main), internal/console (7%; the console runs as its own process, which is not instrumented yet), internal/ca (32%) and internal/grpc (45%).
• task e2e:image locally under TCG, after the fix(init): read the state-key mode from the volume after the first boot #305 rebase: 15 passed, 0 failed, 3 skipped (audit, SNTP, SCEP), both upgrade steps included.
• The branch was first rebased on main after docs: OS-specific commands and risk callouts #277, fix(attest): bind Attest signatures to a versioned attestation context #279 and feat(tpm): hold an RSA CA key in the TPM #282 landed. The docs/image-upgrade.md conflict with docs: OS-specific commands and risk callouts #277 was resolved by keeping its callout and updating the text inside it. The earlier .gitleaksignore commit is dropped: feat(config): switch ACME and EST on and off through machine config #286's own entry (at its rebased commit 205dd64) covers the test fixture, and gitleaks git over the branch finds no leaks.

---

Before merging: add a closing comment summarizing what was actually done in this PR
(not just the checked boxes).

[Bugs5382]

Merging for the v0.1.0 test run: CI green and mergeable at merge time.