Which automated test proves each step of the three production qualifications of v0.11.0 (spec tasks C1, C2, and C3) and each supply-chain item (D2), and which workflow runs it. The matrix is the traceability table behind the release checklist: it says what a green Release qualification run does and does not show. It claims nothing a workflow does not run. A step with no automated proof is a gap, listed with its reason at the end; it is not covered by a manual run unless a gap says so.
| Tier | Workflow | Runs | What it is |
|---|---|---|---|
| Fast gate | CI |
every pull request and main push, Ubuntu, macOS, Windows |
build, unit tests, Pi compatibility, live platform secret store |
| Portable E2E | Portable E2E |
nightly, manually, and in Release qualification, Ubuntu, macOS, Windows | the installed branded command against loopback fixtures (tests/e2e) |
| Reference E2E | Reference E2E |
nightly, manually, and in Release qualification, Ubuntu only | the AcmeCode reference distribution against a live Keycloak, credential broker, and LiteLLM (examples/enterprise-reference, tests/enterprise-reference) |
| Release candidate | Release candidate |
manually and in Release qualification, one runner per target | two builds, reproducibility, attestation, verify-release, tamper rejection, install with the shipped script, and for mypi and devcode the installed offline --smoke and doctor |
| Live provider | Live provider |
manually only | the reference stack routed to a real model provider; never part of qualification |
"Clean machine" in this matrix means what the tests build: an empty temporary home, state directory, install home, bin home, secret store, and (where used) shared file store, on a CI runner or a developer machine. It is not a freshly imaged operating system, and no automated run reaches one.
The flow is tests/e2e/managed-clean-machine.test.ts, "runs from install to uninstall as one flow on one machine" (Portable E2E, three targets, the demo company against loopback fixtures: an identity provider, a credential broker, and a gateway that is not a model), and, on Ubuntu, the same steps against the live stack in examples/enterprise-reference/tests/distribution-flow.test.ts (Reference E2E).
| Step | Portable E2E (fixtures, three targets) | Reference E2E (live stack, Ubuntu) | Release candidate (three targets) |
|---|---|---|---|
| install | the flow's install: archive verified, then the shipped script on a machine with nothing of the distribution; lifecycle-install.test.ts |
"is installed, and refuses to launch before sign-in" | install job: shipped install.sh or install.ps1, then acmecode version |
| launch | the flow's launch: --smoke and models refuse signed out before any service is contacted |
the same test | version only: a managed launch needs a sign-in |
| login | the flow's login: Authorization Code with PKCE, the test acting as the browser (no form, no second factor); security-controls.test.ts identity cases |
"signs Alice in on the Keycloak authorization page with PKCE and a loopback redirect" | none |
| credential acquire | the flow's credential acquire: the fixture broker turns the access token into a runtime credential kept in the platform secret store |
"exchanges the identity for a credential scoped to Alice's models..." | none |
| model request | the flow's authenticated model request: --smoke-model to the fixture gateway, which answers only the brokered credential |
"sends a model request through LiteLLM with the brokered credential" (mock upstream) and "renews a credential the gateway rejected" | none |
| session resume | the flow's session resume; lifecycle-rollback.test.ts keeps the session across a rollback |
the flow, as a whole | none |
| doctor | the flow's doctor: every group reports the state the flow produced |
"reports the identity, credential, gateway, and secret store in doctor" | none (needs a sign-in) |
| update | the flow's update through a signed loopback channel; lifecycle-update.test.ts, lifecycle-integrity.test.ts (tampered archive, forged metadata) |
"updates through a signed channel and keeps the session, the identity, and the credential" | none |
| rollback | the flow's rollback; lifecycle-rollback.test.ts; lifecycle-credentials.test.ts (a revoked secret is never restored) |
"rolls back keeping the session, and the credential the gateway still accepts" | none |
| logout | the flow's logout: revokes at the broker and the provider, clears the store, keeps sessions |
"signs out: revokes the credential and the identity tokens, and clears the secret store" | none |
| uninstall | the flow's uninstall and purge: the machine has nothing of the distribution again |
"uninstalls, keeping the user's sessions, and purges the state, leaving the machine as it was" | none |
What the flow does not prove: an interactive terminal session (every launch is Pi's headless acceptance run; the TUI needs a terminal); a second factor or a sign-in form (the Reference E2E signs in on Keycloak's own page, but not with a human or MFA); a real company identity provider or gateway (the production validation protocol, outside CI); a real model (the stack's upstream is a mock). Sandboxed commands run on Linux and macOS only; on Windows the flow asserts the sandbox is reported as not required and runs no command, because Windows has no sandbox adapter.
Two flows, one per personal example: personal-clean-machine-mypi.test.ts (MyPi delegates its credential to Pi) and personal-clean-machine-local.test.ts (MyPi Local stores its key in its own secret store), both from personal-clean-machine.ts (Portable E2E, three targets, a stand-in model server on loopback).
| Step | Proof | Also |
|---|---|---|
| install | the flows start from an empty home and state and install with the script shipped in the release | Release candidate install job, mypi, three targets; personal-lifecycle.test.ts |
| launch | --smoke starts Pi on the pinned version with the declared resources |
Release candidate: installed mypi --smoke, offline, three targets |
| credential configure or delegate | MyPi: Pi-native sign-in kept in the distribution's own state, and a personal key in the environment or ~/.pi never reaches the model server. MyPi Local: the key is stored in the secret store and appears in exactly one file |
personal-access-modes.test.ts, personal-local-model.test.ts |
| model request | a request to the stand-in server with the right credential, and only that one | no real provider in any CI tier (see gaps) |
| session resume | the next --smoke resumes the same session, including after update and rollback |
personal-lifecycle.test.ts (a damaged session is replaced, not read) |
| doctor | doctor after the update names the verified release and the retained rollback target |
Release candidate: installed mypi doctor, three targets |
| update | a signed loopback channel offers 1.1.0 | personal-lifecycle.test.ts also proves no request leaves the machine but the channel (a trap proxy sees none) |
| rollback | back to 1.0.0 with the same session | |
| uninstall | removes the install and the command, keeps sessions, settings, and the user's ~/.pi |
The developer distribution (examples/developer, devcode) is a qualification target, not a sample. Its end-to-end file is tests/e2e/developer-profile.test.ts (Portable E2E, three targets unless a row says otherwise).
| Step | Proof | Tier and platforms | State |
|---|---|---|---|
| package closure | developer-example.test.ts ("pins every package to an exact version that the lock resolved", "reviews exactly the install scripts the lock contains", "have a committed lock that is current..."); developer-profile.test.ts "vendors exactly the declared packages..."; pi-packages.test.ts (a local and a git package locked, vendored, installed, loaded); Release candidate builds devcode with its per-package vulnerability audit |
fast gate; Portable E2E; Release candidate, three targets | proven |
| dependency dedupe | pi-packages/dedupe.test.ts (Node's resolution unchanged; copies that differ, including in registry integrity, stay package-local; a shared directory name already taken is never mixed into; path length) and developer-profile.test.ts "the Pi package footprint" ("records what became of each closure, and shares identical dependencies"); check:budgets |
fast gate and Portable E2E; budgets on macOS only | proven. Merged in #231 |
| selective bundling | pi-packages/closure.test.ts (bundle-safe closures and every fallback reason) and compatibility-packages.test.ts (Pi's extension loader on shared dependencies and on a bundled closure) |
fast gate; Pi compatibility | proven. None of the six packages is bundle-safe, so no shipped distribution exercises a bundled closure: the developer test asserts each stays vendored. The release-level tests below bundle a fixture closure |
| footprint under release verification | release.test.ts "a release whose Pi packages share dependencies and bundle a closure": a release built by buildRelease with the real dedupePiPackages and bundleClosure output (fixture packages, no registry) verifies from its directory and its archive; a changed shared file, forwarding module, forwarding package.json, bundled module, an added file under .shared, and a removed shared file each fail verify-release and verifyPayload (the check doctor runs) with INTEGRITY_FAILED naming the path; its SBOM, license index, and notices equal those of the same payload built without sharing and without bundling; tests/developer-example.test.ts bounds the shared directory name over every locked package |
fast gate | proven on a fixture release; the real developer release is covered by the Release candidate rows below and by Portable E2E, not by these |
| shared store | tests/e2e/developer-profile.test.ts "the shared file store": the packages' files are placed from an empty store, the installed tree equals the built one byte for byte and equals an install with no store, and the command starts with the store removed; tests/e2e/shared-store.test.ts (install, update, rollback, doctor collection, and an install with an unusable store, on real release archives); packages/core/src/store*.test.ts, security-cases.test.ts |
fast gate (unit); Portable E2E, three targets | proven. The store is off by default (PISHIP_STORE), so no release path depends on it |
| project trust | developer-profile.test.ts "is loaded once policy admits the origin..." and "is loaded by neither PiShip nor pi-code when the origin is unknown and nobody can be asked"; developer-example.test.ts "project trust in the personal profile" and "the managed variant"; governance.test.ts |
Portable E2E, Linux and macOS (the project-trust probes skip on Windows) | proven on two of three targets |
| permissions | developer-profile.test.ts "the seeded permission rules, run by the provider" (allow, ask, deny per command and file), "keeps managed credential-store reads denied by the actual provider", "the launch option that approves every ask for one session"; "has the permission provider as an effective, locked, certified capability" |
Portable E2E; the rule-by-rule run is Linux and macOS only, the capability check runs on all three | rules proven on two targets; the capability on three |
| subagents and extensions | developer-profile.test.ts --smoke: "registers the tools the profile promises, with no name taken twice" (including subagent), "loads every package's extension and the distribution's own skills", "offers the commands of each package, and says where two collide", "never loads the extension that makes requests look like Claude Code's" |
Portable E2E, three targets | proven |
| offline install, smoke, doctor | Release candidate install job: the shipped script, then devcode --smoke and devcode doctor with no model request |
Release candidate, three targets | proven |
| Windows file-count pressure | the Windows installed file count is reported per target by the Release candidate install job (job summary) and measured by hand with the Windows benchmark workflow; the footprint budgets are recorded for darwin-arm64 only and checked on macOS |
Release candidate (reported, not gated) | gap: no Windows budget fails a build |
| update and rollback of the developer release | none | gap (below) |
Release candidate builds acmecode, mypi, and devcode for linux-x64, darwin-arm64, and win32-x64. Every row is a job with that matrix, guarded by tests/ci-guards.test.ts ("release candidate supply-chain evidence"), so removing one fails the unit tier.
| Item | Where | Proof |
|---|---|---|
| reproducibility | build (first and second build on separate runners, same SOURCE_DATE_EPOCH) and reproducibility |
piship reproducibility exits non-zero unless every payload file is equal; the report is uploaded per distribution and target |
| verify-release | verify on a fresh runner that never built the archive |
piship verify-release --sha256: archive, checksums, every payload file against the inventory, manifest and lock, SBOM completeness, notices coverage, recorded scan and test results |
| tamper rejection | verify |
a flipped archive byte, a wrong digest, and a modified payload file, release.json, SBOM, and checksums.txt each fail with INTEGRITY_FAILED at the layer that can prove it, and the exact bytes are restored |
| SBOM | built and self-checked in piship release (verifySbom); checked again by verify-release and by scripts/check-release-evidence.mjs in verify |
the SBOM lists the logical dependency graph, from the lock, the bundle record, and each vendored Pi package's lockfile, not the files on disk (security); supply-chain.test.ts "the logical dependency graph under a smaller file layout" |
| notices | same | every SBOM package has a notices entry and its license text; same test |
| vulnerability scan | piship release (npm audit over the runtime lock and over each vendored Pi package's lockfile) |
a finding at or above failOn fails the build; the verdict and per-package records travel in release.json and vulnerabilities.json; check-release-evidence.mjs requires passed and one audit record per vendored Pi package |
| registry signature | piship release (npm audit signatures over the runtime closure) |
an invalid signature or attestation fails the build. Until this pull request the check reported "found no dependencies to audit" on every release candidate and verified nothing; it now runs (security, finding F1). check-release-evidence.mjs fails on a verdict that audited no package and prints an unavailable verdict (the registry or the Sigstore trust root unreachable) as a warning. The vendored Pi packages are not signature-checked (a documented limit) |
| provenance and attestation | attest (actions/attest-build-provenance on the first build of each target) and verify (gh attestation verify --signer-workflow --source-ref, and that it rejects a tampered archive) |
Sigstore keyless provenance bound to the workflow and the ref; verified on a runner that did not build it |
| install | install job |
the shipped script installs the archive; mypi and devcode then run offline --smoke and doctor |
Each is a step with no automated proof, or less than its row says, with the reason. None is claimed as covered by a manual run.
- Developer update and rollback. No test installs
devcode1.0.0, updates to 1.1.0 through a signed channel, and rolls back. It needs two relocked developer releases (twopiship lockruns against the registry and twopiship releasebuilds with the per-package audits) on every target, which is the heaviest fixture in the suite and could not be validated on Windows here. The update path is distribution-agnostic and is proven onmypiandacmecode; what the developer payload adds is the shared dependency directory and a larger tree. Follow-up in v0.11-followups.md. - Windows file-count budget.
scripts/performance-budgets.jsonrecordsdarwin-arm64only, and a platform without a record passes. The Windows figures for the developer release are not measured in CI; the Release candidateinstalljob now prints them for every target so a first qualification run can recordwin32-x64andlinux-x64budgets. Until then a Windows file-count regression fails nothing. - Project-trust and permission rules on Windows. The provider probes use POSIX paths and skip on Windows, so project trust and the rule-by-rule permission cases are proven on Linux and macOS only. The Windows run proves the capability is effective and locked.
- Personal and developer model requests against a real provider. MyPi and MyPi Local talk to a stand-in server. The manual
Live providerworkflow exercises the reference stack, not the personal or developer examples. - Interactive terminal. Every launch in every tier is Pi's headless acceptance run (
--smoke,--smoke-model); no test drives the TUI. - Managed on Windows: sandbox. No adapter exists, so the sandbox step is not run; the refusal of a required sandbox is asserted in
governance.test.ts. - A real identity provider, gateway, MDM, EDR, or proxy of a company. The Reference E2E is the nearest automated evidence; the production validation protocol is the rest, outside CI.
- A real clean machine, and an air-gapped one. The flows use isolated temporary homes on a runner that has network access. They prove that install and launch contact nothing but the declared hosts (
personal-lifecycle.test.tsandshared-store.test.tsuse a trap proxy and the update host's request log); they are not a packet-level offline proof. - A deduped payload through update and rollback, and a long install home on Windows. The deduped developer release is installed,
--smoked, anddoctored by the Release candidateinstalljob and verified from its archive byrelease.test.ts; no test updates or rolls it back, and none installs it under a long path on Windows. The path-length budget that is tested is the one the repository decides (the shared directory name and a shared file's path against its package-local path); see v0.11-followups.md. - Registry signatures of vendored Pi packages. Only the runtime closure is checked; see the artifact contract's known limitations.