Skip to content

Commit b1cfc91

Browse files
authored
ci(release): promote qualified prereleases to stable (#4611)
* ci(release): promote qualified prereleases to stable Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): preserve qualified images during stable publication Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): run shared release jobs with Nix Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): recheck latest before publishing shared channels Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com>
1 parent 55bf786 commit b1cfc91

17 files changed

Lines changed: 630 additions & 138 deletions

File tree

‎.agents/skills/release-openshell/SKILL.md‎

Lines changed: 39 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,46 @@ metadata:
77

88
# Release OpenShell
99

10-
For the intended release version, pick the latest prerelease whose Release
11-
Qualification job passed. Confirm that its qualification summary matches the
12-
tag and commit. Present the candidate and changes since the previous stable
13-
release to the maintainer; once approved, tag that exact commit as `vX.Y.Z`.
10+
For the intended release version and branch, inspect the latest passing
11+
prerelease and changes since the previous stable release. Present the candidate
12+
to the maintainer. Once approved, dispatch Release from that branch:
13+
14+
```shell
15+
gh workflow run release.yml --ref main -f tag=vX.Y.Z
16+
```
17+
18+
Use the support branch instead of `main` for a previous minor version. Release
19+
selects the highest-numbered `vX.Y.Z-pre.N` reachable from that branch with a
20+
successful Pre-release run triggered by that tag or dispatched on the same branch,
21+
and matching passing `release-tag-v1` qualification evidence. Missing or expired evidence blocks
22+
promotion. Inspect the selected candidate in the promotion job summary.
23+
The entire Pre-release run must succeed, including publication. If qualification
24+
passed but publication failed, rerun the failed jobs before promoting the candidate.
25+
26+
Release creates `vX.Y.Z` at that exact commit, then rebuilds and publishes all
27+
stable artifacts without rerunning qualification. Embedded versions change, so
28+
stable artifacts are not byte-identical to the qualified prerelease. An existing
29+
stable tag may be retried only if it already points to the selected commit.
30+
Stable images are saved as OCI archives during the build and pushed under the
31+
stable version tags during publication. Their SBOMs are verified before updating
32+
`latest`; the qualified prerelease's commit-SHA image tags remain unchanged.
1433

1534
If no prerelease exists, agree on the version and source commit, then create and
16-
push `vX.Y.Z-pre.1`. Wait for qualification to pass before promoting it.
35+
push `vX.Y.Z-pre.1`. Prerelease tag pushes start Pre-release. If the tag was
36+
created using `GITHUB_TOKEN`, dispatch Pre-release explicitly on the intended branch:
37+
38+
```shell
39+
gh workflow run release-prerelease.yml --ref main -f tag=vX.Y.Z-pre.1
40+
```
41+
42+
Wait for the entire Pre-release run to pass before promoting it. Stable tag pushes do not
43+
start Release. Release Auto-Tag creates and dispatches prereleases on `main`.
44+
45+
Stable releases serialize across branches. A release older than the highest
46+
stable Git tag publishes versioned artifacts without moving GHCR `latest`, the
47+
GitHub latest release, npm `latest`, or snap `latest/stable`. Its npm package
48+
uses `release-X.Y`; snap files remain available in its GitHub Release rather than
49+
updating a Snap Store support track. Docs retain an
50+
immutable version and advance latest only for a newer version.
1751

18-
Prerelease tag pushes start Pre-release; stable tag pushes start Release.
19-
Release Auto-Tag uses an explicit dispatch because its `GITHUB_TOKEN` pushes do
20-
not trigger workflows.
2152
Watch the stable release run and report its result.

‎.agents/skills/sbom/SKILL.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ SBOMs are **release artifacts only** -- they are generated on demand and not com
1717

1818
Pushed gateway, sandbox, and supervisor images carry an SPDX SBOM and minimal SLSA provenance as OCI attestations. Branch E2E, Release Dev, Release, and Pre-release image binaries embed cargo-auditable metadata, so their image SBOMs include linked Rust crates.
1919

20+
Stable Release builds retain images and attestations in OCI archive artifacts.
21+
Publication copies every platform and attestation to GHCR with preserved digests,
22+
then runs the SBOM verification before updating `latest`.
23+
2024
## Prerequisites
2125

2226
- `mise install` has been run (installs Syft and other tools)

‎.agents/skills/watch-github-actions/SKILL.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -125,8 +125,9 @@ gh run list --json databaseId,status,headBranch,url --jq '.[] | {id: .databaseId
125125

126126
## View Job Logs
127127

128-
`Release Tag` temporarily omits Codex Security from its `Security Scan` job.
129-
Other scanners and qualification gates still run. Use the standalone
128+
`Pre-release` temporarily omits Codex Security from its `Security Scan` job.
129+
`Release` uses the passing prerelease qualification result. Other scanners and
130+
qualification gates still run. Use the standalone
130131
`Codex Security Release Qualification` workflow for manual investigation.
131132

132133
For `Codex Security`, an exit code of 2 with partial coverage is an incomplete
@@ -167,10 +168,13 @@ During `0.x`, a minor train permits compatibility findings
167168
as warnings; a patch train or no active train rejects them. Compare the current
168169
train's version with the latest stable release; commit messages are irrelevant.
169170
Compilation, baseline, and tool errors remain fatal. The `protobuf_compatibility` suite participates in
170-
the `release-tag-v1` qualification profile. Release and Pre-release call the reusable
171-
`release-qualification.yml` workflow to run the checks and collect their results.
172-
Release builds, qualifies, then publishes. Pre-release builds, then publishes
173-
and qualifies in parallel in the same run. Failed, cancelled, or skipped required suites fail
171+
the `release-tag-v1` qualification profile. Pre-release calls the reusable
172+
`release-qualification.yml` workflow after building, in parallel with publication. Release
173+
selects the latest passing prerelease for the requested version and branch,
174+
verifies its qualification evidence, creates the stable tag at its exact commit,
175+
then rebuilds and publishes without rerunning qualification. The promotion job
176+
summary and `stable-promotion-*` artifact record the candidate and evidence.
177+
Failed, cancelled, or skipped required suites fail
174178
qualification; stable publication is blocked, while published pre-release
175179
artifacts remain available. The Pre-release failure notification includes the
176180
run link. Both callers share `release-build.yml` and `release-publish.yml`.

‎.github/actions/build-docker-image/action.yml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# SPDX-License-Identifier: Apache-2.0
33

44
name: Build Docker Image
5-
description: Build, push, and export a multi-platform OpenShell image
5+
description: Build and export a multi-platform OpenShell image, optionally pushing it
66

77
inputs:
88
component:
@@ -17,6 +17,9 @@ inputs:
1717
image-tag:
1818
description: Docker image tag
1919
required: true
20+
push:
21+
description: Push the image to GHCR during the build
22+
default: "true"
2023
github-token:
2124
description: Token used to push the image
2225
required: true
@@ -29,6 +32,7 @@ runs:
2932
buildkitd-config: /etc/buildkit/buildkitd.toml
3033

3134
- name: Log in to GHCR
35+
if: inputs.push == 'true'
3236
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
3337
with:
3438
registry: ghcr.io
@@ -60,8 +64,13 @@ runs:
6064
env:
6165
COMPONENT: ${{ inputs.component }}
6266
IMAGE_TAG: ${{ inputs.image-tag }}
67+
PUSH_IMAGE: ${{ inputs.push }}
6368
run: |
6469
install -d artifacts/images
70+
push_args=()
71+
if [[ "${PUSH_IMAGE}" == "true" ]]; then
72+
push_args+=(--output "type=image,push=true,oci-mediatypes=true,oci-artifact=true")
73+
fi
6574
docker buildx build \
6675
--builder openshell \
6776
--platform linux/amd64,linux/arm64 \
@@ -72,11 +81,12 @@ runs:
7281
--cache-to type=gha,mode=max,scope=${COMPONENT} \
7382
--provenance=mode=min \
7483
--attest type=sbom \
75-
--output type=image,push=true,oci-mediatypes=true,oci-artifact=true \
76-
--output type=oci,dest=artifacts/images/${COMPONENT}.tar \
84+
"${push_args[@]}" \
85+
--output "type=oci,dest=artifacts/images/${COMPONENT}.tar,oci-mediatypes=true,oci-artifact=true" \
7786
.
7887
7988
- name: Verify SBOM attestation
89+
if: inputs.push == 'true'
8090
shell: bash
8191
env:
8292
IMAGE_REF: ghcr.io/nvidia/openshell/${{ inputs.component }}:${{ inputs.image-tag }}
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: Check Latest Release
5+
description: Check current remote stable tags before updating a shared latest channel
6+
7+
inputs:
8+
release-tag:
9+
description: Stable version being published
10+
required: true
11+
12+
outputs:
13+
make-latest:
14+
description: Whether this is still the highest stable version in the repository
15+
value: ${{ steps.latest.outputs.make-latest }}
16+
17+
runs:
18+
using: composite
19+
steps:
20+
- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
21+
id: latest
22+
env:
23+
RELEASE_TAG: ${{ inputs.release-tag }}
24+
with:
25+
script: |
26+
const tag = process.env.RELEASE_TAG;
27+
const parse = value => /^v([0-9]+)\.([0-9]+)\.([0-9]+)$/.exec(value)?.slice(1).map(BigInt);
28+
const version = parse(tag);
29+
if (!version) {
30+
throw new Error(`Invalid stable release tag: ${tag}`);
31+
}
32+
const tags = await github.paginate(github.rest.repos.listTags, {
33+
...context.repo,
34+
per_page: 100,
35+
});
36+
if (!tags.some(candidate => candidate.name === tag)) {
37+
throw new Error(`Stable release tag ${tag} is missing from the repository`);
38+
}
39+
const newer = tags.find(candidate => {
40+
const other = parse(candidate.name);
41+
if (!other) return false;
42+
const index = other.findIndex((part, i) => part !== version[i]);
43+
return index !== -1 && other[index] > version[index];
44+
});
45+
core.info(newer ? `${tag} is superseded by ${newer.name}; preserving latest` : `${tag} is the latest stable version`);
46+
core.setOutput('make-latest', !newer);

‎.github/workflows/branch-docs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ on:
1212
- "tasks/scripts/check_docs_nav_test.py"
1313
- ".github/workflows/branch-docs.yml"
1414
- ".github/workflows/release-tag.yml"
15+
- ".github/workflows/release.yml"
1516
- ".github/workflows/release-prerelease.yml"
1617
- ".github/workflows/release-publish.yml"
1718

‎.github/workflows/build-images.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@ on:
1414
required: false
1515
type: string
1616
default: ""
17+
push-images:
18+
description: Push images to GHCR during the build
19+
required: false
20+
type: boolean
21+
default: true
1722

1823
permissions:
1924
contents: read
@@ -39,6 +44,7 @@ jobs:
3944
binary: ${{ env.BINARY }}
4045
target-suffix: ${{ env.TARGET_SUFFIX }}
4146
image-tag: ${{ inputs.image-tag || github.sha }}
47+
push: ${{ inputs.push-images }}
4248
github-token: ${{ secrets.GITHUB_TOKEN }}
4349

4450
sandbox:

‎.github/workflows/release-build.yml‎

Lines changed: 35 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,11 @@ on:
1010
description: Git tag to build and publish
1111
required: true
1212
type: string
13+
expected-source-sha:
14+
description: Qualified source commit that the release tag must resolve to
15+
required: false
16+
default: ""
17+
type: string
1318
is-prerelease:
1419
description: Whether the caller is publishing a pre-release
1520
required: true
@@ -47,11 +52,9 @@ jobs:
4752
name: Compute Versions
4853
runs-on: linux-amd64-cpu8
4954
timeout-minutes: 5
50-
container:
51-
image: ghcr.io/nvidia/openshell/ci:9cb72baa2e61a1b5f12407e6e82da7fdba0aa722@sha256:67a9a0c32cb99825e6d3e9d9eec45d67149ea1ff7c11a1b1e1b3480d2d3df684
52-
credentials:
53-
username: ${{ github.actor }}
54-
password: ${{ secrets.GITHUB_TOKEN }}
55+
defaults:
56+
run:
57+
shell: nix develop -c bash -euo pipefail {0}
5558
outputs:
5659
python_version: ${{ steps.v.outputs.python }}
5760
cargo_version: ${{ steps.v.outputs.cargo }}
@@ -71,8 +74,9 @@ jobs:
7174
persist-credentials: false
7275
ref: ${{ inputs.release-tag }}
7376

74-
- name: Mark workspace safe for git
75-
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
77+
- uses: ./.github/actions/setup-nix
78+
with:
79+
prepare-shell: "true"
7680

7781
- name: Fetch tags
7882
run: git fetch --tags --force
@@ -81,8 +85,13 @@ jobs:
8185
id: v
8286
env:
8387
EXPECTED_PRERELEASE: ${{ inputs.is-prerelease }}
88+
EXPECTED_SOURCE_SHA: ${{ inputs.expected-source-sha }}
8489
run: |
8590
set -euo pipefail
91+
if [[ -n "${EXPECTED_SOURCE_SHA}" && "$(git rev-parse HEAD)" != "${EXPECTED_SOURCE_SHA}" ]]; then
92+
echo "Release tag ${RELEASE_TAG} does not match the qualified commit ${EXPECTED_SOURCE_SHA}." >&2
93+
exit 1
94+
fi
8695
if [[ "${RELEASE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[1-9][0-9]*$ ]]; then
8796
is_prerelease=true
8897
elif [[ "${RELEASE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
@@ -97,12 +106,12 @@ jobs:
97106
fi
98107
{
99108
echo "is_prerelease=${is_prerelease}"
100-
echo "python=$(uv run python tasks/scripts/release.py get-version --python)"
101-
echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)"
102-
echo "npm=$(uv run python tasks/scripts/release.py get-version --npm)"
103-
echo "deb=$(uv run python tasks/scripts/release.py get-version --deb)"
104-
echo "rpm_version=$(uv run python tasks/scripts/release.py get-version --rpm-version)"
105-
echo "rpm_release=$(uv run python tasks/scripts/release.py get-version --rpm-release)"
109+
echo "python=$(uv run --no-project python tasks/scripts/release.py get-version --python)"
110+
echo "cargo=$(uv run --no-project python tasks/scripts/release.py get-version --cargo)"
111+
echo "npm=$(uv run --no-project python tasks/scripts/release.py get-version --npm)"
112+
echo "deb=$(uv run --no-project python tasks/scripts/release.py get-version --deb)"
113+
echo "rpm_version=$(uv run --no-project python tasks/scripts/release.py get-version --rpm-version)"
114+
echo "rpm_release=$(uv run --no-project python tasks/scripts/release.py get-version --rpm-release)"
106115
echo "semver=${RELEASE_TAG#v}"
107116
echo "source_sha=$(git rev-parse HEAD)"
108117
} >> "$GITHUB_OUTPUT"
@@ -147,21 +156,19 @@ jobs:
147156
packages: write
148157
uses: ./.github/workflows/build-images.yml
149158
with:
150-
image-tag: ${{ needs.compute-versions.outputs.source_sha }}
159+
image-tag: ${{ inputs.is-prerelease && needs.compute-versions.outputs.source_sha || needs.compute-versions.outputs.semver }}
160+
push-images: ${{ inputs.is-prerelease }}
151161
checkout-ref: ${{ needs.compute-versions.outputs.source_sha }}
152162

153163
build-python-wheel:
154164
name: Build Python Wheel
155165
needs: [compute-versions]
156166
runs-on: linux-amd64-cpu8
157167
timeout-minutes: 20
158-
container:
159-
image: ghcr.io/nvidia/openshell/ci:9cb72baa2e61a1b5f12407e6e82da7fdba0aa722@sha256:67a9a0c32cb99825e6d3e9d9eec45d67149ea1ff7c11a1b1e1b3480d2d3df684
160-
credentials:
161-
username: ${{ github.actor }}
162-
password: ${{ secrets.GITHUB_TOKEN }}
168+
defaults:
169+
run:
170+
shell: nix develop -c bash -euo pipefail {0}
163171
env:
164-
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
165172
SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL: ${{ needs.compute-versions.outputs.python_version }}
166173
steps:
167174
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -170,16 +177,20 @@ jobs:
170177
persist-credentials: false
171178
ref: ${{ needs.compute-versions.outputs.source_sha }}
172179

173-
- name: Mark workspace safe for git
174-
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
180+
- uses: ./.github/actions/setup-nix
181+
with:
182+
prepare-shell: "true"
175183

176184
- name: Sync Python dependencies
177-
run: uv sync
185+
run: uv sync --frozen
178186

179187
- name: Build Python wheel
180188
run: |
181189
set -euo pipefail
182-
mise run python:build
190+
uv run --frozen python tasks/scripts/generate_python_proto.py
191+
uv build --wheel --out-dir target/wheels
192+
uv run --no-project python tasks/scripts/verify-python-wheel.py target/wheels \
193+
--expected-version "${SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPENSHELL}"
183194
ls -la target/wheels/*.whl
184195
185196
- name: Upload wheel artifact

0 commit comments

Comments
 (0)