Skip to content

Commit 54f28c5

Browse files
samuel-williams-shopifysamuel-williams-shopify
committed
Preserve release artifacts across workflow retries.
1 parent f030611 commit 54f28c5

8 files changed

Lines changed: 323 additions & 65 deletions

File tree

‎.github/releasing.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@ Review the entire release diff. Native GitHub rules require the configured revie
66

77
Merging a version increase publishes the actual merged commit through `release-publish.yaml`. There is no second release approval. The public `release.cert` stays in Git; the optional private key belongs in the Actions secret `GEM_SIGNING_KEY`, supplied by the `rubygems` environment or an organization secret. RubyGems authentication uses Trusted Publishing.
88

9-
If a publishing job fails, use **Re-run all jobs** on that same workflow run, or `bundle exec bake gem:github:release:resume run=RUN_ID`. The original artifact is downloaded before another upload is attempted. Do not create a replacement version bump or rebuild an already-published version. Conflicting bytes or tags require investigation; never automatically yank a version or move a tag.
9+
Before uploading to RubyGems, the publisher preserves the verified gem, receipt and both attestation bundles as assets of a draft GitHub release. The draft becomes public after registry verification and tag creation. Registry propagation is retried for up to one minute.
10+
11+
If a publishing job fails, use **Re-run all jobs** on that same workflow run, or `bundle exec bake gem:github:release:resume run=RUN_ID`. The original artifact is restored from the workflow artifact or GitHub release before another upload is attempted. Actions artifacts can disappear on rerun, so the draft release is the durable backup. Do not delete it, create a replacement version bump or rebuild an already-published version. Conflicting bytes or tags require investigation; never automatically yank a version or move a tag. An incomplete draft requires restoring the missing original files before retrying.
1012

1113
Verify both the downloaded gem and `release.json` with `gh attestation verify`, enforcing the publishing workflow's certificate identity and default branch ref. Then check the signed receipt's source commit and gem digest; follow the companion gem's setup guide for the complete commands. `--source-digest` checks the workflow revision, which can differ from the release commit. Release assets include the gem, RubyGems Sigstore bundle, GitHub provenance bundle, and signed source/digest receipt.
1214

‎context/getting-started.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,9 +95,11 @@ Use **Re-run all jobs** on the original publishing run, or:
9595
bundle exec bake gem:github:release:resume run=RUN_ID
9696
```
9797

98-
Rerunning keeps the original event identity. A retained artifact is downloaded and its source identity/digest checked. A matching registry version resumes tag/release finalization; different bytes or a conflicting tag stop. There is no automatic yank, retag, or rebuild of an already-published version. If failure occurs before artifact retention, rebuilding is safe because uploading has not started.
98+
Rerunning keeps the original event identity. A retained artifact is downloaded and its source identity/digest checked. A matching registry version resumes tag/release finalization; different bytes or a conflicting tag stop. There is no automatic yank, retag, or rebuild of an already-published version. Registry propagation is retried every ten seconds for up to one minute; a digest or attestation mismatch fails immediately.
9999

100-
Artifacts are retained for 90 days and copied to GitHub release assets on success. If the workflow artifact expires before finalization, manual recovery from the preserved bytes is required; do not dispatch a replacement build. GitHub concurrency does not guarantee a durable FIFO queue: rerun any publishing run displaced while pending. Resume reruns all jobs, including integrity checks; it does not repeat or second-guess the native review policy or a permitted administrator bypass.
100+
Before uploading to RubyGems, the publisher stores the verified gem, receipt and both attestation bundles in a draft GitHub release targeting the merged commit. It publishes the draft after registry verification and tag creation. Actions artifacts are also retained for 90 days, but can disappear on rerun. Recovery falls back to the draft or published release and verifies the original bytes and attestations. Keep the draft until finalization succeeds. If asset preservation was interrupted and neither backup is complete, restore the missing original files before retrying; conflicting assets are never overwritten.
101+
102+
GitHub concurrency does not guarantee a durable FIFO queue: rerun any publishing run displaced while pending. Resume reruns all jobs, including integrity checks; it does not repeat or second-guess the native review policy or a permitted administrator bypass. Older publishing runs execute their original code; adding this recovery support to the default branch does not change an already-triggered workflow.
101103

102104
## Development and current limits
103105

‎guides/getting-started/readme.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,9 +95,11 @@ Use **Re-run all jobs** on the original publishing run, or:
9595
bundle exec bake gem:github:release:resume run=RUN_ID
9696
```
9797

98-
Rerunning keeps the original event identity. A retained artifact is downloaded and its source identity/digest checked. A matching registry version resumes tag/release finalization; different bytes or a conflicting tag stop. There is no automatic yank, retag, or rebuild of an already-published version. If failure occurs before artifact retention, rebuilding is safe because uploading has not started.
98+
Rerunning keeps the original event identity. A retained artifact is downloaded and its source identity/digest checked. A matching registry version resumes tag/release finalization; different bytes or a conflicting tag stop. There is no automatic yank, retag, or rebuild of an already-published version. Registry propagation is retried every ten seconds for up to one minute; a digest or attestation mismatch fails immediately.
9999

100-
Artifacts are retained for 90 days and copied to GitHub release assets on success. If the workflow artifact expires before finalization, manual recovery from the preserved bytes is required; do not dispatch a replacement build. GitHub concurrency does not guarantee a durable FIFO queue: rerun any publishing run displaced while pending. Resume reruns all jobs, including integrity checks; it does not repeat or second-guess the native review policy or a permitted administrator bypass.
100+
Before uploading to RubyGems, the publisher stores the verified gem, receipt and both attestation bundles in a draft GitHub release targeting the merged commit. It publishes the draft after registry verification and tag creation. Actions artifacts are also retained for 90 days, but can disappear on rerun. Recovery falls back to the draft or published release and verifies the original bytes and attestations. Keep the draft until finalization succeeds. If asset preservation was interrupted and neither backup is complete, restore the missing original files before retrying; conflicting assets are never overwritten.
101+
102+
GitHub concurrency does not guarantee a durable FIFO queue: rerun any publishing run displaced while pending. Resume reruns all jobs, including integrity checks; it does not repeat or second-guess the native review policy or a permitted administrator bypass. Older publishing runs execute their original code; adding this recovery support to the default branch does not change an already-triggered workflow.
101103

102104
## Development and current limits
103105

‎lib/bake/gem/github/publisher.rb‎

Lines changed: 98 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@ module Gem
1313
module GitHub
1414
# Builds one artifact, preserves it before upload, and resumes without moving existing tags.
1515
class Publisher < Project
16+
# A registered package is not yet available from the download service.
17+
class RegistryPending < RuntimeError
18+
end
19+
1620
# Build or restore this workflow run's artifact, after validating the actual merged commit.
1721
def build(number)
1822
guard_environment
@@ -23,24 +27,34 @@ def build(number)
2327
artifact = "release-#{evidence.fetch(:commit)}"
2428
run = ENV.fetch("GITHUB_RUN_ID")
2529
artifacts = api("actions/runs/#{run}/artifacts?per_page=100").fetch("artifacts")
26-
if retained = artifacts.find{|entry| entry.fetch("name") == artifact}
27-
raise "Retained artifact expired; restore it from the GitHub release before retrying." if retained.fetch("expired")
30+
retained = artifacts.find{|entry| entry.fetch("name") == artifact}
31+
if retained && !retained.fetch("expired")
2832
system("gh", "run", "download", run, "--repo", @repository, "--name", artifact, "--dir", path, chdir: @root)
29-
receipt = load_receipt
30-
[:name, :version, :commit, :repository, :pull_request].each do |key|
31-
raise "Retained artifact has different #{key}." unless receipt[key] == evidence[key]
33+
elsif release = github_release("v#{evidence.fetch(:version)}")
34+
guard_release(release, evidence.fetch(:commit))
35+
filename = "#{evidence.fetch(:name)}-#{evidence.fetch(:version)}.gem"
36+
files = release_files(file: filename)
37+
names = release.fetch("assets").map{|asset| asset.fetch("name")}
38+
raise "Retained release is incomplete; restore the original files before retrying." unless files.all?{|file| names.include?(File.basename(file))}
39+
system("gh", "release", "download", release.fetch("tag_name"), "--repo", @repository, "--dir", path, *files.flat_map{|file| ["--pattern", File.basename(file)]}, chdir: @root)
40+
elsif retained
41+
raise "Retained artifact expired and no GitHub release is available. Restore the original files before retrying."
42+
else
43+
filename = "#{evidence.fetch(:name)}-#{evidence.fetch(:version)}.gem"
44+
if registry_digest(evidence.fetch(:name), evidence.fetch(:version))
45+
raise "Version is already published but this run has no retained artifact. Restore the original artifact; do not rebuild."
3246
end
33-
return output(receipt, restored: true)
47+
package = build_package(path)
48+
raise "Unexpected package filename." unless File.basename(package) == filename
49+
receipt = evidence.merge(file: filename, sha256: Digest::SHA256.file(package).hexdigest, run_id: run, signing: @config.fetch("signing"))
50+
File.write(File.join(path, "release.json"), JSON.pretty_generate(receipt) + "\n")
51+
return output(receipt, restored: false)
3452
end
35-
filename = "#{evidence.fetch(:name)}-#{evidence.fetch(:version)}.gem"
36-
if registry_digest(evidence.fetch(:name), evidence.fetch(:version))
37-
raise "Version is already published but this run has no retained artifact. Restore the original artifact; do not rebuild."
53+
receipt = load_receipt
54+
[:name, :version, :commit, :repository, :pull_request].each do |key|
55+
raise "Retained artifact has different #{key}." unless receipt[key] == evidence[key]
3856
end
39-
package = build_package(path)
40-
raise "Unexpected package filename." unless File.basename(package) == filename
41-
receipt = evidence.merge(file: filename, sha256: Digest::SHA256.file(package).hexdigest, run_id: run, signing: @config.fetch("signing"))
42-
File.write(File.join(path, "release.json"), JSON.pretty_generate(receipt) + "\n")
43-
output(receipt, restored: false)
57+
output(receipt, restored: true)
4458
end
4559

4660
# Verify both attestations, upload exactly those bytes, then create only the intended tag and release.
@@ -65,37 +79,19 @@ def publish(number)
6579
remote_digest = registry_digest(receipt.fetch(:name), receipt.fetch(:version))
6680
if remote_digest
6781
raise "Published version has different bytes." unless remote_digest == receipt.fetch(:sha256)
68-
else
82+
end
83+
# Keep verified bytes independently of workflow attempts before uploading:
84+
release = preserve_release(receipt)
85+
unless remote_digest
6986
gem_command("push", package, "--host", "https://rubygems.org", "--attestation", bundle)
7087
end
71-
# A failed read after upload is recoverable by rerunning the same workflow.
72-
raise "Registry artifact does not match; retry after registry propagation." unless registry_digest(receipt.fetch(:name), receipt.fetch(:version)) == receipt.fetch(:sha256)
73-
attestations = registry_get("https://rubygems.org/api/v1/attestations/#{receipt.fetch(:name)}-#{receipt.fetch(:version)}.json")
74-
raise "Registry attestation is missing." unless attestations
75-
registry_bundles = JSON.parse(attestations)
76-
local_bundle = JSON.parse(File.read(bundle))
77-
raise "Registry does not contain this artifact's Sigstore bundle." unless contains_bundle?(registry_bundles, local_bundle)
88+
verify_registry(receipt, bundle)
7889
unless readlines("git", "tag", "--list", tag, chdir: @root).any?
7990
system("git", "tag", tag, receipt.fetch(:commit), chdir: @root)
8091
end
8192
system("git", "-c", "credential.helper=", "-c", "credential.helper=!gh auth git-credential", "push", "origin", "refs/tags/#{tag}", chdir: @root)
82-
releases = JSON.parse(readlines("gh", "release", "list", "--repo", @repository, "--limit", "1000", "--json", "tagName", chdir: @root).join)
83-
unless releases.any?{|entry| entry.fetch("tagName") == tag}
84-
Tempfile.create("release-notes") do |file|
85-
file.write("#{receipt.fetch(:pull_request_url)}\n\nSource: #{receipt.fetch(:commit)}\nSHA256: #{receipt.fetch(:sha256)}\n\nSee releases.md at the release tag for release notes.\n")
86-
file.flush
87-
system("gh", "release", "create", tag, "--repo", @repository, "--verify-tag", "--title", "#{receipt.fetch(:name)} #{tag}", "--notes-file", file.path, chdir: @root)
88-
end
89-
end
90-
# Existing immutable assets are checked before upload; never clobber them.
91-
assets = api("releases/tags/#{tag}").fetch("assets")
92-
[package, bundle, File.join(@root, "pkg", "release.json"), File.join(@root, "pkg", "provenance.sigstore.json")].each do |file|
93-
if existing = assets.find{|asset| asset.fetch("name") == File.basename(file)}
94-
digest = existing.fetch("digest")
95-
raise "Existing release asset differs: #{file}" unless digest == "sha256:#{Digest::SHA256.file(file).hexdigest}"
96-
else
97-
system("gh", "release", "upload", tag, file, "--repo", @repository, chdir: @root)
98-
end
93+
if release.fetch("draft")
94+
system("gh", "release", "edit", tag, "--repo", @repository, "--draft=false", "--verify-tag", chdir: @root)
9995
end
10096
receipt
10197
end
@@ -120,6 +116,68 @@ def guard_tag(tag, commit)
120116

121117
private
122118

119+
def github_release(tag)
120+
pages = JSON.parse(readlines("gh", "api", "--paginate", "--slurp", "repos/#{@repository}/releases?per_page=100", chdir: @root).join)
121+
matches = pages.flatten(1).select{|release| release.fetch("tag_name") == tag}
122+
raise "Multiple GitHub releases have the same tag." if matches.size > 1
123+
matches.first
124+
end
125+
126+
def guard_release(release, commit)
127+
if release.fetch("draft") && release.fetch("target_commitish") != commit
128+
raise "Draft release targets another commit."
129+
end
130+
end
131+
132+
def release_files(receipt)
133+
[receipt.fetch(:file), "#{receipt.fetch(:file)}.sigstore.json", "release.json", "provenance.sigstore.json"].map{|name| File.join(@root, "pkg", name)}
134+
end
135+
136+
def preserve_release(receipt)
137+
tag = "v#{receipt.fetch(:version)}"
138+
release = github_release(tag)
139+
unless release
140+
Tempfile.create("release-notes") do |file|
141+
file.write("#{receipt.fetch(:pull_request_url)}\n\nSource: #{receipt.fetch(:commit)}\nSHA256: #{receipt.fetch(:sha256)}\n\nSee releases.md at the release tag for release notes.\n")
142+
file.flush
143+
system("gh", "release", "create", tag, "--repo", @repository, "--draft", "--target", receipt.fetch(:commit), "--title", "#{receipt.fetch(:name)} #{tag}", "--notes-file", file.path, chdir: @root)
144+
end
145+
release = github_release(tag) or raise "Draft release was not retained."
146+
end
147+
guard_release(release, receipt.fetch(:commit))
148+
assets = release.fetch("assets")
149+
release_files(receipt).each do |file|
150+
if existing = assets.find{|asset| asset.fetch("name") == File.basename(file)}
151+
raise "Existing release asset differs: #{file}" unless existing.fetch("digest") == "sha256:#{Digest::SHA256.file(file).hexdigest}"
152+
else
153+
system("gh", "release", "upload", tag, file, "--repo", @repository, chdir: @root)
154+
end
155+
end
156+
release
157+
end
158+
159+
def verify_registry(receipt, bundle, attempts: 7, delay: 10)
160+
local_bundle = JSON.parse(File.read(bundle))
161+
attempts.times do |attempt|
162+
begin
163+
if digest = registry_digest(receipt.fetch(:name), receipt.fetch(:version))
164+
raise "Published version has different bytes." unless digest == receipt.fetch(:sha256)
165+
if attestations = registry_get("https://rubygems.org/api/v1/attestations/#{receipt.fetch(:name)}-#{receipt.fetch(:version)}.json")
166+
raise "Registry does not contain this artifact's Sigstore bundle." unless contains_bundle?(JSON.parse(attestations), local_bundle)
167+
return
168+
end
169+
end
170+
rescue RegistryPending
171+
# The version API can become visible before the gem download:
172+
end
173+
if attempt < attempts - 1
174+
Console.info(self, "Waiting for RubyGems to serve the release.")
175+
sleep(delay)
176+
end
177+
end
178+
raise "Registry propagation did not complete; rerun this workflow to resume from the retained release."
179+
end
180+
123181
def gem_command(*arguments)
124182
if defined?(::Bundler)
125183
::Bundler.with_unbundled_env{system("gem", *arguments, chdir: @root)}
@@ -193,7 +251,7 @@ def registry_digest(name, version)
193251
# Missing downloads can return 403; use the version API to establish absence.
194252
return nil unless registry_get("https://rubygems.org/api/v2/rubygems/#{name}/versions/#{version}.json?platform=ruby")
195253
body = registry_get("https://rubygems.org/downloads/#{name}-#{version}.gem")
196-
raise "Published gem download is missing; retry after registry propagation." unless body
254+
raise RegistryPending, "Published gem download is missing; retry after registry propagation." unless body
197255
Digest::SHA256.hexdigest(body)
198256
end
199257

‎releases.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
# Releases
22

3+
## Unreleased
4+
5+
- Preserve verified release files in a draft GitHub release before uploading to RubyGems, so reruns can recover when Actions artifacts disappear.
6+
- Wait for RubyGems registry propagation before finalizing releases.
7+
38
## v0.0.1
49

510
- Initial implementation.

0 commit comments

Comments
 (0)