Skip to content

ci: re-enable riscv64 releases, don't let one arch block the manifest - #758

Open
mauro-agent wants to merge 6 commits into
kairos-io:mainfrom
mauro-agent:riscv64-release-manifest-fix
Open

ci: re-enable riscv64 releases, don't let one arch block the manifest#758
mauro-agent wants to merge 6 commits into
kairos-io:mainfrom
mauro-agent:riscv64-release-manifest-fix

Conversation

@mauro-agent

Copy link
Copy Markdown
Contributor

What

Reverts #545's disable of the riscv64 release build, and separately fixes the bug #545's own title named: one failed arch job skipped the final manifest job entirely, leaving the release tag completely unpublished on quay (v0.22.0 never got a manifest, not even for amd64/arm64).

Why revert now

Checked test-riscv64.yml's actual recent run history before reverting: zero riscv64-job failures across the last 20 PR CI runs on the same ubuntu-24.04-riscv runner (the 2 failures in that window were an unrelated build-ui/npm issue that skipped the riscv64 job before it even ran). Whatever made the runner unreliable back in June looks resolved, so this restores build-ui/build-linux-riscv64 unchanged rather than guessing at a different fix for a problem that isn't currently reproducing.

The actual fix

The build (manifest) job now runs regardless of riscv64's outcome, still requiring amd64/arm64 to succeed, and includes riscv64 in the published manifest only if it actually succeeded that run. A riscv64 flake degrades to a two-arch release instead of no release at all.

Also bumped the UI artifact's retention from 1 to 3 days: #545 noted a same-day retry couldn't recover because the artifact had already expired by the time anyone re-ran the failed job.

Test plan

  • A real tag push exercises this end to end (can't fully simulate locally); recommend testing against a pre-release tag before the next real cut
  • Confirm the manifest still publishes correctly if riscv64 is intentionally made to fail

mauro-agent pushed a commit to mauro-agent/kairos-docs that referenced this pull request Aug 28, 2026
Mauro still has real AuroraBoot release work ahead (kairos-io/AuroraBoot#758
needs a real tag push to validate, then an actual release). This post
stays scoped to "here's the image, help me test it" -- the AuroraBoot
story is its own post, next week, once that work has actually landed.

Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
@mauromorales
mauromorales marked this pull request as ready for review August 28, 2026 07:52
@mauromorales
mauromorales requested a review from a team as a code owner August 28, 2026 07:52
Copilot AI lite review requested due to automatic review settings August 28, 2026 07:52
mauro-agent pushed a commit to mauro-agent/kairos-docs that referenced this pull request Aug 28, 2026
Removing only the AuroraBoot staleness caveat left the instructions
themselves in place -- someone following them today still hits the
exact problem the caveat warned about, just without the warning.
Cutting the "long path" section entirely keeps this post to the ISO/raw
image download only. Build-your-own instructions come back once
kairos-io/AuroraBoot#758 has actually landed and released.

Also fixed the frontmatter description, which still promised "build and
boot your own image."

Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Re-enables the riscv64 release image build and adjusts the release image workflow so a single-architecture failure (riscv64) no longer prevents publishing the multi-arch manifest for successful architectures (amd64/arm64).

Changes:

  • Reintroduce build-ui and build-linux-riscv64 jobs for release builds.
  • Make the manifest build job run even if riscv64 fails, while still requiring amd64/arm64 success.
  • Conditionally include the riscv64 image in the manifest only when its build succeeded; increase UI artifact retention to 3 days.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/image.yml Outdated
Comment thread .github/workflows/image.yml Outdated
Comment thread .github/workflows/image.yml Outdated
Comment thread .github/workflows/image.yml Outdated
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.72%. Comparing base (c42c764) to head (2cc002b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #758      +/-   ##
==========================================
+ Coverage   49.71%   49.72%   +0.01%     
==========================================
  Files          77       77              
  Lines        9733     9735       +2     
==========================================
+ Hits         4839     4841       +2     
  Misses       4337     4337              
  Partials      557      557              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mauro-agent pushed a commit to mauro-agent/AuroraBoot that referenced this pull request Aug 28, 2026
- build-ui's comment claimed it's shared by every arch job. It isn't:
  amd64/arm64 build their own UI inline via Dockerfile's js stage, only
  riscv64 needs the separate build-ui job. Corrected.
- setup-node, upload-artifact, and download-artifact were still pinned
  to the versions from before kairos-io#545 disabled this job. Bumped all three
  to v7, matching tests.yml and test-riscv64.yml elsewhere in the repo.

Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
mauromorales and others added 4 commits August 28, 2026 12:09
Reverts kairos-io#545's disable of the riscv64 release build, and separately
fixes the actual bug kairos-io#545's own title named: one failed arch job
skipped the final manifest job entirely, leaving the release tag
completely unpublished on quay (v0.22.0 never got a manifest, not even
for amd64/arm64).

Checked test-riscv64.yml's actual recent run history before reverting:
zero riscv64-job failures across the last 20 PR CI runs on the same
ubuntu-24.04-riscv runner (the only 2 failures were an unrelated
build-ui/npm issue that skipped the riscv64 job before it ran). Whatever
made the runner unreliable in June looks resolved, so this restores the
build-ui/build-linux-riscv64 jobs unchanged rather than guessing at a
different fix for a problem that isn't currently reproducing.

The manifest job now runs regardless of riscv64's outcome (still
requires amd64/arm64), and includes riscv64 in the published manifest
only if it actually succeeded that run -- a riscv64 flake degrades to a
two-arch release instead of no release at all. Also bumped the UI
artifact's retention from 1 to 3 days: kairos-io#545 noted a same-day retry
couldn't recover because the artifact had already expired.

Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Mauro Morales <contact@mauromorales.com>
The inline comments had grown into a retelling of the investigation
(run-history counts, the artifact-TTL story) that belongs in the commit
message, not in code a future editor has to read past. Kept only the
load-bearing why, with issue references for anyone who wants the rest.

Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
- build-ui's comment claimed it's shared by every arch job. It isn't:
  amd64/arm64 build their own UI inline via Dockerfile's js stage, only
  riscv64 needs the separate build-ui job. Corrected.
- setup-node, upload-artifact, and download-artifact were still pinned
  to the versions from before kairos-io#545 disabled this job. Bumped all three
  to v7, matching tests.yml and test-riscv64.yml elsewhere in the repo.

Signed-off-by: Mauro Morales <contact@mauromorales.com>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Mauro Morales <contact@mauromorales.com>
@mauro-agent
mauro-agent force-pushed the riscv64-release-manifest-fix branch from 8837698 to 8ab6ca4 Compare August 28, 2026 10:12
@mauro-agent

Copy link
Copy Markdown
Contributor Author

Timing investigation, recorded before a session restart so it isn't lost.

The native ubuntu-24.04-riscv runner (test-riscv64.yml, same Dockerfile.riscv64) took 42 minutes total, 30 of those in the Docker image build step alone.

Ran the same build locally (16 CPUs / 58GB, docker buildx build --platform linux/riscv64, no cache, plain QEMU emulation, no platform pinning): 24m15s total. Even a modest emulated machine already beats the native runner. Step breakdown showed 77% of that time (18.8 of 24.25 min) in one step: go build, running under full QEMU system emulation because the builder stage isn't pinned to $BUILDPLATFORM.

Tried fixing that properly, pinning with-swagger/without-swagger/builder to --platform=$BUILDPLATFORM and using tonistiigi/xx (xx-go build, xx-apt-get) to cross-compile natively and only cross-link the one CGO dependency (libpcsclite, for go-piv/piv-go/v2 via sbctl). The swag codegen step alone dropped from 196s to 12.6s pinned, confirming the general approach is directionally right. But the actual go build step failed: xx-go didn't correctly enable CGO for the riscv64 target, so go-piv/piv-go and github.com/ThalesGroup/crypto11 fell back to their CGO-disabled stub code paths and hit undefined: pkcs11.Ctx, undefined: scTx, etc. Real toolchain work to fix properly (correct CC, actual cross-linking config), not attempted further this session.

Net result: the plain unpinned emulated build (24m15s) is the only approach actually proven to work end to end. The $BUILDPLATFORM + xx optimization is a real, promising lead (the swag-stage speedup is genuine evidence) but needs someone to actually debug the CGO cross-link failure before it's usable. Not blocking this PR either way, since #758's own fix is unrelated (the manifest-blocking bug, already merged in this branch) -- just recording the finding.

…ted EOF)

Signed-off-by: mauro-agent <mauro-agent@mrls.xyz>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
@mauro-agent

Copy link
Copy Markdown
Contributor Author

Decision: ship the plain, fully-emulated build now. It already works end to end (this PR's own runs prove that). The $BUILDPLATFORM + tonistiigi/xx cross-compile speedup is a separate, real lead but not required to land this fix, and its CGO cross-link problem (see the timing-investigation comment above) needs real toolchain debugging. Picking that back up later, tracked separately.

Retriggered CI above: the failing e2e-tests run was an unrelated flake (unexpected EOF pulling quay.io/kairos/ubuntu:22.04-core-arm64-rpi4-v3.6.1-beta2 from quay.io), not caused by this change.

…building the test image, unrelated to this diff)

Signed-off-by: mauro-agent <mauro-agent@mrls.xyz>
Co-developed-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants