[mnafees]

Description

Publish new set of -fips suffixed images for all of our GHCR images. The Go binaries are linked with the validated Go Cryptography Module (GOFIPS140=v1.0.0, CMVP #4985).

Much inspiration was taken from the go-fips hardened image from Chainguard meaning that we have CGO_ENABLED=0, -ldflags=-w, DefaultGODEBUG=fips140=on.

• FIPS=true build arg on the servers, frontend, and dashboard Dockerfiles, the build stage fails unless go version -m shows the validated module.
• Dashboard image serves the frontend and proxies /api with hatchet-staticfileserver instead of nginx so the whole serving path is Go.
• hack/ci/assert-fips.sh inspects every Go binary in an image. Runs on PRs (fips job), on regular prerelease images (must not link the module) and on -fips prerelease images before manifests are combined.

Type of change

• New feature (non-breaking change which adds functionality)
• CI (any automation pipeline changes)

Checklist

Changes have been:

• Documented (where applicable)
• Added to CHANGELOG (where applicable) -- see Keep a Changelog

---

🤖 AI Disclosure

• I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.

• Details: [e.g. generating tests, writing docs]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hatchet-docs	Ready	Preview	Oct 12, 2026 12:16pm UTC

[greptile-apps]

Confidence Score: 4/5

[High impact] Adds FIPS-validated cryptography build path and enforcement.

The PR does not appear safe to merge until the ARM FIPS job reliably fails when a binary build fails.

Summary

The PR adds FIPS-built image variants, image checks, runtime smoke tests, and deployment guidance. Since the previous review, it adds an ARM FIPS CI job to build and test those images.

_{Reviews (10) · Last reviewed commit: "arm builds" · Reviewed by Greptile}

[github-actions]

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=true). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[greptile-apps]

Comments Outside Diff

These findings could not be posted inline.

•  Previously encrypted secrets become unreadable pkg/encryption/local.go:254 ▶

  When a non-FIPS deployment upgrades from the previous revision of this PR, this code tries to decrypt stored secrets only with the AES-GCM envelope. That revision wrote new secrets with an AES-CTR-HMAC envelope, which the current path cannot read. Webhook credentials and OAuth tokens written by that revision therefore fail to decrypt.

[igor-kupczynski]

At 30% this makes sense to me ✅

• the PR uses the new native Go Cryptographic Module v1.0.0 which is certified by nist
• the right args are passed to go in the dockerfiles

^^^ i have some amount of nickpicks which an agent can probably solve quickly, i'll raise them separately.

One big concern i have (also cc @abelanger5 to chime in) is if we need a fips base image:

• the PR doesn't change the base and uses golang:1.26-alpine for build and alpine for the runtime image
• my interpretation is that the approach in this PR is OK, because of the CGO_ENABLED=0 we don't shell for the native crypto of the base image but rather use fips go module
• the base still has libcrypto3/libssl3 (not fips validated/compliant) which may come up on audit scans

[igor-kupczynski]

Great stuff @mnafees; i've raised a lot of comments, i'd like to see them responded to before merging, so ❌ for now. Let me know WDYT.

[igor-kupczynski]

🤖 Opus 5.5 on behalf of Igor:

Approving. This replaces my earlier changes-requested. Thanks for working through all of it.

I've left two suggestions inline: docs for switching to -fips, and a written position on the Alpine base image. Your call on whether they go in this PR or a follow-up.

Greptile also has an open P1 on the set-password ordering with no reply yet: #4997 (comment). The risk is low in practice, since old bcrypt passwords can't log in on FIPS builds anyway. Writing the hash first and revoking sessions afterwards closes it; if revocation fails, you just re-run the command.

[igor-kupczynski]

🤖 Opus 5.5 on behalf of Igor:

Docs suggestion: once this merges, the next tag publishes -fips:latest. Anyone pointing it at an existing database will hit three things that aren't written down anywhere:

• Users with bcrypt passwords get a 400 on login, and the only hint is a server-side warning. The fix is hatchet-admin user set-password.
• The server refuses to start on an AES-GCM master key. Regenerating with keyset create-local-keys makes the stored secrets unreadable (OAuth tokens, Slack and webhook credentials), so they have to be re-created.
• Regenerating the JWT keysets invalidates every existing API token, so workers need new tokens. Worth a sentence in this error message too.

A short self-hosting page on switching to -fips would cover it.

[igor-kupczynski]

🤖 Opus 5.5 on behalf of Igor:

Following up on the base-image thread, which was resolved without a reply. Dropping openssl and blocking cert generation in quickstart fixes the real problem, so I'm fine with staying on Alpine.

libcrypto3/libssl3 from the Alpine base are still in the image, and scanners will flag them even though nothing loads them (CGO_ENABLED=0). I'd state that in the FIPS docs so support has something to point customers at. Distroless or scratch can be a follow-up if an audit asks for it.

[igor-kupczynski]

🤖 Opus 5.5 on behalf of Igor:

Non-blocking notes, take them or leave them:

• GenerateLocalKeys / HashPassword defaults: data keys now follow the master key type, so creating CTR-HMAC master keys and PBKDF2 hashes by default wouldn't need a migration. Existing installs keep working, and passwords move over as users change them. The real reasons to wait are rollback (older versions can't read either) and PBKDF2's CPU cost on a login endpoint with no rate limit. Fine as a follow-up issue.
• release.yaml: fine to keep it decoupled. A retry loop around docker pull would avoid a red release run when the FIPS pre-release is slower.
• receive.go checks: agreed. With fips140=on they're the only enforcement.
• servers.dockerfile: still uses an untagged FROM alpine, while lite and dashboard use alpine:3.21. Nothing is pinned by digest.
• hatchet-staticfileserver: doesn't call RequireFIPSBuild(). It only proxies plain HTTP to localhost, so probably fine; just noting it.