Skip to content

fix(imageupgrade): serialise upgrade calls and sync the ESP directory - #376

Merged
Bugs5382 merged 2 commits into
mainfrom
fix/373-imageupgrade-serialise
Oct 14, 2026
Merged

Bugs5382 merged 2 commits into
mainfrom
fix/373-imageupgrade-serialise

Conversation

@Bugs5382

Copy link
Copy Markdown
Contributor

What and why

Confirmed in code first, as the issue asked: StageImage, RollbackImage and
ActivateImage took no lock anywhere (internal/grpc, internal/init, and
the realESPMounter that mounts the ESP per call). Two upgrade calls at the
same time could both write EFI/BOOT/BOOTX64.NEW.EFI and leave a
half-written boot image, so this PR covers both halves of the issue.

Serialisation. nodeImageUpgrader (the grpc.ImageUpgrader the node
serves) now holds one sync.Mutex, taken with TryLock at the top of
Stage, Rollback and Activate. A call made while another of the three is
already running is refused immediately with a new sentinel,
grpc.ErrUpgradeInProgress, which the three gRPC handlers map to
FailedPrecondition. TryLock rather than Lock: a concurrent caller is
told outright rather than queued or left racing the one in flight.

ESP directory sync. The activating rename (BOOTX64.NEW.EFI ->
BOOTX64.EFI) is a vfat directory-entry update; a file's own fsync
(already done in WriteFile) doesn't make that entry durable. The FS
interface in internal/imageupgrade gains Sync() error; Stage and
Rollback call it immediately after their renames, and the real ESP
implementation (dirFS in internal/init) opens the mounted root and
fsyncs it.

No docs change: nothing here is a new user-facing behavior, only a
correctness fix to an unlocked race and an unsynced rename that were never
documented as safe.

Refs #373

Verification

  • Lint clean
  • Tests pass
  • Build succeeds
  • Documentation updated (if behavior or API changed) -- not needed, see above

How this was verified

  • go build ./...
  • systemd-run --user --scope -q -p MemoryMax=6G -p MemorySwapMax=0 go test ./... (all packages pass)
  • systemd-run --user --scope -q -p MemoryMax=6G -p MemorySwapMax=0 go test -race ./internal/init/... ./internal/imageupgrade/... ./internal/grpc/...
  • golangci-lint run ./internal/init/... ./internal/imageupgrade/... ./internal/grpc/... (0 issues)
  • gofmt -l . (clean)
  • New tests written first and watched fail for the right reason (undefined symbols, then wrong
    call order / wrong error), then made to pass:
    • TestStage_SyncsTheESPDirectoryAfterTheRename / TestRollback_SyncsTheESPDirectoryAfterTheRename
      assert the directory Sync call lands immediately after the activating/restoring rename.
    • TestStage_ReturnsAnErrorWhenTheDirectorySyncFails / TestRollback_ReturnsAnErrorWhenTheDirectorySyncFails
    • TestUpgrader_StageSerializesAgainstAConcurrentStage drives two real goroutines through Stage
      with a blocking Mount hook: the first holds the lock mid-call, the second gets
      ErrUpgradeInProgress immediately, then the first completes cleanly once released.
    • TestUpgrader_StageRefusesWhileAnUpgradeIsInProgress / RollbackRefusesWhileAnUpgradeIsInProgress /
      ActivateRefusesWhileAnUpgradeIsInProgress hold the lock directly and assert each of the three
      entry points refuses rather than mounting the ESP.

StageImage, RollbackImage and ActivateImage took no lock, so two upgrade
calls at the same time could share BOOTX64.NEW.EFI and leave a
half-written boot image. A shared mutex on the node's image upgrader
now refuses a call made while another of the three is already running
with FailedPrecondition, rather than queuing it or racing it.

The rename that swaps the active image in is also not durable on its
own: vfat needs the directory entry fsynced, which a file's own fsync
does not cover. Stage and Rollback now fsync the ESP's mounted
directory right after that rename, so a power loss right after leaves
the old or the new image, never neither.

Signed-off-by: Bugs5382 <12115015+Bugs5382@users.noreply.github.com>
@Bugs5382
Bugs5382 marked this pull request as ready for review October 14, 2026 15:15
@github-actions github-actions Bot added the fix Bug fix (fix). Patch. label Oct 14, 2026
@Bugs5382
Bugs5382 enabled auto-merge (squash) October 14, 2026 15:55
@Bugs5382
Bugs5382 merged commit 1a82560 into main Oct 14, 2026
15 checks passed
@Bugs5382
Bugs5382 deleted the fix/373-imageupgrade-serialise branch October 14, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug fix (fix). Patch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant