fix(ops): fail rebuild fast when the referenced backup is missing - #10544
fix(ops): fail rebuild fast when the referenced backup is missing#10544weicao wants to merge 3 commits into
Conversation
A RebuildInstance OpsRequest on the in-place path treated a nonexistent rebuildFrom[].backupName as a retryable error: the bare NotFound from the Backup Get in prepareInplaceRebuildHelper leaked out of ReconcileAction, so the ops controller requeued forever and the OpsRequest never left Running. The same applied to a deleted ActionSet referenced by backup.status.backupMethod.actionSetName. Both conditions are deterministic (submit-time validation does not check the backup, and deleted objects do not come back), so classify the two NotFound lookups as intctrlutil.NewFatalError, matching the adjacent backup validations in the same function. The affected instances are now marked Failed and the OpsRequest transitions to Failed. Fixes #10542 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Auto Cherry-pick Instructions CLA Recheck Instructions |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10544 +/- ##
==========================================
- Coverage 63.68% 63.67% -0.01%
==========================================
Files 519 519
Lines 62852 62854 +2
==========================================
- Hits 40027 40023 -4
- Misses 19202 19211 +9
+ Partials 3623 3620 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Address the 2026-07-10 review round: an ActionSet is a cluster-scoped resource supplied by the addon and can be temporarily absent during addon installation/upgrade/reconciliation, so a NotFound on the lookup cannot prove the backup artifact is invalid. Drop the fatal classification for the ActionSet lookup (restoring the plain retryable error) and keep only the backupName fatal, which is supplied directly by the OpsRequest. The regression now covers retry-while-absent and convergence once the ActionSet is installed; it fails on the previous head (fatal marked the instances Failed) and passes here.
|
[P2, non-blocking] The new regression test consumes an ActionSet before its owner has published a valid current-generation result. The test creates an ActionSet with no The previous production issue where ActionSet NotFound was made terminal has been correctly reverted on this head; there is no remaining production blocker from that feedback. |
|
Handled the non-blocking fixture gap in exact head |
Problem
A
RebuildInstanceOpsRequest using the in-place path (rebuildFrom[].inPlace: true) with abackupNamethat references a nonexistent Backup never fails — the ops controller requeues and retries forever, and the OpsRequest staysRunningindefinitely. The same applies when the ActionSet referenced bybackup.status.backupMethod.actionSetNamehas been deleted.Both conditions are deterministic and permanent:
validateRebuildInstancedoes not validate the referenced backup at submit time, and a deleted Backup/ActionSet does not come back.Root cause
In
pkg/operations/rebuild_instance.go,prepareInplaceRebuildHelper:cli.Getof the Backup named byrebuildFrom[].backupNamereturned the bare NotFound error;dputils.GetActionSetByNameforbackup.status.backupMethod.actionSetNamealso returned the bare NotFound error.In the operations framework, a plain error from
ReconcileActionmeans "transient, requeue"; onlyintctrlutil.NewFatalErroris treated as terminal (rebuildInstancesInPlacecatchesErrorTypeFatal, marks the instance progressFailed, and lets the OpsRequest transition toFailed). The adjacent validations in the same function (backup not Full/Incremental, backup not Completed, empty backupMethod) already returnNewFatalError; only these two NotFound lookups leaked through as retryable.Change
intctrlutil.NewFatalError, matching the neighboring checks.intctrlutil.NewFatalError.ReconcileActionend to end: missing backup and missing actionSet both mark the instances' progressFailedwith a "not found" message and move the OpsRequest toFailedon the next reconciliation.Not changed
Actionphase instance lookup fixed by fix(ops): fail rebuild ops for non-existent instances #10510, and submit-time validation are untouched.RED evidence
Both new tests fail on
origin/main(before the fix) with the bare NotFound error escapingReconcileAction:With the fix applied, the focused run and the full
go test ./pkg/operations/... -count=1suite pass;go vetandgofmt -lare clean.Fixes #10542
🤖 Generated with Claude Code