Skip to content

fix(ops): fail rebuild fast when the referenced backup is missing - #10544

Draft
weicao wants to merge 3 commits into
mainfrom
bugfix/rebuild-ops-missing-backup
Draft

fix(ops): fail rebuild fast when the referenced backup is missing#10544
weicao wants to merge 3 commits into
mainfrom
bugfix/rebuild-ops-missing-backup

Conversation

@weicao

@weicao weicao commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Problem

A RebuildInstance OpsRequest using the in-place path (rebuildFrom[].inPlace: true) with a backupName that references a nonexistent Backup never fails — the ops controller requeues and retries forever, and the OpsRequest stays Running indefinitely. The same applies when the ActionSet referenced by backup.status.backupMethod.actionSetName has been deleted.

Both conditions are deterministic and permanent: validateRebuildInstance does not validate the referenced backup at submit time, and a deleted Backup/ActionSet does not come back.

Root cause

In pkg/operations/rebuild_instance.go, prepareInplaceRebuildHelper:

  • the cli.Get of the Backup named by rebuildFrom[].backupName returned the bare NotFound error;
  • dputils.GetActionSetByName for backup.status.backupMethod.actionSetName also returned the bare NotFound error.

In the operations framework, a plain error from ReconcileAction means "transient, requeue"; only intctrlutil.NewFatalError is treated as terminal (rebuildInstancesInPlace catches ErrorTypeFatal, marks the instance progress Failed, and lets the OpsRequest transition to Failed). The adjacent validations in the same function (backup not Full/Incremental, backup not Completed, empty backupMethod) already return NewFatalError; only these two NotFound lookups leaked through as retryable.

Change

  • Classify NotFound on the Backup lookup as intctrlutil.NewFatalError, matching the neighboring checks.
  • Classify NotFound on the ActionSet lookup (same function, same error-flow) as intctrlutil.NewFatalError.
  • Add two regression tests driving ReconcileAction end to end: missing backup and missing actionSet both mark the instances' progress Failed with a "not found" message and move the OpsRequest to Failed on the next reconciliation.

Not changed

RED evidence

Both new tests fail on origin/main (before the fix) with the bare NotFound error escaping ReconcileAction:

[FAIL] ... [It] test rebuild instance in place when the backup does not exist
[FAIL] ... [It] test rebuild instance in place when the actionSet of the backup does not exist
    Unexpected error: backups.dataprotection.kubeblocks.io "backup-not-exist-..." not found
    Unexpected error: actionsets.dataprotection.kubeblocks.io "actionset-not-exist-..." not found
Ran 2 of 114 Specs ... 0 Passed | 2 Failed

With the fix applied, the focused run and the full go test ./pkg/operations/... -count=1 suite pass; go vet and gofmt -l are clean.

Fixes #10542

🤖 Generated with Claude Code

A RebuildInstance OpsRequest on the in-place path treated a nonexistent
rebuildFrom[].backupName as a retryable error: the bare NotFound from
the Backup Get in prepareInplaceRebuildHelper leaked out of
ReconcileAction, so the ops controller requeued forever and the
OpsRequest never left Running. The same applied to a deleted ActionSet
referenced by backup.status.backupMethod.actionSetName.

Both conditions are deterministic (submit-time validation does not
check the backup, and deleted objects do not come back), so classify
the two NotFound lookups as intctrlutil.NewFatalError, matching the
adjacent backup validations in the same function. The affected
instances are now marked Failed and the OpsRequest transitions to
Failed.

Fixes #10542

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@weicao
weicao requested review from a team and wangyelei as code owners July 5, 2026 18:38
@apecloud-bot

Copy link
Copy Markdown
Collaborator

Auto Cherry-pick Instructions

Usage:
  - /nopick: Not auto cherry-pick when PR merged.
  - /pick: release-x.x [release-x.x]: Auto cherry-pick to the specified branch when PR merged.

Example:
  - /nopick
  - /pick release-1.1

CLA Recheck Instructions

Usage:
  - /recheck-cla: Trigger a re-check of CLA status for this pull request.
Example:
  - /recheck-cla

@github-actions github-actions Bot added the size/M Denotes a PR that changes 30-99 lines. label Jul 5, 2026
@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.67%. Comparing base (ecd604d) to head (d8c83ea).
⚠️ Report is 15 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10544      +/-   ##
==========================================
- Coverage   63.68%   63.67%   -0.01%     
==========================================
  Files         519      519              
  Lines       62852    62854       +2     
==========================================
- Hits        40027    40023       -4     
- Misses      19202    19211       +9     
+ Partials     3623     3620       -3     
Flag Coverage Δ
unittests 63.67% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leon-ape leon-ape changed the title fix(operations): fail rebuild fast when the referenced backup is missing fix(ops): fail rebuild fast when the referenced backup is missing Jul 10, 2026
Comment thread pkg/operations/rebuild_instance.go Outdated
Comment thread pkg/operations/rebuild_instance.go
Address the 2026-07-10 review round: an ActionSet is a cluster-scoped
resource supplied by the addon and can be temporarily absent during
addon installation/upgrade/reconciliation, so a NotFound on the lookup
cannot prove the backup artifact is invalid. Drop the fatal
classification for the ActionSet lookup (restoring the plain retryable
error) and keep only the backupName fatal, which is supplied directly
by the OpsRequest. The regression now covers retry-while-absent and
convergence once the ActionSet is installed; it fails on the previous
head (fatal marked the instances Failed) and passes here.
@leon-ape leon-ape added the nopick Not auto cherry-pick when PR merged label Jul 14, 2026
@apecloud-bot apecloud-bot added the approved PR Approved Test label Jul 14, 2026
@leon-ape

Copy link
Copy Markdown
Collaborator

[P2, non-blocking] The new regression test consumes an ActionSet before its owner has published a valid current-generation result.

The test creates an ActionSet with no status.phase or status.observedGeneration and immediately expects rebuild to proceed (pkg/operations/rebuild_instance_test.go:932). ActionSet exposes those status fields specifically so its controller can report whether the current spec is valid. Please make the fixture current-generation Available before asserting that rebuild consumes it; otherwise the test codifies the existing implementation detail that object existence alone is sufficient.

The previous production issue where ActionSet NotFound was made terminal has been correctly reverted on this head; there is no remaining production blocker from that feedback.

@weicao
weicao marked this pull request as draft July 14, 2026 07:30
@apecloud-bot apecloud-bot removed the approved PR Approved Test label Jul 14, 2026
@weicao

weicao commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Handled the non-blocking fixture gap in exact head 4687616203fcff6c214ca002e6db3e58393dc76a. The late-installed ActionSet is now patched to status.phase=Available with status.observedGeneration == metadata.generation before the regression expects rebuild consumption. No production behavior was added; the prior ActionSet-NotFound fatal path remains reverted. Focused regression, full pkg/operations, go vet ./pkg/operations, and diff-check pass locally. The PR remains Draft for exact-head CI/review verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nopick Not auto cherry-pick when PR merged size/M Denotes a PR that changes 30-99 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RebuildInstance ops with nonexistent backupName retries forever in the in-place path

3 participants