Skip to content

fix: make SecretProviderClassPodStatus reconciliation idempotent - #2060

Open
pujitha24 wants to merge 1 commit into
kubernetes-sigs:mainfrom
pujitha24:auto/issue-2058
Open

fix: make SecretProviderClassPodStatus reconciliation idempotent#2060
pujitha24 wants to merge 1 commit into
kubernetes-sigs:mainfrom
pujitha24:auto/issue-2058

Conversation

@pujitha24

Copy link
Copy Markdown

What type of PR is this?

/kind bug

What this PR does / why we need it:

createOrUpdateSecretProviderClassPodStatus unconditionally attempted a Create on every
reconcile, and once the object already existed, unconditionally ran a Get followed by a Put
(Update) even when the label, Status, and OwnerReferences already matched the desired state.
In steady state (repeated NodePublishVolume/republish activity for an unchanged mount), this
produces a POST returning 409 AlreadyExists followed by an unconditional PUT on every
reconcile, adding unnecessary write traffic to the API server without changing any observable
behavior.

This PR makes the helper idempotent: it reads the existing object first (falling back to a direct
API read if the cache doesn't have it, since it may be labeled for a different node and therefore
excluded from this node's cache), compares the node label, Status, and OwnerReferences against
the desired values, and returns without writing when they already match. It only creates when an
authoritative read confirms the object is absent, and only updates when something actually
differs. A bounded retry (k8s.io/client-go/util/retry, retry.OnError with retry.DefaultBackoff)
handles genuine races - AlreadyExists from a concurrent create, or Conflict from a concurrent
update - by re-reading and retrying the compare/write with a fresh object.

User-visible behavior is unchanged: the object still ends up in the same desired state. The benefit
is narrower - fewer redundant write requests against the API server when nothing has changed.

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #

Special notes for your reviewer:

Validation performed locally (no live cluster available in this environment):

  • go build ./... passes.
  • go vet ./pkg/secrets-store/... is clean.
  • go test ./pkg/secrets-store/... -run TestCreateOrUpdateSecretProviderClassPodStatus -v passes,
    including a new regression test,
    TestCreateOrUpdateSecretProviderClassPodStatus_NoOpWhenUpToDate, which seeds a fake client with
    an object that already matches the desired node label, status, and owner references, and wires
    interceptor.Funcs to fail the test if Create or Update is ever called. This test FAILS
    against the pre-fix code (both an unexpected Create attempt on AlreadyExists and an
    unexpected Update call fire) and PASSES against the post-fix code, which is the concrete
    reproduction of the bug described in the issue.
  • The pre-existing TestCreateOrUpdateSecretProviderClassPodStatus create/update-different-node-label
    subtests still pass unmodified, confirming genuinely new or changed objects still take the
    create/update path correctly.
  • Not run: a live cluster / e2e reproduction of the exact POST 409 + PUT API server traffic
    pattern described in the issue. The change is a pure controller-logic fix validated with a
    targeted unit test that reproduces the defect (unconditional write on an unchanged object) and
    proves the fix; it does not depend on cluster or provider-specific runtime behavior.

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests

Fixes #2058

fix: make SecretProviderClassPodStatus reconciliation idempotent

Motivation:
createOrUpdateSecretProviderClassPodStatus unconditionally attempted a
Create on every reconcile and, once the object existed, unconditionally
ran a Get followed by an Update - even when the node label, Status, and
OwnerReferences already matched the desired state. In steady state
(repeated NodePublishVolume/republish activity for an unchanged mount),
this produced a POST returning 409 AlreadyExists followed by an
unconditional PUT on every reconcile, adding unnecessary write traffic
to the API server without changing any observable behavior.

Approach:
Read the existing object first (falling back to a direct API read if
the cache doesn't have it, since it may be labeled for a different
node and excluded from this node's cache), compare the node label,
Status, and OwnerReferences against the desired values, and return
without writing when they already match. Create only when an
authoritative read confirms the object is absent; update only when
something actually differs. Wrap the read/compare/write attempt in a
bounded retry.OnError (k8s.io/client-go/util/retry, retry.DefaultBackoff)
that retries on AlreadyExists (lost create race) or Conflict (lost
update race) with a fresh read on each attempt.

User-visible behavior is unchanged - the object still converges to the
same desired state. The benefit is narrower: fewer redundant write
requests against the API server when nothing has changed.

Validation:
- go build ./... passes.
- go vet ./pkg/secrets-store/... is clean.
- go test ./pkg/secrets-store/... -run TestCreateOrUpdateSecretProviderClassPodStatus -v
  passes, including a new regression test,
  TestCreateOrUpdateSecretProviderClassPodStatus_NoOpWhenUpToDate, which
  seeds a fake client with an object that already matches the desired
  state and fails the test if Create or Update is ever called. This
  test fails against the pre-fix code (both an unexpected Create and an
  unexpected Update fire) and passes against the post-fix code - a
  targeted reproduction of the defect described in the issue.
- The existing create/update-different-node-label subtests in
  TestCreateOrUpdateSecretProviderClassPodStatus still pass unmodified,
  confirming genuinely new or changed objects still take the
  create/update path correctly.
- Not run: a live cluster reproduction of the exact API server request
  sequence described in the issue. This is a controller-logic fix
  validated with a unit test that reproduces the underlying defect
  (unconditional write on an unchanged object); it does not depend on
  cluster or provider-specific runtime behavior.

Report: kubernetes-sigs#2058
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
@kubernetes-prow kubernetes-prow Bot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 9, 2026
@kubernetes-prow

Copy link
Copy Markdown

This issue is currently awaiting triage.

If secrets-store-csi-driver contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 9, 2026
@kubernetes-prow

Copy link
Copy Markdown

Hi @pujitha24. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 9, 2026
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pujitha24
Once this PR has been reviewed and has the lgtm label, please assign aramase for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow
kubernetes-prow Bot requested review from aramase and enj August 9, 2026 10:32
@enj enj added this to SIG Auth Aug 9, 2026
@enj enj moved this to Subprojects - Needs Triage in SIG Auth Aug 9, 2026
@pujitha24

Copy link
Copy Markdown
Author

/assign @aramase

This is ready for review whenever you have a chance — happy to make any adjustments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SecretProviderClassPodStatus reconciliation unconditionally creates and updates existing objects

3 participants