fix: make SecretProviderClassPodStatus reconciliation idempotent - #2060
fix: make SecretProviderClassPodStatus reconciliation idempotent#2060pujitha24 wants to merge 1 commit into
Conversation
Motivation: createOrUpdateSecretProviderClassPodStatus unconditionally attempted a Create on every reconcile and, once the object existed, unconditionally ran a Get followed by an Update - even when the node label, Status, and OwnerReferences already matched the desired state. In steady state (repeated NodePublishVolume/republish activity for an unchanged mount), this produced a POST returning 409 AlreadyExists followed by an unconditional PUT on every reconcile, adding unnecessary write traffic to the API server without changing any observable behavior. Approach: Read the existing object first (falling back to a direct API read if the cache doesn't have it, since it may be labeled for a different node and excluded from this node's cache), compare the node label, Status, and OwnerReferences against the desired values, and return without writing when they already match. Create only when an authoritative read confirms the object is absent; update only when something actually differs. Wrap the read/compare/write attempt in a bounded retry.OnError (k8s.io/client-go/util/retry, retry.DefaultBackoff) that retries on AlreadyExists (lost create race) or Conflict (lost update race) with a fresh read on each attempt. User-visible behavior is unchanged - the object still converges to the same desired state. The benefit is narrower: fewer redundant write requests against the API server when nothing has changed. Validation: - go build ./... passes. - go vet ./pkg/secrets-store/... is clean. - go test ./pkg/secrets-store/... -run TestCreateOrUpdateSecretProviderClassPodStatus -v passes, including a new regression test, TestCreateOrUpdateSecretProviderClassPodStatus_NoOpWhenUpToDate, which seeds a fake client with an object that already matches the desired state and fails the test if Create or Update is ever called. This test fails against the pre-fix code (both an unexpected Create and an unexpected Update fire) and passes against the post-fix code - a targeted reproduction of the defect described in the issue. - The existing create/update-different-node-label subtests in TestCreateOrUpdateSecretProviderClassPodStatus still pass unmodified, confirming genuinely new or changed objects still take the create/update path correctly. - Not run: a live cluster reproduction of the exact API server request sequence described in the issue. This is a controller-logic fix validated with a unit test that reproduces the underlying defect (unconditional write on an unchanged object); it does not depend on cluster or provider-specific runtime behavior. Report: kubernetes-sigs#2058 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
|
This issue is currently awaiting triage. If secrets-store-csi-driver contributors determine this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Hi @pujitha24. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: pujitha24 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/assign @aramase This is ready for review whenever you have a chance — happy to make any adjustments. |
What type of PR is this?
/kind bug
What this PR does / why we need it:
createOrUpdateSecretProviderClassPodStatusunconditionally attempted aCreateon everyreconcile, and once the object already existed, unconditionally ran a
Getfollowed by aPut(
Update) even when the label,Status, andOwnerReferencesalready matched the desired state.In steady state (repeated
NodePublishVolume/republish activity for an unchanged mount), thisproduces a
POSTreturning409 AlreadyExistsfollowed by an unconditionalPUTon everyreconcile, adding unnecessary write traffic to the API server without changing any observable
behavior.
This PR makes the helper idempotent: it reads the existing object first (falling back to a direct
API read if the cache doesn't have it, since it may be labeled for a different node and therefore
excluded from this node's cache), compares the node label,
Status, andOwnerReferencesagainstthe desired values, and returns without writing when they already match. It only creates when an
authoritative read confirms the object is absent, and only updates when something actually
differs. A bounded retry (
k8s.io/client-go/util/retry,retry.OnErrorwithretry.DefaultBackoff)handles genuine races -
AlreadyExistsfrom a concurrent create, orConflictfrom a concurrentupdate - by re-reading and retrying the compare/write with a fresh object.
User-visible behavior is unchanged: the object still ends up in the same desired state. The benefit
is narrower - fewer redundant write requests against the API server when nothing has changed.
Which issue(s) this PR fixes (optional, using
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when the PR gets merged):Fixes #
Special notes for your reviewer:
Validation performed locally (no live cluster available in this environment):
go build ./...passes.go vet ./pkg/secrets-store/...is clean.go test ./pkg/secrets-store/... -run TestCreateOrUpdateSecretProviderClassPodStatus -vpasses,including a new regression test,
TestCreateOrUpdateSecretProviderClassPodStatus_NoOpWhenUpToDate, which seeds a fake client withan object that already matches the desired node label, status, and owner references, and wires
interceptor.Funcsto fail the test ifCreateorUpdateis ever called. This test FAILSagainst the pre-fix code (both an unexpected
Createattempt onAlreadyExistsand anunexpected
Updatecall fire) and PASSES against the post-fix code, which is the concretereproduction of the bug described in the issue.
TestCreateOrUpdateSecretProviderClassPodStatuscreate/update-different-node-labelsubtests still pass unmodified, confirming genuinely new or changed objects still take the
create/update path correctly.
POST 409+PUTAPI server trafficpattern described in the issue. The change is a pure controller-logic fix validated with a
targeted unit test that reproduces the defect (unconditional write on an unchanged object) and
proves the fix; it does not depend on cluster or provider-specific runtime behavior.
TODOs:
Fixes #2058