fix: treat empty existing mount as stale in ensureMountPoint by OS-gustavosalvador · Pull Request #2025 · kubernetes-sigs/secrets-store-csi-driver

OS-gustavosalvador · 2026-04-23T15:45:23Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

On Linux, ensureMountPoint returned mounted=true for any existing mount whose directory could be read; no content check.

If NodePublishVolume was interrupted between mounting the tmpfs and writing the provider's files (driver pod killed, context cancelled), the empty tmpfs persisted. The retry short-circuited on "already mounted" and the pod started with an empty secrets directory. Readiness never recovered without manual intervention.

The fix: if the existing mount is empty, unmount it and return mounted=false so the caller performs a fresh mount. The Windows branch below already enforces the same contract via a different mechanism.

Reproduced in a local k3d cluster by killing the driver pod mid-Mount. Same branch with utils.go reverted to its parent: 0/3 runs recover. With the fix: 5/5 runs Ready within seconds of the next retry, zero restarts.

Which issue(s) this PR fixes:

Fixes #1051

TODOs:

squashed commits
includes documentation
adds unit tests

k8s-ci-robot · 2026-04-23T15:45:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: OS-gustavosalvador
Once this PR has been reviewed and has the lgtm label, please assign aramase for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2026-04-23T15:45:32Z

This issue is currently awaiting triage.

If secrets-store-csi-driver contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2026-04-23T15:45:33Z

Welcome @OS-gustavosalvador!

It looks like this is your first PR to kubernetes-sigs/secrets-store-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/secrets-store-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

linux-foundation-easycla · 2026-04-23T15:45:33Z

The committers listed above are authorized under a signed CLA.

✅ login: OS-gustavosalvador / name: OS-gustavosalvador (8ef87d1)

k8s-ci-robot · 2026-04-23T15:45:34Z

Hi @OS-gustavosalvador. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

stlaz

Pre-review: please reduce the cognitive load of the PR for maintainers by hand-rewriting all of What this PR does / why we need it: and Special notes for your reviewer: and the code comments and the commit message.

Remove issue links from the code comments.

kubernetes-prow · 2026-06-24T07:37:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: OS-gustavosalvador
Once this PR has been reviewed and has the lgtm label, please assign aramase for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

On Linux, when the target is already mounted, check whether the directory is empty and unmount it if so, so the caller performs a fresh mount. Matches the contract the Windows branch already enforces and prevents the retry from reporting success on a stale empty tmpfs left by an interrupted NodePublishVolume. Signed-off-by: Gustavo Salvador <gustavo.salvador@outsystems.com>

OS-gustavosalvador · 2026-06-25T12:45:11Z

thanks @stlaz, adjusted as suggested 👍

stlaz

I'm kind of new to the code base, curious what @aramase thinks here.

stlaz · 2026-06-29T11:36:37Z

/ok-to-test

Signal mount state to the caller instead of unmounting; the caller reuses the existing tmpfs and re-calls the provider. Signed-off-by: Gustavo Salvador <gustavo.salvador@outsystems.com>

OS-gustavosalvador · 2026-07-01T11:24:44Z

/retest

Signed-off-by: OS-gustavosalvador <gustavo.salvador@outsystems.com>

OS-gustavosalvador · 2026-07-01T11:31:13Z

/retest

stlaz

looks good, mostly nits

stlaz · 2026-08-20T11:04:24Z

-	// If it is mounted, it means this is not the first time mount request for this path.
-	isRemountRequest = mounted
+	// An empty existing mount is a previous interrupted call, not a remount.
+	isRemountRequest = mounted && hasContent


it's only remount request if it's mounted, original was more accurate.

also looking at how the var is used elsewhere, adding "hasContent" might be wrong I think?

stlaz · 2026-08-20T11:05:21Z

+	if !rotationEnabled && isRemountRequest {
 		klog.InfoS("target path is already mounted", "targetPath", targetPath, "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName})
 		skipped = true


Suggested change

if !rotationEnabled && isRemountRequest {

klog.InfoS("target path is already mounted", "targetPath", targetPath, "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName})

skipped = true

if !rotationEnabled && mounted && hasContent {

klog.InfoS("target path is already mounted and populated", "targetPath", targetPath, "pod", klog.ObjectRef{Namespace: podNamespace, Name: podName})

skipped = true

@aramase is there any possibility that no content is the actual desired state?

stlaz · 2026-08-20T11:10:10Z

-			return !notMnt, nil
+		entries, readErr := os.ReadDir(target)
+		if readErr == nil {
+			return true, len(entries) > 0, nil


let's maybe keep the original log line but move it to V(4)

stlaz · 2026-08-20T11:13:19Z

-		// instead check if the dir exists for windows and if it's not empty
-		// If there are contents in the dir, then objects are already mounted
-		f, err := os.ReadDir(target)
+		// IsLikelyNotMountPoint always returns notMnt=true on Windows;


keep the original comment

stlaz · 2026-08-20T11:16:38Z

-// ensureMountPoint ensures mount point is valid
-func (ns *nodeServer) ensureMountPoint(target string) (bool, error) {
+// ensureMountPoint reports whether the target is mounted and whether
+// it has any content. The caller decides whether to mount, populate,


don't make assumptions on what the caller wants to do with the return vals in the docs here

Signed-off-by: OS-gustavosalvador <gustavo.salvador@outsystems.com>

OS-gustavosalvador · 2026-08-20T15:11:50Z

thanks again @stlaz, updated the code👍

OS-gustavosalvador · 2026-08-20T15:12:32Z

/retest

kubernetes-prow · 2026-08-20T15:15:12Z

@OS-gustavosalvador: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-secrets-store-csi-driver-image-scan	`fbc5bf3`	link	false	`/test pull-secrets-store-csi-driver-image-scan`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. labels Apr 23, 2026

k8s-ci-robot requested review from aramase and enj April 23, 2026 15:45

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Apr 23, 2026

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 23, 2026

OS-gustavosalvador force-pushed the fix/1051-stale-empty-mount branch from 8ef87d1 to de8cc58 Compare April 23, 2026 16:11

k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Apr 23, 2026

enj moved this to Subprojects - Needs Triage in SIG Auth Apr 28, 2026

enj added this to SIG Auth Apr 28, 2026

raelga mentioned this pull request May 21, 2026

fix(arobit): recover mdsd pods from CSI mount race (AROSLSRE-879) Azure/ARO-HCP#5355

Merged

stlaz suggested changes Jun 24, 2026

View reviewed changes

github-project-automation Bot moved this from Subprojects - Needs Triage to Changes Requested in SIG Auth Jun 24, 2026

OS-gustavosalvador force-pushed the fix/1051-stale-empty-mount branch from de8cc58 to 8c5af2b Compare June 25, 2026 12:44

OS-gustavosalvador requested a review from stlaz June 25, 2026 12:46

stlaz reviewed Jun 29, 2026

View reviewed changes

Comment thread pkg/secrets-store/utils.go Outdated

kubernetes-prow Bot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jun 29, 2026

kubernetes-prow Bot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 29, 2026

refactor: make ensureMountPoint side-effect-free

e9436e0

Signal mount state to the caller instead of unmounting; the caller reuses the existing tmpfs and re-calls the provider. Signed-off-by: Gustavo Salvador <gustavo.salvador@outsystems.com>

fix lint: use 0600 file mode in TestEnsureMountPoint

539013c

Signed-off-by: OS-gustavosalvador <gustavo.salvador@outsystems.com>

OS-gustavosalvador requested a review from stlaz July 7, 2026 13:02

stlaz reviewed Aug 20, 2026

View reviewed changes

fix: review comments

fbc5bf3

Signed-off-by: OS-gustavosalvador <gustavo.salvador@outsystems.com>

Uh oh!

Conversation

OS-gustavosalvador commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 23, 2026

Uh oh!

k8s-ci-robot commented Apr 23, 2026

Uh oh!

k8s-ci-robot commented Apr 23, 2026

Uh oh!

linux-foundation-easycla Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 23, 2026

Uh oh!

stlaz left a comment

Choose a reason for hiding this comment

Uh oh!

kubernetes-prow Bot commented Jun 24, 2026

Uh oh!

OS-gustavosalvador commented Jun 25, 2026

Uh oh!

stlaz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stlaz commented Jun 29, 2026

Uh oh!

OS-gustavosalvador commented Jul 1, 2026

Uh oh!

OS-gustavosalvador commented Jul 1, 2026

Uh oh!

stlaz left a comment

Choose a reason for hiding this comment

Uh oh!

stlaz Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

stlaz Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

stlaz Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stlaz Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

stlaz Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

stlaz Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

stlaz Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

OS-gustavosalvador commented Aug 20, 2026

Uh oh!

OS-gustavosalvador commented Aug 20, 2026

Uh oh!

kubernetes-prow Bot commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

OS-gustavosalvador commented Apr 23, 2026 •

edited

Loading

linux-foundation-easycla Bot commented Apr 23, 2026 •

edited

Loading

stlaz Aug 20, 2026 •

edited

Loading