Provider stalls and stops processing volume create requests when volume deletion fails

[coderabbitai]

Summary

When a volume deletion fails (e.g., due to a Ceph-level error), the provider eventually stalls and stops processing any new volume create requests.

Reported in: #862
Reported by: @afritzler

---

Root Cause Analysis

After reviewing the code, there are two likely root causes, with the first being the primary one.

Root Cause 1 (Primary): Worker Thread Exhaustion via Blocking img.Flatten()

The ImageReconciler uses a fixed pool of 15 worker goroutines to process queue items. The deletion path calls a blocking Ceph RBD operation (img.Flatten()) that has no timeout and no context cancellation support:

reconcileImage
  └─ deleteImage
       └─ deleteImageSnapshots (image_controller.go)
            └─ flattenChildImages (common.go:148)
                 └─ flattenImage (common.go:85)
                      └─ img.Flatten()  ← blocks indefinitely, no context/timeout

If the Ceph cluster is degraded, overloaded, or the flatten operation encounters issues, img.Flatten() can hang indefinitely. Since each stuck worker holds the goroutine for the lifetime of the flatten operation:

1. Concurrent or repeated volume deletion failures cause multiple workers to become stuck inside img.Flatten().
2. Once all 15 workers are occupied in blocking I/O, the workqueue has no available workers to dequeue items.
3. New volume create events are added to the queue but never processed — the provider appears stalled.

Note that the same issue exists in the SnapshotReconciler's deleteSnapshot path.

Root Cause 2 (Secondary): Unbounded Retries Without a Worker Guard

In processNextWorkItem (image_controller.go), failed reconciliations are re-queued using AddRateLimited with no cap on the number of retries:

if err := r.reconcileImage(ctx, id); err != nil {
    log.Error(err, "failed to reconcile image")
    r.queue.AddRateLimited(id)  // retried forever
    return true
}

The DefaultTypedControllerRateLimiter uses exponential backoff with a max delay of 1000 seconds (≈16 minutes), but never gives up. Combined with the blocking flatten issue above, a deletion item can re-enter a worker and block it again after each backoff interval.

Root Cause 3 (Secondary): Silent Event Drop on Unbuffered Watch Channel

In internal/omap/omap.go, the watch channel is unbuffered and events are dispatched with a non-blocking send:

func (s *Store[E]) enqueue(evt store.WatchEvent[E]) {
    for _, handler := range s.watchHandlers() {
        select {
        case handler.events <- evt:
        default:  // silently dropped if no immediate receiver
        }
    }
}

If all workers are busy and the event handler goroutine is momentarily unavailable, new create events can be silently dropped. Because the failing delete item is kept alive via AddRateLimited, it will eventually be retried — but a newly created volume that misses its initial event has no other trigger and may never be processed.

---

Suggested Fixes

1. Add context/timeout to flattenImage: Pass a context.Context with a reasonable timeout to the flatten call chain so that a hanging Ceph operation does not permanently hold a worker goroutine.
2. Add a maximum retry limit in processNextWorkItem: Use queue.NumRequeues(id) to give up after N retries and move the item to a terminal failed state, rather than retrying indefinitely.
3. Buffer the watch channel in omap.Store to reduce the chance of silent event drops for new create requests.
4. Consider dynamically increasing workerSize or separating delete and create work queues so that stuck deletions cannot starve create operations.

---

References

• internal/controllers/image_controller.go — processNextWorkItem, deleteImage, deleteImageSnapshots
• internal/controllers/common.go — flattenChildImages, flattenImage
• internal/omap/omap.go — enqueue, Watch
• internal/controllers/snapshot_controller.go — deleteSnapshot