fix-1828: started work

[wind57]

No description provided.

[wind57]

@ryanjbaxter can you trigger copilot in here please?

[wind57]

in the new IT that I added, there is a need for two replicas, to really test the HA set-up

[wind57]

two properties are needed to enable HA

[wind57]

I've added a single IT, that goes through a cycle of leader / no leader / leader

[wind57]

this is the definition of the resource version store. Methods in this one are executed only by the leader

[wind57]

if HA is enabled and this is the first call, this will create an empty lease. Otherwise, it will read whatever is stored there.

We store in the spring.cloud.kubernetes.configuration.watcher/configmap-resource-version annotation, something like : "default=123,prod=456", so each namespace tracks its own resource version checkpoint.

We need such a store because during a downtime when there is no leader at all, resource version can progress ( meaning configmap is updated ), but since there is no leader, events can get lost. As such, we always increment and "store" ( via this implementation ) the most recent resource version we have observed. This happens in the handlers onAdd / onDelete / onUpdate.

So for example:

• we are now the leader and the resourceVersion is at 1.
• we lose leadership, so our store stays at 1.
• configmap progresses to resourceVersion=2
• another leader is elected, it reads the store, sees that it holds 1
• starts the informer at resourceVersion=1, so it can replay events it has not seen

[wind57]

these events are triggered only when the instance became the leader

[wind57]

after every update that we receive in the handler, write the resource version to the store

[wind57]

With HA enabled:

• @PostConstruct does nothing, so the informers are not started.

• The native leader-election callback emits StartLeadingEvent.

• ConfigurationWatcherHACoordinator receives that event.

• It calls stateStore.readOrCreate()

  • If the HA state Lease does not exist, it creates it and returns an empty state.
  • Otherwise, it reads the stored ConfigMap and Secret resource versions.

• The coordinator calls start(...) on the available detectors, passing:

  • the stored resource versions
  • a writer that persists future resource versions

• Each detector creates its InformerResourceVersionResolver.

• For each watched namespace, the informer sends its initial LIST request:

  • if a stored checkpoint exists, the request starts from that resource version
  • otherwise, it starts normally

• The informer then starts its WATCH.

• Resource events are handled normally:

  • ConfigMap or Secret change is processed
  • refresh or bus notification is triggered
  • the processed resource version is written to the HA state Lease

• Later LIST/WATCH requests use the resource version supplied by the informer, not the stored checkpoint.

• If the leader loses leadership, StopLeadingEvent is emitted.

• The coordinator stops both detectors.

• The next leader reads the persisted state and repeats the process from the stored checkpoints.

[wind57]

TL;DR

This PR adds high-availability support to the Kubernetes client-based Configuration Watcher.

The Configuration Watcher can now run with multiple replicas while ensuring that only one replica actively watches ConfigMaps and Secrets at a time. If the leader fails, another replica acquires leadership, restores the persisted informer checkpoints, and continues processing changes.

How It Works

HA requires both properties to be enabled:

spring:
  cloud:
    kubernetes:
      leader:
        election:
          enabled: true
      configuration:
        watcher:
          ha:
            enabled: true

The two properties have separate responsibilities:

• spring.cloud.kubernetes.leader.election.enabled enables native Kubernetes leader election.
• spring.cloud.kubernetes.configuration.watcher.ha.enabled makes the Configuration Watcher start and stop its informers according to leadership.

When HA is disabled, the existing behavior is unchanged. ConfigMap and Secret informers start during normal bean initialization.

When HA is enabled:

• Informers are not started during initialization.
• The leader-election callback notifies the Configuration Watcher that leadership was acquired.
• The Configuration Watcher reads its persisted state.
• ConfigMap and Secret informers start using the stored resource-version checkpoints.
• When leadership is lost, both informers are stopped.
• The next leader restores the checkpoints and resumes watching.

The HA coordinator is initialized before the leader-election callbacks so that it is ready to receive leadership events.

Persistent State

The Configuration Watcher stores its state in a Kubernetes Lease. The leader-election lock and the Configuration Watcher state lease are separate resources.

The state lease stores the last processed resource version independently for:

• each watched namespace
• ConfigMaps
• Secrets

The values are stored in annotations on the lease. For example:

spring.cloud.kubernetes.configuration.watcher/configmap-resource-version:
  default=123,other-namespace=456

The default configuration is:

spring:
  cloud:
    kubernetes:
      configuration:
        watcher:
          ha:
            enabled: false
            lease-name: configuration-watcher-ha
            lease-namespace: default

The lease is created automatically when the first leader starts, if it does not already exist.

Resource-Version Replay

• When a new leader starts, the persisted resource version is used only for the first informer request for each namespace. This request restores the informer from the exact stored checkpoint.
• After that first request, the informer controls its own resource-version progression. Each subsequent list or watch request uses the resource version supplied by the informer.
• ConfigMap and Secret namespaces are tracked independently, so different namespaces can progress at different resource versions.
• Resource versions are persisted after the corresponding resource event has been handled. This provides at-least-once event processing:
  • If a leader processes an event and persists the checkpoint successfully, the next leader continues after that event.
  • If the leader fails after processing the event but before persisting the checkpoint, the replacement leader may process the event again.

Repeated refresh notifications must therefore be tolerated by refresh targets.

• The watcher does not refresh every ConfigMap or Secret after a leadership change. It only replays events for the configured informer scopes.

• At least two Configuration Watcher replicas are required for failover:

spec:
  replicas: 2

[wind57]

@ryanjbaxter in case you missed this... can you trigger copilot please?

[ryanjbaxter]

Have no missed it, just need to set aside some time to look at it. Probably next week

[wind57]

ok, thank you! as usual, Im available on any ways to answer any questions.

[Copilot]

Pull request overview

Adds High Availability (HA) support for the Kubernetes-client Configuration Watcher by deferring informer startup until leadership is acquired, persisting last-processed informer resourceVersions in a Kubernetes Lease, and replaying missed events after leader loss. This extends the configuration-watcher controller with HA auto-configuration and introduces new integration/unit tests to validate lease-backed checkpointing and leader failover behavior.

Changes:

• Introduce HA coordinator + lease-backed state store to persist and restore informer checkpoints (ConfigMap/Secret resourceVersions).
• Update ConfigMap/Secret event-based change detectors to support HA mode (deferred start, explicit start/stop, checkpoint replay).
• Add/adjust integration and unit tests plus documentation for HA enablement, RBAC, and deployment replicas.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
spring-cloud-kubernetes-test-support/src/main/java/org/springframework/cloud/kubernetes/integration/tests/commons/native_client/NativeClientKubernetesFixture.java	Add HA env vars + configurable replicas; wait for expected replica count.
spring-cloud-kubernetes-test-support/src/main/java/org/springframework/cloud/kubernetes/integration/tests/commons/k3s/NativeClientIntegrationTestExtension.java	Wire scenario HA flags/replicas into fixture calls.
spring-cloud-kubernetes-test-support/src/main/java/org/springframework/cloud/kubernetes/integration/tests/commons/k3s/NativeClientIntegrationTest.java	Extend test annotation with HA + replicas options.
spring-cloud-kubernetes-integration-tests/spring-cloud-kubernetes-k8s-client-reload/src/test/java/org/springframework/cloud/kubernetes/k8s/client/reload/it/K8sClientConfigMapLabelEventTriggeredIT.java	Update asserted log message for informer startup.
spring-cloud-kubernetes-integration-tests/spring-cloud-kubernetes-k8s-client-reload/src/test/java/org/springframework/cloud/kubernetes/k8s/client/reload/it/K8sClientConfigMapEventTriggeredIT.java	Update asserted log message for informer startup.
spring-cloud-kubernetes-integration-tests/spring-cloud-kubernetes-k8s-client-configuration-watcher/src/test/java/org/springframework/cloud/kubernetes/configuration/watcher/KubernetesClientConfigurationWatcherHaIT.java	New k3s-based HA integration test (leader lease + replay).
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/test/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/LeaseConfigurationWatcherStateStoreTests.java	New unit tests for lease read/create/update failure modes.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/test/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConfigurationWatcherHACoordinatorTests.java	New unit tests for leader start/stop wiring and validation.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/test/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConfigurationWatcherHAAutoConfigurationTests.java	New auto-config tests for HA coordinator conditional creation.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/test/java/org/springframework/cloud/kubernetes/configuration/watcher/ConfigurationWatcherConfigurationPropertiesTests.java	Verify new HA properties defaults + binding.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports	Register HA auto-configuration.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/SecretsWatcherChangeDetector.java	Pass HA enabled flag into base change detector.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/LeaseConfigurationWatcherStateStore.java	New lease-backed state store implementation.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConfigurationWatcherStateStore.java	New state store interface.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConfigurationWatcherState.java	New state record (per-namespace checkpoints).
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConfigurationWatcherHaProperties.java	New HA properties (enabled, lease name/namespace).
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConfigurationWatcherHACoordinator.java	New coordinator listening to leader election events.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConfigurationWatcherHAAutoConfiguration.java	New HA auto-config (store + coordinator beans).
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/ConditionalOnConfigurationWatcherHAEnabled.java	New conditional annotation for HA enablement.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ConfigurationWatcherConfigurationProperties.java	Add HA properties to watcher configuration properties.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ConfigMapWatcherChangeDetector.java	Pass HA enabled flag into base change detector.
spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/pom.xml	Add leader-election dependency for HA support.
spring-cloud-kubernetes-client-config/src/test/java/org/springframework/cloud/kubernetes/client/config/reload/KubernetesClientEventBasedSecretsChangeDetectorTests.java	Expand tests for HA deferred start + resourceVersion behavior.
spring-cloud-kubernetes-client-config/src/test/java/org/springframework/cloud/kubernetes/client/config/reload/KubernetesClientEventBasedConfigMapChangeDetectorTests.java	Expand tests for HA deferred start + resourceVersion behavior.
spring-cloud-kubernetes-client-config/src/test/java/org/springframework/cloud/kubernetes/client/config/reload/InformerResourceVersionResolverTests.java	New unit tests for resolver semantics in HA/non-HA.
spring-cloud-kubernetes-client-config/src/main/java/org/springframework/cloud/kubernetes/client/config/reload/SecretResourceEventHandler.java	Add optional resourceVersion writer callback.
spring-cloud-kubernetes-client-config/src/main/java/org/springframework/cloud/kubernetes/client/config/reload/NamespaceAndResourceVersion.java	New value record for persisted checkpoints.
spring-cloud-kubernetes-client-config/src/main/java/org/springframework/cloud/kubernetes/client/config/reload/KubernetesClientEventBasedSecretsChangeDetector.java	Add HA mode: deferred informers + explicit start/stop + replay logic.
spring-cloud-kubernetes-client-config/src/main/java/org/springframework/cloud/kubernetes/client/config/reload/KubernetesClientEventBasedConfigMapChangeDetector.java	Add HA mode: deferred informers + explicit start/stop + replay logic.
spring-cloud-kubernetes-client-config/src/main/java/org/springframework/cloud/kubernetes/client/config/reload/InformerResourceVersionResolver.java	New helper to consume checkpoints once per namespace.
spring-cloud-kubernetes-client-config/src/main/java/org/springframework/cloud/kubernetes/client/config/reload/ConfigMapResourceEventHandler.java	Add optional resourceVersion writer callback.
docs/modules/ROOT/pages/spring-cloud-kubernetes-configuration-watcher.adoc	Document HA enablement, leases, RBAC, and replicas.

Suppressed comments (2)

spring-cloud-kubernetes-client-config/src/main/java/org/springframework/cloud/kubernetes/client/config/reload/KubernetesClientEventBasedSecretsChangeDetector.java:121

• Log message mentions "configmap informer" in the Secrets change detector; should refer to secret informers to avoid confusion in HA mode.

			LOG.info(() -> "config watcher HA is enabled : deferring configmap informer startup "
					+ "until leadership is acquired");

spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configuration-watcher/src/main/java/org/springframework/cloud/kubernetes/configuration/watcher/ha/LeaseConfigurationWatcherStateStore.java:166

• parseResourceVersions() throws "Invalid ConfigMap resource version entry" even when parsing the Secret annotation, which can mislead debugging. The message should be resource-type agnostic.

				throw new IllegalStateException("Invalid ConfigMap resource version entry: " + entry);
			}

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.