Skip to content

xds: Fail CDS cluster when EDS update has no usable endpoints - #13142

Open
lujiajing1126 wants to merge 1 commit into
grpc:masterfrom
lujiajing1126:xds-cds-empty-endpoints-fail
Open

lujiajing1126 wants to merge 1 commit into
grpc:masterfrom
lujiajing1126:xds-cds-empty-endpoints-fail

Conversation

@lujiajing1126

Copy link
Copy Markdown
Contributor

Fixes #13141

When a previously READY EDS cluster gets an update with no usable endpoints (e.g. an empty ClusterLoadAssignment, or all endpoints unhealthy), CdsLoadBalancer2 called delegate.handleNameResolutionError(). The priority/MultiChildLoadBalancer subtree ignores that while READY, so the stale picker kept selecting endpoints the control plane had removed. Before ClusterResolverLoadBalancer was merged into CdsLoadBalancer2 (ebb9420), this case published TRANSIENT_FAILURE and shut down the child policy.

Changes:

  • Use fail() for the "No usable endpoint" case, so the cluster goes to TRANSIENT_FAILURE with an error picker and the child policy is shut down.
  • In fail(), replace the shut-down GracefulSwitchLoadBalancer with a new one, the same way shutdown() does. Without this, a later non-empty update switches to the same priority factory, and GracefulSwitchLoadBalancer hands it to the child policy that was already shut down. This also applies to the existing fail() callers.

Ambient xDS errors are stored separately by XdsDependencyManager and don't reach this branch, so only the authoritative "no usable endpoints" case changes. In-flight RPCs aren't affected, since subchannel shutdown is graceful.

Added ClusterResolverLoadBalancerTest.handleEdsResource_allEndpointsRemovedAfterReady_failsAndRecovers. It covers three steps: READY, then an empty EDS update, which should give TF plus the error picker and shut down the child; then endpoints coming back, which should create a new child with the new addresses.

Note: I wasn't able to run the Gradle build locally (network restrictions in my environment), so I'm relying on CI for the test run. I did validate the fail() change for the empty-endpoint branch end to end against 1.85.1 with a fake ADS server: stale picks went from 20/20 to 0/20, the RPCs failed fast with UNAVAILABLE, and the cluster recovered once endpoints came back.

🤖 Generated with Claude Code

When a previously READY EDS cluster received an update with no usable
endpoints, CdsLoadBalancer2 passed the error to the delegate via
handleNameResolutionError(). The priority/MultiChild subtree ignores that
while READY, so the stale picker kept selecting endpoints the control
plane had removed. Before ClusterResolverLB was merged into CdsLB2, this
case published TRANSIENT_FAILURE and shut down the child policy.

Use fail() for this case, and have fail() replace the shut-down
GracefulSwitchLoadBalancer so a later update creates a fresh child policy
instead of reusing the one that was shut down.

Fixes grpc#13141

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 03:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the fix addressing stale READY pickers after usable endpoints are removed.

No open issues 🎉

What changed in this PR

Fixes #13141 by preventing xDS clusters from retaining a stale READY picker after an authoritative EDS update removes all usable endpoints.

Changes:

  • Fail the cluster and shut down its child policy when no usable endpoints remain.
  • Recreate the graceful-switch delegate after failure so later endpoint updates can recover.
  • Add a regression test covering READY → empty endpoints → recovery with new addresses.
File Description
xds/​src/​test/​java/​io/​grpc/​xds/​ClusterResolverLoadBalancerTest.java Tests failure, child shutdown, and recovery after endpoints return.
xds/​src/​main/​java/​io/​grpc/​xds/​CdsLoadBalancer2.java Uses fail() for empty endpoints and resets the delegate for recovery.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

xds: CdsLoadBalancer2 keeps stale READY picker after EDS update with no usable endpoints

2 participants