xds: CdsLoadBalancer2 keeps stale READY picker after EDS update with no usable endpoints

Ouverte
  • Quelque chose ne va pas ?
#13,141 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@lujiajing1126 y travaille déjà.

Depuis le 28/10/2026.

  • #13142 par @lujiajing1126 — ouverte

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
64/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
java
Domaine
backend, networking

Piste de recherche

Start by reading CdsLoadBalancer2.acceptResolvedAddresses() and trace how the empty-address case updates balancing state; compare it with the existing fail() callers and the earlier ClusterResolverLoadBalancer behavior described in the issue. Add a regression test for an empty EDS update after the cluster is READY, checking that picks fail with UNAVAILABLE and that a later non-empty update recovers.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

What version of gRPC-Java are you using?

1.85.1 (the same code is still on master as of 2026-10-27).

What is your environment?

macOS / Linux, JDK 17. xDS control plane: Istio 1.25 (proxyless), also reproduced with an in-process fake ADS server.

What did you expect to see?

When a cluster that was previously READY receives an EDS update (ACKed) whose ClusterLoadAssignment has no usable endpoints, the CDS policy should go to TRANSIENT_FAILURE with an error picker (UNAVAILABLE: No usable endpoint from cluster: ...) and RPCs routed to that cluster should fail fast.

This is what gRPC-Java did before ClusterResolverLoadBalancer was merged into CdsLoadBalancer2. In 1.56.x, ClusterResolverLoadBalancer.handleEndpointResourceUpdate() did:

if (addresses.isEmpty()) {
  ...
  helper.updateBalancingState(TRANSIENT_FAILURE, new ErrorPicker(endpointNotFound));
  if (childLb != null) {
    childLb.shutdown();
    childLb = null;
  }
  return;
}
What did you see instead?

The cluster keeps its previous READY picker and keeps picking the endpoints that were removed by EDS.

In CdsLoadBalancer2.acceptResolvedAddresses(), the empty-address branch calls handleNameResolutionError on the delegate instead of failing:

if (addresses.isEmpty()) {
  Status status = Status.UNAVAILABLE
      .withDescription("No usable endpoint from cluster: " + clusterName);
  delegate.handleNameResolutionError(status);
  return status;
}

The error goes down to PriorityLoadBalancer -> ... -> MultiChildLoadBalancer.handleNameResolutionError(), which only publishes TRANSIENT_FAILURE when the current state is not READY. So a READY subtree keeps its old picker, even though the endpoint list is now authoritatively empty.

Impact: the stale picker stays in use for as long as the old subchannels stay READY. For example, an endpoint can be removed from EDS but still be alive. This happens with subset/label changes, draining, or Istio scaling a subset to 0 while the connection hasn't been torn down yet. In those cases RPCs keep going to endpoints the control plane has removed, instead of failing fast with UNAVAILABLE.

This looks like it came in with ebb9420948 ("xds: Merge ClusterResolverLB into CdsLB2").

Steps to reproduce the bug
  1. Configure an xDS channel with a route to an EDS cluster with a few endpoints (round_robin; also reproduced with a custom LB policy, so it is independent of the leaf policy).
  2. Make RPCs until the cluster is READY.
  3. From the control plane, push an empty ClusterLoadAssignment for that cluster (cluster_name set, no endpoints). Confirm via XdsClient.getSubscribedResourcesMetadataSnapshot() that it is ACKED with 0 endpoint groups.
  4. Make more RPCs to that cluster.

Result: 20/20 picks still return the old endpoints (same result whether or not the EDS version changes).

With this one-line change, all picks fail fast with UNAVAILABLE, other clusters are unaffected, and a later non-empty EDS update recovers normally:

       if (addresses.isEmpty()) {
         Status status = Status.UNAVAILABLE
             .withDescription("No usable endpoint from cluster: " + clusterName);
-        delegate.handleNameResolutionError(status);
-        return status;
+        return fail(status);
       }

(fail() shuts down the delegate, which tears down the priority children, and publishes TF + an error picker. On the next non-empty update the same PriorityLoadBalancer instance rebuilds its children, which is the same recovery path the existing fail() callers already rely on. RPCs already in flight are not affected, because subchannel shutdown is graceful.)

Ambient errors don't reach this branch, since XdsDependencyManager keeps them separate from the data. So this change only affects the authoritative "no usable endpoints" case.

Happy to send a PR with a test if this approach looks right.

Langage dominant
Java
Étoiles
12.1k
Forks
4k
Merge moyen
2 j 1 h
PR mergées (30 j)
48

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de grpc/grpc-java

Toutes les issues de grpc/grpc-java

Issues similaires

Plus d'issues Java

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.