xds: CdsLoadBalancer2 keeps stale READY picker after EDS update with no usable endpoints

Abierto
  • ¿Algo no está bien?
#13,141 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@lujiajing1126 ya está trabajando en esto.

Desde el 28/10/2026.

  • #13142 de @lujiajing1126 — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
64/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
java
Área
backend, networking

Línea de trabajo

Start by reading CdsLoadBalancer2.acceptResolvedAddresses() and trace how the empty-address case updates balancing state; compare it with the existing fail() callers and the earlier ClusterResolverLoadBalancer behavior described in the issue. Add a regression test for an empty EDS update after the cluster is READY, checking that picks fail with UNAVAILABLE and that a later non-empty update recovers.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
What version of gRPC-Java are you using?

1.85.1 (the same code is still on master as of 2026-10-27).

What is your environment?

macOS / Linux, JDK 17. xDS control plane: Istio 1.25 (proxyless), also reproduced with an in-process fake ADS server.

What did you expect to see?

When a cluster that was previously READY receives an EDS update (ACKed) whose ClusterLoadAssignment has no usable endpoints, the CDS policy should go to TRANSIENT_FAILURE with an error picker (UNAVAILABLE: No usable endpoint from cluster: ...) and RPCs routed to that cluster should fail fast.

This is what gRPC-Java did before ClusterResolverLoadBalancer was merged into CdsLoadBalancer2. In 1.56.x, ClusterResolverLoadBalancer.handleEndpointResourceUpdate() did:

if (addresses.isEmpty()) {
  ...
  helper.updateBalancingState(TRANSIENT_FAILURE, new ErrorPicker(endpointNotFound));
  if (childLb != null) {
    childLb.shutdown();
    childLb = null;
  }
  return;
}
What did you see instead?

The cluster keeps its previous READY picker and keeps picking the endpoints that were removed by EDS.

In CdsLoadBalancer2.acceptResolvedAddresses(), the empty-address branch calls handleNameResolutionError on the delegate instead of failing:

if (addresses.isEmpty()) {
  Status status = Status.UNAVAILABLE
      .withDescription("No usable endpoint from cluster: " + clusterName);
  delegate.handleNameResolutionError(status);
  return status;
}

The error goes down to PriorityLoadBalancer -> ... -> MultiChildLoadBalancer.handleNameResolutionError(), which only publishes TRANSIENT_FAILURE when the current state is not READY. So a READY subtree keeps its old picker, even though the endpoint list is now authoritatively empty.

Impact: the stale picker stays in use for as long as the old subchannels stay READY. For example, an endpoint can be removed from EDS but still be alive. This happens with subset/label changes, draining, or Istio scaling a subset to 0 while the connection hasn't been torn down yet. In those cases RPCs keep going to endpoints the control plane has removed, instead of failing fast with UNAVAILABLE.

This looks like it came in with ebb9420948 ("xds: Merge ClusterResolverLB into CdsLB2").

Steps to reproduce the bug
  1. Configure an xDS channel with a route to an EDS cluster with a few endpoints (round_robin; also reproduced with a custom LB policy, so it is independent of the leaf policy).
  2. Make RPCs until the cluster is READY.
  3. From the control plane, push an empty ClusterLoadAssignment for that cluster (cluster_name set, no endpoints). Confirm via XdsClient.getSubscribedResourcesMetadataSnapshot() that it is ACKED with 0 endpoint groups.
  4. Make more RPCs to that cluster.

Result: 20/20 picks still return the old endpoints (same result whether or not the EDS version changes).

With this one-line change, all picks fail fast with UNAVAILABLE, other clusters are unaffected, and a later non-empty EDS update recovers normally:

       if (addresses.isEmpty()) {
         Status status = Status.UNAVAILABLE
             .withDescription("No usable endpoint from cluster: " + clusterName);
-        delegate.handleNameResolutionError(status);
-        return status;
+        return fail(status);
       }

(fail() shuts down the delegate, which tears down the priority children, and publishes TF + an error picker. On the next non-empty update the same PriorityLoadBalancer instance rebuilds its children, which is the same recovery path the existing fail() callers already rely on. RPCs already in flight are not affected, because subchannel shutdown is graceful.)

Ambient errors don't reach this branch, since XdsDependencyManager keeps them separate from the data. So this change only affects the authoritative "no usable endpoints" case.

Happy to send a PR with a test if this approach looks right.

Lenguaje dominante
Java
Estrellas
12.1k
Forks
4k
Merge medio
1 d 13 h
PR fusionados (30 d)
42

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de grpc/grpc-java

Todos los issues de grpc/grpc-java

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.