xds: CdsLoadBalancer2 keeps stale READY picker after EDS update with no usable endpoints

Aberta
  • Algo errado?
#13,141 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@lujiajing1126 já está trabalhando nisso.

Desde 28/10/2026.

  • #13142 de @lujiajing1126 — aberto

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
64/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
java
Domínio
backend, networking

Direção de pesquisa

Start by reading CdsLoadBalancer2.acceptResolvedAddresses() and trace how the empty-address case updates balancing state; compare it with the existing fail() callers and the earlier ClusterResolverLoadBalancer behavior described in the issue. Add a regression test for an empty EDS update after the cluster is READY, checking that picks fail with UNAVAILABLE and that a later non-empty update recovers.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

What version of gRPC-Java are you using?

1.85.1 (the same code is still on master as of 2026-10-27).

What is your environment?

macOS / Linux, JDK 17. xDS control plane: Istio 1.25 (proxyless), also reproduced with an in-process fake ADS server.

What did you expect to see?

When a cluster that was previously READY receives an EDS update (ACKed) whose ClusterLoadAssignment has no usable endpoints, the CDS policy should go to TRANSIENT_FAILURE with an error picker (UNAVAILABLE: No usable endpoint from cluster: ...) and RPCs routed to that cluster should fail fast.

This is what gRPC-Java did before ClusterResolverLoadBalancer was merged into CdsLoadBalancer2. In 1.56.x, ClusterResolverLoadBalancer.handleEndpointResourceUpdate() did:

if (addresses.isEmpty()) {
  ...
  helper.updateBalancingState(TRANSIENT_FAILURE, new ErrorPicker(endpointNotFound));
  if (childLb != null) {
    childLb.shutdown();
    childLb = null;
  }
  return;
}
What did you see instead?

The cluster keeps its previous READY picker and keeps picking the endpoints that were removed by EDS.

In CdsLoadBalancer2.acceptResolvedAddresses(), the empty-address branch calls handleNameResolutionError on the delegate instead of failing:

if (addresses.isEmpty()) {
  Status status = Status.UNAVAILABLE
      .withDescription("No usable endpoint from cluster: " + clusterName);
  delegate.handleNameResolutionError(status);
  return status;
}

The error goes down to PriorityLoadBalancer -> ... -> MultiChildLoadBalancer.handleNameResolutionError(), which only publishes TRANSIENT_FAILURE when the current state is not READY. So a READY subtree keeps its old picker, even though the endpoint list is now authoritatively empty.

Impact: the stale picker stays in use for as long as the old subchannels stay READY. For example, an endpoint can be removed from EDS but still be alive. This happens with subset/label changes, draining, or Istio scaling a subset to 0 while the connection hasn't been torn down yet. In those cases RPCs keep going to endpoints the control plane has removed, instead of failing fast with UNAVAILABLE.

This looks like it came in with ebb9420948 ("xds: Merge ClusterResolverLB into CdsLB2").

Steps to reproduce the bug
  1. Configure an xDS channel with a route to an EDS cluster with a few endpoints (round_robin; also reproduced with a custom LB policy, so it is independent of the leaf policy).
  2. Make RPCs until the cluster is READY.
  3. From the control plane, push an empty ClusterLoadAssignment for that cluster (cluster_name set, no endpoints). Confirm via XdsClient.getSubscribedResourcesMetadataSnapshot() that it is ACKED with 0 endpoint groups.
  4. Make more RPCs to that cluster.

Result: 20/20 picks still return the old endpoints (same result whether or not the EDS version changes).

With this one-line change, all picks fail fast with UNAVAILABLE, other clusters are unaffected, and a later non-empty EDS update recovers normally:

       if (addresses.isEmpty()) {
         Status status = Status.UNAVAILABLE
             .withDescription("No usable endpoint from cluster: " + clusterName);
-        delegate.handleNameResolutionError(status);
-        return status;
+        return fail(status);
       }

(fail() shuts down the delegate, which tears down the priority children, and publishes TF + an error picker. On the next non-empty update the same PriorityLoadBalancer instance rebuilds its children, which is the same recovery path the existing fail() callers already rely on. RPCs already in flight are not affected, because subchannel shutdown is graceful.)

Ambient errors don't reach this branch, since XdsDependencyManager keeps them separate from the data. So this change only affects the authoritative "no usable endpoints" case.

Happy to send a PR with a test if this approach looks right.

Linguagem predominante
Java
Estrelas
12.1k
Forks
4k
Merge médio
2d 1h
PRs com merge (30d)
48

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de grpc/grpc-java

Todas as issues de grpc/grpc-java

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.