Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

pick_first (legacy): channel permanently stuck in READY with a stale picker after resolver refresh burst during transient network failure; wait-for-ready RPCs black-hole until deadline

Aperta
  • Qualcosa non va?
#12,958 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
java
Ambito
backend-api-design, networking

Direzione di ricerca

Inizia con io.grpc.internal.PickFirstLoadBalancer ed esegui il BurstRepro.java allegato, lo script di controllo e il proxy per riprodurre i blocchi READY e CONNECTING. Confronta il comportamento legacy con i log di PickFirstLeafLoadBalancer in logs/run2-READY-signature/ e logs/run3-newpf-no-wedge/. Il lavoro è completato quando la policy legacy non lascia più un canale READY con un picker non funzionante e la probe wait-for-ready si riprende o fallisce in modo visibile.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

grpc-java-pickfirst-wedge.tar.gz

What version of gRPC-Java are you using?

1.76.2 (grpc-netty, netty 4.1.136.Final)

What is your environment?

Linux, Java 17 (also observed under Java 11 in production).
Load balancing policy: pick_first (single-address target).
Active LB implementation: legacy io.grpc.internal.PickFirstLoadBalancer (its
RequestConnectionPicker appears in the channel logs; the new
PickFirstLeafLoadBalancer was not enabled — see "Additional data point" below).

What did you expect to see?

After a transient network failure heals, a pick_first channel either recovers or
transitions out of READY so that the LB policy or the application (via
notifyWhenStateChanged) can react.

What did you see instead?

The channel is left permanently reporting READY while every RPC — including
withWaitForReady() RPCs — buffers forever and fails only by its own deadline:

DEADLINE_EXCEEDED: ... Name resolution delay 0 ... [buffered_nanos=..., waiting_for_connection]

No channel state transition ever fires again, so nothing can observe that the channel
is broken. ManagedChannel.enterIdle() reliably recovers it.

Mechanism, from FINEST ChannelLogger logs (attached):

  1. The transport to the current address dies (TLS handshakes reset — in production,
    a rolling container upgrade left the port published while the backend reset
    handshakes for ~10 seconds).
  2. gRPC's reconnect logic calls NameResolver.refresh() repeatedly. The resolver
    answers asynchronously with a mix of onError and onResult deliveries (it is
    backed by a discovery service that sits behind the same failing network, so
    deliveries arrive out of order relative to subchannel state changes).
  3. Each onResult causes PickFirstLoadBalancer to create a replacement subchannel
    and schedule delayed shutdown (~5s) of the previous one.
  4. Within the delayed-shutdown window, the old subchannel completes its connection
    against the recovering backend and reaches READY; the LB publishes a READY picker
    referencing it.
  5. The delayed shutdown then fires (SHUTDOWN with UNAVAILABLE(Subchannel shutdown invoked)), terminating the transport — but the published READY picker is never
    replaced. The channel stays READY forever with a picker whose subchannel has no
    transport.
Steps to reproduce the bug

Self-contained repro attached: one Java file (BurstRepro.java, only grpc-java +
netty dependencies, in-process backend server), a small Python TCP proxy that models
the network failure (reset mode: accept + immediately close, emulating TLS resets),
and a driver script. Typically reproduces within a few iterations.

Per iteration: verify a healthy RPC through the proxy, switch the proxy to reset
for a few seconds (driving reconnect/backoff and resolver refreshes), heal the proxy,
then probe with a withWaitForReady() RPC with a deadline.

Two wedge variants observed:

  • logs/run2-READY-signature/ — channel READY, wait-for-ready probe black-holes to
    DEADLINE_EXCEEDED; enterIdle() immediately recovers it. This matches our
    production incident signature.
  • logs/run1-connecting-wedge/ — channel wedged in CONNECTING with no reconnect
    attempt scheduled.
Additional data point: the new PickFirstLeafLoadBalancer does not wedge

Running the identical repro with GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST=true
(confirmed active: PickFirstLeafLoadBalancer in the logs), the permanent black-hole
does not reproduce in 40 iterations. The same post-burst sequence still leaves the
channel claiming READY while the picked transport is dead, but the RPC fails fast
with UNAVAILABLE and gRPC immediately schedules a resolver refresh, after which the
channel recovers (logs/run3-newpf-no-wedge/).

So the catastrophic silent wedge appears specific to the legacy
PickFirstLoadBalancer. Since the new PF also has a known stuck-in-IDLE issue
(#12395) with a draft fix that appears stalled (#12403), may we suggest revisiting
that PR — fixing #12395 and graduating the new PF would retire both wedge variants,
whereas today users are stuck choosing between a legacy PF with this READY wedge and
an experimental PF with a known IDLE wedge.

Related issues
  • #12395 — PickFirstLeafLoadBalancer stuck in IDLE on backend address change (Java
    analog of grpc/grpc-go#8615). Same family: LB internal state diverging from the
    published picker when a resolver update races a subchannel state change. Differs
    in that it wedges in IDLE and affects the new leaf PF; this report is the legacy
    PF wedging in READY, which is strictly worse (wait-for-ready RPCs black-hole and
    no observable state ever changes).
  • #11082 — PF not emitting TRANSIENT_FAILURE (missed state propagation).
  • #11227 — NPE from inconsistent PF internal state after resolution errors (fixed in
    1.66); shows the resolver-error path mutating LB state non-atomically.
Production impact

Our etcd-backed auth-service channel black-holed all RPCs for 60+ seconds (until
process restart) during a rolling upgrade. The server saw zero incoming RPCs during
the window while the client channel claimed READY throughout. We now run an
application-level watchdog that calls enterIdle() when RPCs time out while the
channel claims READY.

Attachment

grpc-java-pickfirst-wedge.tar.gz — repro sources, driver, and FINEST channel logs
for all three runs described above.

Lingua principale
Java
Stelle
12.1k
Fork
4k
Merge medio
2g 3h
PR unite (30g)
30

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di grpc/grpc-java

Tutte le issue di grpc/grpc-java

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.