Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

pick_first (legacy): channel permanently stuck in READY with a stale picker after resolver refresh burst during transient network failure; wait-for-ready RPCs black-hole until deadline

Offen
  • Stimmt etwas nicht?
#12,958 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
45/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Ruhig
Tech-Stack
java
Bereich
backend-api-design, networking

Rechercherichtung

Beginne mit io.grpc.internal.PickFirstLoadBalancer und führe die angehängte BurstRepro.java, das Treiberskript und den Proxy aus, um die READY- und CONNECTING-Blockaden zu reproduzieren. Vergleiche das Legacy-Verhalten mit den PickFirstLeafLoadBalancer-Logs in logs/run2-READY-signature/ und logs/run3-newpf-no-wedge/. Als erledigt gilt die Aufgabe, wenn die Legacy-Richtlinie keinen READY-Kanal mit einem nicht mehr funktionierenden Picker zurücklässt und die wait-for-ready-Probe entweder wiederhergestellt wird oder sichtbar fehlschlägt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

grpc-java-pickfirst-wedge.tar.gz

What version of gRPC-Java are you using?

1.76.2 (grpc-netty, netty 4.1.136.Final)

What is your environment?

Linux, Java 17 (also observed under Java 11 in production).
Load balancing policy: pick_first (single-address target).
Active LB implementation: legacy io.grpc.internal.PickFirstLoadBalancer (its
RequestConnectionPicker appears in the channel logs; the new
PickFirstLeafLoadBalancer was not enabled — see "Additional data point" below).

What did you expect to see?

After a transient network failure heals, a pick_first channel either recovers or
transitions out of READY so that the LB policy or the application (via
notifyWhenStateChanged) can react.

What did you see instead?

The channel is left permanently reporting READY while every RPC — including
withWaitForReady() RPCs — buffers forever and fails only by its own deadline:

DEADLINE_EXCEEDED: ... Name resolution delay 0 ... [buffered_nanos=..., waiting_for_connection]

No channel state transition ever fires again, so nothing can observe that the channel
is broken. ManagedChannel.enterIdle() reliably recovers it.

Mechanism, from FINEST ChannelLogger logs (attached):

  1. The transport to the current address dies (TLS handshakes reset — in production,
    a rolling container upgrade left the port published while the backend reset
    handshakes for ~10 seconds).
  2. gRPC's reconnect logic calls NameResolver.refresh() repeatedly. The resolver
    answers asynchronously with a mix of onError and onResult deliveries (it is
    backed by a discovery service that sits behind the same failing network, so
    deliveries arrive out of order relative to subchannel state changes).
  3. Each onResult causes PickFirstLoadBalancer to create a replacement subchannel
    and schedule delayed shutdown (~5s) of the previous one.
  4. Within the delayed-shutdown window, the old subchannel completes its connection
    against the recovering backend and reaches READY; the LB publishes a READY picker
    referencing it.
  5. The delayed shutdown then fires (SHUTDOWN with UNAVAILABLE(Subchannel shutdown invoked)), terminating the transport — but the published READY picker is never
    replaced. The channel stays READY forever with a picker whose subchannel has no
    transport.
Steps to reproduce the bug

Self-contained repro attached: one Java file (BurstRepro.java, only grpc-java +
netty dependencies, in-process backend server), a small Python TCP proxy that models
the network failure (reset mode: accept + immediately close, emulating TLS resets),
and a driver script. Typically reproduces within a few iterations.

Per iteration: verify a healthy RPC through the proxy, switch the proxy to reset
for a few seconds (driving reconnect/backoff and resolver refreshes), heal the proxy,
then probe with a withWaitForReady() RPC with a deadline.

Two wedge variants observed:

  • logs/run2-READY-signature/ — channel READY, wait-for-ready probe black-holes to
    DEADLINE_EXCEEDED; enterIdle() immediately recovers it. This matches our
    production incident signature.
  • logs/run1-connecting-wedge/ — channel wedged in CONNECTING with no reconnect
    attempt scheduled.
Additional data point: the new PickFirstLeafLoadBalancer does not wedge

Running the identical repro with GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST=true
(confirmed active: PickFirstLeafLoadBalancer in the logs), the permanent black-hole
does not reproduce in 40 iterations. The same post-burst sequence still leaves the
channel claiming READY while the picked transport is dead, but the RPC fails fast
with UNAVAILABLE and gRPC immediately schedules a resolver refresh, after which the
channel recovers (logs/run3-newpf-no-wedge/).

So the catastrophic silent wedge appears specific to the legacy
PickFirstLoadBalancer. Since the new PF also has a known stuck-in-IDLE issue
(#12395) with a draft fix that appears stalled (#12403), may we suggest revisiting
that PR — fixing #12395 and graduating the new PF would retire both wedge variants,
whereas today users are stuck choosing between a legacy PF with this READY wedge and
an experimental PF with a known IDLE wedge.

Related issues
  • #12395 — PickFirstLeafLoadBalancer stuck in IDLE on backend address change (Java
    analog of grpc/grpc-go#8615). Same family: LB internal state diverging from the
    published picker when a resolver update races a subchannel state change. Differs
    in that it wedges in IDLE and affects the new leaf PF; this report is the legacy
    PF wedging in READY, which is strictly worse (wait-for-ready RPCs black-hole and
    no observable state ever changes).
  • #11082 — PF not emitting TRANSIENT_FAILURE (missed state propagation).
  • #11227 — NPE from inconsistent PF internal state after resolution errors (fixed in
    1.66); shows the resolver-error path mutating LB state non-atomically.
Production impact

Our etcd-backed auth-service channel black-holed all RPCs for 60+ seconds (until
process restart) during a rolling upgrade. The server saw zero incoming RPCs during
the window while the client channel claimed READY throughout. We now run an
application-level watchdog that calls enterIdle() when RPCs time out while the
channel claims READY.

Attachment

grpc-java-pickfirst-wedge.tar.gz — repro sources, driver, and FINEST channel logs
for all three runs described above.

Vorherrschende Sprache
Java
Sterne
12.1k
Forks
4k
Ø Merge
2 T. 3 Std.
Gemergte PRs (30 T.)
30

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus grpc/grpc-java

Alle Issues in grpc/grpc-java

Ähnliche Issues

Weitere Issues zu Java

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.