Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug] PD gRPC workers stay held while the leader has no quorum: KV writes, id reads and KV reads wait on the shared hg-grpc pool

Abierto
  • ¿Algo no está bien?
#3,389 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
databases, grpc, java
Área
backend, distributed-systems

Línea de trabajo

Start by reading GRpcServerConfig.configure, RaftKVStore.applyOperation, IdMetaStore.inLeaderTerm, and the ReadIndex paths in KvServiceGrpcImpl; compare how each waits with the shared hg-grpc executor configuration. The issue proposes asynchronous completion, bounded waits, or executor separation, so first trace these paths and decide how they should avoid holding workers during quorum loss. Done means requests needing no quorum remain serviceable during that wait; the payload reports that no quorum-loss test has been run.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Bug Type (问题类型)

logic (逻辑设计问题)

Before submit
  • I have confirmed and searched that there are no similar problems in the historical issue and documents
Environment (环境信息)
  • Server Version: PD on master at ebb8f4e17 (1.8.0-SNAPSHOT), and the KV read change of #3383 at its head db0d771f7
  • Backend: PD's raft metadata store (jraft-core 1.3.9)
  • OS: any; found by reading the code on 2026-10-22, see "How this was found" below
  • Data Size: N/A
Expected & Actual behavior (期望与实际表现)

While a PD leader cannot reach a raft quorum, a request that needs the quorum should fail or be redirected without holding a gRPC worker for long, so the RPCs that need no quorum keep being served.

What happens instead: three request paths wait on a worker of the one shared gRPC executor (hg-grpc) until raft answers, and the executor has a fixed number of workers in front of an unbounded queue.

  1. The executor. GRpcServerConfig.configure (lines 37 to 41) installs HgExecutorUtil.createExecutor("hg-grpc", core, max, queue) for the whole PD gRPC server. PDConfig.ThreadPoolGrpc defaults to thread.pool.grpc.core=600, max=1000 and queue=Integer.MAX_VALUE, which becomes a LinkedBlockingQueue of that capacity (HgExecutorUtil.java lines 101 to 107). A ThreadPoolExecutor adds threads above the core size only when the queue is full, so in practice the server has 600 workers, and every request past them waits in the queue.
  2. KV writes wait without a timeout. Every write of RaftKVStore blocks on applyOperation(...).get() with no deadline (RaftKVStore.java lines 76, 99, 109, 119, 128, 138, 157 and 210). Without a quorum the entry does not commit, and the future completes when jraft fails the pending task at step-down (NodeImpl.stepDown calls ballotBox.clearPendingTasks(), which runs each closure with EPERM "Leader stepped down").
  3. Id reads wait for a ReadIndex. IdMetaStore.inLeaderTerm (line 108) calls waitReadIndexInTerm, added by #3368, which waits up to raft.rpc-timeout (10 s by default; RaftEngine.waitReadIndex, lines 663 and 690 to 713).
  4. KV get and scanPrefix (#3383) wait for the same ReadIndex, also up to raft.rpc-timeout (KvServiceGrpcImpl.java lines 166 and 270 at db0d771f7).

How long a worker is held: PD sets no election timeout, so jraft's default of 1000 ms applies, with a leader lease of 90% of it. A leader that loses its quorum steps down about 1 s later. After that, new requests see isLeader() == false and are redirected without waiting. A write that was waiting is released at the step-down. A ReadIndex that was already waiting (paths 3 and 4) holds its worker until it fails or reaches its 10 s deadline.

So a burst of about 600 requests that arrive in the second before the step-down can occupy every worker. RPCs that need no quorum then queue behind them, for about a second for writes and up to 10 s for reads. Nothing is lost, since the queue is unbounded, but the delay is real. This was raised in the review of #3383 for the KV reads, and paths 2 and 3 show the same holds on master.

One possible direction, for the reads: let the handler return at once and answer from a small executor once the ReadIndex completes. RaftEngine already builds a CompletableFuture for the wait, so it could return that future instead of blocking on it. The write path would need the same treatment, or a deadline. Other directions are a bound on the number of concurrent waits, or a separate executor for the RPCs that need raft.

How this was found

By reading the code at ebb8f4e17 and the jraft-core 1.3.9 sources, on 2026-10-22. No quorum-loss test was run, so the counts and times above come from the configuration defaults and the code paths, not from a measurement.

Vertex/Edge example (问题点 / 边数据举例)

N/A

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

N/A

Lenguaje dominante
Java
Estrellas
3.2k
Forks
640
Merge medio
1 d 11 h
PR fusionados (30 d)
90

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/hugegraph

Todos los issues de apache/hugegraph

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.