Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Bug] PD gRPC workers stay held while the leader has no quorum: KV writes, id reads and KV reads wait on the shared hg-grpc pool

未关闭
  • 哪里不对？
#3,389 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
databases, grpc, java
领域
backend, distributed-systems

调研方向

Start by reading GRpcServerConfig.configure, RaftKVStore.applyOperation, IdMetaStore.inLeaderTerm, and the ReadIndex paths in KvServiceGrpcImpl; compare how each waits with the shared hg-grpc executor configuration. The issue proposes asynchronous completion, bounded waits, or executor separation, so first trace these paths and decide how they should avoid holding workers during quorum loss. Done means requests needing no quorum remain serviceable during that wait; the payload reports that no quorum-loss test has been run.

由索引模型根据 Issue 内容生成。

描述

Bug Type (问题类型)

logic (逻辑设计问题)

Before submit
  • I have confirmed and searched that there are no similar problems in the historical issue and documents
Environment (环境信息)
  • Server Version: PD on master at ebb8f4e17 (1.8.0-SNAPSHOT), and the KV read change of #3383 at its head db0d771f7
  • Backend: PD's raft metadata store (jraft-core 1.3.9)
  • OS: any; found by reading the code on 2026-10-22, see "How this was found" below
  • Data Size: N/A
Expected & Actual behavior (期望与实际表现)

While a PD leader cannot reach a raft quorum, a request that needs the quorum should fail or be redirected without holding a gRPC worker for long, so the RPCs that need no quorum keep being served.

What happens instead: three request paths wait on a worker of the one shared gRPC executor (hg-grpc) until raft answers, and the executor has a fixed number of workers in front of an unbounded queue.

  1. The executor. GRpcServerConfig.configure (lines 37 to 41) installs HgExecutorUtil.createExecutor("hg-grpc", core, max, queue) for the whole PD gRPC server. PDConfig.ThreadPoolGrpc defaults to thread.pool.grpc.core=600, max=1000 and queue=Integer.MAX_VALUE, which becomes a LinkedBlockingQueue of that capacity (HgExecutorUtil.java lines 101 to 107). A ThreadPoolExecutor adds threads above the core size only when the queue is full, so in practice the server has 600 workers, and every request past them waits in the queue.
  2. KV writes wait without a timeout. Every write of RaftKVStore blocks on applyOperation(...).get() with no deadline (RaftKVStore.java lines 76, 99, 109, 119, 128, 138, 157 and 210). Without a quorum the entry does not commit, and the future completes when jraft fails the pending task at step-down (NodeImpl.stepDown calls ballotBox.clearPendingTasks(), which runs each closure with EPERM "Leader stepped down").
  3. Id reads wait for a ReadIndex. IdMetaStore.inLeaderTerm (line 108) calls waitReadIndexInTerm, added by #3368, which waits up to raft.rpc-timeout (10 s by default; RaftEngine.waitReadIndex, lines 663 and 690 to 713).
  4. KV get and scanPrefix (#3383) wait for the same ReadIndex, also up to raft.rpc-timeout (KvServiceGrpcImpl.java lines 166 and 270 at db0d771f7).

How long a worker is held: PD sets no election timeout, so jraft's default of 1000 ms applies, with a leader lease of 90% of it. A leader that loses its quorum steps down about 1 s later. After that, new requests see isLeader() == false and are redirected without waiting. A write that was waiting is released at the step-down. A ReadIndex that was already waiting (paths 3 and 4) holds its worker until it fails or reaches its 10 s deadline.

So a burst of about 600 requests that arrive in the second before the step-down can occupy every worker. RPCs that need no quorum then queue behind them, for about a second for writes and up to 10 s for reads. Nothing is lost, since the queue is unbounded, but the delay is real. This was raised in the review of #3383 for the KV reads, and paths 2 and 3 show the same holds on master.

One possible direction, for the reads: let the handler return at once and answer from a small executor once the ReadIndex completes. RaftEngine already builds a CompletableFuture for the wait, so it could return that future instead of blocking on it. The write path would need the same treatment, or a deadline. Other directions are a bound on the number of concurrent waits, or a separate executor for the RPCs that need raft.

How this was found

By reading the code at ebb8f4e17 and the jraft-core 1.3.9 sources, on 2026-10-22. No quorum-loss test was run, so the counts and times above come from the configuration defaults and the code paths, not from a measurement.

Vertex/Edge example (问题点 / 边数据举例)

N/A

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

N/A

主要语言
Java
星标
3.2k
派生
640
平均合并
1 天 11 小时
30 天内合并 PR
90

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/hugegraph 的其他 Issue

查看 apache/hugegraph 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。