Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Bug] PD gRPC workers stay held while the leader has no quorum: KV writes, id reads and KV reads wait on the shared hg-grpc pool

オープン
  • 何か問題がありますか？
#3,389 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
databases, grpc, java
領域
backend, distributed-systems

調査の方向性

Start by reading GRpcServerConfig.configure, RaftKVStore.applyOperation, IdMetaStore.inLeaderTerm, and the ReadIndex paths in KvServiceGrpcImpl; compare how each waits with the shared hg-grpc executor configuration. The issue proposes asynchronous completion, bounded waits, or executor separation, so first trace these paths and decide how they should avoid holding workers during quorum loss. Done means requests needing no quorum remain serviceable during that wait; the payload reports that no quorum-loss test has been run.

索引モデルが issue の本文から書いたものです。

説明

Bug Type (问题类型)

logic (逻辑设计问题)

Before submit
  • I have confirmed and searched that there are no similar problems in the historical issue and documents
Environment (环境信息)
  • Server Version: PD on master at ebb8f4e17 (1.8.0-SNAPSHOT), and the KV read change of #3383 at its head db0d771f7
  • Backend: PD's raft metadata store (jraft-core 1.3.9)
  • OS: any; found by reading the code on 2026-10-22, see "How this was found" below
  • Data Size: N/A
Expected & Actual behavior (期望与实际表现)

While a PD leader cannot reach a raft quorum, a request that needs the quorum should fail or be redirected without holding a gRPC worker for long, so the RPCs that need no quorum keep being served.

What happens instead: three request paths wait on a worker of the one shared gRPC executor (hg-grpc) until raft answers, and the executor has a fixed number of workers in front of an unbounded queue.

  1. The executor. GRpcServerConfig.configure (lines 37 to 41) installs HgExecutorUtil.createExecutor("hg-grpc", core, max, queue) for the whole PD gRPC server. PDConfig.ThreadPoolGrpc defaults to thread.pool.grpc.core=600, max=1000 and queue=Integer.MAX_VALUE, which becomes a LinkedBlockingQueue of that capacity (HgExecutorUtil.java lines 101 to 107). A ThreadPoolExecutor adds threads above the core size only when the queue is full, so in practice the server has 600 workers, and every request past them waits in the queue.
  2. KV writes wait without a timeout. Every write of RaftKVStore blocks on applyOperation(...).get() with no deadline (RaftKVStore.java lines 76, 99, 109, 119, 128, 138, 157 and 210). Without a quorum the entry does not commit, and the future completes when jraft fails the pending task at step-down (NodeImpl.stepDown calls ballotBox.clearPendingTasks(), which runs each closure with EPERM "Leader stepped down").
  3. Id reads wait for a ReadIndex. IdMetaStore.inLeaderTerm (line 108) calls waitReadIndexInTerm, added by #3368, which waits up to raft.rpc-timeout (10 s by default; RaftEngine.waitReadIndex, lines 663 and 690 to 713).
  4. KV get and scanPrefix (#3383) wait for the same ReadIndex, also up to raft.rpc-timeout (KvServiceGrpcImpl.java lines 166 and 270 at db0d771f7).

How long a worker is held: PD sets no election timeout, so jraft's default of 1000 ms applies, with a leader lease of 90% of it. A leader that loses its quorum steps down about 1 s later. After that, new requests see isLeader() == false and are redirected without waiting. A write that was waiting is released at the step-down. A ReadIndex that was already waiting (paths 3 and 4) holds its worker until it fails or reaches its 10 s deadline.

So a burst of about 600 requests that arrive in the second before the step-down can occupy every worker. RPCs that need no quorum then queue behind them, for about a second for writes and up to 10 s for reads. Nothing is lost, since the queue is unbounded, but the delay is real. This was raised in the review of #3383 for the KV reads, and paths 2 and 3 show the same holds on master.

One possible direction, for the reads: let the handler return at once and answer from a small executor once the ReadIndex completes. RaftEngine already builds a CompletableFuture for the wait, so it could return that future instead of blocking on it. The write path would need the same treatment, or a deadline. Other directions are a bound on the number of concurrent waits, or a separate executor for the RPCs that need raft.

How this was found

By reading the code at ebb8f4e17 and the jraft-core 1.3.9 sources, on 2026-10-22. No quorum-loss test was run, so the counts and times above come from the configuration defaults and the code paths, not from a measurement.

Vertex/Edge example (问题点 / 边数据举例)

N/A

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

N/A

主要言語
Java
スター
3.2k
フォーク
640
平均マージ
1日 11時間
マージ済み PR（30日）
90

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/hugegraph のほかの issue

apache/hugegraph の issue をすべて見る

似ている issue

Java の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。