[Improve] benchmark ordered range scans and identify fan-out bottlenecks

Open
#3,158 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
grpc, java
Domain
databases, distributed-systems, performance

Research direction

Start with the ordered HStore range-index scan behavior and the Store-local/client-side K-way merge established by #3140. Build the stated matrix across partition counts, cache and latency conditions, then record first-result and total time, RPCs, bytes, heap, and Store CPU; done means reproducible results, a documented bottleneck and recommendation, with follow-up implementation work split into focused issues.

Written by the indexing model from the issue text.

Description

improvement

Context

Follow-up to #3140. That PR establishes correctness for ordered HStore range-index scans through Store-local and client-side K-way merge. This issue is limited to measuring the performance characteristics and deciding which optimization, if any, is justified.

Benchmark scope

Build a reproducible benchmark matrix covering:

  • Small limits with many partitions
  • Medium limits with sparse matches
  • Large limits and page/cursor continuation
  • 1, 4, 16, and 64 partitions
  • Warm and cold cache, plus representative network latency

Record time to first result, total query time, RPC/open-stream count, transferred bytes, client heap usage, and Store CPU.

Decisions after the baseline

Use the results to decide whether to open focused implementation issues for:

  1. Partition pruning, when range metadata can safely eliminate non-overlapping partitions.
  2. Client-driven initial and subsequent page-demand sizing, when it reduces unused data materially.
  3. Prefetch/pipelining, after characterizing existing gRPC streaming and buffering behavior.

Store-local merge pushdown is already delivered by #3140 and is not duplicated here.

Acceptance criteria

  • Benchmark configuration and results are reproducible.
  • The bottleneck and recommended next optimization are documented.
  • Further implementation work is split into focused issues only when supported by the measurements.

I will follow up on this issue.

Dominant language
Java
Stars
3.2k
Forks
637
Avg merge
3d 18h
Merged PRs (30d)
23

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/hugegraph

All issues in apache/hugegraph

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.