[Question] 十亿点边如何进行快速导入

Open

#2,607 opened on Jul 31, 2024 6 comments 1 reaction 1 assignee View on GitHub

@JackyYangPassion is already working on this.

Since Jul 31, 2024.

Assessment

This issue has not been assessed yet.

Description

hbase question

Problem Type (问题类型)

None

Before submit

我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

Server Version: 1.0.0 (Apache Release Version)
Backend: RocksDB x nodes, HDD or SSD
OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
Data Size: xx vertices, xx edges

Your Question (问题描述)

每天的增量数据大概有10亿条左右，底层的存储用的是hbase。通过loader工具导入的时候，运行一段时间会出现rest server is too busy to write的情况。尝试着限制了导入的速率（rate_limit.write=10000），和调大了批量写入的线程占比（batch.max_write_ratio=90），这样导入不会报错，但是导入性能会变的很差。针对大批量数据的导入有什么调优建议吗。在保证导入不报错的前提下，最大化的提高导入的性能。

Vertex/Edge example (问题点 / 边数据举例)

No response

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

No response

Dominant language: Java
Stars: 3.2k
Forks: 637
Avg merge: 3d 18h
Merged PRs (30d): 23

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

More from apache/hugegraph

[Bug] Prometheus metrics format bug Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 64/100

apache/hugegraph#3142 · 7 comments · Opened Aug 7, 2026
[Bug] PD reports "Started" and keeps serving after failing to open its RocksDB (lock held by the previous instance): every request fails, /v1/health stays 200, no retry, no exit Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

apache/hugegraph#3226 · Opened Sep 21, 2026
[Bug] HStore accepts writes with the default serializer=text and then fails every read with BufferUnderflowException; the backend should require serializer=binary Open

Difficulty 3/5 1-2 days Newbie friendliness 64/100

apache/hugegraph#3225 · Opened Sep 21, 2026
[Bug] HStore: data written by 1.7.0 under the missing graph id 0xFFFE becomes unreachable once master allocates a real graph id (raft log replay at the upgrade restart, or the first batch write); no migration path for #3153 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

apache/hugegraph#3224 · Opened Sep 21, 2026
[Rust roadmap] OLTP and storage workload baselines plus rolling-upgrade/rollback requirements for HStore and PD Open

Difficulty 5/5 Over a week Newbie friendliness 28/100

apache/hugegraph#3223 · 1 comment · Opened Sep 20, 2026

All issues in apache/hugegraph

Similar issues

Document Hot Rod migration version compatibility for Data Grid 8.6 Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

infinispan/infinispan#18150 · Opened Sep 21, 2026
[BUG] Route53Resolver: CreateResolverEndpoint requires IpAddressRequests, but the AWS member is IpAddresses, so no SDK client can call it Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

floci-io/floci#4108 · Opened Sep 21, 2026
[BUG] RestGetModelHandlerIT.testGetModel_whenModelIDIsInValid_thenFail depends on test order and fails when run alone Open

untriaged

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

opensearch-project/k-NN#3597 · Opened Sep 21, 2026
Concurrent builder skips 'verify' phase when running 'mvn install' Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

apache/maven#13230 · 1 comment · Opened Sep 21, 2026
ComponentGeNRTL leaves lnGamma stale after computing gamma Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

equinor/neqsim#3899 · 1 comment · Opened Sep 21, 2026

More Java issues

A short digest of beginner-friendly GitHub issues.