[Feature] Implement Cloud Storage Support for HugeGraph
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start with the linked RFC and discussion thread, then inspect the HStore+CloudStorage branch in the referenced fork. Compare the proposal with HugeGraph's existing storage architecture and identify the relevant extension entry points. Done requires an agreed SPI-based design, cloud-provider support, offloading and hydration behavior, configuration, and tests.
Written by the indexing model from the issue text.
Description
Feature Description (功能描述)
Motivation
in modern cloud-native environments, compute instances are ephemeral—frequently created and destroyed based on autoscaling policies. HugeGraph currently relies on local disk storage (RocksDB), which creates significant risks for data durability and complicates operational lifecycle management in these dynamic environments. We propose integrating a pluggable cloud-native storage tier to offload data to object storage, effectively decoupling compute from storage to ensure data persistence, reliability, and operational flexibility.
High-Level Requirements:
- Decoupled Storage: Enable offloading and retrieving HugeGraph data from cloud object storage (e.g., S3, GCS, Azure Blob).
- Extensible Architecture: Implement a provider-agnostic framework using an SPI (Service Provider Interface) mechanism. This design ensures the core storage engine remains decoupled, allowing the community to add new storage backends without requiring modifications to the core codebase.
- Initial Cloud Support: Provide out-of-the-box support for few popular cloud object storage services e.g. AWS S3, Azure Data Lake Storage Gen2. Google Cloud Storage (GCS) etc.
Functional Requirements:
- Automated Data Offloading: Mechanism to persist local SST data to configured cloud object storage.
- Intelligent Hydration: Ability to recover/hydrate data from cloud storage to local compute nodes upon startup or during read-miss scenarios.
- Configuration: Standardized property-based configuration for buckets, endpoints, and caching strategies.
Non-Functional Requirements:
- Performance: Minimize read/write latency overhead via local caching layers.
- Scalability: Support elastic scaling of compute nodes without requiring complex manual data migration.
- Maintainability: Enable the addition of new storage backends via an SPI-based plugin model to minimize technical debt.
Discussion thread : https://github.com/apache/hugegraph/discussions/3080
RFC Document : RFC: Cloud Storage Support for HugeGraph
PoC Code: In Branch HStore+CloudStorage in my fork https://github.com/vaijosh/hugegraph
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 637
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 23
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/hugegraph
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Difficulty 3/5 1-2 days Newbie friendliness 64/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
All issues in apache/hugegraph
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100