milvus-io/milvus

Support HDFS on Milvus 2.0

Open

#2,745 opened on Jul 5, 2020

View on GitHub
 (13 comments) (0 reactions) (2 assignees)Go (44,298 stars) (4,000 forks)batch import
difficulty/intermediategood first issuekind/featurepriority/backlog

Description

Is your feature request related to a problem? Please describe. If we could let Milvus support to save data in a distributed file system, that's will be convenient to use/save huge data.

Describe the solution you'd like Re-implement some functions about I/O to support HDFS.

Describe alternatives you've considered N/A

Additional context For "storage" add new source files and head files HDFSWriter / HDFSReader... and re-implement some functions. For "codec" add some files about HDFS. ...

Contributor guide

Tech stack
go
Domain
backenddatabaseinfrastructure
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Go programmingMilvus I/O layerHDFS APIDistributed file systems
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Investigate the current storage and codec subsystems in Milvus to understand existing file I/O interfaces. Review the HDFS API (Hadoop libraries) to design the integration. Follow the issue's suggestion to create HDFSWriter/HDFSReader classes in the storage layer and corresponding codec modules. Since the issue has assignees, coordinate with them to avoid duplication. Check the Milvus 2.0 architecture documentation for data persistence.