milvus-io/milvus

[Feature]: Support publishing data on hdfs into milvus in pyspark

Open

#20,001 opened on Oct 24, 2022

View on GitHub
 (4 comments) (0 reactions) (1 assignee)Go (4,000 forks)batch import
good first issuekind/feature

Repository metrics

Stars
 (44,298 stars)
PR merge metrics
 (Avg merge 7d) (314 merged PRs in 30d)

Description

Is there an existing issue for this?

  • I have searched the existing issues

Is your feature request related to a problem? Please describe.

Now, I publish mode:

  1. Start a pod on k8s.
  2. Pull all parquet files into the pod from hdfs.
  3. Read the parquet files in pandas, then insert data into milvus.

This process consumes huge amount of memory. And if mem can not release, it will affect other process in the pod.

Describe the solution you'd like.

I want to publish data on hdfs into milvus in pyspark

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

Contributor guide

Research direction
Examine the existing PySpark connector for Milvus and understand how to extend it to read from HDFS. Review Milvus data import API and HDFS integration patterns.
Tech stack
pythongo
Domain
backenddatadistributed systems
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PySparkHDFSMilvusKubernetes
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30