milvus-io/milvus
GitHub で見る[Feature]: Support publishing data on hdfs into milvus in pyspark
Open
#20,001 opened on 2022年10月24日
good first issuekind/feature
Repository metrics
- Stars
- (44,298 stars)
- PR merge metrics
- (平均マージ 7d) (30d で 314 merged PRs)
説明
Is there an existing issue for this?
- I have searched the existing issues
Is your feature request related to a problem? Please describe.
Now, I publish mode:
- Start a pod on k8s.
- Pull all parquet files into the pod from hdfs.
- Read the parquet files in pandas, then insert data into milvus.
This process consumes huge amount of memory. And if mem can not release, it will affect other process in the pod.
Describe the solution you'd like.
I want to publish data on hdfs into milvus in pyspark
Describe an alternate solution.
No response
Anything else? (Additional Context)
No response