[Feature]: Support publishing data on hdfs into milvus in pyspark

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

Now, I publish mode:

1. Start a pod on k8s.
2. Pull all parquet files into the pod from hdfs.
3. Read the parquet files in pandas, then insert data into milvus.

This process consumes huge amount of memory. And if mem can not release, it will affect other process in the pod.

Describe the solution you'd like.

I want to publish data on hdfs into milvus in pyspark

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response