milvus-io/milvus

[Feature]: Implement a preprocessing and post processing module on proxy

Open

#27,469 创建于 2023年10月3日

在 GitHub 查看
 (11 评论) (2 反应) (0 负责人)Go (44,298 star) (4,000 fork)batch import
good first issuehacktoberfestkind/feature

描述

Is there an existing issue for this?

  • I have searched the existing issues

Is your feature request related to a problem? Please describe.

Description: The proxy component in Milvus plays a crucial role in handling client requests and managing data flow. Enhancing this component with preprocessing and postprocessing modules can significantly improve the convenience of data/result processing. Here's a detailed breakdown of the proposed implementation:

Preprocessing Module typical functions:

Normalization: Automatically normalize vector data to a standard form during insertion, ensuring consistent data format which is crucial for accurate similarity searches.

Dimension Reduction: Apply dimensionality reduction techniques to transform high-dimensional data into a lower-dimensional space, reducing storage requirements and potentially improving the efficiency of subsequent operations.

Scalar Transformation: Perform scalar transformations on vector data to conform to specific scales or ranges, aiding in more meaningful analysis and comparisons.

Postprocessing Module typical functions:

Search Result Handling: Develop functionalities to handle and manipulate search query results, such as sorting, filtering, or clustering, to provide more insightful and organized output to the users.

Ranking and Re-ranking: Implement ranking and re-ranking mechanisms to order the search results based on certain criteria, ensuring the most relevant results are presented at the top.

Modular and Extensible Architecture:

Design the modules in a modular and extensible manner, allowing for easy addition of new preprocessing and postprocessing operations in the future as per the community's needs.

User-Friendly Configuration:

Allow users to easily configure the preprocessing and postprocessing operations through a user-friendly interface, possibly integrating with the existing Milvus CLI or developing a dedicated interface. Comprehensive Documentation and Tutorials:

Create detailed documentation and tutorials demonstrating the usage and benefits of these modules, ensuring users can effectively leverage these new capabilities.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

贡献者指南

技术栈
go
领域
backenddatamachine learning
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
5
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
over 1 week
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
active
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
Go programmingUnderstanding of proxy architecture in MilvusExperience with vector database internals
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
10
研究方向
The issue requests adding preprocessing and postprocessing modules to Milvus proxy. The implementation will involve the Go based proxy component; refer to the existing proxy code in cmd/proxy and internal/proxy directories. The 11 comments likely discuss high level design; review them for any concrete proposals. Maintainers may have provided guidance on modular architecture and extension points. The feature is large and requires deep knowledge of Milvus internals and vector processing, so it's unsuitable for first time contributors.
[Feature]: Implement a preprocessing and post processing module on proxy · milvus-io/milvus#27469 | Good First Issue