Support reusing a training result across multiple vector index writers
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 45/100
调研方向
首先，跟踪 VectorIndexTrainer::train、提议的 training.create_writer() 入口点，以及 IVF-PQ 在所有受支持索引类型中的现有 from_trained 路径。然后检查统一 API 及其 C、C++、Java/JNI 和 Python 绑定。完成的标志是：独立的 writer 可以复用一个已完成的训练结果，同时保留空 payload、现有的消费 API 和当前的磁盘格式。
由索引模型根据 Issue 内容生成。
描述
Motivation
Callers that build one vector index file per segment currently need a separate training pass for every writer. The trained centroids, quantizers, and rotations can often be shared when those segments come from the same table or data distribution, but the unified API consumes VectorIndexTraining when it creates a writer.
For N segments, this makes the lifecycle effectively:
train -> writer 1
train -> writer 2
...
train -> writer N
IVF-PQ already has an internal from_trained path, but this capability is not available through the unified API or consistently across the supported index types and language bindings.
Proposed API
Allow a completed training result to create multiple independent writers:
let training = VectorIndexTrainer::train(config, training_vectors, training_count)?;
let first = training.create_writer();
let second = training.create_writer();
Each writer should start with the same trained model and an empty vector/ID payload. The existing consuming APIs should remain unchanged for one-shot callers.
Equivalent opt-in entry points should be exposed through C, C++, Java/JNI, and Python.
Expected benefit
In a release-mode synthetic build with 128-dimensional vectors, 8,192 training rows, and eight segments of 2,048 vectors each, training once reduced writer preparation time by 84-87% across IVF-Flat, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN. Including vector insertion and serialization, elapsed time fell by 73-84%.
The benefit comes from reducing N training passes to one; it does not change the index algorithms themselves.
Compatibility and scope
- No default behavior change: callers opt in to reusable training explicitly.
- Existing consuming APIs remain available.
- No on-disk format change.
- No cross-process training-model serialization or index merging in this issue.
- 主要语言
- Rust
- 星标
- 25
- 派生
- 24
- 平均合并
- 1 小时 45 分钟
- 30 天内合并 PR
- 2
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/paimon-vector-index 的其他 Issue
-
Add optional GPU acceleration for IVF-SQ training and construction可能已有人在做 @TheR1sing3un 于 52 天前认领。 未关闭
难度 5/5 一周以上 新手友好度 35/100
apache/paimon-vector-index#100 ·
-
难度 5/5 一周以上 新手友好度 35/100
apache/paimon-vector-index#97 · 2 条评论 ·
-
[Feature] Add CI benchmarks to compare vector index PRs with their base可能已有人在做 @wangzhigang1999 于 55 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 38/100
查看 apache/paimon-vector-index 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
libnativeapi/nativeapi#21 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
longbridge/nsh#804 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复