[Bug] Deleted rows take vector search top-k slots in data-evolution tables
維護者通常 3 天內回覆
還沒有人認領這個 Issue。
評估
研究方向
Start with LiveRowGlobalIndexReader::VisitVectorSearch and compare its vector path with the full-text path that loads live rows before searching shards. The issue suggests combining VectorSearch.pre_filter with shard live rows through ReplacePreFilter, using GlobalIndexLiveRowFilter::LiveRows, or loading live rows before fan-out. Add coverage for the reproduction with three deleted nearest rows and verify that a limit of 3 returns 3 live rows when enough matches pass the filters.
由索引模型根據 Issue 內容生成。
描述
Search before asking
- I searched in the issues and found nothing similar.
Paimon-cpp version
main at commit 50ef548ec87cc444a153fb2a03789f9be768d610, with #448 applied.
Minimal reproduce step
- Create a data-evolution table with row tracking and
deletion-vectors.enabled = true. Write 10 rows with a vector column, then build a vector global index on that column. - Delete the three rows nearest to a query vector, using deletion vectors.
- Search for that vector with
limit = 3throughGlobalIndexScan::CreateReader→VisitVectorSearch. - Pass the result to
ScanContextBuilder::SetGlobalIndexResult, then scan and read.
What doesn't meet your expectations?
The index still ranks the deleted rows, so they take all three top-k slots. The read then drops them, because deletion vectors are applied only at read time. The query returns 0 rows instead of 3.
#448 (for #403) excludes deleted rows only from full-text ranking: LiveRowGlobalIndexReader::VisitVectorSearch delegates unchanged, and live rows are loaded only before a full-text search fans out to the shards. Java filters vector search with the same live rows (apache/paimon#8459).
Expected: limit = k returns k live rows whenever at least k live rows pass the search's pre_filter and predicate.
Anything else?
Follow-up of #403 / #448.
Suggestions:
VectorSearchalready carries apre_filterover global row ids, soLiveRowGlobalIndexReader::VisitVectorSearchcan combine it with the live rows of the shard throughReplacePreFilter, reusingGlobalIndexLiveRowFilter::LiveRows, and pass the search on unchanged when the whole shard is live.- Load live rows on the visiting thread before a vector search fans out to the shards, as
LiveRowUnionGlobalIndexReaderdoes for full-text search.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- 主要語言
- C++
- 星號
- 70
- 分支
- 36
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 38
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
apache/paimon-cpp 的其他 Issue
-
[Feature] Split the table hierarchy and materialize the store a table is served by可能已有人在做 @lucasfang 於 5 天前認領。 未關閉
難度 5/5 一週以上 新手友好度 20/100
apache/paimon-cpp#459 ·
維護者通常 3 天內回覆
-
enhancement
難度 5/5 一週以上 新手友好度 35/100
apache/paimon-cpp#452 ·
維護者通常 3 天內回覆
-
難度 5/5 一週以上 新手友好度 8/100
apache/paimon-cpp#435 · 2 則留言 ·
維護者通常 3 天內回覆
-
enhancement
難度 5/5 一週以上 新手友好度 35/100
apache/paimon-cpp#410 ·
維護者通常 3 天內回覆
-
enhancement
難度 5/5 一週以上 新手友好度 25/100
apache/paimon-cpp#409 ·
維護者通常 3 天內回覆
查看 apache/paimon-cpp 的全部 Issue
相似的 Issue
-
難度 2/5 1-3 小時 新手友好度 76/100
k2-fsa/sherpa-onnx#4100 ·
維護者通常 2 天內回覆
-
難度 2/5 1-3 小時 新手友好度 78/100
維護者通常 1 天內回覆
-
Mod: Part Design Topic: User Interface
難度 2/5 1-3 小時 新手友好度 68/100
維護者通常 1 天內回覆
-
A-tools P-medium S-ready-to-start task
難度 2/5 1-3 小時 新手友好度 77/100
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 76/100
sistemicorp/scripts#62 ·