quickwit-oss/tantivy

Optimize Segment Readers Alive Doc Iterator

Open

#296 建立於 2018年5月8日

在 GitHub 查看
 (3 留言) (0 反應) (0 負責人)Rust (8,354 star) (499 fork)batch import
enhancementgood first issueperformance

描述

Noted in PR #293

Use the same trick as what is used in the [BitSetDocSet] (https://github.com/tantivy-search/tantivy/blob/78673172d001a0c4c7c73cdd3d9923fc43fc0312/src/query/bitset/mod.rs) which relies on TinySet

In a nutshell : instead of looping on the bitset to find the next thing alive, it uses a special cpu instruction to get the next bit set to 1 in a u64. The instruction is not free, but except if your bitset is EXTREMELY saturated it is generally faster.

https://github.com/tantivy-search/tantivy/pull/293#discussion-diff-186488306R373

貢獻者指南

技術棧
rust
領域
backendperformance
議題類型
performance
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-3 hours
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
Rust proficiencyunderstanding of bitsets and low level optimizationfamiliarity with tantivy's query engine
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
30
研究方向
The issue suggests optimizing the segment readers alive doc iterator by adopting the same technique used in BitSetDocSet (https://github.com/tantivy search/tantivy/blob/78673172d001a0c4c7c73cdd3d9923fc43fc0312/src/query/bitset/mod.rs), which relies on TinySet. The key improvement is to use a CPU instruction (likely x86 BSF or TZCNT) to find the next set bit in a u64, instead of looping. The contributor should examine PR #293 (https://github.com/tantivy search/tantivy/pull/293) and the referenced discussion diff. The implementation should replace the current iteration logic in the segment reader's alive doc iterator with a TinySet based approach, ensuring correctness and performance gains.