Qualify Tantivy pruning for small-window ordinary ANY searches

Đang mở
  • Có gì không ổn?
#73 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@kylebernhardy đang làm issue này rồi.

Từ ngày 1/11/2026.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
25/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust
Lĩnh vực
performance, search

Hướng nghiên cứu

Start with src/engine.rs at commit e1cc2c160a32a5f0bdbf97f8f547942de11f2e2b and compare its stable collector routing with Tantivy's pruned TopDocs and Count paths. Build representative benchmarks covering the listed query, segment, deletion, tie, offset, and ID cases; verify result and error semantics before measuring interleaved optimized runs. Done when a reproducible benefit justifies a routing change, or the retained decision and its evidence are documented.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Outcome

Determine whether small-window ordinary ANY queries can reuse Tantivy's pruned top-hit collection while preserving stable results.

Evidence and limits

Code-trace evidence from the reviewed engine: multi-clause unfiltered ANY queries choose the exhaustive stable-ID collector. Tantivy's score-only TopDocs uses specialized pruning; the wrapper already uses it for other ordinary requests with stable-tie fallback. The collector comparison recorded in Upgrade fulltext to Tantivy 0.27 with safe index rebuilds compares stable composite sorting, and does not establish the best small-window strategy. No measured regression or production blocker is claimed.

Acceptance

  • Compare current stable collection with supported Tantivy TopDocs plus stable-tie fallback for small windows and approximate totals. Include common/rare terms, large match sets, ties, multiple segments, offset boundaries, Unicode IDs and deletes.
  • Prove identical IDs, score bits, stable order and total-relation semantics. Preserve scorer/for_each equivalence and missing-ID error policy.
  • Measure p50/p95/p99, CPU, allocations and matching visits with interleaved optimized runs. Keep separate top-hit/count and combined-collector candidates: combining can lose specialized pruning/count paths.
  • Change routing only for a reproducible benefit beyond baseline noise; otherwise document the retained decision. Preserve bounded memory and ordinary-query progress.

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API. Separate from Reduce repeated matching work during deep ranked traversal.

Priority P3: speculative optimization without a committed roadmap lane or measured incident. Effort Medium: existing collectors can be reused, but equivalence and representative benchmarks need dedicated work. Task; no milestone, customer context or security incident.

Review-Deferral: 4aec767f4ec6f3d4 severity=minor category=performance

Preserve mandatory ID-presence checks across the retained window. Investigate narrowing dictionary string resolution to the returned page and boundary-straddling score-tie groups, with an explicit policy for dangling ordinals and invalid UTF-8 outside that subset. Existing local optimized macOS no-version deep-page p50 was 0.767 ms before / 0.786 ms after the version-hydration change; this bounds existing cost, not a measured win for the proposed ID optimization.

Retained-window ID validation

The planning review for Upgrade fulltext to Tantivy 0.27 with safe index rebuilds identified an additional candidate: resolve IDs only for equal-score groups intersecting the requested page, rather than the entire retained TopDocs window. This remains unimplemented and unmeasured.

  • Evaluate score-group-restricted ID resolution with a byte-exact comparator and explicit treatment of signed zero/non-finite scores. Compare long IDs as well as long optional versions.
  • Decide and document mandatory-ID corruption/error coverage before narrowing reads. The existing ordinary path validates IDs across the retained window; stable collection checks every match for a missing ID ordinal, then resolves and validates dictionary strings for retained hits. Do not silently reduce that coverage merely to improve timings.

Exact-total collector comparison

Current routing uses the stable collectors' match counts when available, reuses an already exact bounded result, and otherwise runs a separate native Count. Tantivy's standalone Count delegates to Weight::count; deletion masks change that path, so comparisons must cover both clean and deletion-bearing segments. A tuple uses generic combined collection and loses the score-only collector's specialized pruning path.

  • Compare separate native TopDocs::order_by_score() then Count against the native (TopDocs, Count) tuple on exact-total requests. Include ordinary single-clause, filtered/candidate and multi-clause ANY workloads, shallow/deep windows, rare/common terms, deletes and multiple segments.
  • Measure scan visits, CPU, allocations and latency distributions: the tuple avoids a second search but uses general collection instead of the score collector's specialized pruning. Preserve count accuracy, score bits, stable ID order, timeout/error coverage and bounded resources.
  • Keep approximate-total routing independent; prove any routing threshold with interleaved optimized measurements rather than assuming one pass is faster. Record the chosen policy even if no change wins.

Exact-total routing remains deferred from the upgrade PR; no unmeasured collector switch is included.

Comment generated by kAIle (GPT-6)

Ngôn ngữ chính
Rust
Star
3
Fork
0
Merge trung bình
15 giờ 12 phút
Pull request đã merge (30 ngày)
11

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của HarperFast/fulltext

Tất cả issue của HarperFast/fulltext

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.