Qualify Tantivy pruning for small-window ordinary ANY searches

Aperta
  • Qualcosa non va?
#73 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@kylebernhardy ci sta già lavorando.

Dal 1/11/2026.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
performance, search

Direzione di ricerca

Start with src/engine.rs at commit e1cc2c160a32a5f0bdbf97f8f547942de11f2e2b and compare its stable collector routing with Tantivy's pruned TopDocs and Count paths. Build representative benchmarks covering the listed query, segment, deletion, tie, offset, and ID cases; verify result and error semantics before measuring interleaved optimized runs. Done when a reproducible benefit justifies a routing change, or the retained decision and its evidence are documented.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Outcome

Determine whether small-window ordinary ANY queries can reuse Tantivy's pruned top-hit collection while preserving stable results.

Evidence and limits

Code-trace evidence from the reviewed engine: multi-clause unfiltered ANY queries choose the exhaustive stable-ID collector. Tantivy's score-only TopDocs uses specialized pruning; the wrapper already uses it for other ordinary requests with stable-tie fallback. The collector comparison recorded in Upgrade fulltext to Tantivy 0.27 with safe index rebuilds compares stable composite sorting, and does not establish the best small-window strategy. No measured regression or production blocker is claimed.

Acceptance

  • Compare current stable collection with supported Tantivy TopDocs plus stable-tie fallback for small windows and approximate totals. Include common/rare terms, large match sets, ties, multiple segments, offset boundaries, Unicode IDs and deletes.
  • Prove identical IDs, score bits, stable order and total-relation semantics. Preserve scorer/for_each equivalence and missing-ID error policy.
  • Measure p50/p95/p99, CPU, allocations and matching visits with interleaved optimized runs. Keep separate top-hit/count and combined-collector candidates: combining can lose specialized pruning/count paths.
  • Change routing only for a reproducible benefit beyond baseline noise; otherwise document the retained decision. Preserve bounded memory and ordinary-query progress.

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API. Separate from Reduce repeated matching work during deep ranked traversal.

Priority P3: speculative optimization without a committed roadmap lane or measured incident. Effort Medium: existing collectors can be reused, but equivalence and representative benchmarks need dedicated work. Task; no milestone, customer context or security incident.

Review-Deferral: 4aec767f4ec6f3d4 severity=minor category=performance

Preserve mandatory ID-presence checks across the retained window. Investigate narrowing dictionary string resolution to the returned page and boundary-straddling score-tie groups, with an explicit policy for dangling ordinals and invalid UTF-8 outside that subset. Existing local optimized macOS no-version deep-page p50 was 0.767 ms before / 0.786 ms after the version-hydration change; this bounds existing cost, not a measured win for the proposed ID optimization.

Retained-window ID validation

The planning review for Upgrade fulltext to Tantivy 0.27 with safe index rebuilds identified an additional candidate: resolve IDs only for equal-score groups intersecting the requested page, rather than the entire retained TopDocs window. This remains unimplemented and unmeasured.

  • Evaluate score-group-restricted ID resolution with a byte-exact comparator and explicit treatment of signed zero/non-finite scores. Compare long IDs as well as long optional versions.
  • Decide and document mandatory-ID corruption/error coverage before narrowing reads. The existing ordinary path validates IDs across the retained window; stable collection checks every match for a missing ID ordinal, then resolves and validates dictionary strings for retained hits. Do not silently reduce that coverage merely to improve timings.

Exact-total collector comparison

Current routing uses the stable collectors' match counts when available, reuses an already exact bounded result, and otherwise runs a separate native Count. Tantivy's standalone Count delegates to Weight::count; deletion masks change that path, so comparisons must cover both clean and deletion-bearing segments. A tuple uses generic combined collection and loses the score-only collector's specialized pruning path.

  • Compare separate native TopDocs::order_by_score() then Count against the native (TopDocs, Count) tuple on exact-total requests. Include ordinary single-clause, filtered/candidate and multi-clause ANY workloads, shallow/deep windows, rare/common terms, deletes and multiple segments.
  • Measure scan visits, CPU, allocations and latency distributions: the tuple avoids a second search but uses general collection instead of the score collector's specialized pruning. Preserve count accuracy, score bits, stable ID order, timeout/error coverage and bounded resources.
  • Keep approximate-total routing independent; prove any routing threshold with interleaved optimized measurements rather than assuming one pass is faster. Record the chosen policy even if no change wins.

Exact-total routing remains deferred from the upgrade PR; no unmeasured collector switch is included.

Comment generated by kAIle (GPT-6)

Lingua principale
Rust
Stelle
3
Fork
0
Merge medio
15h 12m
PR unite (30g)
11

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di HarperFast/fulltext

Tutte le issue di HarperFast/fulltext

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.