Qualify Tantivy pruning for small-window ordinary ANY searches

Abierto
  • ¿Algo no está bien?
#73 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@kylebernhardy ya está trabajando en esto.

Desde el 1/11/2026.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust
Área
performance, search

Línea de trabajo

Start with src/engine.rs at commit e1cc2c160a32a5f0bdbf97f8f547942de11f2e2b and compare its stable collector routing with Tantivy's pruned TopDocs and Count paths. Build representative benchmarks covering the listed query, segment, deletion, tie, offset, and ID cases; verify result and error semantics before measuring interleaved optimized runs. Done when a reproducible benefit justifies a routing change, or the retained decision and its evidence are documented.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Outcome

Determine whether small-window ordinary ANY queries can reuse Tantivy's pruned top-hit collection while preserving stable results.

Evidence and limits

Code-trace evidence from the reviewed engine: multi-clause unfiltered ANY queries choose the exhaustive stable-ID collector. Tantivy's score-only TopDocs uses specialized pruning; the wrapper already uses it for other ordinary requests with stable-tie fallback. The collector comparison recorded in Upgrade fulltext to Tantivy 0.27 with safe index rebuilds compares stable composite sorting, and does not establish the best small-window strategy. No measured regression or production blocker is claimed.

Acceptance

  • Compare current stable collection with supported Tantivy TopDocs plus stable-tie fallback for small windows and approximate totals. Include common/rare terms, large match sets, ties, multiple segments, offset boundaries, Unicode IDs and deletes.
  • Prove identical IDs, score bits, stable order and total-relation semantics. Preserve scorer/for_each equivalence and missing-ID error policy.
  • Measure p50/p95/p99, CPU, allocations and matching visits with interleaved optimized runs. Keep separate top-hit/count and combined-collector candidates: combining can lose specialized pruning/count paths.
  • Change routing only for a reproducible benefit beyond baseline noise; otherwise document the retained decision. Preserve bounded memory and ordinary-query progress.

Parent: Epic: Extend fulltext search capabilities beyond the initial wrapper API. Separate from Reduce repeated matching work during deep ranked traversal.

Priority P3: speculative optimization without a committed roadmap lane or measured incident. Effort Medium: existing collectors can be reused, but equivalence and representative benchmarks need dedicated work. Task; no milestone, customer context or security incident.

Review-Deferral: 4aec767f4ec6f3d4 severity=minor category=performance

Preserve mandatory ID-presence checks across the retained window. Investigate narrowing dictionary string resolution to the returned page and boundary-straddling score-tie groups, with an explicit policy for dangling ordinals and invalid UTF-8 outside that subset. Existing local optimized macOS no-version deep-page p50 was 0.767 ms before / 0.786 ms after the version-hydration change; this bounds existing cost, not a measured win for the proposed ID optimization.

Retained-window ID validation

The planning review for Upgrade fulltext to Tantivy 0.27 with safe index rebuilds identified an additional candidate: resolve IDs only for equal-score groups intersecting the requested page, rather than the entire retained TopDocs window. This remains unimplemented and unmeasured.

  • Evaluate score-group-restricted ID resolution with a byte-exact comparator and explicit treatment of signed zero/non-finite scores. Compare long IDs as well as long optional versions.
  • Decide and document mandatory-ID corruption/error coverage before narrowing reads. The existing ordinary path validates IDs across the retained window; stable collection checks every match for a missing ID ordinal, then resolves and validates dictionary strings for retained hits. Do not silently reduce that coverage merely to improve timings.

Exact-total collector comparison

Current routing uses the stable collectors' match counts when available, reuses an already exact bounded result, and otherwise runs a separate native Count. Tantivy's standalone Count delegates to Weight::count; deletion masks change that path, so comparisons must cover both clean and deletion-bearing segments. A tuple uses generic combined collection and loses the score-only collector's specialized pruning path.

  • Compare separate native TopDocs::order_by_score() then Count against the native (TopDocs, Count) tuple on exact-total requests. Include ordinary single-clause, filtered/candidate and multi-clause ANY workloads, shallow/deep windows, rare/common terms, deletes and multiple segments.
  • Measure scan visits, CPU, allocations and latency distributions: the tuple avoids a second search but uses general collection instead of the score collector's specialized pruning. Preserve count accuracy, score bits, stable ID order, timeout/error coverage and bounded resources.
  • Keep approximate-total routing independent; prove any routing threshold with interleaved optimized measurements rather than assuming one pass is faster. Record the chosen policy even if no change wins.

Exact-total routing remains deferred from the upgrade PR; no unmeasured collector switch is included.

Comment generated by kAIle (GPT-6)

Lenguaje dominante
Rust
Estrellas
3
Forks
0
Merge medio
15 h 12 min
PR fusionados (30 d)
11

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de HarperFast/fulltext

Todos los issues de HarperFast/fulltext

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.