quickwit-oss/tantivy

Missing search results with in memory index

Open

#1,824 opened on Jan 23, 2023

View on GitHub
 (7 comments) (0 reactions) (0 assignees)Rust (8,354 stars) (499 forks)batch import
good first issuehelp wanted

Description

I've been working on a small command line app to merge bibtex files and have been using tantivy to index paper titles. I noticed some strange behavior where if I (from python extension):

  1. Create an in memory index
  2. Call writer.add_document in a for loop
  3. Call writer.commit() outside the for loop The search results I'd expect to be exact matches (duplicate detection) are not showing up. I reran the same code several times and every so often I do get the correct results, which suggested to me that on commit, some index entries were getting lost. If I have writer.commit inside the for loop, the issue goes away. Likewise, if I use a file-based index, the issue also goes away.

Reading the docs, it looks like even with an in memory index, calling writer.commit() after the for loop and not in it should work, is there something I'm missing or might this be a bug?

If it's not a known issue already and you cant reproduce it, once I finish with my cli I can share the relevant code.

Contributor guide

Tech stack
rustpython
Domain
backend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
Familiarity with tantivy indexingBasic understanding of RustKnowledge of Python bindings (optional)
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Investigate the in memory index commit behavior in tantivy. The issue reporter observed that calling `writer.commit()` after a loop of `add document` calls leads to missing search results, while committing inside the loop works. Check if there is a known race condition or bug in the in memory index's commit logic. Review the tantivy source code, particularly the `IndexWriter::commit` method and how it handles in memory vs file based indexes. Look at related issues or pull requests in the repository for similar reports.
Missing search results with in memory index · quickwit-oss/tantivy#1824 | Good First Issue