quickwit-oss/tantivy

Too many small segments

Open

#194 opened on Jun 20, 2017

View on GitHub
 (3 comments) (2 reactions) (0 assignees)Rust (8,354 stars) (499 forks)batch import
enhancementgood first issuehelp wantedlow priority

Description

Avoid creating 8 segments if commit after 8 documents just because we use 8 threads. Maybe pack documents in blocks of 100 docs?

Contributor guide

Tech stack
rust
Domain
backendperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Rust programmingConcurrent indexingTantivy internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Investigate the Tantivy indexing code, particularly the multithreaded indexer that creates segments. Look at the logic that decides when to flush a new segment. Consider implementing a block size limit (e.g., 100 documents) to reduce the number of small segments. Review any comments in the issue for additional context. Check the repository's contribution guidelines for testing procedures.