quickwit-oss/tantivy

Too many small segments

Open

#194 opened on Jun 20, 2017

View on GitHub
 (3 comments) (2 reactions) (0 assignees)Rust (499 forks)batch import
enhancementgood first issuehelp wantedlow priority

Repository metrics

Stars
 (8,354 stars)
PR merge metrics
 (Avg merge 12d 14h) (20 merged PRs in 30d)

Description

Avoid creating 8 segments if commit after 8 documents just because we use 8 threads. Maybe pack documents in blocks of 100 docs?

Contributor guide

Research direction
Investigate how Tantivy creates segments during indexing and propose batching documents into blocks of 100 to reduce the number of small segments.
Tech stack
rust
Domain
backend
Issue type
Performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Rust
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60