quickwit-oss/tantivy

Generate meaningful SegmentIDs instead of pure random

Open

#971 opened on Jan 7, 2021

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Rust (8,354 stars) (499 forks)batch import
enhancementgood first issuehigh priorityquickwit

Description

Is your feature request related to a problem? Please describe. Related to #969 I would like to suggest a cheap feature which will help debugging in the future. Now SegmentIDs are generated randomly but it is a waste of 16 bytes which could be used to embed debugging info otherwise.

Describe the solution you'd like Generate SegmentID containing the following info:

  • timestamp of segment creation
  • segment origin (merging or writing new data)
  • hash of hostname, probably useful for those who will implement sharding/replication paired with Tantivy.

Additionally here we should ensure that there is left enough randomness to avoid any possibility of collisions on the one hand, and that names are not too long to avoid metadata bloating (? not sure if it is actual, the number of segments is supposed to be relatively low by design afaik) on the other hand.

Contributor guide

Tech stack
rust
Domain
backenddatabase
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
RustTantivy basicsSegment concept
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
55
Research direction
The SegmentID generation logic is likely in the segment module (e.g., segment id.rs). Examine the current random generation and design a new struct that embeds a timestamp, origin flag, and a hash of the hostname while retaining enough randomness. Check issue #969 and the comments in this issue for additional context on requirements.