apache/pinot

Messures and Doc for index freshness/update

Open

#14371 opened on Nov 4, 2024

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Java (4,937 stars) (1,234 forks)batch import
documentationgood first issue

Description

A part of Pinots secret sauce for speed are indexes. Is there any info available what index freshness/delay is achievable for newly ingested data / just changed data (upsert) for the different kinds of indexes? on how this could be improved/reduced? On how index updating works for each type?

of course I found

but no overview / table summing this topic up.

Contributor guide

Tech stack
java
Domain
documentationdatabaseperformance
Issue type
documentation
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
basic understanding of Pinot indexingaccess to documentation and issues
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
Review the existing documentation links and related issues (e.g., #6431, #13504) to understand current index freshness information. Then, study the code for each index type (e.g., text, inverted, star tree) to describe update mechanisms. Compile findings into a table or overview section in the documentation.