apache/pinot

What's the root cause to wiki's recommendation: "Pause consumption when add a new column"?

Open

#12652 opened on Mar 14, 2024

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Java (4,937 stars) (1,234 forks)batch import
documentationgood first issuetroubleshooting

Description

In the ingestion transformation wiki page, it mentioned that

If a new column is added to table or schema configuration during ingestion, incorrect data may appear in the consuming segment.

To ensure accurate values are reloaded, do the following: Pause consumption (and wait for pause status success)

Is it only limited to table which has ingestion transform config? If not, it seems a breaking change since we need to pause consumption for every schema update.

Another question: In which scenario the data will incorrectly appears?

Contributor guide

Tech stack
javasql
Domain
databasedocumentation
Issue type
documentation
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Apache Pinot basicsIngestion transformations
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Start by examining the ingestion transformation code in Apache Pinot, particularly the handling of schema changes during segment consumption. Look at the IngestionTransformations class and related consumer logic to understand the root cause of the recommendation to pause consumption. Check for any race conditions or assumptions about schema stability that could lead to incorrect data.