What's the root cause to wiki's recommendation: "Pause consumption when add a new column"? · apache/pinot#12652

(4 comments) (0 reactions) (0 assignees)Java (4,937 stars) (1,234 forks)batch import

documentationgood first issuetroubleshooting

説明

In the ingestion transformation wiki page, it mentioned that

If a new column is added to table or schema configuration during ingestion, incorrect data may appear in the consuming segment.

To ensure accurate values are reloaded, do the following: Pause consumption (and wait for pause status success)

Is it only limited to table which has ingestion transform config? If not, it seems a breaking change since we need to pause consumption for every schema update.

Another question: In which scenario the data will incorrectly appears?

コントリビューターガイド

技術スタック: javasql
領域: databasedocumentation
Issue 種別: documentation
難度: 3
推定時間: 1-3 hours
活動状況: active
明確さ: clear
前提条件: Apache Pinot basicsIngestion transformations
初心者向け度: 40
調査方針: Start by examining the ingestion transformation code in Apache Pinot, particularly the handling of schema changes during segment consumption. Look at the IngestionTransformations class and related consumer logic to understand the root cause of the recommendation to pause consumption. Check for any race conditions or assumptions about schema stability that could lead to incorrect data.