What's the root cause to wiki's recommendation: "Pause consumption when add a new column"? · apache/pinot#12652

(4 评论) (0 反应) (0 负责人)Java (4,937 star) (1,234 fork)batch import

documentationgood first issuetroubleshooting

描述

In the ingestion transformation wiki page, it mentioned that

If a new column is added to table or schema configuration during ingestion, incorrect data may appear in the consuming segment.

To ensure accurate values are reloaded, do the following: Pause consumption (and wait for pause status success)

Is it only limited to table which has ingestion transform config? If not, it seems a breaking change since we need to pause consumption for every schema update.

Another question: In which scenario the data will incorrectly appears?

贡献者指南

技术栈: javasql
领域: databasedocumentation
议题类型: documentation
难度: 3
预计时间: 1-3 hours
活动状态: active
清晰度: clear
前置要求: Apache Pinot basicsIngestion transformations
新手友好度: 40
研究方向: Start by examining the ingestion transformation code in Apache Pinot, particularly the handling of schema changes during segment consumption. Look at the IngestionTransformations class and related consumer logic to understand the root cause of the recommendation to pause consumption. Check for any race conditions or assumptions about schema stability that could lead to incorrect data.