apache/pinot

What's the root cause to wiki's recommendation: "Pause consumption when add a new column"?

Open

#12,652 创建于 2024年3月14日

在 GitHub 查看
 (4 评论) (0 反应) (0 负责人)Java (1,234 fork)batch import
documentationgood first issuetroubleshooting

仓库指标

Star
 (4,937 star)
PR 合并指标
 (平均合并 6天 7小时) (30 天内合并 186 个 PR)

描述

In the ingestion transformation wiki page, it mentioned that

If a new column is added to table or schema configuration during ingestion, incorrect data may appear in the consuming segment.

To ensure accurate values are reloaded, do the following: Pause consumption (and wait for pause status success)

Is it only limited to table which has ingestion transform config? If not, it seems a breaking change since we need to pause consumption for every schema update.

Another question: In which scenario the data will incorrectly appears?

贡献者指南