apache/beam

Improve UnboundedSourceShard structure

开放

#18,122 创建于 2022年6月3日

 (0 条评论) (0 个反应) (0 位负责人)Java (4,097 个派生)batch import
P3directgood first issueimprovementrunners

仓库指标

星标
 (7,313 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.

Imported from Jira BEAM-1659. Original Jira may contain additional context. Reported by: tgroh.

贡献者指南