apache/beam

Improve UnboundedSourceShard structure

Open

#18,122 建立於 2022年6月3日

在 GitHub 查看
 (0 留言) (0 反應) (0 負責人)Java (4,097 fork)batch import
P3directgood first issueimprovementrunners

倉庫指標

Star
 (7,313 star)
PR 合併指標
 (平均合併 6天) (30 天內合併 243 個 PR)

描述

Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.

Imported from Jira BEAM-1659. Original Jira may contain additional context. Reported by: tgroh.

貢獻者指南