apache/beam

Improve UnboundedSourceShard structure

開放

#18,122 建立於 2022年6月3日

 (0 則留言) (0 個反應) (0 位負責人)Java (4,097 個分叉)batch import
P3directgood first issueimprovementrunners

倉庫指標

星標
 (7,313 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.

Imported from Jira BEAM-1659. Original Jira may contain additional context. Reported by: tgroh.

貢獻者指南