apache/beam

Improve UnboundedSourceShard structure

オープン

#18,122 opened on 2022/06/03

 (0 件のコメント) (0 件のリアクション) (0 人の担当者)Java (4,097 件のフォーク)batch import
P3directgood first issueimprovementrunners

Repository metrics

Stars
 (7,313 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.

Imported from Jira BEAM-1659. Original Jira may contain additional context. Reported by: tgroh.

コントリビューターガイド