apache/beam

Improve UnboundedSourceShard structure

Open

#18,122 opened on 2022年6月3日

GitHub で見る
 (0 comments) (0 reactions) (0 assignees)Java (4,097 forks)batch import
P3directgood first issueimprovementrunners

Repository metrics

Stars
 (7,313 stars)
PR merge metrics
 (平均マージ 6d) (30d で 243 merged PRs)

説明

Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.

Imported from Jira BEAM-1659. Original Jira may contain additional context. Reported by: tgroh.

コントリビューターガイド