Kafka Streams runner: finalize unbounded source checkpoints

Open
  • Is something wrong?
#39,634 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
java, kafka
Domain
stream-processing

Research direction

Start in the Kafka Streams runner at the checkpoint persistence and Kafka Streams commit path, then trace CheckpointMark.finalizeCheckpoint(). Done means unbounded source marks are finalized only after durable commit, and bounded splits are committed only after complete processing and downstream flush.

Written by the indexing model from the issue text.

Description

The Kafka Streams runner stores an unbounded source's checkpoint mark so a restart resumes from it, but never calls CheckpointMark.finalizeCheckpoint(). Sources that rely on finalization to acknowledge or release data will not see it.

A mark should only be finalized once it is durably committed, which needs a hook that runs before Kafka Streams commits. The runner does not have one — the same gap that prevents bundles being bounded by time. A related point from review: a bounded split should likewise only be committed after it has been completely processed and flushed downstream.

Part of #18479.

Dominant language
Java
Stars
8.7k
Forks
4.8k
Avg merge
2d 9h
Merged PRs (30d)
368

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/beam

All issues in apache/beam

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.