Kafka Streams runner: side inputs
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- java, kafka
- Domain
- stream-processing
Research direction
Start with the Kafka Streams runner code that currently uses StateRequestHandler.unsupported(), then read BatchSideInputHandlerFactory and StateRequestHandlers.forSideInputHandlerFactory as the Flink reference. The work is done when side input state is served to the SDK harness during an active bundle and the stage watermark waits for each side input to be ready.
Written by the indexing model from the issue text.
Description
The Kafka Streams runner executes stages with StateRequestHandler.unsupported(), so a pipeline using side inputs cannot run.
Supporting them means serving side input state to the SDK harness while a bundle is in flight, and holding the stage's watermark until each side input is ready. BatchSideInputHandlerFactory and StateRequestHandlers.forSideInputHandlerFactory are how the Flink runner does it.
Part of #18479.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 368
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/beam
-
awaiting triage bug P3
Difficulty 1/5 Under an hour Newbie friendliness 67/100
apache/beam#41197 · 2 comments ·
Maintainers usually reply within 1 day
-
[Bug]: [Python] WriteToJson silently rounds floats to 10 decimal placesPossibly taken @Abhi2939 claimed this 1 day ago. Openio P2 python yaml
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apache/beam#41102 · 4 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/beam#39624 · 2 reactions ·
Maintainers usually reply within 1 day
-
[Bug]: PubsubIO used in batch incorrect batch cutoff sizePossibly taken @1fanwang claimed this 78 days ago. Openbug io P3 pinned pubsub
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/beam#28011 · 4 comments ·
Maintainers usually reply within 1 day
-
build P3 sub-task
Difficulty 1/5 Under an hour Newbie friendliness 68/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
SasanLabs/VulnerableApp#912 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
desktop edt not responding
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
datafusion-contrib/StreamFusion#653 ·
Maintainers usually reply within 1 day
-
[Bug] Absolute row/column index 0 is ignored for images, hyperlinks and commentsPossibly taken @fanOfSunny claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day