Kafka Streams runner: read a source's splits in parallel
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- java, kafka
- Domain
- data, distributed-systems, stream-processing
Research direction
Start with ReadTranslator, where split(1, options) currently requires exactly one split, and read the topic-based shuffle work tracked in #18479. The change is done when bounded and unbounded sources can use multiple splits with separate readers and watermarks, distributed across Kafka Streams instances.
Written by the indexing model from the issue text.
Description
Summary
The Kafka Streams runner splits a source into exactly one part and reads it with a single reader, so a source that could be read in parallel is not.
ReadTranslator calls split(1, options) and requires exactly one split back. A source returning more is rejected at translation rather than having the extra splits silently dropped, since that would be data loss, but the effect is that such a source cannot be read at all.
Reading several splits means giving each one its own reader and its own watermark, and distributing them across instances, which belongs with the topic-based shuffle work. Bounded and unbounded sources are both affected.
Part of #18479.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 368
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/beam
-
awaiting triage bug P3
Difficulty 1/5 Under an hour Newbie friendliness 67/100
apache/beam#41197 · 2 comments ·
Maintainers usually reply within 1 day
-
[Bug]: [Python] WriteToJson silently rounds floats to 10 decimal placesPossibly taken @Abhi2939 claimed this 1 day ago. Openio P2 python yaml
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apache/beam#41102 · 4 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/beam#39624 · 2 reactions ·
Maintainers usually reply within 1 day
-
[Bug]: PubsubIO used in batch incorrect batch cutoff sizePossibly taken @1fanwang claimed this 78 days ago. Openbug io P3 pinned pubsub
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/beam#28011 · 4 comments ·
Maintainers usually reply within 1 day
-
build P3 sub-task
Difficulty 1/5 Under an hour Newbie friendliness 68/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
SasanLabs/VulnerableApp#912 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
desktop edt not responding
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
datafusion-contrib/StreamFusion#653 ·
Maintainers usually reply within 1 day
-
[Bug] Absolute row/column index 0 is ignored for images, hyperlinks and commentsPossibly taken @fanOfSunny claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day