When a Kafka batch doesn't contain data in all partitions, subsequent ingestions will fail
Maintainers usually reply within 1 day
A pull request for this has already been merged.
- #821 by @yruslan — merged
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- kafka, scala, spark
- Domain
- data-engineering, stream-processing
Research direction
The error text 'Cannot compare Kafka offsets with different number of partitions' is the place to start: search the repository for it to find where the offset maps from two batches are compared. Reproduce by running two consecutive ingestions where the second batch has no data for some partitions, then check how the stored offset map is read back. Done means the second ingestion succeeds with a partition gap and an existing test or a new one covers that case.
Written by the indexing model from the issue text.
Description
Describe the bug
When a Kafka batch doesn't contain data in all partitions, subsequent ingestions will fail:
Caused by java.lang.IllegalArgumentException: Cannot compare Kafka offsets with different
number of partitions: 9 and 6
({"0":1623,"1":1541,"2":1815,"3":1273,"4":1422,"5":1562,"6":1373,"7":1458,"8":1436}
vs
{"0":1702,"1":1733,"2":1945,"4":1637,"5":1681,"7":1597}).
Code and/or configuration snippet that caused the issue
--
Expected behavior
Kafka ingestion should work in presence of gaps in partition for a given batch.
Context
- Pramen/pramen-py version: 1.14.9
- Dominant language
- Scala
- Stars
- 33
- Forks
- 5
- Avg merge
- 6h
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from AbsaOSS/pramen
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 43/100
Maintainers usually reply within 1 day
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 28/100
Maintainers usually reply within 1 day
-
Add a parallelism mode that allows parallel job runs but never the same job for different info datesOpenenhancement
Difficulty 4/5 3-5 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
Jdbc Count doesn't use additional options from JDBC configPossibly taken @qwerty44-U claimed this 115 days ago. Openbug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
Similar issues
-
AI Auto-validate
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
ProjectSidewalk/SidewalkWebpage#5788 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
scalameta/metals-zed#99 ·
-
area:ops priority:P2 status:idea
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
KD5UJC-XYZ/scarab#112 ·
-
Bound wrong-magic P2P diagnostics to parsed header bytesPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
ergoplatform/ergo#2736 ·
Maintainers usually reply within 1 day
-
perf: use extend when buffering partition row indices in the native shuffle writerPossibly taken @HarshilMital claimed this 1 day ago. Openarea:shuffle enhancement good first issue performance requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-comet#7072 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day