Incremental ingestion based on datetime field missed 2 hours
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 28/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- postgresql, scala, spark
- Domain
- data-engineering, databases
Research direction
The payload names no source files, so begin by finding where Pramen stores and reads the incremental offset for a datetime column from a PostgreSQL source. Reproduce the 2-hour gap by running an incremental ingestion with the JVM and session timezone set to Africa/Johannesburg (UTC+2) and comparing stored offsets against the source rows. Done means consecutive runs leave no gap, with a test covering the timezone case.
Written by the indexing model from the issue text.
Description
Describe the bug
It is noticed that if an incremental pipeline uses 'datetime' column from PostgreSQL, several consecutive ingestions have gaps of 2 hours.
It is suspected it is related to timezone conversion between UTC and Johannesburg when storing offerts.
Code and/or configuration snippet that caused the issue
--
Expected behavior
Incremental ingestion should ingest contiguous data when a datetime column is used as an offset.
Context
- Pramen/pramen-py version: 1.14.9
- Dominant language
- Scala
- Stars
- 33
- Forks
- 5
- Avg merge
- 6h
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from AbsaOSS/pramen
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 43/100
Maintainers usually reply within 1 day
-
Add a parallelism mode that allows parallel job runs but never the same job for different info datesOpenenhancement
Difficulty 4/5 3-5 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
Jdbc Count doesn't use additional options from JDBC configPossibly taken @qwerty44-U claimed this 118 days ago. Openbug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Add ability to track number of records deviation and report it as warnings in email notificationsOpenenhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
Similar issues
-
area:scale audit priority:P2
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
KD5UJC-XYZ/scarab#231 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
SpinalHDL/VexiiRiscv#198 ·
Maintainers usually reply within 3 days
-
nativeLink relinks every build: IO.write leaves stale trailing bytes in build-checksumPossibly taken @kimmoeklund claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
scala-native/scala-native#5111 ·
Maintainers usually reply within 1 day
-
maintenance
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apex-dev-tools/apex-ls#603 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 2 days