Add timeout based on expected execution time of JDBC ingestion
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- scala, spark
- Domain
- data-engineering, databases
Research direction
The issue identifies JDBC ingestion in the Scala and Spark data pipeline but names no files, tests, or entry points. First locate the JDBC ingestion path and its existing execution-time and record-count handling, then clarify the default timeout calculation and whether expiry stops or restarts ingestion. Done means the timeout behavior and expiry action are agreed and covered by tests.
Written by the indexing model from the issue text.
Description
Background
Sometime JDBC connections hang when running Pramen ingestion job.
It can result 1000 records from a DB to be loaded in hours.
Feature
Add timeout based on expected execution time of JDBC ingestion.
Example [Optional]
A simple example if applicable.
Proposed Solution [Optional]
Define minimum execution time for JDBC insertions (even for 1 record), say 10 minutes by default.
Define minimum throughput, say 10 records per second by default.
When the record count is known in advance set the timeout based on above parameters.
Should the ingestion be restarted, or just stopped when the timer has expired - to be decided
- Dominant language
- Scala
- Stars
- 33
- Forks
- 5
- Avg merge
- 6h 22m
- Merged PRs (30d)
- 14
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from AbsaOSS/pramen
-
When a Kafka batch doesn't contain data in all partitions, subsequent ingestions will failPossibly taken @yruslan claimed this 1 day ago. Openbug
Difficulty 3/5 1-2 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 28/100
Maintainers usually reply within 1 day
-
Add a parallelism mode that allows parallel job runs but never the same job for different info datesOpenenhancement
Difficulty 4/5 3-5 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
Jdbc Count doesn't use additional options from JDBC configPossibly taken @qwerty44-U claimed this 114 days ago. Openbug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Add ability to track number of records deviation and report it as warnings in email notificationsOpenenhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
Similar issues
-
agent-ready area/dev-tooling kind/docs-only layer/docs size/XS
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
marola-dev/marola#855 ·
Maintainers usually reply within 1 day
-
llm4s-ollama: bind a base-URL variable for chat sections, and align the documented variable nameOpen
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
scylladb/scylla-migrator#413 ·
-
Difficulty 2/5 Under an hour Newbie friendliness 70/100
-
dependencies
Difficulty 1/5 Under an hour Newbie friendliness 78/100
apache/pekko-http#1354 ·
Maintainers usually reply within 1 day