"Total elapsed time" in the JOB DONE line is ~1.79 billion seconds (perf_counter overwrites the time.time() start)
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- python
- Domain
- backend, data-engineering
Research direction
Read pipeline_core.py, jobs/stages/download.py, and the elapsed-time logging in local_runner._log_done and prefect_flow.py. Trace how timer_start is initialized and then reassigned, and check the reported duration on both runner paths. Done means successful jobs report a duration close to their actual runtime, including the Prefect duration_seconds value.
Written by the indexing model from the issue text.
Description
Every successful job on main ef6b6a4 ends with:
DATAPUSHER+ v3.0.0a0 JOB DONE! Total elapsed time: 1,792,617,918.06 seconds.
for a job that took about two seconds.
Cause: pipeline_core.py initialises the context with timer_start=time.time(), but DownloadStage.process (jobs/stages/download.py) reassigns context.timer_start = time.perf_counter() to measure the fetch and never restores it. perf_counter() is a small monotonic value, so the final time.time() - runtime.timer_start in local_runner._log_done (and the equivalent lines in prefect_flow.py, including duration_seconds reported to Prefect) is roughly the Unix epoch.
Suggested fix: give the download stage its own local start variable (or a separate fetch_start field on the context) and leave timer_start untouched.
Observed with prefect_enabled = false on CKAN 2.13.0a0; the same subtraction is used on the Prefect path.
- Dominant language
- Python
- Stars
- 47
- Forks
- 33
- Avg merge
- 2h 28m
- Merged PRs (30d)
- 10
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dathere/datapusher-plus
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
dathere/datapusher-plus#359 ·
Maintainers usually reply within 1 day
-
FormulaProcessing stage always skipped on CKAN 2.10+: ckan.plugins is a list, stage calls .split() on itPossibly taken @tino097 claimed this 14 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
dathere/datapusher-plus#354 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
dathere/datapusher-plus#340 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 53/100
dathere/datapusher-plus#338 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
dathere/datapusher-plus#337 · 1 comment ·
Maintainers usually reply within 1 day
All issues in dathere/datapusher-plus
Similar issues
-
documentation good first issue priority: P2 status: ready
Difficulty 1/5 Under an hour Newbie friendliness 88/100
RhinoLink-Tech/data-fabric#500 ·
Maintainers usually reply within 1 day
-
backlog
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
EchoTools/nevr-runtime#2384 ·
Maintainers usually reply within 1 day
-
[Bug]: The H2 series' and the H3C's fan 3 is labelled "Chamber Fan", Bambu calls it the exhaust fanOpenarea:ui
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
btclib-org/btclib#3062 ·
Maintainers usually reply within 1 day
-
area:evals area:infra enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
jse-avl/support-bot#4 · 1 comment ·