Merging diff files
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- cpp
- Domain
- data-engineering
Research direction
No files, tests, or entry points are named in the issue. Start by locating the existing minutely, hourly, and daily diff processing, then clarify the aggregation, reset, empty-output, and partial-work requirements. Done means the behavior is agreed and implemented for missed intervals and safe regeneration.
Written by the indexing model from the issue text.
Description
To fully replace Osmosis, we need to merge minutely diff files into hourly and daily diffs and I am trying to figure out exactly how this should be done.
In normal operation the program will run
- once an hour (shortly after the full hour) to assemble 60 minutely diffs into one hourly diff and
- once a day (shortly after midnight) to assemble 24 hourly diffs into one daily diff.
So far that's easy enough. But the question is: What should happen in special situations, say when the program didn't run for some reason and there are older minutelies lying around etc.
What I am bit unclear about is how osmdbt should figure out which minutelies to assemble into an hourly. If if just takes the next 60 it hasn't processed yet, the result might not correspond to full hours, because there might be minutelies missing. But it can't really look at the timestamps either, because its a bit fuzzy how they are generated. The Osmosis code is really hard to read so I haven't figured out yet what it is doing exactly.
In addition to that I would expect something like the following behaviour:
- For every hour fully in the past, if there is at least one minutely, create the hourly from all minutelies in that hour. And the analogue for the daylies.
- Independently of when it is run, hours that are not "done" should not be processed. So if the program is run at half past, it should ignore the 30 minutely files lying around. Same for days.
- Don't create empty diff files. This could happen if the server is down for some extended period.
- It should be possible to "reset" the program to some earlier point in time, so that if we have created defective hourly or daily files for some reason, we can re-generate them easily.
- The diff and state files should be generated outside the directory tree and synced before being moved in place, so that if the program dies for some reason, no partial output is generated. The next run should ignore partial work done before and do the right thing.
Any opinions on these issues?
- Dominant language
- C++
- Stars
- 27
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openstreetmap/osmdbt
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
openstreetmap/osmdbt#50 · 5 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
openstreetmap/osmdbt#48 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 58/100
openstreetmap/osmdbt#46 · 2 comments ·
-
osmdbt and pgoutputOpen
Difficulty 4/5 3-5 days Newbie friendliness 25/100
openstreetmap/osmdbt#38 · 3 comments · 1 reaction ·
All issues in openstreetmap/osmdbt
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
useless statementOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
grizzlypeak3d/DJV#962 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day