Reuse arrangements for FULL reads in occurrence-bound runs
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 22/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- c
- Domain
- databases, performance
Research direction
Start by profiling the insert route with the overlapping CSPA workload to confirm how much time goes to rebuilding join hashes. The skip logic is in wirelog/columnar/join.c near lines 1356, 1769 and 1818, reached from wl_columnar_eval_tdd_run_bound_slice_begin() in wirelog/columnar/eval.c. Done means the arrangement-staleness test passes, the insert route takes at most half the full evaluation time at W=1 with identical tuple counts, and the three named tests stay green.
Written by the indexing model from the issue text.
Description
Summary
The occurrence-bound runner used by the #2114 insert route (wl_columnar_eval_tdd_run_bound_slice_begin(), wirelog/columnar/eval.c; join paths in wirelog/columnar/join.c) skips the arrangement and materialisation caches whenever sess->tdd_input_run is set (join.c around lines 1356, 1769, 1818). So every FULL read of a large relation -- a recursive head or a big EDB relation -- rebuilds its join hash in every instance and every round. With overlapping CSPA inserts the insert route takes 2.2-2.3 s (see #2122); these rebuilt hashes are a likely large part of that, though the split has not been profiled yet -- profiling it is the first step.
Proposed change
Let a bound FULL read of a registered, unpartitioned coordinator relation reuse that relation's primary arrangement (or build it once per round and share it across the round's instances), keyed by the relation's view and storage generation so a head that changed at the end of the previous round is never read through a stale index. DELTA reads keep building their own.
Acceptance
- Arrangement-staleness test: an instance in round i+1 joining against a head that gained rows at the end of round i sees those rows.
- The overlapping-insert CSPA measurement at W=1 improves measurably (target: insert route at most half the full evaluation's time), with identical tuple counts.
tests/test_tdd_insert_run.c,tests/test_tdd_insert_stratum.c,tests/test_incremental_snapshot_recursive.cstay green; no change for the #1984 manifest forms.
- Dominant language
- C
- Stars
- 18
- Forks
- 2
- Avg merge
- 9h 48m
- Merged PRs (30d)
- 165
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from semantic-reasoning/wirelog
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
semantic-reasoning/wirelog#2229 ·
Maintainers usually reply within 1 day
-
Doc: insert executor and incremental consolidation leave sorted runs, not globally sorted headsOpen
Difficulty 1/5 Under an hour Newbie friendliness 83/100
semantic-reasoning/wirelog#2133 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
semantic-reasoning/wirelog#2042 ·
Maintainers usually reply within 1 day
-
p2-medium
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
semantic-reasoning/wirelog#1914 ·
Maintainers usually reply within 1 day
-
enhancement track/semantics
Difficulty 5/5 Over a week Newbie friendliness 12/100
semantic-reasoning/wirelog#2305 ·
Maintainers usually reply within 1 day
All issues in semantic-reasoning/wirelog
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 70/100
ly05010419/MacNativeGames#1340 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
InukaAsith/TwinDesk#84 ·
Maintainers usually reply within 2 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day