bug: MemWAL index catch-up checks stored bitmaps, never proves coverage under tagged FRI
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 52/100
Research direction
Start with covers_read_version in rust/lance-table/src/transaction/manifest_build.rs, then compare its coverage check with how load_indices derives coverage from the union of segments. Add a regression test for a tagged FRI history with deferred compaction; done means catch-up proves coverage using live fragments and trims the SSTables.
Written by the indexing model from the issue text.
Description
MemWAL index catch-up decides whether an index covers the read version by testing stored fragment_bitmaps against the fragments at that version (covers_read_version in rust/lance-table/src/transaction/manifest_build.rs).
Under a tagged FRI history (FRI index version 1), a segment's stored bitmap is provenance. It holds the retired source fragment ids and never the destinations. So after any deferred compaction, coverage is never proven, and SSTables are never trimmed.
Not confirmed by a test; this is from reading the code.
Expected (per #10090): use live coverage, i.e. the bitmap mapped through the FRI and intersected with the live fragments. Under a tagged history, derive it for the whole index from the union of its segments, the way load_indices does.
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 900
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 366
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 78/100
lance-format/lance#10225 ·
Maintainers usually reply within 3 days
-
feature performance
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
lance-format/lance#10137 ·
Maintainers usually reply within 3 days
-
After initialize_mem_wal, every FTS query logs "ERROR Index __lance_mem_wal has no fields"Possibly taken @lance-gatefixer claimed this 24 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
lance-format/lance#9655 ·
Maintainers usually reply within 3 days
-
Upgrade DataFusion for the unparser stack-overflow fixMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
lance-format/lance#8225 ·
Maintainers usually reply within 3 days
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#6425 · 2 comments ·
Maintainers usually reply within 3 days
All issues in lance-format/lance
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
manuscript-lang/manuscript#1364 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
fable-compiler/Fable#5157 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
C-bug needs-triage O-UEFI T-libs
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day