perf: index coverage silently cleared once the manifest at dataset_version is cleaned up
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Start with prune_stale_segment_coverage in rust/lance/src/index.rs, then trace the merge-coverage history lookups and legacy-segment handling in rust/lance/src/dataset/optimize.rs. Compare those freshness checks with the current-manifest metadata named in the proposal. Done means coverage is retained after the old manifest is cleaned up, without pinning its referenced data files.
Written by the indexing model from the issue text.
Description
Several index paths check out the manifest at a segment's dataset_version to decide which fragments are still fresh. If cleanup_old_versions has removed that manifest, they silently clear the segment's coverage, and queries fall back to flat scans.
Affected paths:
prune_stale_segment_coverage(rust/lance/src/index.rs), used when committing staged segments and in mergeshistory.atlookups in the merge-coverage code (index.rs)- legacy segments without a bitmap (
rust/lance/src/dataset/optimize.rs)
This is likely in practice. Delete, Update, and immediate-remap compaction leave dataset_version unchanged, so it can age past the cleanup retention window. A distributed index build that runs longer than the retention window hits it too.
Results stay correct, but coverage is lost with no warning.
Proposed fix: decide freshness from metadata in the current manifest (overlay committed_versions, FRI version stamps, and per-fragment last-modified versions) instead of diffing against an old manifest. Keeping the old manifest during cleanup was rejected, because it would pin every data file that manifest references.
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 900
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 366
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 78/100
lance-format/lance#10225 ·
Maintainers usually reply within 3 days
-
feature performance
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
lance-format/lance#10137 ·
Maintainers usually reply within 3 days
-
After initialize_mem_wal, every FTS query logs "ERROR Index __lance_mem_wal has no fields"Possibly taken @lance-gatefixer claimed this 24 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
lance-format/lance#9655 ·
Maintainers usually reply within 3 days
-
Upgrade DataFusion for the unparser stack-overflow fixMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
lance-format/lance#8225 ·
Maintainers usually reply within 3 days
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#6425 · 2 comments ·
Maintainers usually reply within 3 days
All issues in lance-format/lance
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 67/100
rust-bitcoin/rust-bitcoin#7089 · 1 comment · 1 reaction ·
Maintainers usually reply within 2 days
-
area/tools bug session-analysis
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
[Bug]: Python OpenTelemetrySectionConfig has no file_sinksMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
NVIDIA/NeMo-Relay#1514 ·
Maintainers usually reply within 1 day
-
prompting proposal tool triaged
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
can1357/oh-my-pi#17041 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
nix-community/fenix#262 ·