feature: use fields and covering_fields in index dependency checks before flag 1<<13
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start with rust/lance-table/src/format/overlay/staleness.rs and the reserved flag in rust/lance-table/src/feature_flags.rs; trace the listed pruning, rewrite, withdrawal, and remap_index checks. Compare their dependency logic with the covering-field contract in #10090. Done means each check considers the union of fields and covering_fields before support for the flag ships.
Written by the indexing model from the issue text.
Description
Index staleness and pruning checks look only at IndexMetadata.fields:
- overlay exclusion (
rust/lance-table/src/format/overlay/staleness.rs) prune_updated_fragment_ids_from_indicesregister_pure_rewrite_rows_update_frags_in_indicesprune_overlay_stale_fields_from_indiceswithdraw_rewritten_coverage- the
remap_indexoverlay check
This is correct today. FLAG_INDEPENDENT_COVERING_FIELDS (1 << 13) is reserved and unsupported (rust/lance-table/src/feature_flags.rs), and under the legacy suffix contract covering_fields is a suffix of fields.
Once flag 1 << 13 ships, fields holds only key columns. These checks would then miss updates and overlays on carried columns, and the index would serve stale carried values.
Expected (per #10090 and the existing covering-field contract): every dependency check uses the union of fields and covering_fields. This should land with, or before, support for flag 1 << 13.
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 900
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 371
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 78/100
lance-format/lance#10225 ·
Maintainers usually reply within 3 days
-
feature performance
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
lance-format/lance#10137 ·
Maintainers usually reply within 3 days
-
After initialize_mem_wal, every FTS query logs "ERROR Index __lance_mem_wal has no fields"Possibly taken @lance-gatefixer claimed this 24 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
lance-format/lance#9655 ·
Maintainers usually reply within 3 days
-
Upgrade DataFusion for the unparser stack-overflow fixMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
lance-format/lance#8225 ·
Maintainers usually reply within 3 days
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#6425 · 2 comments ·
Maintainers usually reply within 3 days
All issues in lance-format/lance
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
FavioVazquez/mollify#62 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 61/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
fendouai/awesome-deepseek-harness#125 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
rust-embedded/heapless#694 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
Maintainers usually reply within 1 day