Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

bug: compaction drops BTREE coverage on stable-row-id tables since #9481

Open
  • Is something wrong?
#10,013 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

  • #10122 by @Tyagiquamar — closed without merging

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python, rust
Domain
databases

Research direction

Start by running the reproduction script to confirm describe_indices() shows empty fragment coverage after compact_files(). The branch to read is results_are_row_addrs && !deferred_remap in rust/lance-table/src/transaction/manifest_build.rs, together with frag_reuse_with_stable_row_ids_enabled(). Done means BTREE keeps its fragment coverage after compaction on stable-row-id tables in a release build, without requiring the debug-only path.

Written by the indexing model from the issue text.

Description

bug

On a table with stable row ids, a compaction now leaves a BTREE index covering none of the fragments it rewrote, so filters on that column fall back to a full scan until the next optimize_indices. Answers stay correct. This started with #9481; 14.0.0b15 keeps the coverage.

import shutil

import lance
import pyarrow as pa

path = "/tmp/btree_coverage.lance"
shutil.rmtree(path, ignore_errors=True)
ds = lance.write_dataset(
    pa.table({"v": pa.array(range(1000))}),
    path,
    max_rows_per_file=100,
    enable_stable_row_ids=True,
)
ds.create_scalar_index("v", "BTREE")
ds.optimize.compact_files()

ds = lance.dataset(path)
print([sorted(s.fragment_ids) for s in ds.describe_indices()[0].segments])
print("ScalarIndexQuery" in ds.scanner(filter="v = 7").explain_plan())

With the published wheels this prints [[10]] / True on 14.0.0b15 and [[]] / False on 14.0.0b16. Without stable row ids, b16 prints [[10]] / True.

#9481 moved BTREE to the row-address domain. When a stable-row-id table is compacted without deferring index remap, the manifest build moves a row-id-domain index's coverage to the new fragments but drops the rewritten fragments from an address-domain index's coverage, since that index's stored addresses point into fragments the rewrite removed (the results_are_row_addrs && !deferred_remap branch in rust/lance-table/src/transaction/manifest_build.rs, from #8084). BTREE used to take the first branch and now takes the second.

The way to keep the coverage is to defer the remap through a fragment reuse index, and for stable-row-id tables that is enabled only in debug builds (frag_reuse_with_stable_row_ids_enabled() returns cfg!(debug_assertions)). A release build therefore cannot compact such a table without losing it: compact_files(defer_index_remap=True) on b16 fails with "recording their row-address moves in a fragment reuse index is not supported by this build of Lance".

The cost grows with the table. Using the same script and machine with the two published wheels, a point lookup on 2M rows compacted from 100 fragments to 4 took 0.20 ms on b15 and 4.35 ms on b16 after compaction. On b16 it returned to 0.22 ms after an optimize_indices, which took 66 ms.

#9814, #9856 and #9858 move Bitmap, LabelList, JSON, NGram, RTree and MinHash LSH to the row-address domain, so once they land those indices will hit the same branch.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.