Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

perf: row-id index fragments added after dataset_version are dropped by freshness checks

Open
  • Is something wrong?
#10,095 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
databases

Research direction

Start by tracing fresh_fragments and build_provenance in rust/lance/src/index.rs, then compare their freshness logic with the bitmap updates in rust/lance-table/src/transaction/index_maintenance.rs and manifest_build.rs; #10090 gives the intended semantics. Find and run the existing tests for FRI freshness, segment merges, and staged commits; done means fragments added for rows from covered fragments remain fresh when no indexed field changed.

Written by the indexing model from the issue text.

Description

performance

With stable row ids, some operations add new fragment ids to a row-id-domain index's fragment_bitmap without changing dataset_version:

  • an update that moves rows to new fragments (register_pure_rewrite_rows_update_frags_in_indices, rust/lance-table/src/transaction/index_maintenance.rs)
  • a non-deferred compaction (recalculate_fragment_bitmap in the Rewrite arm of manifest_build.rs)

Results stay correct. But those fragments didn't exist at the segment's dataset_version, so freshness checks that trace fragments back to that version can't find them and drop them:

  • fresh_fragments / build_provenance (rust/lance/src/index.rs)
  • prune_stale_segment_coverage with prune_historically_missing

That happens during segment merges or staged commits whenever an FRI applies. The affected rows then fall back to flat scans.

Expected (per #10090): for row-id-domain indices the bitmap means "fragments whose rows, by stable row id, are represented", and the watermark applies to row values. Freshness checks should treat a fragment added this way as fresh if its rows came from covered fragments and no indexed field changed.

Related: #9543.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.