bug(python): read_transaction raises TypeError on a Rewrite with rewritten indices

Open Beginner friendly
#11,117 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@Borg93 is already working on this.

Since Nov 15, 2026.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, rust
Domain
api

Research direction

Start in python/src/transaction.rs at impl IntoPyObject for PyLance<&RewrittenIndex>; it passes only old and new index ids. Compare the LanceOperation.RewrittenIndex dataclass and the matching FromPyObject impl, which reads all five fields, then update the conversion to supply the missing three. Reproduce with the issue snippet after rebuilding, or run the Python transaction tests; done means read_transaction succeeds for a BTREE-indexed compaction and rewritten index data round-trips.

Written by the indexing model from the issue text.

Description

bug

Bug

LanceDataset.read_transaction raises TypeError on a Rewrite that rewrote an index, for example a compaction of a stable-row-id table that has a BTREE index.

import tempfile, lance, pyarrow as pa

uri = tempfile.mkdtemp()
ds = lance.write_dataset(pa.table({"id": pa.array(range(64), pa.int64())}), uri,
                         enable_stable_row_ids=True, max_rows_per_file=16)
ds.create_scalar_index("id", "BTREE")
ds.optimize.compact_files(target_rows_per_fragment=64)
ds = lance.dataset(uri)
ds.read_transaction(ds.version)
# TypeError: LanceOperation.RewrittenIndex.__init__() missing 3 required positional
# arguments: 'new_details_type_url', 'new_details_value', and 'new_index_version'

Observed on pylance 14.0.3 (PyPI, aarch64 Linux, Python 3.13). The same code is on main (a33c468).

Cause

impl IntoPyObject for PyLance<&RewrittenIndex> in python/src/transaction.rs calls the class with (old_id, new_id) only. The LanceOperation.RewrittenIndex dataclass requires five fields, and the matching FromPyObject impl already reads all five.

With BITMAP, LABEL_LIST, INVERTED or no index the same compaction's transaction reads back, because its rewritten_indices is empty.

Impact

Any reader of a compaction commit's transaction fails on such a table. That includes a catalog that judges staged commits under managed versioning, and anything that walks read_transaction over a table's history.

Dominant language
Rust
Stars
7.3k
Forks
929
Avg merge
2d 22h
Merged PRs (30d)
572

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.