bug(python): read_transaction raises TypeError on a Rewrite with rewritten indices
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
Research direction
Start in python/src/transaction.rs at impl IntoPyObject for PyLance<&RewrittenIndex>; it passes only old and new index ids. Compare the LanceOperation.RewrittenIndex dataclass and the matching FromPyObject impl, which reads all five fields, then update the conversion to supply the missing three. Reproduce with the issue snippet after rebuilding, or run the Python transaction tests; done means read_transaction succeeds for a BTREE-indexed compaction and rewritten index data round-trips.
Written by the indexing model from the issue text.
Description
Bug
LanceDataset.read_transaction raises TypeError on a Rewrite that rewrote an index, for example a compaction of a stable-row-id table that has a BTREE index.
import tempfile, lance, pyarrow as pa
uri = tempfile.mkdtemp()
ds = lance.write_dataset(pa.table({"id": pa.array(range(64), pa.int64())}), uri,
enable_stable_row_ids=True, max_rows_per_file=16)
ds.create_scalar_index("id", "BTREE")
ds.optimize.compact_files(target_rows_per_fragment=64)
ds = lance.dataset(uri)
ds.read_transaction(ds.version)
# TypeError: LanceOperation.RewrittenIndex.__init__() missing 3 required positional
# arguments: 'new_details_type_url', 'new_details_value', and 'new_index_version'
Observed on pylance 14.0.3 (PyPI, aarch64 Linux, Python 3.13). The same code is on main (a33c468).
Cause
impl IntoPyObject for PyLance<&RewrittenIndex> in python/src/transaction.rs calls the class with (old_id, new_id) only. The LanceOperation.RewrittenIndex dataclass requires five fields, and the matching FromPyObject impl already reads all five.
With BITMAP, LABEL_LIST, INVERTED or no index the same compaction's transaction reads back, because its rewritten_indices is empty.
Impact
Any reader of a compaction commit's transaction fails on such a table. That includes a catalog that judges staged commits under managed versioning, and anything that walks read_transaction over a table's history.
- Dominant language
- Rust
- Stars
- 7.3k
- Forks
- 929
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 572
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
lance-format/lance#10782 ·
Maintainers usually reply within 2 days
-
Upgrade DataFusion for the unparser stack-overflow fixMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
lance-format/lance#8225 ·
Maintainers usually reply within 2 days
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#6425 · 2 comments ·
Maintainers usually reply within 2 days
-
Expose compute_pq_distanceMay be free again A pull request for this issue was closed without being merged. Openenhancement
Difficulty 1/5 1-3 hours Newbie friendliness 62/100
lance-format/lance#5675 · 1 comment ·
Maintainers usually reply within 2 days
-
documentation
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
lance-format/lance#5240 ·
Maintainers usually reply within 2 days
All issues in lance-format/lance
Similar issues
-
comp-rbac
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
ClickHouse/ClickHouse#131875 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
storytold/photocraft#15232 · 1 comment ·
Maintainers usually reply within 1 day
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)Possibly taken @Ynyesto claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
foundry-rs/foundry#18485 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
bondybondbond/rushcut#238 · 1 comment ·
-
area: app type: bug
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day