UPDATE that matches no rows still commits a new version
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 53/100
Research direction
Start in UpdateJob::commit_impl and trace how num_updated_rows and removed, updated, or new fragments determine whether a commit occurs. The issue identifies no_op_update_ignores_explicit_exact_version as an existing test; extend it or add a focused case that checks the returned dataset version when no rows match. Done means the no-op update returns the input dataset without creating a version, while updates that affect rows still commit.
Written by the indexing model from the issue text.
Description
Summary
UpdateBuilder::...execute() commits a new dataset version even when the predicate matches no rows. The commit carries an Operation::Update with no removed, updated or new fragments: it changes nothing, but it still takes a manifest CAS, adds a version to the history, and conflicts with (pre-empts) concurrent writers such as compaction.
Why it matters
Conditional ("compare-and-set") UPDATEs are a common pattern for leases and fenced status writes, e.g.
UPDATE t SET owner = 'pod-a', deadline = 1234
WHERE id IN (...) AND (owner IS NULL OR deadline < now)
When several writers race for the same rows, every loser's UPDATE matches 0 rows and still commits a version. In our deployment (lance 12, GCS), a 356-file ingest burst produced 26 such zero-row Update commits on one table, alongside the real ones. Every commit of that burst pre-empted the table's compaction, so compaction never landed until the burst ended (fragments went from 40 to 384).
Reproduction
let before = dataset.manifest.version;
let result = UpdateBuilder::new(dataset.clone())
.update_where("id < 0")? // matches nothing
.set("name", "'bar'")?
.build()?
.execute()
.await?;
assert_eq!(result.rows_updated, 0);
assert_eq!(result.new_dataset.manifest.version, before); // fails: before + 1
The existing test no_op_update_ignores_explicit_exact_version exercises the same path and passes because it never checks the version.
Proposed fix
In UpdateJob::commit_impl, when num_updated_rows == 0 and there are no removed, updated or new fragments, return the input dataset with rows_updated: 0 instead of committing. Nothing is written in that case, so there is nothing to clean up, and a transaction with no affected rows can never conflict, so skipping it cannot change any outcome except the version number.
Still present on main (checked at v14.0.0-beta.6). A patch with a test, against v12.0.0, is at https://github.com/rotty3000/lance/commit/d21067c02 — happy to open a PR against main.
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 900
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 392
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
feature performance
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
lance-format/lance#10137 ·
Maintainers usually reply within 3 days
-
Upgrade DataFusion for the unparser stack-overflow fixMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
lance-format/lance#8225 ·
Maintainers usually reply within 3 days
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#6425 · 2 comments ·
Maintainers usually reply within 3 days
-
Expose compute_pq_distanceMay be free again A pull request for this issue was closed without being merged. Openenhancement
Difficulty 1/5 1-3 hours Newbie friendliness 62/100
lance-format/lance#5675 · 1 comment ·
Maintainers usually reply within 3 days
-
documentation
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
lance-format/lance#5240 ·
Maintainers usually reply within 3 days
All issues in lance-format/lance
Similar issues
-
area:io kernel-paused needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
bherbruck/solvecraft#7854 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
transit-audio/transit#159 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
HakanSeven12/OpenCADStudio#2257 · 1 comment ·
Maintainers usually reply within 1 day
-
priority: low ui
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
storytold/photocraft#9884 · 1 comment ·
Maintainers usually reply within 1 day
-
triage: new type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Jagalite/superseedr#376 ·
Maintainers usually reply within 1 day