bug: merge_insert does not detect concurrent inserts of the same FixedSizeBinary primary key

Open Beginner friendly
  • Is something wrong?
#10,811 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@lance-gatefixer is already working on this.

Since Nov 6, 2026.

  • #10813 by @lance-gatefixer — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
databases

Research direction

Start with extract_key_value in rust/lance-table/src/format/key_existence.rs and compare its handling of FixedSizeBinary with Binary and LargeBinary. Reproduce the issue using the concurrency test pattern in merge_insert/tests/concurrency.rs, then confirm that concurrent inserts of the same fixed-size key produce a retryable conflict.

Written by the indexing model from the issue text.

Description

bug
Description

A declared primary key of type FixedSizeBinary (for example a 32-byte content hash) is never added to a merge_insert's inserted_rows_filter. Two writers that insert the same new key against one version therefore both commit, and the key ends up duplicated. Binary and LargeBinary keys are detected correctly.

Root cause: extract_key_value in rust/lance-table/src/format/key_existence.rs has no FixedSizeBinary arm and falls through to _ => return None. The key is skipped, the transaction carries an empty Bloom filter, and check_update_txn in the conflict resolver finds no intersection. This happens on both the indexed path (after #10606) and the v2 path, because both build the filter through this function.

Mapping the type to KeyValue::Binary, as Binary and LargeBinary already are, fixes it. I have a patch with tests and can open a PR if that's welcome.

Steps to reproduce

This reproduces on main at 9b311133d and on v15.0.0-beta.12. Test, in the style of merge_insert/tests/concurrency.rs:

// schema: id FixedSizeBinary(32) with lance-schema:unenforced-primary-key=true, value UInt32
// 1. create a dataset with key [1; 32]; optionally create a BTree index on id
// 2. from the same dataset version, stage two merge_inserts of key [5; 32]
//    (when_matched DoNothing, when_not_matched InsertAll) via execute_uncommitted_batches
// 3. commit the first staged transaction, then the second with CommitBuilder
// 4. the second commit succeeds and the dataset holds two rows with key [5; 32]

The same test with id declared as Binary gets a retryable conflict on the second commit, as expected.

Expected behavior

The second commit is refused with a retryable conflict, the same as for Binary, Utf8 and integer keys.

Lance version

15.0.0-beta.12; also main at 9b311133d

Language binding

Rust

Environment

macOS (arm64), local filesystem and memory://

Dominant language
Rust
Stars
7.3k
Forks
924
Avg merge
3d 18h
Merged PRs (30d)
575

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.