bug: binary_quantization appends a spurious byte when the length is a multiple of 8

Open Beginner friendly
#9,028 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
machine-learning

Research direction

Start in rust/lance-index/src/vector/bq.rs at the binary_quantization helper and trace how BinaryQuantization::transform flattens rows. Verify the 16-value example and the three-row dimension-8 case, then ensure dimensions divisible by 8 produce ceil(dim / 8) bytes with no extra tail byte.

Written by the indexing model from the issue text.

Description

bug

binary_quantization packs the sign bits of a float slice into bytes and appends the tail byte unconditionally (rust/lance-index/src/vector/bq.rs:69):

let iter = data.chunks_exact(8);
iter.clone()
    .map(|c| { ... })
    .chain(once(0).map(move |_| {
        let mut bits: u8 = 0;
        iter.remainder().iter().enumerate().for_each(...);
        bits
    }))

When data.len() is a multiple of 8 the remainder is empty, so the chained byte is always zero and the output is floor(len / 8) + 1 bytes instead of ceil(len / 8).

The public entry point BinaryQuantization::transform flattens every row through this helper into one UInt8Array, so the per-row width a caller has to assume, ceil(dim / 8), disagrees with what it gets. For a dimension that is a multiple of 8, every row carries one extra zero byte.

Reproduced on 36bd4e27a: 16 values give [85, 85, 0] where [85, 85] is correct, and transform over 3 rows of dimension 8 returns 6 bytes instead of 3.

BinaryQuantization has no callers in this repository, so this is latent rather than a live failure, and RabitQuantizer is what the IVF_RQ path uses. Filing it because the type is public API (lance_index::vector::bq::BinaryQuantization) and the helper predates RaBitQ (#1988).

Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
252

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.