bug: raw index commits can mix incompatible vector segments under one name

Aperta
#10,728 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
rust
Ambito
databases

Direzione di ricerca

Read rust/lance/src/index/lifecycle.rs around commit_existing_index_segments and rust/lance-table/src/transaction/manifest_build.rs around the CreateIndex manifest builder. Trace how raw CreateIndex commits and Transaction V2 AddIndexSegment actions reach the commit path, then compare those checks with validate_vector_query_compatibility. Done means incompatible vector segments sharing an index name are rejected on every commit path or when opened for search; an open linked pull request (#10761) is already addressing this issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug critical-fix

A transaction committed directly can add a vector index segment whose metric, sub-index, or quantization differs from the other segments under the same index name. Nothing rejects it, and a search then unions the segments and ranks part of the data by the wrong distance, without an error.

The index APIs guard against this. commit_existing_index_segments opens every segment that will share the name, including those already in the read version, and checks them before it builds the CreateIndex transaction:

https://github.com/lance-format/lance/blob/7f8d4d0b22421d2a6293a018eb839291630be8d2/rust/lance/src/index/lifecycle.rs#L1023-L1027

Building the manifest for a CreateIndex doesn't repeat that check. It validates covering fields and row-id/row-address consistency, but not vector compatibility. So a Transaction with Operation::CreateIndex committed through CommitBuilder, or an AddIndexSegment action under Transaction V2 (#8645), can put an L2 segment and a Dot segment under one name. Queries don't check either: validate_vector_query_compatibility runs only on commit, optimize, and merge paths.

Repro

  1. Build two ivf_flat segments on different fragments with execute_uncommitted, one with DistanceType::L2 and one with DistanceType::Dot.
  2. Commit the L2 segment under name shared with a raw CreateIndex transaction, then commit the Dot segment the same way from the new version.
  3. dataset.scan().nearest("vector", ..) succeeds and mixes the two distance rankings.

Possible fix

lance-table can't decode index details without depending on lance-index, so the check can't live in manifest building as-is. One option is a compatibility hook that lance supplies to the commit path, dispatching by details type, so every commit path checks the final segment set rather than only the API paths. An alternative or complement is checking segment compatibility when a search opens them.

Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.