bug: raw index commits can mix incompatible vector segments under one name

Abierto
#10,728 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
rust
Área
databases

Línea de trabajo

Read rust/lance/src/index/lifecycle.rs around commit_existing_index_segments and rust/lance-table/src/transaction/manifest_build.rs around the CreateIndex manifest builder. Trace how raw CreateIndex commits and Transaction V2 AddIndexSegment actions reach the commit path, then compare those checks with validate_vector_query_compatibility. Done means incompatible vector segments sharing an index name are rejected on every commit path or when opened for search; an open linked pull request (#10761) is already addressing this issue.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug critical-fix

A transaction committed directly can add a vector index segment whose metric, sub-index, or quantization differs from the other segments under the same index name. Nothing rejects it, and a search then unions the segments and ranks part of the data by the wrong distance, without an error.

The index APIs guard against this. commit_existing_index_segments opens every segment that will share the name, including those already in the read version, and checks them before it builds the CreateIndex transaction:

https://github.com/lance-format/lance/blob/7f8d4d0b22421d2a6293a018eb839291630be8d2/rust/lance/src/index/lifecycle.rs#L1023-L1027

Building the manifest for a CreateIndex doesn't repeat that check. It validates covering fields and row-id/row-address consistency, but not vector compatibility. So a Transaction with Operation::CreateIndex committed through CommitBuilder, or an AddIndexSegment action under Transaction V2 (#8645), can put an L2 segment and a Dot segment under one name. Queries don't check either: validate_vector_query_compatibility runs only on commit, optimize, and merge paths.

Repro

  1. Build two ivf_flat segments on different fragments with execute_uncommitted, one with DistanceType::L2 and one with DistanceType::Dot.
  2. Commit the L2 segment under name shared with a raw CreateIndex transaction, then commit the Dot segment the same way from the new version.
  3. dataset.scan().nearest("vector", ..) succeeds and mixes the two distance rankings.

Possible fix

lance-table can't decode index details without depending on lance-index, so the check can't live in manifest building as-is. One option is a compatibility hook that lance supplies to the commit path, dispatching by details type, so every commit path checks the final segment set rather than only the API paths. An alternative or complement is checking segment compatibility when a search opens them.

Lenguaje dominante
Rust
Estrellas
7.3k
Forks
929
Merge medio
2 d 22 h
PR fusionados (30 d)
572

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de lance-format/lance

Todos los issues de lance-format/lance

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.