merge_insert RewriteColumns lists the join key in fields_modified, withdrawing exact index coverage over the key

Abierto
#10,781 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust
Área
databases

Línea de trabajo

Start by reading select_write_sink and its comment, then trace how MergeInsertWriteMode::RewriteColumns populates Operation::Update::fields_modified and how Transaction::prepare_indices handles rewritten_physical_columns and withdraw_in_place_rewrites. Reproduce the behavior with a BTREE index on the join key and inspect the fragment bitmap after commit. Done when a matched-row rewrite preserves exact index coverage, with tests confirming the behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

A merge_insert with MergeInsertWriteMode::RewriteColumns writes the join key into the patch data file and lists the key's field id in Operation::Update::fields_modified. Transaction::prepare_indices (through rewritten_physical_columns and withdraw_in_place_rewrites) then withdraws every touched fragment from every index over the key. The key values are byte-identical to the stored ones, because the join matched them exactly, and a column patch moves no row, so each row keeps its stable row id. The withdrawn coverage was exact.

The doc comment in select_write_sink already describes this cost ("the patch would reproduce the key column byte for byte, and still cost a full-fragment column file plus the invalidation of every index over the key").

Reproduction (Lance 13.0.0)
  1. Create a dataset with stable row ids: id: Utf8 (the join key) with a BTREE index, score: Int32, plus any other columns.
  2. Stage MergeInsertBuilder::try_new(ds, ["id"]), when_matched(UpdateAll), when_not_matched(DoNothing), write_mode(RewriteColumns), with a source of (id, score) that matches some rows.
  3. Observe that fields_modified contains the field id of id, and that after commit the id index's fragment_bitmap no longer contains the touched fragment.

Removing the key's field id from fields_modified before commit keeps the coverage. Equality lookups on the key then return every row exactly once, with the patched values.

Expected

For RewriteColumns, either do not list the join key in fields_modified, or omit the key from the patch file. Both are correct because the key values cannot change in a matched-row update. The second also saves writing a full-fragment copy of the key column.

Context

Omnigraph stages column patches for narrow updates of wide rows and currently removes the key from fields_modified itself, pinned by a surface test: https://github.com/ModernRelay/omnigraph/pull/1272. An upstream fix would let that adapter go.

Lenguaje dominante
Rust
Estrellas
7.3k
Forks
929
Merge medio
2 d 22 h
PR fusionados (30 d)
572

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de lance-format/lance

Todos los issues de lance-format/lance

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.