merge_insert RewriteColumns lists the join key in fields_modified, withdrawing exact index coverage over the key

Aperta
#10,781 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
databases

Direzione di ricerca

Start by reading select_write_sink and its comment, then trace how MergeInsertWriteMode::RewriteColumns populates Operation::Update::fields_modified and how Transaction::prepare_indices handles rewritten_physical_columns and withdraw_in_place_rewrites. Reproduce the behavior with a BTREE index on the join key and inspect the fragment bitmap after commit. Done when a matched-row rewrite preserves exact index coverage, with tests confirming the behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

A merge_insert with MergeInsertWriteMode::RewriteColumns writes the join key into the patch data file and lists the key's field id in Operation::Update::fields_modified. Transaction::prepare_indices (through rewritten_physical_columns and withdraw_in_place_rewrites) then withdraws every touched fragment from every index over the key. The key values are byte-identical to the stored ones, because the join matched them exactly, and a column patch moves no row, so each row keeps its stable row id. The withdrawn coverage was exact.

The doc comment in select_write_sink already describes this cost ("the patch would reproduce the key column byte for byte, and still cost a full-fragment column file plus the invalidation of every index over the key").

Reproduction (Lance 13.0.0)
  1. Create a dataset with stable row ids: id: Utf8 (the join key) with a BTREE index, score: Int32, plus any other columns.
  2. Stage MergeInsertBuilder::try_new(ds, ["id"]), when_matched(UpdateAll), when_not_matched(DoNothing), write_mode(RewriteColumns), with a source of (id, score) that matches some rows.
  3. Observe that fields_modified contains the field id of id, and that after commit the id index's fragment_bitmap no longer contains the touched fragment.

Removing the key's field id from fields_modified before commit keeps the coverage. Equality lookups on the key then return every row exactly once, with the patched values.

Expected

For RewriteColumns, either do not list the join key in fields_modified, or omit the key from the patch file. Both are correct because the key values cannot change in a matched-row update. The second also saves writing a full-fragment copy of the key column.

Context

Omnigraph stages column patches for narrow updates of wide rows and currently removes the key from fields_modified itself, pinned by a surface test: https://github.com/ModernRelay/omnigraph/pull/1272. An upstream fix would let that adapter go.

Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.