perf: tagged fragment reuse histories can't tell that an index segment was built after a rewrite

Aperta
#10,715 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust

Direzione di ricerca

Read rust/lance/src/dataset/index/frag_reuse.rs and rust/lance/src/index/frag_reuse_reader.rs to compare legacy and tagged transition handling. The issue describes a possible persisted metadata convention, but says that convention needs a decision; first resolve how per-transition versions should be stored and how older writers should behave. Done means rebuilt segments can use stored addresses after compaction without repeated translation or full scans, while older metadata remains safe.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

performance

A legacy (v0) fragment reuse history stores a dataset_version on each version record. is_index_remap_caught_up (rust/lance/src/dataset/index/frag_reuse.rs:264) uses it: a segment built after a rewrite is caught up and its stored addresses are used as-is.

Tagged (v1) transitions don't record the version they were committed at. The tagged read path, FragmentReuseIndex::may_need_translation (rust/lance/src/index/frag_reuse_reader.rs:211), only checks overlap. A segment needs translation whenever its fragment bitmap contains any fragment a transition names, sources or destinations.

So a segment rebuilt or optimized after a compaction is never treated as caught up, because it covers that compaction's destination fragments. This lasts until trim drops the transition:

  • Indices that can translate run an identity translation on every load. It reads the row maps and returns the addresses unchanged.
  • Indices that can't translate are excluded and their rows are scanned (#10693, FM).

Results are correct; the cost is query-time IO and CPU, or full scans.

#10706 adds a version check to the tagged path, but it has to use the FRI entry's dataset_version as an upper bound for every transition's version. Every new rewrite or trim re-stamps the entry, so under continuous compaction a rebuilt segment gets the direct read only until the next rewrite.

A fix that doesn't change the format would record a per-transition version in the entry's IndexMetadata.metadata map, keyed by transition identity (for example the row-map id). Older writers that drop or copy the map fall back to the entry version, which stays safe. This is a new persisted convention and needs a decision.

Related: #10693, #10697.

Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.