feature: repack fragments whose files carry many dead column slots

Aperta
#11,035 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
15/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
data

Direzione di ricerca

Start with Dataset::column_layout_stats and its dead_slot_ratio, then the compact_files planner that uses max_data_files_per_fragment, and see how UpdateMode::RewriteColumns and drop_columns leave dead slots. The open questions are design decisions (slots vs bytes, one-file rewrites, option naming and combination), so the first step is a maintainer decision, not a patch. Done means those choices are settled in the issue and a follow-up scopes the implementation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature

Motivation

#9604 adds column repack tasks to compact_files. The file count is the only trigger: a fragment is repacked when it holds its columns in more than max_data_files_per_fragment data files. In review, a question came up about other conditions that should plan a repack, such as the dead weight an in-place column rewrite leaves behind.

Two operations leave slots that are dead in the latest version but sit in a file that still holds live columns:

  • an in-place rewrite of a column that shares a file with others (UpdateMode::RewriteColumns, or a DataReplacement with data_change: true) tombstones it in that file and writes the new values to a new file;
  • drop_columns leaves the dropped field's id in the file that held it.

Dataset::column_layout_stats already reports this per fragment as dead_slot_ratio. Today those bytes are reclaimed only when something else rewrites the fragment, when a repack happens to empty the file, or, for a file holding spilled row lineage next to user columns that are all dead, by the existing rewrite trigger for that case.

Open questions

  1. Slots or bytes. The manifest records only whole-file sizes, so the ratio counts dead slots, not dead bytes: a dead int32 column and a dead 1024-dim embedding move it by about the same amount. Per-column sizes would mean reading file metadata at plan time, which the planner avoids today, or recording them in the manifest.
  2. One-file rewrites. A repack only merges files it can empty, at least two at a time, so every task lowers the file count. Reclaiming a dead slot in a file that keeps live columns takes a task that rewrites that one file. The new file holds no dead slots, so the trigger cannot fire on it again.
  3. Options. A name and default (say max_dead_slot_ratio, off by default), how it combines with max_data_files_per_fragment in one task, and whether a fragment that also qualifies for a rewrite (deletions, overlays) should simply be rewritten.
Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.