perf: writer emits one page per inserted array for large per-value values (multi-GB pages, single-threaded compression)

Aperta
#10,526 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
performance

Direzione di ricerca

Begin in the Rust writer at AccumulationQueue::insert_with_pages, examining how maybe_encode keeps one RepDefBuilder per array. Run the Python repro to confirm a single multi-GB page in metadata. Done means an inserted array over max_page_bytes splits into multiple bounded pages, each compressed in parallel, as checked by the page size list in the repro.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

performance

Problem

For variable-width values above 32 KiB the structural primitive encoder's accumulation queue (AccumulationQueue::insert_with_pages) emits at most one page boundary per inserted array. An array larger than max_page_bytes (32 MiB) becomes a single page of whatever size the caller handed in. Writing a column of 180 KB JSON strings from one 25k-row batch produces one 4.5 GB full-zip page per column; from 2 048-row batches it produces 370 MB pages.

Consequences measured on a 64-core box (lance 15.0.0-beta.7 main at bf867011a):

  • Write parallelism is one task per page, so the 11 GB column pair of two fragments compressed on 1.2 cores for 42 s (perf: 60% in ZSTD_compressBlock_doubleFast on a single lance-cpu thread); a 156 GB table rewrite to 2.2 took 553 s. Chunking the per-value loop inside the page (#10531) is a mitigation: the page is still 4.5 GB.
  • Readers must handle multi-gigabyte pages: #10160 / #10168 add chunked loads for exactly these oversized full-zip pages.
  • LANCE_FILE_WRITER_MAX_PAGE_BYTES has no effect on such columns.

#8405 tried to split oversized primitive batches into row-aligned pages and was closed; the repdef/validity bookkeeping per slice is the hard part, since maybe_encode keeps one RepDefBuilder per inserted array.

Expected

An inserted array larger than max_page_bytes is split at value boundaries into several pages, each encoded by its own task, so page size is bounded by the writer option and compression parallelism follows from the page count.

Repro

import lance, pyarrow as pa
tbl = pa.table({"v": pa.array(["x" * 180_000] * 25_000, pa.large_string())})
lance.write_dataset(tbl, "/tmp/big", data_storage_version="2.2")
from lance.file import LanceFileReader
m = LanceFileReader("/tmp/big/data/" + lance.dataset("/tmp/big").get_fragments()[0].metadata.files[0].path).metadata()
print([sum(b.size for b in p.buffers) for p in m.columns[0].pages])   # one ~4.5 GB page
Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.