Offset overflow - JVM columnar shuffle cuts row-to-Arrow batches by row count only

Aperta
  • Qualcosa non va?
#7,497 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, scala, spark
Ambito
backend, data-engineering

Direzione di ricerca

Start with native/shuffle/src/spark_unsafe/row.rs, especially process_sorted_row_partition at lines 1579–1588 and its existing row size handling; compare it with process_sorted_partitions at line 1686. Add byte budget coverage alongside the row limit, then follow the issue’s test plan: use a small budget and verify several blocks are written and the same rows read back.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:shuffle bug priority:high requires-triage

Part of #7267.

Problem

The JVM columnar shuffle hands sorted UnsafeRows to native code, which converts them to Arrow in batches of spark.comet.shuffle.jvm.batchSize rows, capped by spark.comet.batchSize. Both entry points cut batches by row count only: process_sorted_partitions splits each partition's rows with run.chunks(batch_size) (native/shuffle/src/spark_unsafe/row.rs:1686), and process_sorted_row_partition does the same at :1579-1588. Strings and binaries are appended with arrow's append_value (:1173-1203), which panics with byte array offset overflow once a column passes 2 GiB. Strings inside arrays, maps and structs go through the same builders.

So a batch fails when one string or binary column holds more than 2 GiB, which is more than about 256 KiB per value at 8192 rows. When the column is dictionary-encoded, the writer succeeds and the reader fails instead (#7496). When the dictionary doesn't pay off, the cast back to plain values returns OffsetOverflowError (row.rs:1777). Spark shuffles rows and has no batch to overflow.

#7269 covers the native shuffle writer, which is separate code.

Line references are for main at 4e6b12290f.

Workaround

Lower spark.comet.shuffle.jvm.batchSize.

Proposed fix

Cut batches by bytes as well as rows. The row sizes are known before conversion (row_sizes), and a row's size bounds the bytes it adds to any column, so ending a batch before the sum of its row sizes passes i32::MAX keeps every column valid without looking at the values.

Tests

Inject a small byte budget and check that process_sorted_row_partition writes several blocks for rows that would otherwise share one, with the same rows read back.

Lingua principale
Scala
Stelle
1.3k
Fork
414
Merge medio
3g 5h
PR unite (30g)
210

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.