Offset overflow - Lower-risk sites (opt-in features, unrealistic sizes, upstream panics)

Aperta
  • Qualcosa non va?
#7,503 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, rust, scala, spark
Ambito
backend, databases, performance

Direzione di ricerca

Start with the specific code locations listed in the issue, such as native/core/src/execution/operators/iceberg_write.rs, ArrowWriters.scala, or the named upstream Arrow, Parquet, and DataFusion files. Read the linked parent issue #7267 and identify one overflow case and its existing tests before choosing a fix. Done means the selected case returns a safe error or handles the size correctly, with a regression test; the issue groups many separate cases, so its scope needs narrowing.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug priority:low

Part of #7267.

These came out of the same audit as the other new sub-issues. Each needs an opt-in feature, sizes that are unlikely in practice, or a fix upstream, so they are collected here instead of filed one by one.

Line references are for main at 4e6b12290f, DataFusion 55.2.0 and arrow 59.3.2.

Comet

  • The native Iceberg writer (opt-in) concatenates pending pieces into 1000-row units (native/core/src/execution/operators/iceberg_write.rs:1897-1909, ROWS_DIVISOR at :727). Values averaging more than about 2.1 MB in one column overflow a unit with OffsetOverflowError. Iceberg's Java writer writes one row at a time.
  • FixedWidthArrowFieldWriter.writeArrayElements calls ensureCapacity(count + numElements) with Int arithmetic (ArrowWriters.scala:980-1005). Past 2^31 elements in one batch the sum goes negative, the buffer is not grown, and copyMemory writes past its end. Comet sets arrow.enable_unsafe_memory_access=true (NativeBase.java:228), so Arrow's own bounds checks don't catch it. Math.addExact would.
  • Importing JVM strings with invalid UTF-8 decodes them with U+FFFD, up to 3x the bytes, through a builder that panics on overflow (native/common/src/utf8.rs:236-245).
  • DataFusion UDF results of type Utf8View/BinaryView are cast back with arrow's cast_view_to_byte (native/core/src/execution/planner.rs:2227-2244), which panics on overflow instead of returning an error.
  • Past 2^31 list elements in one batch, these panic instead of returning an error: array_insert (array_insert.rs:309), arrays_zip (arrays_zip.rs:295), percentile (percentile.rs:205-206), pivot_first (pivot_first.rs:536-537).
  • CometScalaUDFCodegen.estimatedOutputBytes sums input sizes as an Int (CometScalaUDFCodegen.scala:294-308), so inputs over 2 GiB give a wrong preallocation size.
  • Utils.narrowOffsets (Utils.scala:791-805) fails broadcast and cache serialization for a large-offset column over 2 GiB, for example from mapInArrow with large variable-width types.
  • ConstantColumnVectors.materialize (ArrowWriters.scala:129-143) writes a constant once per row with no size limit, the JVM side of #7247. In practice these constants are small partition and default values.

Upstream

  • arrow's MapBuilder::append pushes key_builder.len() as i32 without a check (arrow-array src/builder/map_builder.rs:176), and finish builds the array unchecked (:277). Past 2^31 entries in one batch the offsets wrap silently. Comet reaches it through str_to_map (datafusion-spark) and the JVM shuffle's map conversion (native/shuffle/src/spark_unsafe/row.rs:1234).
  • parquet's DELTA_LENGTH_BYTE_ARRAY decoder panics on offset overflow (src/arrow/array_reader/byte_array.rs:514), where the other decoders return an error.
  • arrow-csv builds each string column with collect::<StringArray>() (src/reader/mod.rs:872-879), which panics past 2 GiB. Comet's native CSV scan is opt-in.
  • DataFusion's sort-merge join pushes to its BatchCoalescer with .expect(...) (src/joins/sort_merge_join/materializing_stream.rs:465, :505, :516), so a coalescer overflow (#7271) panics instead of failing with an error.
Lingua principale
Scala
Stelle
1.3k
Fork
414
Merge medio
3g 5h
PR unite (30g)
210

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.