Offset overflow - Lower-risk sites (opt-in features, unrealistic sizes, upstream panics)

Abierto
  • ¿Algo no está bien?
#7,503 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
java, rust, scala, spark
Área
backend, databases, performance

Línea de trabajo

Start with the specific code locations listed in the issue, such as native/core/src/execution/operators/iceberg_write.rs, ArrowWriters.scala, or the named upstream Arrow, Parquet, and DataFusion files. Read the linked parent issue #7267 and identify one overflow case and its existing tests before choosing a fix. Done means the selected case returns a safe error or handles the size correctly, with a regression test; the issue groups many separate cases, so its scope needs narrowing.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug priority:low

Part of #7267.

These came out of the same audit as the other new sub-issues. Each needs an opt-in feature, sizes that are unlikely in practice, or a fix upstream, so they are collected here instead of filed one by one.

Line references are for main at 4e6b12290f, DataFusion 55.2.0 and arrow 59.3.2.

Comet

  • The native Iceberg writer (opt-in) concatenates pending pieces into 1000-row units (native/core/src/execution/operators/iceberg_write.rs:1897-1909, ROWS_DIVISOR at :727). Values averaging more than about 2.1 MB in one column overflow a unit with OffsetOverflowError. Iceberg's Java writer writes one row at a time.
  • FixedWidthArrowFieldWriter.writeArrayElements calls ensureCapacity(count + numElements) with Int arithmetic (ArrowWriters.scala:980-1005). Past 2^31 elements in one batch the sum goes negative, the buffer is not grown, and copyMemory writes past its end. Comet sets arrow.enable_unsafe_memory_access=true (NativeBase.java:228), so Arrow's own bounds checks don't catch it. Math.addExact would.
  • Importing JVM strings with invalid UTF-8 decodes them with U+FFFD, up to 3x the bytes, through a builder that panics on overflow (native/common/src/utf8.rs:236-245).
  • DataFusion UDF results of type Utf8View/BinaryView are cast back with arrow's cast_view_to_byte (native/core/src/execution/planner.rs:2227-2244), which panics on overflow instead of returning an error.
  • Past 2^31 list elements in one batch, these panic instead of returning an error: array_insert (array_insert.rs:309), arrays_zip (arrays_zip.rs:295), percentile (percentile.rs:205-206), pivot_first (pivot_first.rs:536-537).
  • CometScalaUDFCodegen.estimatedOutputBytes sums input sizes as an Int (CometScalaUDFCodegen.scala:294-308), so inputs over 2 GiB give a wrong preallocation size.
  • Utils.narrowOffsets (Utils.scala:791-805) fails broadcast and cache serialization for a large-offset column over 2 GiB, for example from mapInArrow with large variable-width types.
  • ConstantColumnVectors.materialize (ArrowWriters.scala:129-143) writes a constant once per row with no size limit, the JVM side of #7247. In practice these constants are small partition and default values.

Upstream

  • arrow's MapBuilder::append pushes key_builder.len() as i32 without a check (arrow-array src/builder/map_builder.rs:176), and finish builds the array unchecked (:277). Past 2^31 entries in one batch the offsets wrap silently. Comet reaches it through str_to_map (datafusion-spark) and the JVM shuffle's map conversion (native/shuffle/src/spark_unsafe/row.rs:1234).
  • parquet's DELTA_LENGTH_BYTE_ARRAY decoder panics on offset overflow (src/arrow/array_reader/byte_array.rs:514), where the other decoders return an error.
  • arrow-csv builds each string column with collect::<StringArray>() (src/reader/mod.rs:872-879), which panics past 2 GiB. Comet's native CSV scan is opt-in.
  • DataFusion's sort-merge join pushes to its BatchCoalescer with .expect(...) (src/joins/sort_merge_join/materializing_stream.rs:465, :505, :516), so a coalescer overflow (#7271) panics instead of failing with an error.
Lenguaje dominante
Scala
Estrellas
1.3k
Forks
414
Merge medio
3 d 5 h
PR fusionados (30 d)
210

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.