Offset overflow - Lower-risk sites (opt-in features, unrealistic sizes, upstream panics)

オープン
  • 何か問題がありますか？
#7,503 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
java, rust, scala, spark
領域
backend, databases, performance

調査の方向性

Start with the specific code locations listed in the issue, such as native/core/src/execution/operators/iceberg_write.rs, ArrowWriters.scala, or the named upstream Arrow, Parquet, and DataFusion files. Read the linked parent issue #7267 and identify one overflow case and its existing tests before choosing a fix. Done means the selected case returns a safe error or handles the size correctly, with a regression test; the issue groups many separate cases, so its scope needs narrowing.

索引モデルが issue の本文から書いたものです。

説明

bug priority:low

Part of #7267.

These came out of the same audit as the other new sub-issues. Each needs an opt-in feature, sizes that are unlikely in practice, or a fix upstream, so they are collected here instead of filed one by one.

Line references are for main at 4e6b12290f, DataFusion 55.2.0 and arrow 59.3.2.

Comet

  • The native Iceberg writer (opt-in) concatenates pending pieces into 1000-row units (native/core/src/execution/operators/iceberg_write.rs:1897-1909, ROWS_DIVISOR at :727). Values averaging more than about 2.1 MB in one column overflow a unit with OffsetOverflowError. Iceberg's Java writer writes one row at a time.
  • FixedWidthArrowFieldWriter.writeArrayElements calls ensureCapacity(count + numElements) with Int arithmetic (ArrowWriters.scala:980-1005). Past 2^31 elements in one batch the sum goes negative, the buffer is not grown, and copyMemory writes past its end. Comet sets arrow.enable_unsafe_memory_access=true (NativeBase.java:228), so Arrow's own bounds checks don't catch it. Math.addExact would.
  • Importing JVM strings with invalid UTF-8 decodes them with U+FFFD, up to 3x the bytes, through a builder that panics on overflow (native/common/src/utf8.rs:236-245).
  • DataFusion UDF results of type Utf8View/BinaryView are cast back with arrow's cast_view_to_byte (native/core/src/execution/planner.rs:2227-2244), which panics on overflow instead of returning an error.
  • Past 2^31 list elements in one batch, these panic instead of returning an error: array_insert (array_insert.rs:309), arrays_zip (arrays_zip.rs:295), percentile (percentile.rs:205-206), pivot_first (pivot_first.rs:536-537).
  • CometScalaUDFCodegen.estimatedOutputBytes sums input sizes as an Int (CometScalaUDFCodegen.scala:294-308), so inputs over 2 GiB give a wrong preallocation size.
  • Utils.narrowOffsets (Utils.scala:791-805) fails broadcast and cache serialization for a large-offset column over 2 GiB, for example from mapInArrow with large variable-width types.
  • ConstantColumnVectors.materialize (ArrowWriters.scala:129-143) writes a constant once per row with no size limit, the JVM side of #7247. In practice these constants are small partition and default values.

Upstream

  • arrow's MapBuilder::append pushes key_builder.len() as i32 without a check (arrow-array src/builder/map_builder.rs:176), and finish builds the array unchecked (:277). Past 2^31 entries in one batch the offsets wrap silently. Comet reaches it through str_to_map (datafusion-spark) and the JVM shuffle's map conversion (native/shuffle/src/spark_unsafe/row.rs:1234).
  • parquet's DELTA_LENGTH_BYTE_ARRAY decoder panics on offset overflow (src/arrow/array_reader/byte_array.rs:514), where the other decoders return an error.
  • arrow-csv builds each string column with collect::<StringArray>() (src/reader/mod.rs:872-879), which panics past 2 GiB. Comet's native CSV scan is opt-in.
  • DataFusion's sort-merge join pushes to its BatchCoalescer with .expect(...) (src/joins/sort_merge_join/materializing_stream.rs:465, :505, :516), so a coalescer overflow (#7271) panics instead of failing with an error.
主要言語
Scala
スター
1.3k
フォーク
414
平均マージ
3日 5時間
マージ済み PR（30日）
210

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/datafusion-comet のほかの issue

apache/datafusion-comet の issue をすべて見る

似ている issue

Scala の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。