`ScalarValue::to_array_of_size` panics when a `Utf8` or `Binary` scalar repeated `size` times passes `i32::MAX` bytes

Offen
  • Stimmt etwas nicht?
#27,277 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@ETS2K7 arbeitet bereits daran.

Seit 10.11.2026.

  • #27291 von @ETS2K7 — offen

Bewertung

Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Veraltet
Tech-Stack
rust
Bereich
backend, databases

Rechercherichtung

Start in datafusion/common/src/scalar/mod.rs at ScalarValue::to_array_of_size, focusing on the Utf8 and Binary arms and comparing them with the Utf8View arm. Check how repeated values handle offset overflow and look for relevant tests. Done means oversized values return an error instead of panicking; a linked pull request is already open.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

bug

Describe the bug

ScalarValue::to_array_of_size returns a Result, but for a Utf8 or Binary scalar it panics when the value's length times size is more than i32::MAX bytes. The Utf8 and Binary arms call GenericByteArray::new_repeated, which builds its offsets with OffsetBuffer::from_repeated_length, and that panics on offset overflow. The Utf8View and BinaryView arms use the fallible try_append_value_n instead.

A query reaches it: a projection turns a scalar result into an array of the batch's rows through ColumnarValue::into_array_of_size, which calls to_array_of_size. The panic happens before the value bytes are allocated, so it needs no memory beyond the scalar.

To Reproduce

use datafusion_common::ScalarValue;

fn main() {
    // 2^20 bytes in each of 2^11 rows is 2^31 bytes, one past i32::MAX
    let value = "a".repeat(1 << 20);

    let view = ScalarValue::Utf8View(Some(value.clone())).to_array_of_size(1 << 11);
    println!("Utf8View: {:?}", view.map(|a| a.len()));
    // Utf8View: Ok(2048)

    let utf8 = ScalarValue::Utf8(Some(value)).to_array_of_size(1 << 11);
    // panics: offset overflow: 2147483648 does not fit in i32
    println!("Utf8: {:?}", utf8.map(|a| a.len()));
}

The same panic from SQL, through ProjectionExec:

use datafusion::prelude::SessionContext;

#[tokio::main]
async fn main() {
    let ctx = SessionContext::new();
    let r = ctx
        .sql("SELECT repeat('a', 1048576) AS s FROM generate_series(1, 2048)")
        .await
        .unwrap()
        .collect()
        .await;
    // panics: offset overflow: 2147483648 does not fit in i32
    println!("{:?}", r.map(|b| b.iter().map(|b| b.num_rows()).sum::<usize>()));
}

The backtrace of the second runs through Projector::project_batch, ColumnarValue::into_array_of_size, ScalarValue::to_array_of_size and StringArray::new_repeated.

Expected behavior

to_array_of_size returns an error, as it does for Utf8View. The query fails with that error instead of panicking. GenericByteArray::new_repeated has no fallible form, so the Utf8 and Binary arms could check value.len().checked_mul(size) against i32::MAX before calling it.

Additional context

Property-based tests found this on main at 14b3edd9a3, when a test broadcast a scalar result of repeat to the call's rows. Both reproductions panic on main at 14b3edd9a3, which builds on arrow 60.1.0. The LargeUtf8 and LargeBinary arms have i64 offsets, so for them the size limit is the allocation, not the offsets.

Vorherrschende Sprache
Rust
Sterne
9.6k
Forks
2.6k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
345

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/datafusion

Alle Issues in apache/datafusion

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.