`ScalarValue::to_array_of_size` panics when a `Utf8` or `Binary` scalar repeated `size` times passes `i32::MAX` bytes

Aperta
  • Qualcosa non va?
#27,277 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@ETS2K7 ci sta già lavorando.

Dal 10/11/2026.

  • #27291 di @ETS2K7 — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
rust
Ambito
backend, databases

Direzione di ricerca

Start in datafusion/common/src/scalar/mod.rs at ScalarValue::to_array_of_size, focusing on the Utf8 and Binary arms and comparing them with the Utf8View arm. Check how repeated values handle offset overflow and look for relevant tests. Done means oversized values return an error instead of panicking; a linked pull request is already open.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Describe the bug

ScalarValue::to_array_of_size returns a Result, but for a Utf8 or Binary scalar it panics when the value's length times size is more than i32::MAX bytes. The Utf8 and Binary arms call GenericByteArray::new_repeated, which builds its offsets with OffsetBuffer::from_repeated_length, and that panics on offset overflow. The Utf8View and BinaryView arms use the fallible try_append_value_n instead.

A query reaches it: a projection turns a scalar result into an array of the batch's rows through ColumnarValue::into_array_of_size, which calls to_array_of_size. The panic happens before the value bytes are allocated, so it needs no memory beyond the scalar.

To Reproduce

use datafusion_common::ScalarValue;

fn main() {
    // 2^20 bytes in each of 2^11 rows is 2^31 bytes, one past i32::MAX
    let value = "a".repeat(1 << 20);

    let view = ScalarValue::Utf8View(Some(value.clone())).to_array_of_size(1 << 11);
    println!("Utf8View: {:?}", view.map(|a| a.len()));
    // Utf8View: Ok(2048)

    let utf8 = ScalarValue::Utf8(Some(value)).to_array_of_size(1 << 11);
    // panics: offset overflow: 2147483648 does not fit in i32
    println!("Utf8: {:?}", utf8.map(|a| a.len()));
}

The same panic from SQL, through ProjectionExec:

use datafusion::prelude::SessionContext;

#[tokio::main]
async fn main() {
    let ctx = SessionContext::new();
    let r = ctx
        .sql("SELECT repeat('a', 1048576) AS s FROM generate_series(1, 2048)")
        .await
        .unwrap()
        .collect()
        .await;
    // panics: offset overflow: 2147483648 does not fit in i32
    println!("{:?}", r.map(|b| b.iter().map(|b| b.num_rows()).sum::<usize>()));
}

The backtrace of the second runs through Projector::project_batch, ColumnarValue::into_array_of_size, ScalarValue::to_array_of_size and StringArray::new_repeated.

Expected behavior

to_array_of_size returns an error, as it does for Utf8View. The query fails with that error instead of panicking. GenericByteArray::new_repeated has no fallible form, so the Utf8 and Binary arms could check value.len().checked_mul(size) against i32::MAX before calling it.

Additional context

Property-based tests found this on main at 14b3edd9a3, when a test broadcast a scalar result of repeat to the call's rows. Both reproductions panic on main at 14b3edd9a3, which builds on arrow 60.1.0. The LargeUtf8 and LargeBinary arms have i64 offsets, so for them the size limit is the allocation, not the offsets.

Lingua principale
Rust
Stelle
9.6k
Fork
2.6k
Merge medio
3g 4h
PR unite (30g)
345

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion

Tutte le issue di apache/datafusion

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.