`ScalarValue::to_array_of_size` panics when a `Utf8` or `Binary` scalar repeated `size` times passes `i32::MAX` bytes

Ouverte
  • Quelque chose ne va pas ?
#27,277 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@ETS2K7 y travaille déjà.

Depuis le 10/11/2026.

  • #27291 par @ETS2K7 — ouverte

Évaluation

Difficulté
2/5
Temps estimé
1-3 heures
Accessibilité débutants
35/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
À l'abandon
Stack technique
rust
Domaine
backend, databases

Piste de recherche

Start in datafusion/common/src/scalar/mod.rs at ScalarValue::to_array_of_size, focusing on the Utf8 and Binary arms and comparing them with the Utf8View arm. Check how repeated values handle offset overflow and look for relevant tests. Done means oversized values return an error instead of panicking; a linked pull request is already open.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

bug

Describe the bug

ScalarValue::to_array_of_size returns a Result, but for a Utf8 or Binary scalar it panics when the value's length times size is more than i32::MAX bytes. The Utf8 and Binary arms call GenericByteArray::new_repeated, which builds its offsets with OffsetBuffer::from_repeated_length, and that panics on offset overflow. The Utf8View and BinaryView arms use the fallible try_append_value_n instead.

A query reaches it: a projection turns a scalar result into an array of the batch's rows through ColumnarValue::into_array_of_size, which calls to_array_of_size. The panic happens before the value bytes are allocated, so it needs no memory beyond the scalar.

To Reproduce

use datafusion_common::ScalarValue;

fn main() {
    // 2^20 bytes in each of 2^11 rows is 2^31 bytes, one past i32::MAX
    let value = "a".repeat(1 << 20);

    let view = ScalarValue::Utf8View(Some(value.clone())).to_array_of_size(1 << 11);
    println!("Utf8View: {:?}", view.map(|a| a.len()));
    // Utf8View: Ok(2048)

    let utf8 = ScalarValue::Utf8(Some(value)).to_array_of_size(1 << 11);
    // panics: offset overflow: 2147483648 does not fit in i32
    println!("Utf8: {:?}", utf8.map(|a| a.len()));
}

The same panic from SQL, through ProjectionExec:

use datafusion::prelude::SessionContext;

#[tokio::main]
async fn main() {
    let ctx = SessionContext::new();
    let r = ctx
        .sql("SELECT repeat('a', 1048576) AS s FROM generate_series(1, 2048)")
        .await
        .unwrap()
        .collect()
        .await;
    // panics: offset overflow: 2147483648 does not fit in i32
    println!("{:?}", r.map(|b| b.iter().map(|b| b.num_rows()).sum::<usize>()));
}

The backtrace of the second runs through Projector::project_batch, ColumnarValue::into_array_of_size, ScalarValue::to_array_of_size and StringArray::new_repeated.

Expected behavior

to_array_of_size returns an error, as it does for Utf8View. The query fails with that error instead of panicking. GenericByteArray::new_repeated has no fallible form, so the Utf8 and Binary arms could check value.len().checked_mul(size) against i32::MAX before calling it.

Additional context

Property-based tests found this on main at 14b3edd9a3, when a test broadcast a scalar result of repeat to the call's rows. Both reproductions panic on main at 14b3edd9a3, which builds on arrow 60.1.0. The LargeUtf8 and LargeBinary arms have i64 offsets, so for them the size limit is the allocation, not the offsets.

Langage dominant
Rust
Étoiles
9.6k
Forks
2.6k
Merge moyen
3 j 4 h
PR mergées (30 j)
345

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/datafusion

Toutes les issues de apache/datafusion

Issues similaires

Plus d'issues Rust

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.