Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

ArrowArrayConcatenator throws on a zero-length string/binary array with an empty offsets buffer

Chiusa Adatta ai principianti
  • Qualcosa non va?
#462 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
90/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
data-engineering

Direzione di ricerca

Inizia da src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs, leggendo ConcatenateVariableBinaryValueBuffer() e ConcatenateLargeVariableBinaryValueBuffer() insieme ai percorsi vicini che ignorano gli input di lunghezza zero. Verifica che la concatenazione di una stringa vuota o di un array binario vuoto con un buffer di offset vuoto non generi più un’eccezione e che i valori dell’altro input siano preservati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

ArrowArrayConcatenator.Concatenate throws ArgumentOutOfRangeException when one of the inputs is a zero-length string, binary, large_string or large_binary array whose offsets buffer is empty (0 bytes).

That input is valid, and it arrives through IPC. For a length-0 variable-size array, Arrow C++ accepts an absent or empty offsets buffer (ARROW-544, ValidateOffsetsAndSizes in cpp/src/arrow/array/validate.cc: "For length 0, an empty offsets buffer is accepted"). Its IPC writer emits that buffer as it is (GetZeroBasedValueOffsets returns array.value_offsets() unchanged when length() == 0). nanoarrow's IPC writer does the same. Apache.Arrow's ArrowStreamReader reads such a batch fine. Only concatenating it fails.

Seen on Apache.Arrow 23.0.0 (.NET 10, Linux arm64). The code is unchanged on main as of 117da35.

Cause

ConcatenateVariableBinaryValueBuffer() (src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:972) and ConcatenateLargeVariableBinaryValueBuffer() (:869) slice Length + 1 offsets from every input:

var offsets = arrayData.Buffers[1].Span.CastTo<int>().Slice(arrayData.Offset, arrayData.Length + 1);

For a length-0 input with an empty offsets buffer, that slices one element out of an empty span and throws. The neighbouring list paths (:758) and the offset-buffer concatenators ConcatenateOffsetBuffer() (:995) and ConcatenateLargeOffsetBuffer() already skip such inputs with if (arrayData.Length == 0) continue;. Those two functions are missing the same guard. A length-0 input contributes no value bytes, so skipping it is correct.

Repro (pure C#)

using Apache.Arrow;
using Apache.Arrow.Types;

// A zero-length string array with an empty (0-byte) offsets buffer, as read from IPC
var empty = new StringArray(new ArrayData(StringType.Default, length: 0, nullCount: 0, offset: 0,
    new[] { ArrowBuffer.Empty, ArrowBuffer.Empty, ArrowBuffer.Empty }));
var other = new StringArray.Builder().Append("a").Build();

Console.WriteLine(ArrowArrayConcatenator.Concatenate(new IArrowArray[] { empty, other }).Length);

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.ConcatenateVariableBinaryValueBuffer()
   at Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.ConcatenateVariableBinaryArrayData(IArrowType type)
   at Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.Visit(StringType type)

Repro (IPC from pyarrow)

This writes a stream whose string, large_string and list<int32> columns have 0-byte offsets buffers. pyarrow 26.0.0 validates it with validate(full=True) before writing and after reading it back:

import pyarrow as pa, pyarrow.ipc as ipc

cols = {
    "s": pa.Array.from_buffers(pa.string(), 0, [None, None, pa.py_buffer(b"")]),
    "ls": pa.Array.from_buffers(pa.large_string(), 0, [None, None, pa.py_buffer(b"")]),
    "l": pa.Array.from_buffers(pa.list_(pa.int32()), 0, [None, None],
                               children=[pa.array([], pa.int32())]),
}
batch = pa.record_batch(list(cols.values()), names=list(cols))
with ipc.new_stream("empty_offsets.arrows", batch.schema) as w:
    w.write_batch(batch)

Reading it with ArrowStreamReader and calling ArrowArrayConcatenator.Concatenate(new[] { col, col }) on each column gives:

column 0 utf8 length=0
  ArgumentOutOfRangeException at ...ConcatenateVariableBinaryValueBuffer()
column 1 large_utf8 length=0
  ArgumentOutOfRangeException at ...ConcatenateLargeVariableBinaryValueBuffer()
column 2 list length=0
  Concatenate([col, col]) length=0      <- lists already skip empty inputs

Expected: the empty input contributes nothing, and the concatenation succeeds.

Suggested fix: add if (arrayData.Length == 0) continue; to the loops in ConcatenateVariableBinaryValueBuffer() and ConcatenateLargeVariableBinaryValueBuffer(), as the other offset-reading paths in the file already do.

Context

I hit this when a DuckDB extension started writing Arrow IPC with nanoarrow, and a .NET worker using Apache.Arrow concatenated the incoming batches. The value array of an empty MAP arrived as a zero-length string with an empty offsets buffer.

Lingua principale
C#
Stelle
42
Fork
31
Merge medio
1g 3h
PR unite (30g)
21

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/arrow-dotnet

Tutte le issue di apache/arrow-dotnet

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.