Imported Arrow strings lose nulls when exported in nonzero-offset record batches

Aperta
  • Qualcosa non va?
#7,800 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
Mezza giornata
Idoneità per principianti
66/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
data

Direzione di ricerca

The bug is in the null-count loop of ArrowStringDataFrameColumn.GetNullCount in src/Microsoft.Data.Analysis/DataFrameColumns/ArrowStringDataFrameColumn.cs, which uses i < numberOfRows where it should scan from startIndex to startIndex + numberOfRows. Check the same pattern in PrimitiveDataFrameColumn<T>.GetNullCount. Done when a small unit test with a nonzero-offset slice (not the multi-GB repro) shows the correct NullCount and IsNull for the final row.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area-DataFrame bug untriaged

Description

An Arrow record imported with DataFrame.FromArrowRecordBatch() can export a null string as a valid empty string when ToArrowRecordBatches() divides the DataFrame into multiple batches.

Reproduced on Microsoft.Data.Analysis 0.23.0, Apache.Arrow 14.0.2, Windows x64, .NET 10.

Reproduction

Create a console app, add Microsoft.Data.Analysis version 0.23.0, and run:

using System;
using System.Linq;
using Apache.Arrow;
using Microsoft.Data.Analysis;

const int rows = 268_304_384;
var builder = new StringArray.Builder().Reserve(rows);
builder.Append("a");
builder.AppendRange(Enumerable.Repeat("", rows - 2));
builder.AppendNull();
var input = new RecordBatch.Builder()
    .Append("strings", true, builder.Build())
    .Build();
var frame = DataFrame.FromArrowRecordBatch(input);

var numbers = new Int64DataFrameColumn("numbers");
numbers.AppendMany(42, rows);
frame.Columns.Add(numbers);

Console.WriteLine($"Imported rows={frame.Rows.Count}; NullCount={frame.Columns["strings"].NullCount}; final string is null={frame[rows - 1, 0] is null}");
int index = 0;
foreach (var batch in frame.ToArrowRecordBatches())
{
    var strings = batch.Arrays.OfType<StringArray>().Last();
    Console.WriteLine($"Batch {index++}: rows={batch.Length}; last string array length={strings.Length}; NullCount={strings.NullCount}");
    if (strings.Length == 1)
        Console.WriteLine($"Final exported string={(strings.GetString(0) is { } value ? $"\"{value}\"" : "null")}; IsNull={strings.IsNull(0)}");
}

This deliberately large input needs several GB of RAM. The numeric column naturally occupies two buffers and forces two record batches. The strings remain in one buffer; their final null is exported as a nonzero-offset, one-row slice. A small, single-batch input would not reach the faulty loop bounds.

Actual output
Imported rows=268304384; NullCount=1; final string is null=True
Batch 0: rows=268304383; last string array length=268304383; NullCount=0
Batch 1: rows=1; last string array length=1; NullCount=0
Final exported string=""; IsNull=False
Expected behavior

The final string array should have NullCount == 1, IsNull(0) == true, and GetString(0) == null.

The repro selects the last string array because there is a separate bug where ToArrowRecordBatches() includes previous batches' arrays in later batches (https://github.com/dotnet/machinelearning/issues/7799). This does not cause the newly exported string array's incorrect null metadata.

Cause

ArrowStringDataFrameColumn.GetNullCount loops with i < numberOfRows, treating a row count as an exclusive end index. It should scan [startIndex, startIndex + numberOfRows).

For the final batch, startIndex == 268304383 and numberOfRows == 1, so no validity bits are checked and the null count is incorrectly zero. PrimitiveDataFrameColumn<T>.GetNullCount contains the same loop-bound pattern and should also be considered when fixing this.

Lingua principale
C#
Stelle
9.4k
Fork
2k
Merge medio
2g 18h
PR unite (30g)
64

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di dotnet/machinelearning

Tutte le issue di dotnet/machinelearning

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.