Imported Arrow strings lose nulls when exported in nonzero-offset record batches

Offen
  • Stimmt etwas nicht?
#7,800 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
Ein halber Tag
Anfängerfreundlichkeit
66/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
csharp
Bereich
data

Rechercherichtung

The bug is in the null-count loop of ArrowStringDataFrameColumn.GetNullCount in src/Microsoft.Data.Analysis/DataFrameColumns/ArrowStringDataFrameColumn.cs, which uses i < numberOfRows where it should scan from startIndex to startIndex + numberOfRows. Check the same pattern in PrimitiveDataFrameColumn<T>.GetNullCount. Done when a small unit test with a nonzero-offset slice (not the multi-GB repro) shows the correct NullCount and IsNull for the final row.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

area-DataFrame bug untriaged

Description

An Arrow record imported with DataFrame.FromArrowRecordBatch() can export a null string as a valid empty string when ToArrowRecordBatches() divides the DataFrame into multiple batches.

Reproduced on Microsoft.Data.Analysis 0.23.0, Apache.Arrow 14.0.2, Windows x64, .NET 10.

Reproduction

Create a console app, add Microsoft.Data.Analysis version 0.23.0, and run:

using System;
using System.Linq;
using Apache.Arrow;
using Microsoft.Data.Analysis;

const int rows = 268_304_384;
var builder = new StringArray.Builder().Reserve(rows);
builder.Append("a");
builder.AppendRange(Enumerable.Repeat("", rows - 2));
builder.AppendNull();
var input = new RecordBatch.Builder()
    .Append("strings", true, builder.Build())
    .Build();
var frame = DataFrame.FromArrowRecordBatch(input);

var numbers = new Int64DataFrameColumn("numbers");
numbers.AppendMany(42, rows);
frame.Columns.Add(numbers);

Console.WriteLine($"Imported rows={frame.Rows.Count}; NullCount={frame.Columns["strings"].NullCount}; final string is null={frame[rows - 1, 0] is null}");
int index = 0;
foreach (var batch in frame.ToArrowRecordBatches())
{
    var strings = batch.Arrays.OfType<StringArray>().Last();
    Console.WriteLine($"Batch {index++}: rows={batch.Length}; last string array length={strings.Length}; NullCount={strings.NullCount}");
    if (strings.Length == 1)
        Console.WriteLine($"Final exported string={(strings.GetString(0) is { } value ? $"\"{value}\"" : "null")}; IsNull={strings.IsNull(0)}");
}

This deliberately large input needs several GB of RAM. The numeric column naturally occupies two buffers and forces two record batches. The strings remain in one buffer; their final null is exported as a nonzero-offset, one-row slice. A small, single-batch input would not reach the faulty loop bounds.

Actual output
Imported rows=268304384; NullCount=1; final string is null=True
Batch 0: rows=268304383; last string array length=268304383; NullCount=0
Batch 1: rows=1; last string array length=1; NullCount=0
Final exported string=""; IsNull=False
Expected behavior

The final string array should have NullCount == 1, IsNull(0) == true, and GetString(0) == null.

The repro selects the last string array because there is a separate bug where ToArrowRecordBatches() includes previous batches' arrays in later batches (https://github.com/dotnet/machinelearning/issues/7799). This does not cause the newly exported string array's incorrect null metadata.

Cause

ArrowStringDataFrameColumn.GetNullCount loops with i < numberOfRows, treating a row count as an exclusive end index. It should scan [startIndex, startIndex + numberOfRows).

For the final batch, startIndex == 268304383 and numberOfRows == 1, so no validity bits are checked and the null count is incorrectly zero. PrimitiveDataFrameColumn<T>.GetNullCount contains the same loop-bound pattern and should also be considered when fixing this.

Vorherrschende Sprache
C#
Sterne
9.4k
Forks
2k
Ø Merge
2 T. 18 Std.
Gemergte PRs (30 T.)
64

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus dotnet/machinelearning

Alle Issues in dotnet/machinelearning

Ähnliche Issues

Weitere Issues zu C#

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.