Imported Arrow strings lose nulls when exported in nonzero-offset record batches

Ouverte
  • Quelque chose ne va pas ?
#7,800 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
Une demi-journée
Accessibilité débutants
66/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
csharp
Domaine
data

Piste de recherche

The bug is in the null-count loop of ArrowStringDataFrameColumn.GetNullCount in src/Microsoft.Data.Analysis/DataFrameColumns/ArrowStringDataFrameColumn.cs, which uses i < numberOfRows where it should scan from startIndex to startIndex + numberOfRows. Check the same pattern in PrimitiveDataFrameColumn<T>.GetNullCount. Done when a small unit test with a nonzero-offset slice (not the multi-GB repro) shows the correct NullCount and IsNull for the final row.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

area-DataFrame bug untriaged

Description

An Arrow record imported with DataFrame.FromArrowRecordBatch() can export a null string as a valid empty string when ToArrowRecordBatches() divides the DataFrame into multiple batches.

Reproduced on Microsoft.Data.Analysis 0.23.0, Apache.Arrow 14.0.2, Windows x64, .NET 10.

Reproduction

Create a console app, add Microsoft.Data.Analysis version 0.23.0, and run:

using System;
using System.Linq;
using Apache.Arrow;
using Microsoft.Data.Analysis;

const int rows = 268_304_384;
var builder = new StringArray.Builder().Reserve(rows);
builder.Append("a");
builder.AppendRange(Enumerable.Repeat("", rows - 2));
builder.AppendNull();
var input = new RecordBatch.Builder()
    .Append("strings", true, builder.Build())
    .Build();
var frame = DataFrame.FromArrowRecordBatch(input);

var numbers = new Int64DataFrameColumn("numbers");
numbers.AppendMany(42, rows);
frame.Columns.Add(numbers);

Console.WriteLine($"Imported rows={frame.Rows.Count}; NullCount={frame.Columns["strings"].NullCount}; final string is null={frame[rows - 1, 0] is null}");
int index = 0;
foreach (var batch in frame.ToArrowRecordBatches())
{
    var strings = batch.Arrays.OfType<StringArray>().Last();
    Console.WriteLine($"Batch {index++}: rows={batch.Length}; last string array length={strings.Length}; NullCount={strings.NullCount}");
    if (strings.Length == 1)
        Console.WriteLine($"Final exported string={(strings.GetString(0) is { } value ? $"\"{value}\"" : "null")}; IsNull={strings.IsNull(0)}");
}

This deliberately large input needs several GB of RAM. The numeric column naturally occupies two buffers and forces two record batches. The strings remain in one buffer; their final null is exported as a nonzero-offset, one-row slice. A small, single-batch input would not reach the faulty loop bounds.

Actual output
Imported rows=268304384; NullCount=1; final string is null=True
Batch 0: rows=268304383; last string array length=268304383; NullCount=0
Batch 1: rows=1; last string array length=1; NullCount=0
Final exported string=""; IsNull=False
Expected behavior

The final string array should have NullCount == 1, IsNull(0) == true, and GetString(0) == null.

The repro selects the last string array because there is a separate bug where ToArrowRecordBatches() includes previous batches' arrays in later batches (https://github.com/dotnet/machinelearning/issues/7799). This does not cause the newly exported string array's incorrect null metadata.

Cause

ArrowStringDataFrameColumn.GetNullCount loops with i < numberOfRows, treating a row count as an exclusive end index. It should scan [startIndex, startIndex + numberOfRows).

For the final batch, startIndex == 268304383 and numberOfRows == 1, so no validity bits are checked and the null count is incorrectly zero. PrimitiveDataFrameColumn<T>.GetNullCount contains the same loop-bound pattern and should also be considered when fixing this.

Langage dominant
C#
Étoiles
9.4k
Forks
2k
Merge moyen
2 j 18 h
PR mergées (30 j)
64

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de dotnet/machinelearning

Toutes les issues de dotnet/machinelearning

Issues similaires

Plus d'issues C#

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.