Imported Arrow strings lose nulls when exported in nonzero-offset record batches

Abierto
  • ¿Algo no está bien?
#7,800 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
Medio día
Aptitud para principiantes
66/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
csharp
Área
data

Línea de trabajo

The bug is in the null-count loop of ArrowStringDataFrameColumn.GetNullCount in src/Microsoft.Data.Analysis/DataFrameColumns/ArrowStringDataFrameColumn.cs, which uses i < numberOfRows where it should scan from startIndex to startIndex + numberOfRows. Check the same pattern in PrimitiveDataFrameColumn<T>.GetNullCount. Done when a small unit test with a nonzero-offset slice (not the multi-GB repro) shows the correct NullCount and IsNull for the final row.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area-DataFrame bug untriaged

Description

An Arrow record imported with DataFrame.FromArrowRecordBatch() can export a null string as a valid empty string when ToArrowRecordBatches() divides the DataFrame into multiple batches.

Reproduced on Microsoft.Data.Analysis 0.23.0, Apache.Arrow 14.0.2, Windows x64, .NET 10.

Reproduction

Create a console app, add Microsoft.Data.Analysis version 0.23.0, and run:

using System;
using System.Linq;
using Apache.Arrow;
using Microsoft.Data.Analysis;

const int rows = 268_304_384;
var builder = new StringArray.Builder().Reserve(rows);
builder.Append("a");
builder.AppendRange(Enumerable.Repeat("", rows - 2));
builder.AppendNull();
var input = new RecordBatch.Builder()
    .Append("strings", true, builder.Build())
    .Build();
var frame = DataFrame.FromArrowRecordBatch(input);

var numbers = new Int64DataFrameColumn("numbers");
numbers.AppendMany(42, rows);
frame.Columns.Add(numbers);

Console.WriteLine($"Imported rows={frame.Rows.Count}; NullCount={frame.Columns["strings"].NullCount}; final string is null={frame[rows - 1, 0] is null}");
int index = 0;
foreach (var batch in frame.ToArrowRecordBatches())
{
    var strings = batch.Arrays.OfType<StringArray>().Last();
    Console.WriteLine($"Batch {index++}: rows={batch.Length}; last string array length={strings.Length}; NullCount={strings.NullCount}");
    if (strings.Length == 1)
        Console.WriteLine($"Final exported string={(strings.GetString(0) is { } value ? $"\"{value}\"" : "null")}; IsNull={strings.IsNull(0)}");
}

This deliberately large input needs several GB of RAM. The numeric column naturally occupies two buffers and forces two record batches. The strings remain in one buffer; their final null is exported as a nonzero-offset, one-row slice. A small, single-batch input would not reach the faulty loop bounds.

Actual output
Imported rows=268304384; NullCount=1; final string is null=True
Batch 0: rows=268304383; last string array length=268304383; NullCount=0
Batch 1: rows=1; last string array length=1; NullCount=0
Final exported string=""; IsNull=False
Expected behavior

The final string array should have NullCount == 1, IsNull(0) == true, and GetString(0) == null.

The repro selects the last string array because there is a separate bug where ToArrowRecordBatches() includes previous batches' arrays in later batches (https://github.com/dotnet/machinelearning/issues/7799). This does not cause the newly exported string array's incorrect null metadata.

Cause

ArrowStringDataFrameColumn.GetNullCount loops with i < numberOfRows, treating a row count as an exclusive end index. It should scan [startIndex, startIndex + numberOfRows).

For the final batch, startIndex == 268304383 and numberOfRows == 1, so no validity bits are checked and the null count is incorrectly zero. PrimitiveDataFrameColumn<T>.GetNullCount contains the same loop-bound pattern and should also be considered when fixing this.

Lenguaje dominante
C#
Estrellas
9.4k
Forks
2k
Merge medio
2 d 18 h
PR fusionados (30 d)
64

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotnet/machinelearning

Todos los issues de dotnet/machinelearning

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.