ToArrowRecordBatches accumulates arrays from previous batches, producing invalid record batches
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 62/100
Research direction
The bug lives in ToArrowRecordBatches() in src/Microsoft.Data.Analysis/DataFrame.Arrow.cs, where the arrays list is declared outside the batching loop and never reset. Read that loop, then check the reproduction against DataFrame.FromArrowRecordBatch. Done means each batch holds only its own column arrays, and the repro's batch 1 reimports as a one-row, one-column frame containing 42; the test to extend is in the Microsoft.Data.Analysis tests for Arrow round-trips.
Written by the indexing model from the issue text.
Description
Description
DataFrame.ToArrowRecordBatches() accumulates arrays from previous batches. A one-column DataFrame exports its second record batch with one schema field but two arrays, including the first batch's stale array.
This invalid batch cannot be imported using DataFrame.FromArrowRecordBatch(), which throws ArgumentOutOfRangeException.
Reproduced on Microsoft.Data.Analysis 0.23.0, Apache.Arrow 14.0.2, Windows x64, .NET 10. The reproduction uses only public, high-level DataFrame APIs and is independent of Arrow string null-count bugs (https://github.com/dotnet/machinelearning/issues/7800 and https://github.com/dotnet/machinelearning/issues/7801).
Reproduction
Create a console app, add Microsoft.Data.Analysis version 0.23.0, and run:
using System;
using System.Linq;
using Microsoft.Data.Analysis;
var numbers = new Int64DataFrameColumn("numbers");
numbers.AppendMany(42, 268_304_384);
var frame = new DataFrame(numbers);
var batches = frame.ToArrowRecordBatches();
foreach (var (i, batch) in batches.Index())
{
Console.WriteLine(
$"Batch {i}: rows={batch.Length}; fields={batch.Schema.FieldsList.Count}; arrays={batch.Arrays.Count()}; array lengths=[{string.Join(", ", batch.Arrays.Select(array => array.Length))}]");
try
{
var roundTrip = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine($"Reimported batch {i}: rows={roundTrip.Rows.Count}; columns={roundTrip.Columns.Count}");
}
catch (ArgumentOutOfRangeException e)
{
Console.WriteLine($"Reimporting batch {i}: {e.GetType().Name}: {e.Message}");
}
}
Requires slightly over 2 GB for the numeric buffers and validity bitmaps. The row count naturally creates two numeric buffers and forces two record batches.
Actual output
Batch 0: rows=268304383; fields=1; arrays=1; array lengths=[268304383]
Reimported batch 0: rows=268304383; columns=1
Batch 1: rows=1; fields=1; arrays=2; array lengths=[268304383, 1]
Reimporting batch 1: ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
Expected behavior
Each batch should contain exactly one array matching its single schema field. Batch 1 should contain only its one-row array, and reimport should produce a one-row, one-column DataFrame containing 42.
Cause
DataFrame.ToArrowRecordBatches() creates the arrays list outside its batching loop and continually appends each batch's arrays without resetting that list.
Each record batch should receive a fresh per-batch collection containing only the current batch's column arrays. Also ensure that continuing enumeration cannot mutate an already yielded batch.
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 51
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dotnet/machinelearning
-
area-Native untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
dotnet/machinelearning#7837 ·
Maintainers usually reply within 2 days
-
area-TensorFlow untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
dotnet/machinelearning#7836 ·
Maintainers usually reply within 2 days
-
LlamaTokenizer encodes some runs of repeated characters differently from SentencePiecePossibly taken @Laurianti claimed this 20 days ago. Openuntriaged
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
dotnet/machinelearning#7751 · 1 comment ·
Maintainers usually reply within 2 days
-
BertTokenizer merges words separated only by `\n`, `\t` or `\r`Possibly taken @rosebyte claimed this 28 days ago. Openarea-Tokenizers green
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
dotnet/machinelearning#7724 · 3 comments · 1 assignee ·
Maintainers usually reply within 2 days
-
area-TimeSeries documentation untriaged
Difficulty 1/5 Under an hour Newbie friendliness 90/100
dotnet/machinelearning#7205 ·
Maintainers usually reply within 2 days
All issues in dotnet/machinelearning
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
bug priority: P2
Difficulty 1/5 Under an hour Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
IridiumIO/CompactGUI#699 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day