Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

CArrowArrayStreamExporter never disposes exported record batches, leaving native memory to the finalizer

Aperta
  • Qualcosa non va?
#464 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@ump45nose ci sta già lavorando.

Dal 20/10/2026.

  • #483 di @ump45nose — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend

Direzione di ricerca

Start by reading CArrowArrayStreamExporter.GetNext, CArrowArrayExporter.ExportRecordBatch, and the ownership behavior in ArrowBuffer.TryExport and ExportedAllocationOwner; check how NativeMemoryManager behaves when pinned. The issue describes the leak and candidate approaches, but choosing a safe ownership strategy is the main design work. Done means releasing an exported array also releases the batch's buffers without relying on finalization, including the non-shared-buffer case.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

CArrowArrayStreamExporter.GetNext reads a RecordBatch from the underlying IArrowArrayStream, exports it with CArrowArrayExporter.ExportRecordBatch, and then never disposes it:

RecordBatch recordBatch = arrayStream.ArrowArrayStream.ReadNextRecordBatchAsync().Result;
if (recordBatch != null)
{
    CArrowArrayExporter.ExportRecordBatch(recordBatch, cArray);
}

For allocator-backed buffers, ArrowBuffer.TryExport takes its own reference (_handle.Retain()), so the exported ArrowArray keeps the memory alive independently of the batch. When the consumer calls release, only the exported reference is dropped. The batch's own reference is never released, so with the default NativeMemoryAllocator the native memory is only freed when the NativeMemoryManager finalizer eventually runs.

In practice this means every batch exported through the C stream interface depends on finalization to free its buffers. Reclamation is driven entirely by GC activity in the exporting runtime, which is mostly the AddMemoryPressure calls in NativeMemoryAllocator and ExportedAllocationOwner.

This is especially visible in NativeAOT-compiled libraries (for example, ADBC drivers). They have their own private GC heap, and nothing collects that heap once the library stops allocating.

Measurements

I used a NativeAOT-compiled ADBC driver loaded by a C host. The host drains a stream of 6 batches × 4M rows twice and releases every array, the stream and the statement/connection/database. All numbers are process private bytes; the baseline is about 3 MB.

as-is stream disposes each batch after export
after release, no GC 395 MB 335 MB
after GC.Collect(2, Aggressive, true, true) 138 MB 5.8 MB
after Collect + WaitForPendingFinalizers + Aggressive 5.5 MB —

The 138 MB left behind in the "as-is" column is native memory that only the finalizer queue frees.

Suggested fix

Have GetNext dispose the batch once it has been exported. Disposing is safe for buffers that TryExport retains (_handle != null).

There is one caveat. Buffers without a shared handle are exported by pinning _memory, and some MemoryManager implementations (e.g. NativeMemoryManager) throw when disposed while pinned. A complete fix therefore needs one of these:

  • make the export path always take an owning reference, or
  • only dispose the batch when every buffer was exported by reference, or
  • move ownership of the batch into the ExportedAllocationOwner, so it is disposed when the exported array is released.

The third option is probably the cleanest, since it also covers the non-shared case.

Component(s)

C#

Lingua principale
C#
Stelle
42
Fork
31
Merge medio
1g 1h
PR unite (30g)
24

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/arrow-dotnet

Tutte le issue di apache/arrow-dotnet

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.