Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

CArrowArrayStreamExporter never disposes exported record batches, leaving native memory to the finalizer

クローズ
  • 何か問題がありますか？
#464 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

関連するプルリクエストがすでにマージされています。

  • #483 @ump45nose による — マージ済み

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
38/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
csharp
領域
backend

調査の方向性

Start by reading CArrowArrayStreamExporter.GetNext, CArrowArrayExporter.ExportRecordBatch, and the ownership behavior in ArrowBuffer.TryExport and ExportedAllocationOwner; check how NativeMemoryManager behaves when pinned. The issue describes the leak and candidate approaches, but choosing a safe ownership strategy is the main design work. Done means releasing an exported array also releases the batch's buffers without relying on finalization, including the non-shared-buffer case.

索引モデルが issue の本文から書いたものです。

説明

Describe the bug

CArrowArrayStreamExporter.GetNext reads a RecordBatch from the underlying IArrowArrayStream, exports it with CArrowArrayExporter.ExportRecordBatch, and then never disposes it:

RecordBatch recordBatch = arrayStream.ArrowArrayStream.ReadNextRecordBatchAsync().Result;
if (recordBatch != null)
{
    CArrowArrayExporter.ExportRecordBatch(recordBatch, cArray);
}

For allocator-backed buffers, ArrowBuffer.TryExport takes its own reference (_handle.Retain()), so the exported ArrowArray keeps the memory alive independently of the batch. When the consumer calls release, only the exported reference is dropped. The batch's own reference is never released, so with the default NativeMemoryAllocator the native memory is only freed when the NativeMemoryManager finalizer eventually runs.

In practice this means every batch exported through the C stream interface depends on finalization to free its buffers. Reclamation is driven entirely by GC activity in the exporting runtime, which is mostly the AddMemoryPressure calls in NativeMemoryAllocator and ExportedAllocationOwner.

This is especially visible in NativeAOT-compiled libraries (for example, ADBC drivers). They have their own private GC heap, and nothing collects that heap once the library stops allocating.

Measurements

I used a NativeAOT-compiled ADBC driver loaded by a C host. The host drains a stream of 6 batches × 4M rows twice and releases every array, the stream and the statement/connection/database. All numbers are process private bytes; the baseline is about 3 MB.

as-is stream disposes each batch after export
after release, no GC 395 MB 335 MB
after GC.Collect(2, Aggressive, true, true) 138 MB 5.8 MB
after Collect + WaitForPendingFinalizers + Aggressive 5.5 MB —

The 138 MB left behind in the "as-is" column is native memory that only the finalizer queue frees.

Suggested fix

Have GetNext dispose the batch once it has been exported. Disposing is safe for buffers that TryExport retains (_handle != null).

There is one caveat. Buffers without a shared handle are exported by pinning _memory, and some MemoryManager implementations (e.g. NativeMemoryManager) throw when disposed while pinned. A complete fix therefore needs one of these:

  • make the export path always take an owning reference, or
  • only dispose the batch when every buffer was exported by reference, or
  • move ownership of the batch into the ExportedAllocationOwner, so it is disposed when the exported array is released.

The third option is probably the cleanest, since it also covers the non-shared case.

Component(s)

C#

主要言語
C#
スター
42
フォーク
31
平均マージ
1日 3時間
マージ済み PR（30日）
25

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/arrow-dotnet のほかの issue

apache/arrow-dotnet の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。