Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

CArrowArrayStreamExporter never disposes exported record batches, leaving native memory to the finalizer

未关闭
  • 哪里不对？
#464 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@ump45nose 已经在做这个了。

开始于 2026年10月20日。

  • #483 来自 @ump45nose —— 未关闭

评估

难度
4/5
预计耗时
3-5 天
新手友好度
38/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
csharp
领域
backend

调研方向

Start by reading CArrowArrayStreamExporter.GetNext, CArrowArrayExporter.ExportRecordBatch, and the ownership behavior in ArrowBuffer.TryExport and ExportedAllocationOwner; check how NativeMemoryManager behaves when pinned. The issue describes the leak and candidate approaches, but choosing a safe ownership strategy is the main design work. Done means releasing an exported array also releases the batch's buffers without relying on finalization, including the non-shared-buffer case.

由索引模型根据 Issue 内容生成。

描述

Describe the bug

CArrowArrayStreamExporter.GetNext reads a RecordBatch from the underlying IArrowArrayStream, exports it with CArrowArrayExporter.ExportRecordBatch, and then never disposes it:

RecordBatch recordBatch = arrayStream.ArrowArrayStream.ReadNextRecordBatchAsync().Result;
if (recordBatch != null)
{
    CArrowArrayExporter.ExportRecordBatch(recordBatch, cArray);
}

For allocator-backed buffers, ArrowBuffer.TryExport takes its own reference (_handle.Retain()), so the exported ArrowArray keeps the memory alive independently of the batch. When the consumer calls release, only the exported reference is dropped. The batch's own reference is never released, so with the default NativeMemoryAllocator the native memory is only freed when the NativeMemoryManager finalizer eventually runs.

In practice this means every batch exported through the C stream interface depends on finalization to free its buffers. Reclamation is driven entirely by GC activity in the exporting runtime, which is mostly the AddMemoryPressure calls in NativeMemoryAllocator and ExportedAllocationOwner.

This is especially visible in NativeAOT-compiled libraries (for example, ADBC drivers). They have their own private GC heap, and nothing collects that heap once the library stops allocating.

Measurements

I used a NativeAOT-compiled ADBC driver loaded by a C host. The host drains a stream of 6 batches × 4M rows twice and releases every array, the stream and the statement/connection/database. All numbers are process private bytes; the baseline is about 3 MB.

as-is stream disposes each batch after export
after release, no GC 395 MB 335 MB
after GC.Collect(2, Aggressive, true, true) 138 MB 5.8 MB
after Collect + WaitForPendingFinalizers + Aggressive 5.5 MB —

The 138 MB left behind in the "as-is" column is native memory that only the finalizer queue frees.

Suggested fix

Have GetNext dispose the batch once it has been exported. Disposing is safe for buffers that TryExport retains (_handle != null).

There is one caveat. Buffers without a shared handle are exported by pinning _memory, and some MemoryManager implementations (e.g. NativeMemoryManager) throw when disposed while pinned. A complete fix therefore needs one of these:

  • make the export path always take an owning reference, or
  • only dispose the batch when every buffer was exported by reference, or
  • move ownership of the batch into the ExportedAllocationOwner, so it is disposed when the exported array is released.

The third option is probably the cleanest, since it also covers the non-shared case.

Component(s)

C#

主要语言
C#
星标
42
派生
31
平均合并
1 天 1 小时
30 天内合并 PR
24

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/arrow-dotnet 的其他 Issue

查看 apache/arrow-dotnet 的全部 Issue

相似的 Issue

更多 C# Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。