Hacktoberfest 2026:維護者為十月標記出來的 issue,仍然開放、適合新手。 瀏覽 Hacktoberfest issue

[Java] VectorUnloader has multiple memory leak issues

未關閉
#1,234 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

評估

難度
4/5
預估耗時
3-5 天
新手友好度
62/100
Issue 類型
缺陷
描述清晰度
描述清楚
活躍度
冷清
技術堆疊
java
領域
data

研究方向

從 VectorUnloader 類別開始,重點檢查 getRecordBatch() 和 appendNodes(),尤其是 issue 中描述的 retain()、compress() 以及遞迴子節點處理路徑。驗證每個位置發生失敗時都會釋放所有保留的 ArrowBuf,包括先前反覆執行中的緩衝區,並確認正常的 record-batch 序列化仍然運作。

由索引模型根據 Issue 內容生成。

描述

Type: bug
Describe the bug, including details regarding any error messages, version, and platform.
Problem Description

The VectorUnloader class has serious memory leak issues when serializing Vector data. Any exception thrown during the process will cause previously retained ArrowBuf instances to never be released.
Root Cause
ArrowBuf uses reference counting for memory management. After calling retain(), a corresponding release() must be called to free the buffer. The code has exception-safety defects in multiple locations:


Bug 1: OOM during compression causes leak
Location: appendNodes() method, around line 75
buf.getReferenceManager().retain();
buffers.add(codec.compress(vector.getAllocator(), buf)); // OOM here causes leak
Problem: If codec.compress() throws OutOfMemoryError, retain() has already been called but buffers.add() never executes, making the ArrowBuf unreleasable.

Bug 2: Compression failure in loop causes partial leak
Location: appendNodes() method, around lines 78-85
for (ArrowBuf buf : fieldBuffers) {
    buf.getReferenceManager().retain();
    buffers.add(codec.compress(vector.getAllocator(), buf)); // Any failure leaks all previously retained buffers
}
Problem: Any compression failure in the loop causes all previously retained buffers to leak.

Bug 3: Exception during recursive child vector processing causes leak
Location: appendNodes() method, around lines 87-89
for (FieldVector child : vector.getChildrenFromFields()) {
    appendNodes(child, nodes, buffers, variadicBufferCounts); // Exception here leaks buffers from current level
}
Problem: An exception during recursive child processing prevents release of buffers already retained at the current level.

Bug 4: Iteration failure in top-level loop causes cumulative leaks
Location: getRecordBatch() method, around lines 53-58
for (FieldVector vector : root.getFieldVectors()) {
    appendNodes(vector, nodes, buffers, variadicBufferCounts); // Any failure leaks all previously retained buffers
}
Problem: Each iteration retains buffers, and any failure causes cumulative leaks of all previously retained buffers. This is an accumulated leak.

Impact
  • All code using VectorUnloader.getRecordBatch()
  • Leak probability increases significantly when processing large datasets or when compressor runs low on memory
  • Leaked ArrowBuf instances persist until process termination
    Suggested Fix
  1. Use try-finally to ensure release() is called even when exceptions occur
  2. Or add buffers to the list before calling retain() and compress()
  3. Use transactional error handling for the entire getRecordBatch() operation, rolling back all buffers on failure
主要語言
Java
星號
95
分支
154
平均合併
2 天 10 小時
30 天內合併 PR
11

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

apache/arrow-java 的其他 Issue

查看 apache/arrow-java 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。