[Java] VectorUnloader has multiple memory leak issues
还没有人认领这个 Issue。
评估
调研方向
从 VectorUnloader 类开始,重点检查 getRecordBatch() 和 appendNodes(),尤其是 issue 中描述的 retain()、compress() 以及递归子节点处理路径。验证每个位置发生失败时都会释放所有保留的 ArrowBuf,包括之前迭代中的缓冲区,并确认正常的 record-batch 序列化仍然有效。
由索引模型根据 Issue 内容生成。
描述
Describe the bug, including details regarding any error messages, version, and platform.
Problem Description
The VectorUnloader class has serious memory leak issues when serializing Vector data. Any exception thrown during the process will cause previously retained ArrowBuf instances to never be released.
Root Cause
ArrowBuf uses reference counting for memory management. After calling retain(), a corresponding release() must be called to free the buffer. The code has exception-safety defects in multiple locations:
Bug 1: OOM during compression causes leak
Location: appendNodes() method, around line 75
buf.getReferenceManager().retain();
buffers.add(codec.compress(vector.getAllocator(), buf)); // OOM here causes leak
Problem: If codec.compress() throws OutOfMemoryError, retain() has already been called but buffers.add() never executes, making the ArrowBuf unreleasable.
Bug 2: Compression failure in loop causes partial leak
Location: appendNodes() method, around lines 78-85
for (ArrowBuf buf : fieldBuffers) {
buf.getReferenceManager().retain();
buffers.add(codec.compress(vector.getAllocator(), buf)); // Any failure leaks all previously retained buffers
}
Problem: Any compression failure in the loop causes all previously retained buffers to leak.
Bug 3: Exception during recursive child vector processing causes leak
Location: appendNodes() method, around lines 87-89
for (FieldVector child : vector.getChildrenFromFields()) {
appendNodes(child, nodes, buffers, variadicBufferCounts); // Exception here leaks buffers from current level
}
Problem: An exception during recursive child processing prevents release of buffers already retained at the current level.
Bug 4: Iteration failure in top-level loop causes cumulative leaks
Location: getRecordBatch() method, around lines 53-58
for (FieldVector vector : root.getFieldVectors()) {
appendNodes(vector, nodes, buffers, variadicBufferCounts); // Any failure leaks all previously retained buffers
}
Problem: Each iteration retains buffers, and any failure causes cumulative leaks of all previously retained buffers. This is an accumulated leak.
Impact
- All code using VectorUnloader.getRecordBatch()
- Leak probability increases significantly when processing large datasets or when compressor runs low on memory
- Leaked ArrowBuf instances persist until process termination
Suggested Fix
- Use try-finally to ensure release() is called even when exceptions occur
- Or add buffers to the list before calling retain() and compress()
- Use transactional error handling for the entire getRecordBatch() operation, rolling back all buffers on failure
- 主要语言
- Java
- 星标
- 95
- 派生
- 154
- 平均合并
- 2 天 10 小时
- 30 天内合并 PR
- 11
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/arrow-java 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 74/100
apache/arrow-java#1261 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
apache/arrow-java#1236 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
apache/arrow-java#1230 ·
-
Type: bug
难度 2/5 1-3 小时 新手友好度 85/100
apache/arrow-java#1205 ·
-
Type: bug
难度 2/5 1-3 小时 新手友好度 68/100
apache/arrow-java#1196 · 1 条评论 ·
查看 apache/arrow-java 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 75/100
elastic/gradle-plugins#157 ·
-
enhancement Tools
难度 1/5 1 小时以内 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 70/100
apache/rocketmq-dashboard#5008 ·
-
bug
难度 2/5 1-3 小时 新手友好度 75/100
-
DETECT_PARAMETER_NAMES=false silently disables @ConstructorProperties-based Creator detection too 未关闭
难度 2/5 1-3 小时 新手友好度 70/100
FasterXML/jackson-databind#6229 ·