Java compression codecs do not release compressed ArrowBuf in decompress, causing allocator leaks

Open
#1,037 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
java
Domain
data

Research direction

Locate the compression codec decompression path containing loadFieldBuffers, the decompression loop, and ownBuffers. Move the loop under the existing try-finally cleanup, then verify that a failure after earlier buffers decompress leaves the BufferAllocator at zero allocated memory while the success path still retains buffers in the field vector.

Written by the indexing model from the issue text.

Description

Type: bug
bug

When using Arrow Java’s compression codecs (e.g. ZSTD) from downstream projects, we observed that an OutOfMemoryError: Direct buffer memory during decompression can leave the associated BufferAllocator in a leaked state (its allocated memory never returns to zero after the failing operation).

In our case this surfaced in Apache Fluss (see apache/fluss#2646), but after investigation the root cause appears to be in Arrow Java’s compression codec implementation:

Image
version

18.3.0

Solve

In the original Arrow implementation, the decompression loop runs outside the try-finally block that guards loadFieldBuffers. This means if decompression succeeds for the first N buffers of a field but fails on the (N+1)-th buffer, the already-decompressed buffers in ownBuffers are never closed, leaking Direct Memory.

To fix it, move the decompression loop inside the try block so that the finally clause always closes every buffer in ownBuffers, regardless of whether the load succeeds or fails:

  • Success path: loadFieldBuffers retains each buffer (ref count +1), then the finally close decrements it back (ref count -1). The field vector still holds the buffer.
  • Error path: The finally close decrements each already-decompressed buffer's ref count to 0, immediately freeing the Direct Memory.
Dominant language
Java
Stars
95
Forks
154
Avg merge
2d 16h
Merged PRs (30d)
9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/arrow-java

All issues in apache/arrow-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.