Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Java] VectorUnloader has multiple memory leak issues

Aperta
#1,234 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
62/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
java
Ambito
data

Direzione di ricerca

Inizia con la classe VectorUnloader, concentrandoti su getRecordBatch() e appendNodes(), in particolare sui percorsi di retain(), compress() e dell’elaborazione ricorsiva dei figli descritti nella issue. Verifica che gli errori in ogni punto rilascino ogni ArrowBuf mantenuto, inclusi i buffer delle iterazioni precedenti, e conferma che la serializzazione normale dei record batch continui a funzionare.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Type: bug
Describe the bug, including details regarding any error messages, version, and platform.
Problem Description

The VectorUnloader class has serious memory leak issues when serializing Vector data. Any exception thrown during the process will cause previously retained ArrowBuf instances to never be released.
Root Cause
ArrowBuf uses reference counting for memory management. After calling retain(), a corresponding release() must be called to free the buffer. The code has exception-safety defects in multiple locations:


Bug 1: OOM during compression causes leak
Location: appendNodes() method, around line 75
buf.getReferenceManager().retain();
buffers.add(codec.compress(vector.getAllocator(), buf)); // OOM here causes leak
Problem: If codec.compress() throws OutOfMemoryError, retain() has already been called but buffers.add() never executes, making the ArrowBuf unreleasable.

Bug 2: Compression failure in loop causes partial leak
Location: appendNodes() method, around lines 78-85
for (ArrowBuf buf : fieldBuffers) {
    buf.getReferenceManager().retain();
    buffers.add(codec.compress(vector.getAllocator(), buf)); // Any failure leaks all previously retained buffers
}
Problem: Any compression failure in the loop causes all previously retained buffers to leak.

Bug 3: Exception during recursive child vector processing causes leak
Location: appendNodes() method, around lines 87-89
for (FieldVector child : vector.getChildrenFromFields()) {
    appendNodes(child, nodes, buffers, variadicBufferCounts); // Exception here leaks buffers from current level
}
Problem: An exception during recursive child processing prevents release of buffers already retained at the current level.

Bug 4: Iteration failure in top-level loop causes cumulative leaks
Location: getRecordBatch() method, around lines 53-58
for (FieldVector vector : root.getFieldVectors()) {
    appendNodes(vector, nodes, buffers, variadicBufferCounts); // Any failure leaks all previously retained buffers
}
Problem: Each iteration retains buffers, and any failure causes cumulative leaks of all previously retained buffers. This is an accumulated leak.

Impact
  • All code using VectorUnloader.getRecordBatch()
  • Leak probability increases significantly when processing large datasets or when compressor runs low on memory
  • Leaked ArrowBuf instances persist until process termination
    Suggested Fix
  1. Use try-finally to ensure release() is called even when exceptions occur
  2. Or add buffers to the list before calling retain() and compress()
  3. Use transactional error handling for the entire getRecordBatch() operation, rolling back all buffers on failure
Lingua principale
Java
Stelle
95
Fork
154
Merge medio
2g 10h
PR unite (30g)
11

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/arrow-java

Tutte le issue di apache/arrow-java

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.