[Bug] RowFormatReaderFactory leaks the input stream when reading the footer fails
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- java
- Domain
- data-engineering
Research direction
Start at RowFormatReaderFactory.createReader and trace ownership of the stream after opening it, especially the footer and block-index parsing paths. Use the truncated-tail or corrupt-footer scenario described in the issue; done means failures close the input stream before being rethrown, including errors from seek, read, footer parsing, and block-index parsing.
Written by the indexing model from the issue text.
Description
Search before asking
- I searched in the issues and found nothing similar.
Paimon version
master, 142f8239b
Compute Engine
Engine independent.
Minimal reproduce step
Found by code inspection rather than from a failing job. Read a row format file whose tail is truncated, or whose footer is corrupt, so that footer or block index parsing throws.
What doesn't meet your expectations?
RowFormatReaderFactory.createReader opens the stream first and only transfers ownership to RowFormatReader on the last line:
SeekableInputStream in = fileIO.newInputStream(path);
int tailSize = (int) Math.min(TAIL_PREFETCH_SIZE, fileSize);
long tailOffset = fileSize - tailSize;
in.seek(tailOffset);
byte[] tailBuf = new byte[tailSize];
IOUtils.readFully(in, tailBuf);
RowFileFooter footer = RowFileFooter.readFrom(tailBuf, tailSize - RowFileFooter.FOOTER_SIZE);
RowBlockIndex blockIndex;
...
blockIndex = RowBlockIndex.readFrom(in, footer.indexOffset, footer.indexLength);
return new RowFormatReader(in, path, footer, blockIndex, rowType, projection, context.selection());
There is no try / catch anywhere in the method, so a throw from in.seek, IOUtils.readFully, RowFileFooter.readFrom or RowBlockIndex.readFrom leaves in open with no owner. Scanning a set of files where several are corrupt leaks one stream per file.
Anything else?
Same interaction as the sibling report on CachingSeekableInputStream: with the lifetime tracking added in #8962, a stream that is never closed holds its lease forever, so that entry's FileIO is never released. That matches today's behaviour rather than regressing it, but it does keep the fix from reaching the affected entries.
Fix shape: close in before rethrowing.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 468
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/paimon
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
[Bug] [Hive] IndexOutOfBoundsException when converting an unavailable dynamic BETWEEN predicate Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100