ParquetFileWriter.end() flushes an incomplete file to storage when finalizing fails
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- java
- Domain
- data-engineering
Research direction
Start by locating ParquetFileWriter.end() and withAbortOnFailure in the parquet-hadoop writer path, then trace the finally block and failure handling around serialization and close(). Check existing writer-finalization tests, if present. Done means serialization failures, including non-IOException failures, abort without flushing an incomplete file to the final path.
Written by the indexing model from the issue text.
Description
Describe the bug, including details regarding any error messages, version, and platform.
close() in end()'s finally runs before the writer is marked aborted, so a failure serializing indexes/bloom filters/footer flushes the incomplete stream. And withAbortOnFailure only catches IOException, so a RuntimeException/Error (e.g. OOM) never aborts at all. On non-rename storage this leaves a corrupt file at the final path that a retried task silently reuses. Affects the parquet-hadoop writer path on master, any platform.
Component(s)
Core
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 28
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/parquet-java
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/parquet-java#3792 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/parquet-java#3767 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/parquet-java#3695 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/parquet-java#3667 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/parquet-java#3587 ·
All issues in apache/parquet-java
Similar issues
-
awaiting triage bug Causes friction Hop Gui P1 P2 Transforms
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/flink-agents#1152 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
jenkinsci/blueocean-plugin#5417 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
objectionary/eo-graphs#75 ·