Write truncated parquet footer

Open

#3,069 opened on Nov 18, 2024 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 4/5
Estimated time: 3-5 days
Newbie friendliness: 32/100

Issue type: Bug
Clarity: Needs clarification
Activity status: Stale
Tech stack: aws, gcp, java
Domain: cloud, data-engineering

Research direction

No source file, test, or entry point is named. Reproduce the corrupt output on GCP with version 1.14.3 and compare its seek behavior with AWS; done means the writer consistently produces a file ending in PAR1 that can be read without EOFException.

Written by the indexing model from the issue text.

Description

Type: bug

Describe the bug, including details regarding any error messages, version, and platform.

Sometimes a file is written that is missing the last byte, so it ends in .PAR when it should be .PAR1. This causes EOFException when attempting to read the file.

$ hexdump -C good.snappy.parquet| tail -n 10
004fff70  6b 2e 6c 65 67 61 63 79  44 61 74 65 54 69 6d 65  |k.legacyDateTime|
004fff80  18 00 00 18 4a 70 61 72  71 75 65 74 2d 6d 72 20  |....Jparquet-mr |
004fff90  76 65 72 73 69 6f 6e 20  31 2e 31 32 2e 33 20 28  |version 1.12.3 (|
004fffa0  62 75 69 6c 64 20 66 38  64 63 65 64 31 38 32 63  |build f8dced182c|
004fffb0  34 63 31 66 62 64 65 63  36 63 63 62 33 31 38 35  |4c1fbdec6ccb3185|
004fffc0  35 33 37 62 35 61 30 31  65 36 65 64 36 62 29 19  |537b5a01e6ed6b).|
004fffd0  dc 1c 00 00 1c 00 00 1c  00 00 1c 00 00 1c 00 00  |................|
004fffe0  1c 00 00 1c 00 00 1c 00  00 1c 00 00 1c 00 00 1c  |................|
004ffff0  00 00 1c 00 00 1c 00 00  00 e7 0f 00 00 50 41 52  |.............PAR|
00500000

This might be related - we are seeing this issue only on GCP, not AWS. For GCP we do disk seeks randomly and on AWS we do disk seeks sequentially.

We can rerun a job that writes the corrupt parquet file, and it will succeed the second time, so it seems to be nondeterministic.

This is on version 1.14.3.

Component(s)

No response

Dominant language: Java
Stars: 3.1k
Forks: 1.6k
Avg merge: 6d 16h
Merged PRs (30d): 36

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

chore: aspectjweaver(EPL-2.0)와 AGPL-3.0 호환성 정리 Open

documentation

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

inu-appcenter/memorIN-backend#288 · Opened Sep 24, 2026
Vulnerability Nessus 12085 - Apache Tomcat Default Files (with killbill/killbill:0.24.21) Open

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

killbill/killbill#2314 · 1 comment · Opened Sep 24, 2026
Work list, Cases and Compliance overlap, and nothing says which one to use Open

frontend maui-pilot pilot-ask question

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

Taleef7/workwell#698 · Opened Sep 24, 2026
[Bug][UI] Replace the status of the button in the Sessions Tab Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

Netcracker/qubership-integration-platform#918 · Opened Sep 24, 2026
executions.Query — startDate and timeRange filters are sent with inverted comparison operators Open

area/plugin

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

kestra-io/plugin-kestra#190 · Opened Sep 24, 2026

More Java issues