Parquet TIME_MILLIS statistics are not converted to microseconds
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
Research direction
Start with ParquetConversions and find the tests covering Parquet statistics conversions. Check how min/max statistics for TIME_MILLIS are handled compared with the reader's conversion to microseconds. Done when those statistics are converted by multiplying by 1,000 and the relevant tests pass.
Written by the indexing model from the issue text.
Description
Apache Iceberg version
main (development)
Query engine
Other
Please describe the bug 🐞
Follow-up to #18173 and the review on #18321.
The Parquet reader converts TIME_MILLIS values from milliseconds to microseconds, but ParquetConversions leaves min/max statistics unchanged. Since Iceberg time values use microseconds, row-group filtering and file metrics can use statistics in the wrong unit.
The statistics conversion should multiply TIME_MILLIS values by 1,000 to match the reader.
Willingness to contribute
- I can contribute a fix for this bug independently
- Dominant language
- Java
- Stars
- 9.3k
- Forks
- 3.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 134
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg
-
Temporal transforms put pre-epoch timestamps at `.999999` into the previous unitPossibly taken @gitedmond claimed this 14 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg#18371 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
OpenAPI: Remove deprecated TrueExpression and FalseExpressionPossibly taken @zhang-arvin claimed this 10 days ago. Openimprovement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg#18351 · 3 comments ·
Maintainers usually reply within 1 day
-
Parquet: StringReader decodes every occurrence of a dictionary value into a new StringPossibly taken @Neuw84 claimed this 24 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg#18259 · 2 comments ·
Maintainers usually reply within 1 day
-
RCK tests fail to bind port 8181 when run alongside the kafka-connect integration stackPossibly taken @Kurtiscwright claimed this 26 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Core: SchemaParser rejects integral JSON numbers for floating-point defaultsPossibly taken @laserninja claimed this 34 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
Similar issues
-
bug follow-up lane:backlog
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Sanne/incus-spawn#1576 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
checkstyle/checkstyle#22069 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
it-at-m/eNoVa-Editor#79 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day