null_count is omitted for large columns in parquet files

Open Beginner friendly
#3,574 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java
Domain
data-engineering

Research direction

Start in parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java at the guard around lines 800-807. Check how the size limit currently suppresses statistics, then ensure null_count is retained even when min/max are omitted; done means large-column Parquet files include null_count metadata and avoid the reported downstream error.

Written by the indexing model from the issue text.

Description

Type: bug
Describe the bug, including details regarding any error messages, version, and platform.

Currently in ParquetMetadataConverter.java, there is a guard that prevents the writing of statistics such as min/max AND null_count when the stats are larger than the max allowed size under truncation. The rationale for this makes sense for omitting min/max, however null_count can be written on the file despite the size of its content. See the code below:

https://github.com/apache/parquet-java/blob/7be05b4702df78ae0c0c6b44adc6b7b7af2d931f/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L800-L807

The missing null_count metadata sometimes causes downstream consumers of the parquet files to error. For example in Snowflake we are seeing the following kind of error:

non-nullable column without default has null values according to file statistics
Component(s)

Core

Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 23h
Merged PRs (30d)
30

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/parquet-java

All issues in apache/parquet-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.