Parquet: notNaN filter crashes on columns absent from older files

Start with ParquetDictionaryRowGroupFilter.java, especially EvalVisitor#notNaN and the logic around line 178, then review TestDictionaryRowGroupFilter#testColumnNotInFile. Run the targeted Gradle test command from the issue and confirm that notNaN on an absent column retains the row group without an exception, while existing-column behavior remains covered.

Apache Iceberg version

1.11.0 (latest release)

Query engine

Other

Please describe the bug 🐞

Summary

A Parquet read using a notNaN filter crashes when the referenced float or double column exists in the projected Iceberg schema but is absent from an older Parquet file.

I reproduced this through Iceberg's generic Java Parquet reader rather than a specific query engine. Spark and Flink can also produce this predicate when translating comparisons against NaN.

This is a normal schema-evolution scenario:

1. Write a Parquet file with schema { required long id }.
2. Add an optional float column such as added_float.
3. Read the old file using the evolved schema and filter with notNaN("added_float").

The added column should read as null for old rows. Iceberg defines null as not NaN, so the row group must remain readable. Instead, reader initialization throws a NullPointerException.

I reproduced this on main at commit:

1ec15051dc09ef6a0d61fa332eb8f2503f3170f8

The same faulty code is present in the apache-iceberg-1.11.0 tag.

Minimal reproduction

In TestDictionaryRowGroupFilter#testColumnNotInFile, add notNaN("not_in_file") to the existing exprs array:

Expression[] exprs =
    new Expression[] {
      lessThan("not_in_file", 1.0f),
      lessThanOrEqual("not_in_file", 1.0f),
      equal("not_in_file", 1.0f),
      greaterThan("not_in_file", 1.0f),
      greaterThanOrEqual("not_in_file", 1.0f),
      notNull("not_in_file"),
      isNull("not_in_file"),
      notEqual("not_in_file", 1.0f),
      notNaN("not_in_file")
    };

Then run:

./gradlew --no-build-cache \
  :iceberg-parquet:test \
  --tests org.apache.iceberg.parquet.TestDictionaryRowGroupFilter \
  --no-daemon

I also reproduced the failure twice through the production Parquet.read(...) path by writing an old-schema file and reading it with the evolved schema.

Expected behavior

The reader should retain the row group. Because the column is absent from the file, its values are null, and null values satisfy Iceberg's notNaN expression semantics.

Actual behavior

The read fails with:

Cannot invoke "java.lang.Boolean.booleanValue()" because
the return value of "java.util.Map.get(Object)" is null

The exception originates from ParquetDictionaryRowGroupFilter.java:178 and is reached through ReadConf.java:109.

Root cause

ParquetDictionaryRowGroupFilter.EvalVisitor#notNaN evaluates:

if (mayContainNulls.get(id)) {
  return ROWS_MIGHT_MATCH;
}

before checking whether the field ID exists in the file metadata:

Boolean hasNonDictPage = isFallback.get(id);
if (hasNonDictPage == null || hasNonDictPage) {
  return ROWS_MIGHT_MATCH;
}

For a column absent from the file, neither map has an entry for the field ID. The first lookup returns null and is unboxed as a boolean, causing the exception.

Moving the existing absent/fallback-column guard before the nullability lookup should preserve behavior for present columns while conservatively retaining row groups for absent columns.

Related history

• #6431 added the nullability check for physically present columns containing nulls, but did not cover columns absent from the file.
• #16692 addresses an initial-default problem in ParquetMetricsRowGroupFilter and does not change this dictionary-filter path.

I searched existing open and closed issues and pull requests using combinations of notNaN, ParquetDictionaryRowGroupFilter, missing column, column not in file, and schema evolution, and did not find an active fix for this crash.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time