rapidsai/cudf

[FEA] Allow Hybrid Scan PQ reader to prune row groups with length-absent bloom filters

Ouverte

#23 515 ouverte le 3 août 2026

 (2 commentaires) (0 réaction) (0 personne assignée)C++ (735 forks)batch import
0 - BacklogcuIOfeature requestgood first issue

Métriques du dépôt

Stars
 (6 000 étoiles)
Métriques de merge PR
 (Merge moyen 17j 21h) (230 PRs mergées en 30 j)

Description

Is your feature request related to a problem?

parquet::fetch_bloom_filters_to_device treats any size-zero range as “no filter” and skips it. Consequently, these chunks are never bloom-pruned. This behavior is correct but suboptimal: pruning is silently lost because the device caster conservatively returns membership true for empty spans in bloom_filter_reader.cu.

The default read_parquet path already handles this case. read_bloom_filters substitutes a speculative 256-byte initial read, clamped to the datasource size, using speculative_read_size in:

https://github.com/rapidsai/cudf/blob/3b3cea4d9669b9db1e35769ea727514eac3cb3e7/cpp/src/io/parquet/bloom_filter_reader.cu#L363-L366

Describe the solution you’d like

Handle this case in fetch_bloom_filters_to_device_impl in cpp/src/io/parquet/io_utils/parquet_io_utils.cpp

This function owns the datasources and can therefore clamp reads safely. The hybrid-scan metadata layer cannot do so because it has no access to the datasources.

Phase 1:

Distinguish between:

  • {offset > 0, 0}: A filter is present, but its length is unknown.
  • {0, 0}: No filter is present.

An offset of zero is impossible for a real filter because Parquet files begin with PAR1.

For {offset > 0, 0}, use the following initial read size:

std::min(speculative_read_size, datasource.size() - offset)

Here, speculative_read_size is 256 bytes.

Phases 2–3:

No changes are needed:

  • Header parsing already recovers the actual bitset size.
  • The deferred read already fetches any data not covered by the speculative read.

Move the speculative_read_size constant next to the fetch helper so it can be shared.

Backward compatibility

  • The default reader never sends {offset > 0, 0} because it substitutes the speculative read size upstream.
  • Handling of {0, 0} remains unchanged.

Addition context

This metadata is produced by older Spark/parquet-mr writers. One example is Apache’s data_index_bloom_encoding_stats.parquet.

Guide contributeur