rapidsai/cudf

[FEA] Allow Hybrid Scan PQ reader to prune row groups with length-absent bloom filters

オープン

#23,515 opened on 2026/08/03

 (2 件のコメント) (0 件のリアクション) (0 人の担当者)C++ (735 件のフォーク)batch import
0 - BacklogcuIOfeature requestgood first issue

Repository metrics

Stars
 (6,000 個のスター)
PR merge metrics
 (平均マージ 17d 21h) (30d で 230 merged PRs)

説明

Is your feature request related to a problem?

parquet::fetch_bloom_filters_to_device treats any size-zero range as “no filter” and skips it. Consequently, these chunks are never bloom-pruned. This behavior is correct but suboptimal: pruning is silently lost because the device caster conservatively returns membership true for empty spans in bloom_filter_reader.cu.

The default read_parquet path already handles this case. read_bloom_filters substitutes a speculative 256-byte initial read, clamped to the datasource size, using speculative_read_size in:

https://github.com/rapidsai/cudf/blob/3b3cea4d9669b9db1e35769ea727514eac3cb3e7/cpp/src/io/parquet/bloom_filter_reader.cu#L363-L366

Describe the solution you’d like

Handle this case in fetch_bloom_filters_to_device_impl in cpp/src/io/parquet/io_utils/parquet_io_utils.cpp

This function owns the datasources and can therefore clamp reads safely. The hybrid-scan metadata layer cannot do so because it has no access to the datasources.

Phase 1:

Distinguish between:

  • {offset > 0, 0}: A filter is present, but its length is unknown.
  • {0, 0}: No filter is present.

An offset of zero is impossible for a real filter because Parquet files begin with PAR1.

For {offset > 0, 0}, use the following initial read size:

std::min(speculative_read_size, datasource.size() - offset)

Here, speculative_read_size is 256 bytes.

Phases 2–3:

No changes are needed:

  • Header parsing already recovers the actual bitset size.
  • The deferred read already fetches any data not covered by the speculative read.

Move the speculative_read_size constant next to the fetch helper so it can be shared.

Backward compatibility

  • The default reader never sends {offset > 0, 0} because it substitutes the speculative read size upstream.
  • Handling of {0, 0} remains unchanged.

Addition context

This metadata is produced by older Spark/parquet-mr writers. One example is Apache’s data_index_bloom_encoding_stats.parquet.

コントリビューターガイド