[FEA] Allow Hybrid Scan PQ reader to prune row groups with length-absent bloom filters
#23,515 opened on 2026/08/03
Repository metrics
- Stars
- (6,000 個のスター)
- PR merge metrics
- (平均マージ 17d 21h) (30d で 230 merged PRs)
説明
Is your feature request related to a problem?
parquet::fetch_bloom_filters_to_device treats any size-zero range as “no filter” and skips it. Consequently, these chunks are never bloom-pruned. This behavior is correct but suboptimal: pruning is silently lost because the device caster conservatively returns membership true for empty spans in bloom_filter_reader.cu.
The default read_parquet path already handles this case. read_bloom_filters substitutes a speculative 256-byte initial read, clamped to the datasource size, using speculative_read_size in:
Describe the solution you’d like
Handle this case in fetch_bloom_filters_to_device_impl in cpp/src/io/parquet/io_utils/parquet_io_utils.cpp
This function owns the datasources and can therefore clamp reads safely. The hybrid-scan metadata layer cannot do so because it has no access to the datasources.
Phase 1:
Distinguish between:
{offset > 0, 0}: A filter is present, but its length is unknown.{0, 0}: No filter is present.
An offset of zero is impossible for a real filter because Parquet files begin with PAR1.
For {offset > 0, 0}, use the following initial read size:
std::min(speculative_read_size, datasource.size() - offset)
Here, speculative_read_size is 256 bytes.
Phases 2–3:
No changes are needed:
- Header parsing already recovers the actual bitset size.
- The deferred read already fetches any data not covered by the speculative read.
Move the speculative_read_size constant next to the fetch helper so it can be shared.
Backward compatibility
- The default reader never sends
{offset > 0, 0}because it substitutes the speculative read size upstream. - Handling of
{0, 0}remains unchanged.
Addition context
This metadata is produced by older Spark/parquet-mr writers. One example is Apache’s data_index_bloom_encoding_stats.parquet.