apache/hudi

Limit the amount of partitions considered for GlobalBloomIndex

Closed

#14,484 opened on Nov 30, 2025

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Java (2,431 forks)batch import
area:indexfrom-jiragood first issuepriority:hightype:improvement

Repository metrics

Stars
 (4,823 stars)
PR merge metrics
 (Avg merge 16d 21h) (127 merged PRs in 30d)

Description

Currently, global bloom index will check inputs against files in all partitions.. In lot of cases, the user may know a range of partitions actually impacted from updates clearly (e.g upstream system drops updates older than a year, ... ).. In such a scenario,it may make sense to support an option for Global bloom to control how many partitions you want to match against, to gain performance.

JIRA info

Contributor guide

Research direction
Explore how to add a configuration parameter to limit the number of partitions considered in GlobalBloomIndex. Look at existing partition filtering mechanisms in Hudi.
Tech stack
java
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Apache HudiBloom filters
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40