apache/hudi

Incremental Cleaning should not revert to brute force all-partition scanning in any cases

Open

#14569 opened on Nov 30, 2025

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (4,823 stars) (2,431 forks)batch import
component:cleaningfrom-jiragood first issuehelp-wantedpriority:hightype:feature

Description

After [https://github.com/apache/incubator-hudi/pull/1576] . Incremental Cleaning would still resort to full partition scan when  no previous clean operation was done in the dataset. This ticket is to design and implement a safe solution which would avoid full scanning in all cases.

JIRA info

Comments

09/Jul/20 13:10;309637554;i am happy to take this issue.;;;

Contributor guide

Tech stack
java
Domain
backenddatabaseperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Understanding of Hudi's cleaning mechanismJava development experience
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Review the linked PR #1576 to understand the current incremental cleaning behavior. Investigate the Hudi codebase to identify where full partition scans occur when no previous clean operation exists. Propose a design that safely avoids full scanning in all cases, possibly by maintaining metadata about cleaned partitions. The JIRA ticket HUDI 854 may contain additional requirements.