delta-io/delta

[Feature Request][SPARK] Make Vacuum DRY RUN list all the files along with the the size of the files

Open

#3,247 opened on Jun 10, 2024

View on GitHub
 (4 comments) (0 reactions) (1 assignee)Scala (8,807 stars) (2,100 forks)batch import
enhancementgood first issue

Description

Feature request

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Overview

As of now, Vacuum dry run mode returns a dataframe containing a list of first 1000 files to be deleted. Instead, it's more useful to return all the eligible files to be deleted along with the size of these files.

Motivation

Today, users run Vacuum in dry run mode to understand the eligible files and the size of the data that is going to be deleted. By returning only a sample of 1000 files, users are not able to achieve their objective.

Further details

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

  • Yes. I can contribute this feature independently.
  • Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
  • No. I cannot contribute this feature at this time.

Contributor guide

Tech stack
scalasql
Domain
datadatabase
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Delta Lake basicsSpark SQLScala
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
The issue requests modifying Vacuum's dry run mode to return all eligible files with their sizes instead of the current 1000 file limit. Start by locating the Vacuum command implementation in the Delta Lake Spark source code (likely in `src/main/scala/org/apache/spark/sql/delta/commands/VacuumCommand.scala`). Identify where the dry run result is truncated (e.g., `take(1000)` or `limit(1000)`). Replace that with the full list and add file size information, possibly by accessing `FileStatus` for each file. Ensure backward compatibility and update any documentation. Look at existing tests for Vacuum to understand expected behavior.
[Feature Request][SPARK] Make Vacuum DRY RUN list all the files along with the the size of the files · delta-io/delta#3247 | Good First Issue