delta-io/delta

[Kernel][Data skipping] Add the STARTS_WITH expression and support data skipping for it

Open

#2,539 opened on Jan 18, 2024

View on GitHub
 (8 comments) (0 reactions) (1 assignee)Scala (8,807 stars) (2,100 forks)batch import
enhancementgood first issuekernel

Description

Feature request

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Overview

Currently Kernel supports a limited set of expressions. We should 1) add the STARTS_WITH expression and 2) use file statistics to prune files based on the expression.

Motivation

Better file pruning.

Further details

This means we should

  1. add STARTS_WITH to the Kernel Predicate and support it in the kernel-defaults project
  2. Generate a data skipping filter according to the same rules we use in delta-spark

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

  • Yes. I can contribute this feature independently.
  • Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
  • No. I cannot contribute this feature at this time.

Contributor guide

Tech stack
java
Domain
databaseperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaDelta Kernel basicsData skipping concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The issue requires implementing the STARTS WITH expression in Kernel's Predicate.java (see https://github.com/delta io/delta/blob/master/kernel/kernel api/src/main/java/io/delta/kernel/expressions/Predicate.java) and supporting data skipping in kernel defaults. Refer to the data skipping filter logic in Delta Spark (DataSkippingReader.scala, line 542) for the rules to generate the filter. The implementation must integrate with existing file statistics pruning. Review the linked pull requests and comments for additional context.