delta-io/delta

[Kernel][Data skipping] Add the STARTS_WITH expression and support data skipping for it

Open

#2,539 建立於 2024年1月18日

在 GitHub 查看
 (8 留言) (0 反應) (1 負責人)Scala (8,807 star) (2,100 fork)batch import
enhancementgood first issuekernel

描述

Feature request

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Overview

Currently Kernel supports a limited set of expressions. We should 1) add the STARTS_WITH expression and 2) use file statistics to prune files based on the expression.

Motivation

Better file pruning.

Further details

This means we should

  1. add STARTS_WITH to the Kernel Predicate and support it in the kernel-defaults project
  2. Generate a data skipping filter according to the same rules we use in delta-spark

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

  • Yes. I can contribute this feature independently.
  • Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
  • No. I cannot contribute this feature at this time.

貢獻者指南

技術棧
java
領域
databaseperformance
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
4
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
3-5 days
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
blocked
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
JavaDelta Kernel basicsData skipping concepts
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
20
研究方向
The issue requires implementing the STARTS WITH expression in Kernel's Predicate.java (see https://github.com/delta io/delta/blob/master/kernel/kernel api/src/main/java/io/delta/kernel/expressions/Predicate.java) and supporting data skipping in kernel defaults. Refer to the data skipping filter logic in Delta Spark (DataSkippingReader.scala, line 542) for the rules to generate the filter. The implementation must integrate with existing file statistics pruning. Review the linked pull requests and comments for additional context.