[Feature Request] Support additional generation expressions for automatic data skipping · delta-io/delta#1442

(5 comments) (0 reactions) (0 assignees)Scala (2,100 forks)batch import

enhancementgood first issue

Repository metrics

Stars: (8,807 stars)
PR merge metrics: (Avg merge 7d 1h) (142 merged PRs in 30d)

Description

Overview

For partition columns that are generated columns, we are able to automatically generate partition filters when we see a data filter on its generating columns. Right now we automatically generate these for a small subset of possible generation expressions (defined here.)

Details

We can add support for additional expressions. Here are the supported generation expressions in Delta.

A few specific expressions that would make sense to add include:

Ceil: https://github.com/delta-io/delta/issues/1443
Floor: https://github.com/delta-io/delta/issues/1444
TruncDate: https://github.com/delta-io/delta/issues/1446
TruncTimestamp: https://github.com/delta-io/delta/issues/1445

These are just a few, feel free to comment or create a new sub-issue for any additional expressions you think would be beneficial.

What to update?

Contributor guide

Research direction: Study the existing generation expressions in optimizablePartitionExpressions.scala and the handling in GeneratedColumn.scala. For each new expression (Ceil, Floor, TruncDate, TruncTimestamp), find the corresponding Spark SQL expression class, add it to the list of supported expressions, and update the test suite in OptimizeGeneratedColumnSuite.scala.
Tech stack: scalaspark
Domain: backend
Issue type: Feature
Difficulty: 3
Estimated time: 1-3 hours
Activity status: Fresh
Clarity: Clear
Prerequisites: ScalaApache Spark
Newbie friendliness: 75