delta-io/delta

[Feature Request][Spark] Add test coverage for clustering on generated columns

Open

#3,248 opened on Jun 10, 2024

View on GitHub
 (8 comments) (0 reactions) (1 assignee)Scala (8,807 stars) (2,100 forks)batch import
enhancementgood first issue

Description

Feature request

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Overview

Clustered tables support columns with stats collected on them, so generated column should work. We should add testing to cover this scenario.

Further details

We can add tests for generated columns in ClusteredTableDDLSuite.scala

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

  • Yes. I can contribute this feature independently.
  • Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
  • No. I cannot contribute this feature at this time.

Contributor guide

Tech stack
scala
Domain
testingdatabase
Issue type
test
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
ScalaApache SparkDelta Lake
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Review ClusteredTableDDLSuite.scala in the delta repository. Add test cases that verify generated columns work with clustering, including scenarios where generated columns are used as clustering keys. Refer to existing tests in the same suite for patterns. Ensure the tests cover both creation and querying of clustered tables with generated columns.