delta-io/delta

[Kernel] Validate readSchema is a subset of the table schema in ScanBuilder

Open

#2,149 opened on Oct 6, 2023

View on GitHub
 (5 comments) (0 reactions) (1 assignee)Scala (8,807 stars) (2,100 forks)batch import
enhancementgood first issuekernel

Description

Feature request

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Overview

Validate that the read schema provided in ScanBuilder is a subset of the snapshot schema. https://github.com/delta-io/delta/blob/master/kernel/kernel-api/src/main/java/io/delta/kernel/internal/ScanBuilderImpl.java#L80

Motivation

Better validation / error messages.

Contributor guide

Tech stack
java
Domain
backenddatabase
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaDelta Kernel basicsScanBuilderImpl.java
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue is to add validation in `ScanBuilderImpl.java` (line 80) to ensure the provided read schema is a subset of the snapshot table schema. Currently, an invalid read schema may cause obscure errors later. The fix involves checking each column in the read schema against the table schema and throwing an appropriate error message if the read schema contains columns not present in the table schema. The issue has an assignee, so check if that assignee has a branch or ongoing work before starting.