`SELECT *` fails after a column is added until the next write to the table

Open
  • Is something wrong?
#42 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
58/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust, sql
Domain
backend, databases

Research direction

Start by reproducing the query after the schema-only column addition, then read crates/datafusion/src/table/mod.rs at the provider declarations and iceberg-rust's TableScanBuilder::build, which binds scans to the snapshot schema. Done means the fresh provider can run SELECT * and return the existing value with NULL for the newly added column without a new table write.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

IcebergTableProvider and IcebergStaticTableProvider::try_new_from_table declare the table's current schema (table/mod.rs#L94-L97, table/mod.rs#L258-L262). iceberg-rust's scan binds the projection and filters to the schema of the snapshot it reads instead: TableScanBuilder::build uses snapshot.schema(...). After a schema change that has no snapshot of its own, the provider declares columns the scan can't find.

To Reproduce
  1. Create t(a INT NOT NULL) and insert one row.
  2. Add an optional column c STRING, for example with Transaction::update_schema().add_column(AddColumn::optional("c", ...)) or with ALTER TABLE ... ADD COLUMN from Spark. Don't write any new data.
  3. Build a fresh provider and run SELECT * FROM t:
DataInvalid => Column c not found in table. Schema: table {
  1: a: required int
}
Expected behavior

The query returns a = 1, c = NULL.

Additional context

The root cause is the iceberg-rust scan behavior tracked in apache/iceberg-rust#2565 and apache/iceberg-rust#2905. Until that's fixed upstream, this crate could declare the scanned snapshot's schema, or fill columns missing from it with nulls.

Dominant language
Rust
Stars
30
Forks
30
Avg merge
3d 19h
Merged PRs (30d)
15

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-iceberg

All issues in apache/datafusion-iceberg

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.