iceberg-catalog-glue pushes all historical schema fields to Glue, causing duplicate columns in Glue UI

Open Beginner friendly
#2,749 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
aws, rust
Domain
databases

Research direction

Start in crates/catalog/glue/src/schema.rs at from_iceberg and the visit_schema loop around lines 57-63. Verify how current and historical schemas become Glue TableInput.storage_descriptor.columns; done means Glue receives only the current schema's fields while historical schemas remain in Iceberg metadata.

Written by the indexing model from the issue text.

Description

bug
Apache Iceberg Rust version

None

Describe the bug

When using iceberg-catalog-glue, every call to update_table pushes ALL historical schema versions' fields to AWS Glue's column definition. This causes Glue Console to display duplicate columns — one entry per schema version for every field that existed in any past schema.

To Reproduce
  1. Create an Iceberg table via the Glue catalog
  2. Insert data with new fields, triggering schema evolution (AddSchema + SetCurrentSchema)
  3. Observe the table in AWS Glue Console → Table → Schema
Expected behavior

Only the current schema's fields should be pushed to Glue's column definition. Historical schema versions are already preserved in the Iceberg metadata file on S3.

https://github.com/apache/iceberg-rust/blob/main/crates/catalog/glue/src/schema.rs#L57-L63

for schema in metadata.schemas_iter() {
    if schema.schema_id() == current_schema.schema_id() {
        continue;
    }
    visit_schema(schema, &mut builder)?;
}

The from_iceberg function visits the current schema (marked as current=true), then iterates over all historical schemas and visits them too (marked as current=false). All of these are included in the Glue TableInput.storage_descriptor.columns. Glue's API does not filter by iceberg.field.current, so the UI displays all of them.

Willingness to contribute

None

Dominant language
Rust
Stars
1.4k
Forks
574
Avg merge
1d 18h
Merged PRs (30d)
84

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-rust

All issues in apache/iceberg-rust

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.