UpgradeFormatVersion fails when last-column-id is lower than an existing field ID

Open Beginner friendly
  • Is something wrong?
#2,578 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@Shruthirdy is already working on this.

Since Nov 12, 2026.

  • #2579 by @Shruthirdy — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
databases

Research direction

Start by reading the format-version upgrade path through MetadataBuilderFromBase, SetFormatVersion, and validateLastColumnID, then compare how #2453 applies columnIDFloor in NewUpdateSchema and AddSchema. The repro uses ExampleTableMetadataV2 with last-column-id 2 while field z has ID 3. Done when upgrading to version 3 succeeds and raises last-column-id to 3, with a regression test for that metadata.

Written by the indexing model from the issue text.

Description

Apache Iceberg version

main (development)

Please describe the bug 🐞

If a table's last-column-id is lower than a field ID already used in its schemas, UpgradeFormatVersion fails, while other updates on the same table work.

#2453 fixed this for schema updates: NewUpdateSchema and AddSchema now take the max of last-column-id and every schema's highest field ID (columnIDFloor). MetadataBuilderFromBase still copies last-column-id from metadata as is, and SetFormatVersion checks that value in validateLastColumnID, so the upgrade is rejected.

Repro

Same metadata as #2452: ExampleTableMetadataV2 with last-column-id changed from 3 to 2 (field z has ID 3).

data := strings.Replace(table.ExampleTableMetadataV2,
	`"last-column-id": 3`, `"last-column-id": 2`, 1)
meta, _ := table.ParseMetadataString(data)

tbl := table.New([]string{"db", "t"}, meta, "", nil, nil)
txn := tbl.NewTransaction()
err := txn.UpgradeFormatVersion(3)
fmt.Println(err)

Output:

invalid metadata: last-column-id 2 is less than the highest field ID 3 in schemas

Expected:

The upgrade succeeds and last-column-id is raised to 3, as schema updates already do since #2453.

I'd like to work on this issue.

Dominant language
Go
Stars
477
Forks
262
Avg merge
1d 23h
Merged PRs (30d)
303

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-go

All issues in apache/iceberg-go

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.