Core: Discard changes when suppressing historical snapshots in CatalogHandlers
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start in core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java at loadTable and its snapshot-loading-mode=refs branch; compare the metadata rebuild with RESTSessionCatalog.loadTable. Reproduce the failure through RESTCatalogAdapter using a historical snapshot with a statistics file, then verify that GET .../tables/{table}?snapshots=refs succeeds without the metadata-location exception.
Written by the indexing model from the issue text.
Description
Apache Iceberg version
1.11.0 (latest release)
Query engine
None
Please describe the bug 🐞
Behavior
In snapshot-loading-mode=refs, CatalogHandlers.loadTable fails with IllegalArgumentException: Cannot set metadata location with changes to table metadata: 1 changes when serving GET .../tables/{table}?snapshots=refs for a table that has a statistics file (or partition statistics file) attached to a historical (unreferenced) snapshot.
Cause
The REFS branch builds the response metadata like this (CatalogHandlers.java#L526-L531):
metadata =
TableMetadata.buildFrom(loadedMetadata)
.withMetadataLocation(loadedMetadata.metadataFileLocation())
.suppressHistoricalSnapshots()
.build();
suppressHistoricalSnapshots() does not record RemoveSnapshots changes, but it removes the suppressed snapshots' statistics via removeStatistics(...) / removePartitionStatistics(...), which do record MetadataUpdate.RemoveStatistics / RemovePartitionStatistics changes (TableMetadata.java, rewriteSnapshotsInternal). build() then rejects the combination of pending changes and a set metadata location.
The client-side equivalent in RESTSessionCatalog.loadTable already prevents this with by calling .discardChanges() when rebuilding metadata; CatalogHandlers performs the change-recording suppression without that measure.
To Reproduce
E.g. via RESTCatalogAdapter with snapshot-loading-mode=refs on the client:
- Create a table, commit snapshot A, attach a
StatisticsFileto A (e.g.UpdateStatistics). - Commit snapshot B so A becomes historical (only reachable via parent chain, not via refs).
loadTablewithsnapshots=refs→ 500 /IllegalArgumentException: Cannot set metadata location with changes to table metadata: 1 changes.
Encountered in practice testing Trino's REST catalog against the in-memory test server with snapshot-loading-mode=refs: Trino writes statistics on INSERT by default, so the first loadTable after a stats-bearing snapshot becomes historical reliably fails.
Suggested Fix
Add .discardChanges() to the builder chain in CatalogHandlers.loadTable (matching the client-side wrapper), e.g.
TableMetadata.buildFrom(loadedMetadata)
.withMetadataLocation(loadedMetadata.metadataFileLocation())
.suppressHistoricalSnapshots()
.discardChanges()
.build();
This would be the minimal fix, keeping the general behavior as is. Alternatively, the suppress-parameter that already prevents MetadataUpdate.RemoveSnapshots changes from being created here could be incorporated into removeStatistics() and removePartitionStatistics() to prevent change creation there at the root.
AI assistance
Found this while working on a trino patch. This root-cause analysis was AI-assisted. I'm quite confident this is an actual bug and checked this in detail before opening the issue. I'll link to the related PR at trino shortly, whose test triggers this behavior.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
- Dominant language
- Java
- Stars
- 9.3k
- Forks
- 3.6k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 156
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
improvement
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
apache/rocketmq-dashboard#5358 ·
Maintainers usually reply within 3 days
-
area:cpan-port area:database bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
fglock/PerlOnJava#1605 ·
Maintainers usually reply within 1 day
-
1.0.0-rc2
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
wso2/dpdp-accelerator#377 ·
Maintainers usually reply within 1 day
-
area/dependencies backport/26.4 kind/cve severity/high source/scan-dependencies status/triage
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day