table.inspect metadata tables silently drop (or misattribute) partition values after a partition field rename

Open
  • Is something wrong?
#4,362 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Ankitkj1999 is already working on this.

Since Nov 13, 2026.

  • #4363 by @Ankitkj1999 — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
data

Research direction

Start with TableMetadata.specs_struct() and the inspect code that builds partition dicts from each file's historical PartitionSpec by field.name; key them by field_id instead. Run the reproduction script in the issue to see None values after the rename. Done when pre-rename partitions show {'p_renamed': 'a'} and {'p_renamed': 'b'} and the v1 remove/re-add case stays separate. Open PR #4363 already targets this, so check it before starting.

Written by the indexing model from the issue text.

Description

kind:bug
Apache Iceberg version

main (development)

Please describe the bug 🐞
Describe the bug

When a table evolves its partition spec by renaming a partition field:

  1. TableMetadata.specs_struct() adopts the latest field name for that partition field_id (e.g. 1000: p -> 1000: p_renamed).
  2. However, table.inspect.partitions(), files(), data_files(), all_files(), all_data_files(), and entries() build partition dictionaries using field.name from each file's historical PartitionSpec directly ({"p": "a"}).
  3. When PyArrow converts these rows to the schema defined by specs_struct() (which has field "p_renamed"), the "p" key is ignored and "p_renamed" is filled with None. All pre-rename partition values are silently replaced with None ({'p_renamed': None}).

Similarly, on format-version 1 tables, removing a partition field (remove_field("p")) and subsequently re-adding it (add_identity("p")) automatically renames the voided field (1000: p_1000) and creates 1001: p. Because inspect keys by historical field.name == "p", historical spec 0 data files written under field_id=1000 are silently misattributed to field_id=1001 ({'p_1000': None, 'p': 'a'}).

To Reproduce
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

with tempfile.TemporaryDirectory() as tmp:
    cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
    cat.create_namespace("ns")
    tbl = cat.create_table(
        "ns.t_rename",
        schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
        partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
        properties={"format-version": "2"},
    )
    tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
    tbl.update_spec().rename_field("p", "p_renamed").commit()
    tbl.append(pa.table({"id": [3], "p": ["c"]}))

    # Historical values 'a' and 'b' silently become None:
    print(tbl.inspect.partitions().to_pydict()["partition"])
    # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
    # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]

    print([r["partition"] for r in tbl.inspect.files().to_pylist()])
    # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
    # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]
Expected behavior

table.inspect tables should map partition field values using the canonical field names from specs_struct() based on field_id, preserving historical partition values across spec renames and correctly separating partition entries per specification layout.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Dominant language
Python
Stars
1.2k
Forks
664
Avg merge
2d 10h
Merged PRs (30d)
131

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-python

All issues in apache/iceberg-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.