table.inspect metadata tables silently drop (or misattribute) partition values after a partition field rename

未關閉
  • 哪裡不對？
#4,362 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

@Ankitkj1999 已經在處理了。

開始於 2026年11月13日。

  • #4363 來自 @Ankitkj1999 —— 未關閉

評估

這個 Issue 還沒有評估資料。

描述

kind:bug
Apache Iceberg version

main (development)

Please describe the bug 🐞
Describe the bug

When a table evolves its partition spec by renaming a partition field:

  1. TableMetadata.specs_struct() adopts the latest field name for that partition field_id (e.g. 1000: p -> 1000: p_renamed).
  2. However, table.inspect.partitions(), files(), data_files(), all_files(), all_data_files(), and entries() build partition dictionaries using field.name from each file's historical PartitionSpec directly ({"p": "a"}).
  3. When PyArrow converts these rows to the schema defined by specs_struct() (which has field "p_renamed"), the "p" key is ignored and "p_renamed" is filled with None. All pre-rename partition values are silently replaced with None ({'p_renamed': None}).

Similarly, on format-version 1 tables, removing a partition field (remove_field("p")) and subsequently re-adding it (add_identity("p")) automatically renames the voided field (1000: p_1000) and creates 1001: p. Because inspect keys by historical field.name == "p", historical spec 0 data files written under field_id=1000 are silently misattributed to field_id=1001 ({'p_1000': None, 'p': 'a'}).

To Reproduce
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

with tempfile.TemporaryDirectory() as tmp:
    cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
    cat.create_namespace("ns")
    tbl = cat.create_table(
        "ns.t_rename",
        schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
        partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
        properties={"format-version": "2"},
    )
    tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
    tbl.update_spec().rename_field("p", "p_renamed").commit()
    tbl.append(pa.table({"id": [3], "p": ["c"]}))

    # Historical values 'a' and 'b' silently become None:
    print(tbl.inspect.partitions().to_pydict()["partition"])
    # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
    # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]

    print([r["partition"] for r in tbl.inspect.files().to_pylist()])
    # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
    # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]
Expected behavior

table.inspect tables should map partition field values using the canonical field names from specs_struct() based on field_id, preserving historical partition values across spec renames and correctly separating partition entries per specification layout.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
主要語言
Python
星號
1.2k
分支
654
平均合併
2 天 11 小時
30 天內合併 PR
122

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 有 Pull Request 範本
  • 沒有貢獻指南

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

apache/iceberg-python 的其他 Issue

查看 apache/iceberg-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。