table.inspect metadata tables fail (or lose manifest bounds) after dropping an old partition column

未关闭
  • 哪里不对？
#4,355 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@Ankitkj1999 已经在做这个了。

开始于 2026年11月12日。

  • #4356 来自 @Ankitkj1999 —— 未关闭

评估

难度
3/5
预计耗时
1-2 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python
领域
data

调研方向

Start with TableMetadata.specs_struct() and _partition_summaries_to_rows(), which call spec.partition_type(...) against the current schema, and with the InspectTable methods that use them. Run the reproduction in the issue to see the ValueError and the 'null' bounds. Done means the inspect partitions/files calls and manifest partition summary bounds work after a source column is dropped. Open PR #4356 already targets this, so check it before starting.

由索引模型根据 Issue 内容生成。

描述

kind:bug
Apache Iceberg version

main (development)

Please describe the bug 🐞
Describe the bug

If a table stops partitioning by a column (update_spec().remove_field(...)) and later drops that source column (update_schema().delete_column(...)):

  1. Calling table.inspect.partitions(), entries(), files(), data_files(), delete_files(), all_files(), all_data_files(), or all_delete_files() raises ValueError: Could not find field with id: ... inside TableMetadata.specs_struct().
  2. Calling table.inspect.manifests() / all_manifests() succeeds, but renders lower_bound and upper_bound in partition_summaries as 'null' because _partition_summaries_to_rows() evaluates spec.partition_type(self.tbl.schema()), which falls back to UnknownType().

(Note: open PR #4115 fixes delete() manifest rewrites in pyiceberg/avro/file.py and pyiceberg/manifest.py, whereas this failure occurs in TableMetadata.specs_struct() and InspectTable even without calling delete().)

To Reproduce
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

with tempfile.TemporaryDirectory() as tmp:
    cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
    cat.create_namespace("ns")
    tbl = cat.create_table(
        "ns.t_v2",
        schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
        partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
        properties={"format-version": "2"},
    )
    tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
    with tbl.update_spec() as spec_update:
        spec_update.remove_field("p")
    with tbl.update_schema() as schema_update:
        schema_update.delete_column("p")

    print(tbl.inspect.manifests().to_pydict()["partition_summaries"])
    # [[{'contains_null': False, 'contains_nan': False, 'lower_bound': 'null', 'upper_bound': 'null'}]]

    tbl.inspect.partitions()
    # ValueError: Could not find field with id: 2
Expected behavior

TableMetadata.specs_struct() and table.inspect.* should resolve partition source field types from historical table schemas when the source column was dropped from the current schema, keeping metadata inspection tables and manifest partition summary bounds readable.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
主要语言
Python
星标
1.2k
派生
654
平均合并
2 天 11 小时
30 天内合并 PR
122

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/iceberg-python 的其他 Issue

查看 apache/iceberg-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。