table.inspect metadata tables fail (or lose manifest bounds) after dropping an old partition column

Open
  • Is something wrong?
#4,355 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Ankitkj1999 is already working on this.

Since Nov 12, 2026.

  • #4356 by @Ankitkj1999 — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data

Research direction

Start with TableMetadata.specs_struct() and _partition_summaries_to_rows(), which call spec.partition_type(...) against the current schema, and with the InspectTable methods that use them. Run the reproduction in the issue to see the ValueError and the 'null' bounds. Done means the inspect partitions/files calls and manifest partition summary bounds work after a source column is dropped. Open PR #4356 already targets this, so check it before starting.

Written by the indexing model from the issue text.

Description

kind:bug
Apache Iceberg version

main (development)

Please describe the bug 🐞
Describe the bug

If a table stops partitioning by a column (update_spec().remove_field(...)) and later drops that source column (update_schema().delete_column(...)):

  1. Calling table.inspect.partitions(), entries(), files(), data_files(), delete_files(), all_files(), all_data_files(), or all_delete_files() raises ValueError: Could not find field with id: ... inside TableMetadata.specs_struct().
  2. Calling table.inspect.manifests() / all_manifests() succeeds, but renders lower_bound and upper_bound in partition_summaries as 'null' because _partition_summaries_to_rows() evaluates spec.partition_type(self.tbl.schema()), which falls back to UnknownType().

(Note: open PR #4115 fixes delete() manifest rewrites in pyiceberg/avro/file.py and pyiceberg/manifest.py, whereas this failure occurs in TableMetadata.specs_struct() and InspectTable even without calling delete().)

To Reproduce
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

with tempfile.TemporaryDirectory() as tmp:
    cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
    cat.create_namespace("ns")
    tbl = cat.create_table(
        "ns.t_v2",
        schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
        partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
        properties={"format-version": "2"},
    )
    tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
    with tbl.update_spec() as spec_update:
        spec_update.remove_field("p")
    with tbl.update_schema() as schema_update:
        schema_update.delete_column("p")

    print(tbl.inspect.manifests().to_pydict()["partition_summaries"])
    # [[{'contains_null': False, 'contains_nan': False, 'lower_bound': 'null', 'upper_bound': 'null'}]]

    tbl.inspect.partitions()
    # ValueError: Could not find field with id: 2
Expected behavior

TableMetadata.specs_struct() and table.inspect.* should resolve partition source field types from historical table schemas when the source column was dropped from the current schema, keeping metadata inspection tables and manifest partition summary bounds readable.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Dominant language
Python
Stars
1.2k
Forks
664
Avg merge
2d 10h
Merged PRs (30d)
131

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-python

All issues in apache/iceberg-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.