table.inspect metadata tables fail (or lose manifest bounds) after dropping an old partition column

Offen
  • Stimmt etwas nicht?
#4,355 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@Ankitkj1999 arbeitet bereits daran.

Seit 12.11.2026.

  • #4356 von @Ankitkj1999 — offen

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
data

Rechercherichtung

Start with TableMetadata.specs_struct() and _partition_summaries_to_rows(), which call spec.partition_type(...) against the current schema, and with the InspectTable methods that use them. Run the reproduction in the issue to see the ValueError and the 'null' bounds. Done means the inspect partitions/files calls and manifest partition summary bounds work after a source column is dropped. Open PR #4356 already targets this, so check it before starting.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

kind:bug
Apache Iceberg version

main (development)

Please describe the bug 🐞
Describe the bug

If a table stops partitioning by a column (update_spec().remove_field(...)) and later drops that source column (update_schema().delete_column(...)):

  1. Calling table.inspect.partitions(), entries(), files(), data_files(), delete_files(), all_files(), all_data_files(), or all_delete_files() raises ValueError: Could not find field with id: ... inside TableMetadata.specs_struct().
  2. Calling table.inspect.manifests() / all_manifests() succeeds, but renders lower_bound and upper_bound in partition_summaries as 'null' because _partition_summaries_to_rows() evaluates spec.partition_type(self.tbl.schema()), which falls back to UnknownType().

(Note: open PR #4115 fixes delete() manifest rewrites in pyiceberg/avro/file.py and pyiceberg/manifest.py, whereas this failure occurs in TableMetadata.specs_struct() and InspectTable even without calling delete().)

To Reproduce
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

with tempfile.TemporaryDirectory() as tmp:
    cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
    cat.create_namespace("ns")
    tbl = cat.create_table(
        "ns.t_v2",
        schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
        partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
        properties={"format-version": "2"},
    )
    tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
    with tbl.update_spec() as spec_update:
        spec_update.remove_field("p")
    with tbl.update_schema() as schema_update:
        schema_update.delete_column("p")

    print(tbl.inspect.manifests().to_pydict()["partition_summaries"])
    # [[{'contains_null': False, 'contains_nan': False, 'lower_bound': 'null', 'upper_bound': 'null'}]]

    tbl.inspect.partitions()
    # ValueError: Could not find field with id: 2
Expected behavior

TableMetadata.specs_struct() and table.inspect.* should resolve partition source field types from historical table schemas when the source column was dropped from the current schema, keeping metadata inspection tables and manifest partition summary bounds readable.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Vorherrschende Sprache
Python
Sterne
1.2k
Forks
654
Ø Merge
2 T. 11 Std.
Gemergte PRs (30 T.)
122

Entwicklungsumgebung

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/iceberg-python

Alle Issues in apache/iceberg-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.