table.inspect metadata tables silently drop (or misattribute) partition values after a partition field rename

Abierto
  • ¿Algo no está bien?
#4,362 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@Ankitkj1999 ya está trabajando en esto.

Desde el 13/11/2026.

  • #4363 de @Ankitkj1999 — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

kind:bug
Apache Iceberg version

main (development)

Please describe the bug 🐞
Describe the bug

When a table evolves its partition spec by renaming a partition field:

  1. TableMetadata.specs_struct() adopts the latest field name for that partition field_id (e.g. 1000: p -> 1000: p_renamed).
  2. However, table.inspect.partitions(), files(), data_files(), all_files(), all_data_files(), and entries() build partition dictionaries using field.name from each file's historical PartitionSpec directly ({"p": "a"}).
  3. When PyArrow converts these rows to the schema defined by specs_struct() (which has field "p_renamed"), the "p" key is ignored and "p_renamed" is filled with None. All pre-rename partition values are silently replaced with None ({'p_renamed': None}).

Similarly, on format-version 1 tables, removing a partition field (remove_field("p")) and subsequently re-adding it (add_identity("p")) automatically renames the voided field (1000: p_1000) and creates 1001: p. Because inspect keys by historical field.name == "p", historical spec 0 data files written under field_id=1000 are silently misattributed to field_id=1001 ({'p_1000': None, 'p': 'a'}).

To Reproduce
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

with tempfile.TemporaryDirectory() as tmp:
    cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
    cat.create_namespace("ns")
    tbl = cat.create_table(
        "ns.t_rename",
        schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
        partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
        properties={"format-version": "2"},
    )
    tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
    tbl.update_spec().rename_field("p", "p_renamed").commit()
    tbl.append(pa.table({"id": [3], "p": ["c"]}))

    # Historical values 'a' and 'b' silently become None:
    print(tbl.inspect.partitions().to_pydict()["partition"])
    # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
    # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]

    print([r["partition"] for r in tbl.inspect.files().to_pylist()])
    # Current output: [{'p_renamed': 'c'}, {'p_renamed': None}, {'p_renamed': None}]
    # Expected output: [{'p_renamed': 'c'}, {'p_renamed': 'a'}, {'p_renamed': 'b'}]
Expected behavior

table.inspect tables should map partition field values using the canonical field names from specs_struct() based on field_id, preserving historical partition values across spec renames and correctly separating partition entries per specification layout.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Lenguaje dominante
Python
Estrellas
1.2k
Forks
654
Merge medio
2 d 11 h
PR fusionados (30 d)
122

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/iceberg-python

Todos los issues de apache/iceberg-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.