`add_column` changes the values of an existing column for files without field IDs

Ouverte
  • Quelque chose ne va pas ?
#4,318 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@Anurag-S1ngh y travaille déjà.

Depuis le 10/11/2026.

  • #4332 par @Anurag-S1ngh — ouverte

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
57/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
postgresql, python
Domaine
database

Piste de recherche

Start with _remove_reassigned_names in pyiceberg/table/name_mapping.py and trace how schema updates change name mappings for files without field IDs. Reproduce the report using the supplied add_files, rename, and add_column sequence, then check both scan results and IsNull("b") filtering. Done means the existing column keeps its values and the filter returns the matching rows after adding the reused name.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Apache Iceberg version

0.12.0 (latest release); the same logic is on main

Please describe the bug 🐞

Adding a column changes what an existing column reads. Files added with add_files have no field IDs and are read through the name mapping. After a rename, the renamed column reads correctly. Adding a column that reuses the old name then moves that name to the new field in the mapping (_remove_reassigned_names in pyiceberg/table/name_mapping.py). From then on, the renamed column reads as null for those files, and the new column gets its values.

The manifest statistics of those files still describe the original field, so planning also prunes with statistics that no longer match what is read: below, IsNull("b") skips the file although both rows read b as null.

import tempfile

import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import IsNull
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField

d = tempfile.mkdtemp()
catalog = SqlCatalog("c", uri=f"sqlite:///{d}/c.db", warehouse=f"file://{d}")
catalog.create_namespace("ns")
table = catalog.create_table("ns.t", Schema(NestedField(1, "a", LongType())))

# a Parquet file without field IDs, read through the name mapping
pq.write_table(pa.table({"a": [1, 3]}), f"{d}/imported.parquet")
table.add_files([f"file://{d}/imported.parquet"])

with table.update_schema() as update:
    update.rename_column("a", "b")
print(table.scan().to_arrow().to_pydict())
# {'b': [1, 3]}

with table.update_schema() as update:
    update.add_column("a", LongType())
print(table.scan().to_arrow().to_pydict())
# {'b': [None, None], 'a': [1, 3]}: expected {'b': [1, 3], 'a': [None, None]}

print(table.scan(row_filter=IsNull("b")).to_arrow().num_rows)
# 0: the file is pruned by the statistics of `b`, which hold its original values

Expected: adding a column leaves the values of existing columns unchanged, as it does for files with field IDs. The name was written for the original field in those files, so either the mapping keeps it there, or adding a column whose name files without field IDs still use is refused.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Langage dominant
Python
Étoiles
1.2k
Forks
654
Merge moyen
2 j 11 h
PR mergées (30 j)
122

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/iceberg-python

Toutes les issues de apache/iceberg-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.