Delta Kernel source: physicalName not propagated to Iceberg name mapping; renamed column reads NULL for files written before column mapping

Read DeltaKernelSchemaExtractor.toInternalSchema around lines 155–181 and compare its InternalField construction with DeltaSchemaExtractor; the issue identifies the missing storageName propagation. Then inspect IcebergSchemaExtractor and the name mapping update in IcebergConversionTarget. Add regression coverage in ITConversionController for enabling column mapping after the initial write and then renaming a column; done when the old rows resolve in the Iceberg target in both sync modes.

Search before asking

• I had searched in the issues and found no similar issues.

Please describe the bug 🐞

Search before asking

• I had searched in the issues and found no similar issues.

Please describe the bug 🐞

When delta.columnMapping.mode=name is enabled on an existing Delta table and a column is then renamed, the Iceberg target produced by the Delta Kernel conversion source (useDeltaKernel=true) cannot resolve the renamed column in the files written before column mapping was enabled. Delta reads all three values. From the Iceberg target, Spark 4.1.1 / Iceberg 1.11.0 and DuckDB 1.5.6 return NULL for the two old rows, and PyIceberg 0.12.0 fails:

ValueError: Could not find field with name: name

expected (Delta source):   (1, USA, John Smith)  (2, USA, Mary Johnson)  (6, Italy, Luca Verdi)
actual (Spark, DuckDB):    (1, USA, NULL)        (2, USA, NULL)          (6, Italy, Luca Verdi)
actual (PyIceberg):        read fails on the old physical column `name`

Reproduced on unmodified main 9d9f9b3 (0.5.0-SNAPSHOT) in both FULL and INCREMENTAL mode, and on 0.4.0-incubating. The renamed column keeps Iceberg field ID 3 across the incremental sync and the table UUID is unchanged, so this is not the field-ID change of #711; the generated name mapping simply lacks the old physical name. The sync reports SUCCESS.

Why this is not a duplicate of #711 or #765

#711 reports that an incremental sync turns a Delta rename into drop + add (field ID 2 → 5 there). I see that on 0.3.0-incubating (3 → 4, all names NULL), but on 0.4.0-incubating and main the ID is preserved (3 → 3) and the failure is a missing name-mapping alias, a different mechanism.

#765 / #766 fixed column-mapped Delta → Iceberg conversion by adding InternalField.storageName and writing the physical names into the Iceberg name mapping. That fix is in the Standalone DeltaSchemaExtractor and is not present in the Kernel source path. #911 covers nested field IDs under column mapping and notes that Iceberg applies a name mapping only to files that carry no field IDs, which is exactly the pre-mapping files in this fixture. #936 is the target-side counterpart (Kernel target and physicalName). This is the "fix that only exists on the Standalone path" case that #886 prerequisite 2 asks to verify before Kernel becomes the default.

Root cause

DeltaKernelSchemaExtractor.toInternalSchema reads delta.columnMapping.physicalName into a local storageName (L155-158) and uses it only to build nestedIdPath (L173); the InternalField.builder() call (L174-181) sets name, fieldId, parentPath, schema, defaultValue and never storageName:

https://github.com/apache/incubator-xtable/blob/9d9f9b3ced22adea03001b4b5e547b51fa7c8adc/xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelSchemaExtractor.java#L155-L181

The Standalone DeltaSchemaExtractor calls .storageName(storageName) at the same position (L191), added by #766:

https://github.com/apache/incubator-xtable/blob/9d9f9b3ced22adea03001b4b5e547b51fa7c8adc/xtable-core/src/main/java/org/apache/xtable/delta/DeltaSchemaExtractor.java#L188-L196

IcebergSchemaExtractor records an id → storage-name entry only when getStorageName() is non-null, and IcebergConversionTarget updates the Iceberg name mapping from that map. With the Kernel source the map stays empty, so the mapping for the renamed column is {"field-id":3,"names":["new_name"]}, while the two old Parquet files carry no field IDs and the physical column name (Delta keeps the existing name as physical name when mapping is enabled late). The file written after the rename carries field IDs and resolves by ID, which is why only the old rows are NULL.

Diagnostic, not a repair: appending name to field 3's names in an in-memory copy of the metadata makes PyIceberg return all three rows; no stored metadata was modified. Control: with column mapping enabled at table creation, all three names resolve on main and 0.4.0 (the files carry Parquet field IDs), but every partition value is NULL, which is #A.

Complete schema history and decoded name mapping of the main incremental target:

{"current-schema-id":2,
 "schemas":[{"type":"struct","schema-id":0,"fields":[]},
  {"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","type":"int"},{"id":2,"name":"country","type":"string"},{"id":3,"name":"name","type":"string"}]},
  {"type":"struct","schema-id":2,"fields":[{"id":1,"name":"id","type":"int"},{"id":2,"name":"country","type":"string"},{"id":3,"name":"new_name","type":"string"}]}],
 "schema.name-mapping.default":[{"field-id":1,"names":["id"]},{"field-id":2,"names":["country"]},{"field-id":3,"names":["new_name"]}]}

To Reproduce

Environment: as in #A (sync host Spark 3.5.5 / JDK 17, main 9d9f9b3 built with Java 11, Delta Kernel 4.0.0, delta-storage 3.3.0, Iceberg 1.9.2, Avro 1.12.0; source written with Delta 4.2.0 / Spark 4.1.1, deletion vectors disabled; local path-based tables).

CREATE TABLE delta_rename (id INT, country STRING, name STRING) USING DELTA
PARTITIONED BY (country) LOCATION '<path>'
TBLPROPERTIES ('delta.enableDeletionVectors'='false');

INSERT INTO delta_rename (id, name, country) VALUES (1, 'John Smith', 'USA'), (2, 'Mary Johnson', 'USA');
-- incremental case: sync here (returns FULL); the two-row target reads correctly in all three readers

ALTER TABLE delta_rename SET TBLPROPERTIES ('delta.columnMapping.mode'='name');
ALTER TABLE delta_rename RENAME COLUMN name TO new_name;
INSERT INTO delta_rename (id, new_name, country) VALUES (6, 'Luca Verdi', 'Italy');
-- sync: first-ever sync returns FULL; a second sync returns INCREMENTAL

targets = spark._jvm.java.util.ArrayList(); targets.add("ICEBERG")
spec = (spark._jvm.org.apache.xtable.spark.TableSyncSpec.builder()
        .key("delta_rename").basePath(path).sourceFormat("DELTA")
        .targets(targets).useDeltaKernel(True).build())
r = spark._jvm.org.apache.xtable.spark.XTableSyncService().sync(spec, spark.sparkContext._jsc.hadoopConfiguration())
print(r.get("ICEBERG").getMode(), r.get("ICEBERG").getTableFormatSyncStatus().getStatusCode())

Observed: SUCCESS in both modes; rows (1, USA, NULL), (2, USA, NULL), (6, Italy, Luca Verdi) in Spark and DuckDB; PyIceberg raises Could not find field with name: name; field ID 3 unchanged. Contrast: the pre-rename incremental baseline (before SET TBLPROPERTIES) is correct in all three readers; with column mapping set at creation the names are correct and the partitions are NULL (#A).

Expected behavior

The Kernel source sets storageName on struct fields so that IcebergConversionTarget adds the physical name to the name mapping, matching the Standalone source after #766; the three rows then read identically from Delta and Iceberg. A regression case in ITConversionController for "column mapping enabled after the initial write, then rename" on both sources would keep the two paths aligned (#903 is adding Kernel coverage there).

Affected versions

main 9d9f9b3 (0.5.0-SNAPSHOT): ID preserved, alias missing. 0.4.0-incubating spark-runtime: same. 0.3.0-incubating utilities: additionally changes the field ID (3 → 4) and loses all three names on incremental sync. Standalone source on main not run; by code reading it sets storageName.

Are you willing to submit PR?

• I am willing to submit a PR!
• I am willing to submit a PR but need help getting started!

Code of Conduct

• I agree to follow this project's Code of Conduct