`IcebergWriteExec` and `CometIcebergWriteExec` are marked with `!` (missing input) in EXPLAIN

Abierto Apto para principiantes
  • ¿Algo no está bien?
#7,622 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@Visorgood ya está trabajando en esto.

Desde el 14/11/2026.

  • #7627 de @Visorgood — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
65/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
scala
Área
data-engineering

Línea de trabajo

Los dos nodos de escritura son IcebergWriteExec y CometIcebergWriteExec; el issue indica que ambos reciben output como argumento del constructor sin sobrescribir producedAttributes. Revisa CometProjectExec para ver cómo ya se hace esa sobrescritura. Luego reproduce el problema con los comandos EXPLAIN del issue y comprueba que missingInput esté vacío. Está terminado cuando desaparece el prefijo ! y una aserción sobre missingInput.isEmpty pasa en una prueba de escritura existente de CometIcebergWriteActionSuite.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug requires-triage
Describe the bug

In the split Iceberg write plan, both write nodes are printed with Spark's ! prefix, which marks a node whose input attributes are missing:

IcebergCommit IcebergBatchWrite(table=local.db.dst, format=PARQUET), ...
+- AdaptiveSparkPlan isFinalPlan=false
   +- !CometIcebergWrite [iceberg_commit_message#303], ...

Both nodes take output (the iceberg_commit_message attribute) as a constructor argument but don't override producedAttributes. Spark therefore counts that attribute in references, and since no child produces it, missingInput is non-empty. The write itself runs correctly; the plan only looks broken.

This is the same class of issue as #2214 (CometHashAggregate) and #2789 (CometWindowExec).

Steps to reproduce

Comet main at 4de670439, Spark 4.1.3, Iceberg 1.11.0, spark-shell with a Hadoop catalog named local and spark.comet.write.iceberg.enabled=true:

spark.sql("CREATE TABLE local.db.src (id INT, amount INT) USING iceberg")
spark.sql("INSERT INTO local.db.src VALUES (1, 10), (2, 20)")
spark.sql("CREATE TABLE local.db.dst (id INT, amount INT) USING iceberg")
spark.sql("CREATE TABLE local.db.dst_orc (id INT, amount INT) USING iceberg TBLPROPERTIES ('write.format.default'='orc')")

// native write
spark.sql("EXPLAIN INSERT INTO local.db.dst SELECT * FROM local.db.src")
// ORC is not written natively, so the write stays on iceberg-java
spark.sql("EXPLAIN INSERT INTO local.db.dst_orc SELECT * FROM local.db.src")

Native write:

== Physical Plan ==
IcebergCommit IcebergBatchWrite(table=local.db.dst, format=PARQUET), IcebergWrite(table=local.db.dst, format=PARQUET), org.apache.comet.iceberg.IcebergWriteStrategy$$Lambda/0x00000070021c8ba8@1b766d44
+- !CometIcebergWrite [iceberg_commit_message#303], /tmp/iceberg-warehouse/db/dst/data, ICEBERG_WRITER_UNPARTITIONED
   +- CometIcebergNativeScan [id#292, amount#293], /tmp/iceberg-warehouse/db/src/metadata/v2.metadata.json, IcebergScan(table=local.db.src, schemaId=0, snapshotId=7721505751242735037, branch=null, filters=, runtimeFilters=, groupedBy=), 2

Fallback to iceberg-java:

== Physical Plan ==
IcebergCommit IcebergBatchWrite(table=local.db.dst_orc, format=ORC), IcebergWrite(table=local.db.dst_orc, format=ORC), org.apache.comet.iceberg.IcebergWriteStrategy$$Lambda/0x00000070021c8ba8@4ba48bc7
+- !IcebergWrite IcebergBatchWrite(table=local.db.dst_orc, format=ORC), [iceberg_commit_message#322], PlainIcebergWrite
   +- *(1) CometColumnarToRow
      +- CometIcebergNativeScan [id#311, amount#312], /tmp/iceberg-warehouse/db/src/metadata/v2.metadata.json, IcebergScan(table=local.db.src, schemaId=0, snapshotId=7721505751242735037, branch=null, filters=, runtimeFilters=, groupedBy=), 2
Expected behavior

No ! on IcebergWrite and CometIcebergWrite: the commit-message attribute is produced by the node itself, so missingInput should be empty.

Additional context

Proposed fix: override def producedAttributes: AttributeSet = outputSet in IcebergWriteExec and CometIcebergWriteExec, as CometProjectExec already does, plus a test that asserts missingInput.isEmpty for both nodes in an existing CometIcebergWriteActionSuite write.

Willingness to contribute

I can contribute a fix for this bug independently

Lenguaje dominante
Scala
Estrellas
1.3k
Forks
418
Merge medio
2 d 23 h
PR fusionados (30 d)
158

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.