Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Casting to a struct type with repeated field names fails with "ArrowArray struct has 2 children (expected 1)"

Abierto Apto para principiantes
  • ¿Algo no está bien?
#6,591 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@Samearth17 ya está trabajando en esto.

Desde el 5/10/2026.

  • #6671 de @Samearth17 — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
68/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust, scala
Área
backend

Línea de trabajo

Lee la rama struct de CometCast.isSupported y compárala con CometCreateNamedStruct, que ya rechaza nombres de campo repetidos; las ramas array y map recursan en él, así que una comprobación cubre los casos anidados en el SQL de repro. Reproduce con las tres consultas CAST fallidas más los dos controles que pasan, luego confirma que la corrección hace que el cast haga fallback para que los resultados coincidan con Spark. Terminado se ve como un nuevo caso que cubre nombres de campo struct repetidos en la suite de pruebas de cast y ninguna IllegalStateException de NativeUtil.importVector.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:expressions area:ffi bug good first issue priority:high
Describe the bug

A cast whose target struct type repeats a field name fails the task when Comet runs it natively. Spark allows the cast as long as each pair of fields casts, so CAST(s AS STRUCT<x: INT, x: INT>) over a STRUCT<p: INT, q: INT> column returns {1, 10}. Comet's native cast_struct_to_struct builds the result with the target fields, and the projection itself works: to_json over the cast matches Spark with {"x":1,"x":10}. Importing the result on the JVM then fails, because Java Arrow keys struct children by name and the two x children collapse into one:

java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1)
	at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:562)
	at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:92)
	at org.apache.arrow.c.ArrayImporter.importChild(ArrayImporter.java:83)
	at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:101)
	at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java:68)
	at org.apache.arrow.c.ArrowImporter.importVector(ArrowImporter.java:62)
	at org.apache.comet.vector.NativeUtil.importVector(NativeUtil.scala:264)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:211)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:236)

The same happens when the struct is nested in an array or a map value.

Steps to reproduce

Default configs, reproduced on main at fef94f6cd with Spark 4.1.3:

CREATE TABLE ts (
  s STRUCT<p: INT, q: INT>,
  arr ARRAY<STRUCT<p: INT, q: INT>>,
  m MAP<STRING, STRUCT<p: INT, q: INT>>) USING parquet;

INSERT INTO ts VALUES
  (named_struct('p', 1, 'q', 10), array(named_struct('p', 1, 'q', 10)),
   map('k', named_struct('p', 1, 'q', 10))),
  (named_struct('p', 2, 'q', NULL), array(), map()),
  (NULL, NULL, NULL);

SELECT CAST(s AS STRUCT<x: INT, x: INT>) FROM ts;               -- IllegalStateException
SELECT CAST(arr AS ARRAY<STRUCT<x: INT, x: INT>>) FROM ts;      -- IllegalStateException
SELECT CAST(m AS MAP<STRING, STRUCT<x: INT, x: INT>>) FROM ts;  -- IllegalStateException
SELECT CAST(s AS STRUCT<x: INT, y: INT>) FROM ts;               -- matches Spark
SELECT to_json(CAST(s AS STRUCT<x: INT, x: INT>)) FROM ts;      -- matches Spark

The plan is CometProject over CometNativeScan. Spark returns {1, 10}, {2, null} and null for the first query.

Expected behavior

Either match Spark or fall back. The smallest fix is probably for the struct branch of CometCast.isSupported to return Unsupported when the target struct repeats a field name, the same way CometCreateNamedStruct declines repeated names. The array and map branches already recurse into the struct branch, so one check there also covers the nested cases.

Additional context

This is the same Java Arrow limitation as #6251 (arrays_zip, fix in #6324), #5605 and #1015. Main declines structs with repeated field names wherever they would cross into Java Arrow: the shuffle and row conversion checks (#5866), the codegen dispatcher (#5766) and the cache serializer (#6004). Those checks guard the paths that bring a struct into Comet, though, and a native cast creates one. The broadcast exchange admits these types too (CometSink.convert calls supportedDataType with the default allowDuplicateStructFieldNames = true), so declining in the serde that produces the struct is what keeps it out.

Found while checking whether anything in #5603 was worth keeping.

Lenguaje dominante
Scala
Estrellas
1.3k
Forks
383
Merge medio
2 d 58 min
PR fusionados (30 d)
399

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.