Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Casting to a struct type with repeated field names fails with "ArrowArray struct has 2 children (expected 1)"

Open Beginner friendly
  • Is something wrong?
#6,591 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Samearth17 is already working on this.

Since Oct 5, 2026.

  • #6671 by @Samearth17 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust, scala
Domain
backend

Research direction

Read the struct branch of CometCast.isSupported and compare it with CometCreateNamedStruct, which already declines repeated field names; the array and map branches recurse into it, so one check covers the nested cases in the repro SQL. Reproduce with the three failing CAST queries plus the two passing controls, then confirm the fix makes the cast fall back so the results match Spark. Done looks like a new case covering repeated struct field names in the cast test suite and no IllegalStateException from NativeUtil.importVector.

Written by the indexing model from the issue text.

Description

area:expressions area:ffi bug good first issue priority:high
Describe the bug

A cast whose target struct type repeats a field name fails the task when Comet runs it natively. Spark allows the cast as long as each pair of fields casts, so CAST(s AS STRUCT<x: INT, x: INT>) over a STRUCT<p: INT, q: INT> column returns {1, 10}. Comet's native cast_struct_to_struct builds the result with the target fields, and the projection itself works: to_json over the cast matches Spark with {"x":1,"x":10}. Importing the result on the JVM then fails, because Java Arrow keys struct children by name and the two x children collapse into one:

java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1)
	at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:562)
	at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:92)
	at org.apache.arrow.c.ArrayImporter.importChild(ArrayImporter.java:83)
	at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:101)
	at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java:68)
	at org.apache.arrow.c.ArrowImporter.importVector(ArrowImporter.java:62)
	at org.apache.comet.vector.NativeUtil.importVector(NativeUtil.scala:264)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:211)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:236)

The same happens when the struct is nested in an array or a map value.

Steps to reproduce

Default configs, reproduced on main at fef94f6cd with Spark 4.1.3:

CREATE TABLE ts (
  s STRUCT<p: INT, q: INT>,
  arr ARRAY<STRUCT<p: INT, q: INT>>,
  m MAP<STRING, STRUCT<p: INT, q: INT>>) USING parquet;

INSERT INTO ts VALUES
  (named_struct('p', 1, 'q', 10), array(named_struct('p', 1, 'q', 10)),
   map('k', named_struct('p', 1, 'q', 10))),
  (named_struct('p', 2, 'q', NULL), array(), map()),
  (NULL, NULL, NULL);

SELECT CAST(s AS STRUCT<x: INT, x: INT>) FROM ts;               -- IllegalStateException
SELECT CAST(arr AS ARRAY<STRUCT<x: INT, x: INT>>) FROM ts;      -- IllegalStateException
SELECT CAST(m AS MAP<STRING, STRUCT<x: INT, x: INT>>) FROM ts;  -- IllegalStateException
SELECT CAST(s AS STRUCT<x: INT, y: INT>) FROM ts;               -- matches Spark
SELECT to_json(CAST(s AS STRUCT<x: INT, x: INT>)) FROM ts;      -- matches Spark

The plan is CometProject over CometNativeScan. Spark returns {1, 10}, {2, null} and null for the first query.

Expected behavior

Either match Spark or fall back. The smallest fix is probably for the struct branch of CometCast.isSupported to return Unsupported when the target struct repeats a field name, the same way CometCreateNamedStruct declines repeated names. The array and map branches already recurse into the struct branch, so one check there also covers the nested cases.

Additional context

This is the same Java Arrow limitation as #6251 (arrays_zip, fix in #6324), #5605 and #1015. Main declines structs with repeated field names wherever they would cross into Java Arrow: the shuffle and row conversion checks (#5866), the codegen dispatcher (#5766) and the cache serializer (#6004). Those checks guard the paths that bring a struct into Comet, though, and a native cast creates one. The broadcast exchange admits these types too (CometSink.convert calls supportedDataType with the default allowDuplicateStructFieldNames = true), so declining in the serde that produces the struct is what keeps it out.

Found while checking whether anything in #5603 was worth keeping.

Dominant language
Scala
Stars
1.3k
Forks
383
Avg merge
2d 58m
Merged PRs (30d)
399

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-comet

All issues in apache/datafusion-comet

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.