Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Native from_json with a schema that repeats a field name fails with "ArrowArray struct has 2 children (expected 1)"

Open Beginner friendly
  • Is something wrong?
#6,592 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Samearth17 is already working on this.

Since Oct 5, 2026.

  • #6651 by @Samearth17 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
70/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
scala
Domain
backend

Research direction

Begin at CometJsonToStructs.isSupportedSchema, which validates field types but not names; the failure surfaces in NativeUtil.importVector because Java Arrow keys struct children by name. Add a recursive check that rejects structs with repeated field names at any depth so the query falls back to the dispatcher, which already declines them (#5766), and cover it in the JsonToStructs test suite with the SQL from the report. Done means from_json(j, 'a INT, a INT') returns Spark's {null, 1} instead of throwing IllegalStateException.

Written by the indexing model from the issue text.

Description

area:expressions area:ffi bug good first issue priority:medium
Describe the bug

With the native from_json path enabled (spark.comet.expression.JsonToStructs.allowIncompatible=true), a schema that repeats a field name fails the task once the result reaches the JVM. Spark accepts such a schema and fills the last field with that name, so from_json('{"a":1}', 'a INT, a INT') returns {null, 1}. Comet's native projection builds the same struct, and to_json over it matches Spark, but importing the result on the JVM fails because Java Arrow keys struct children by name:

java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1)
	at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:562)
	at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:92)
	at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java:68)
	at org.apache.arrow.c.ArrowImporter.importVector(ArrowImporter.java:62)
	at org.apache.comet.vector.NativeUtil.importVector(NativeUtil.scala:264)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:211)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:236)

By default from_json runs through the codegen dispatcher, which declines structs with repeated field names (#5766), so only the opt-in native path is affected.

Steps to reproduce

Reproduced on main at fef94f6cd with Spark 4.1.3:

SET spark.comet.expression.JsonToStructs.allowIncompatible=true;

CREATE TABLE tj (j STRING) USING parquet;
INSERT INTO tj VALUES ('{"a":1}'), ('{"a":2,"b":3}'), (NULL);

SELECT from_json(j, 'a INT, a INT') FROM tj;            -- IllegalStateException
SELECT to_json(from_json(j, 'a INT, a INT')) FROM tj;   -- matches Spark
SELECT from_json(j, 'a INT, b INT') FROM tj;            -- matches Spark

The plan is CometProject over CometNativeScan.

Expected behavior

Either match Spark or stay off the native path. CometJsonToStructs.isSupportedSchema checks the field types but not the names. If it rejected a struct with repeated field names at any depth, these schemas would go to the dispatcher, which already declines them, and the query would fall back to Spark.

Additional context

This is the same Java Arrow limitation as #6591 (struct casts) and #6251 (arrays_zip). Found while checking whether anything in #5603 was worth keeping.

Dominant language
Scala
Stars
1.3k
Forks
383
Avg merge
2d 58m
Merged PRs (30d)
399

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-comet

All issues in apache/datafusion-comet

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.