Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Native hashing of NullType and ANSI interval values fails with "Unsupported data type in hasher"

Abierto Apto para principiantes
  • ¿Algo no está bien?
#7,063 0 comentarios 0 reacciones 0 asignados Ver en GitHub

@ump45nose ya está trabajando en esto.

Desde el 17/10/2026.

  • #7084 de @ump45nose — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
62/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust, scala, spark
Área
data-engineering

Línea de trabajo

La rama que falla es el catch-all de create_hashes_internal! en native/spark-expr/src/hash_funcs/utils.rs, y el mismo kernel lo usa native/shuffle/src/partitioners/multi_partition.rs. Empieza reproduciendo la consulta hash(a, null) y leyendo cómo están escritos los brazos existentes para enteros primitivos int y long. Se considera terminado cuando los valores Null dejan la semilla sin cambios, Interval(YearMonth) y Duration(Microsecond) producen hashes como int y long, y las consultas del issue coinciden con Spark.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:expressions area:shuffle bug priority:high
Describe the bug

The native Murmur3 and xxHash64 hashers have no branch for Arrow Null, Interval(YearMonth) or Duration(Microsecond), the types Comet maps Spark's NullType, YearMonthIntervalType and DayTimeIntervalType to. A value of any of them falls through to the catch-all in create_hashes_internal! and fails the query:

org.apache.comet.CometNativeException: Unsupported data type in hasher: Null.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:260)

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/spark-expr/src/hash_funcs/utils.rs#L1272-L1278

Two Scala gates let these types reach it. hash.scala checks a hash expression's children with QueryPlanSerde.supportedDataType, which accepts NullType at any depth. So with the default configuration, hash(a, null), xxhash64(a, null) and hash(named_struct('x', a, 'n', null)) run natively and fail. For hash(a, null) Spark returns the same value as hash(a). The ANSI interval types are rejected there, so hashing them falls back to Spark.

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/spark/src/main/scala/org/apache/comet/serde/hash.scala#L136-L147

Native round-robin shuffle places rows by hashing every data column with the same kernel, and the native shuffle's type check accepts NullType and both ANSI interval types as data columns. So with spark.comet.shuffle.native.partitioning.roundrobin.enabled=true, a repartition(n) over a native plan with a column of any of the three types, at any depth, fails with Error inserting batch: Internal error: Unsupported data type in hasher: .... With the config off, which is the default, those shuffles run on JVM columnar shuffle or Spark shuffle and work.

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala#L662-L673

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/shuffle/src/partitioners/multi_partition.rs#L419-L434

Steps to reproduce
CREATE TABLE t (a INT) USING parquet;
INSERT INTO t SELECT CAST(id AS INT) FROM range(1000);

-- default configuration
SELECT a, hash(a, null) AS h FROM t;
SELECT a, xxhash64(a, null) AS h FROM t;
SELECT a, hash(named_struct('x', a, 'n', null)) AS h FROM t;

-- native round robin
SET spark.comet.shuffle.native.partitioning.roundrobin.enabled=true;
SELECT /*+ REPARTITION(4) */ a, null AS n FROM t;
SELECT /*+ REPARTITION(4) */ a, make_ym_interval(a % 5, 1) AS ym FROM t;
SELECT /*+ REPARTITION(4) */ a, make_dt_interval(a % 5, 1, 2, 3.5) AS dt FROM t;

Every query fails. I reproduced them on c3316fbe39 with the Spark 4.1 profile, and the code involved is unchanged on main at 3a4a159409.

Expected behavior

The queries should return the same results as Spark. Spark's HashExpression leaves the hash unchanged for NullType, hashes YearMonthIntervalType as an int and hashes DayTimeIntervalType as a long. The native side can match that exactly with a no-op arm for Null and the existing primitive int and long paths for Interval(YearMonth) and Duration(Microsecond), which fixes both entry points. With those arms in place, hash.scala could also accept the ANSI interval types, which #5061 lists as falling back today.

Additional context

#5059 was the same failure for Interval(MonthDayNano), fixed in #5135.

Lenguaje dominante
Scala
Estrellas
1.3k
Forks
391
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.