Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Native hashing of NullType and ANSI interval values fails with "Unsupported data type in hasher"

Aperta Adatta ai principianti
  • Qualcosa non va?
#7,063 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@ump45nose ci sta già lavorando.

Dal 17/10/2026.

  • #7084 di @ump45nose — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
62/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, scala, spark
Ambito
data-engineering

Direzione di ricerca

Il ramo che fallisce è il catch-all in create_hashes_internal! in native/spark-expr/src/hash_funcs/utils.rs, e lo stesso kernel è usato anche da native/shuffle/src/partitioners/multi_partition.rs. Inizia riproducendo la query hash(a, null) e leggendo come sono scritti i rami esistenti per i primitivi int e long. Il lavoro è completo quando i valori Null lasciano invariato il seed, Interval(YearMonth) e Duration(Microsecond) producono hash come int e long, e le query della issue corrispondono a Spark.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:expressions area:shuffle bug priority:high requires-triage
Describe the bug

The native Murmur3 and xxHash64 hashers have no branch for Arrow Null, Interval(YearMonth) or Duration(Microsecond), the types Comet maps Spark's NullType, YearMonthIntervalType and DayTimeIntervalType to. A value of any of them falls through to the catch-all in create_hashes_internal! and fails the query:

org.apache.comet.CometNativeException: Unsupported data type in hasher: Null.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:260)

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/spark-expr/src/hash_funcs/utils.rs#L1272-L1278

Two Scala gates let these types reach it. hash.scala checks a hash expression's children with QueryPlanSerde.supportedDataType, which accepts NullType at any depth. So with the default configuration, hash(a, null), xxhash64(a, null) and hash(named_struct('x', a, 'n', null)) run natively and fail. For hash(a, null) Spark returns the same value as hash(a). The ANSI interval types are rejected there, so hashing them falls back to Spark.

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/spark/src/main/scala/org/apache/comet/serde/hash.scala#L136-L147

Native round-robin shuffle places rows by hashing every data column with the same kernel, and the native shuffle's type check accepts NullType and both ANSI interval types as data columns. So with spark.comet.shuffle.native.partitioning.roundrobin.enabled=true, a repartition(n) over a native plan with a column of any of the three types, at any depth, fails with Error inserting batch: Internal error: Unsupported data type in hasher: .... With the config off, which is the default, those shuffles run on JVM columnar shuffle or Spark shuffle and work.

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala#L662-L673

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/shuffle/src/partitioners/multi_partition.rs#L419-L434

Steps to reproduce
CREATE TABLE t (a INT) USING parquet;
INSERT INTO t SELECT CAST(id AS INT) FROM range(1000);

-- default configuration
SELECT a, hash(a, null) AS h FROM t;
SELECT a, xxhash64(a, null) AS h FROM t;
SELECT a, hash(named_struct('x', a, 'n', null)) AS h FROM t;

-- native round robin
SET spark.comet.shuffle.native.partitioning.roundrobin.enabled=true;
SELECT /*+ REPARTITION(4) */ a, null AS n FROM t;
SELECT /*+ REPARTITION(4) */ a, make_ym_interval(a % 5, 1) AS ym FROM t;
SELECT /*+ REPARTITION(4) */ a, make_dt_interval(a % 5, 1, 2, 3.5) AS dt FROM t;

Every query fails. I reproduced them on c3316fbe39 with the Spark 4.1 profile, and the code involved is unchanged on main at 3a4a159409.

Expected behavior

The queries should return the same results as Spark. Spark's HashExpression leaves the hash unchanged for NullType, hashes YearMonthIntervalType as an int and hashes DayTimeIntervalType as a long. The native side can match that exactly with a no-op arm for Null and the existing primitive int and long paths for Interval(YearMonth) and Duration(Microsecond), which fixes both entry points. With those arms in place, hash.scala could also accept the ANSI interval types, which #5061 lists as falling back today.

Additional context

#5059 was the same failure for Interval(MonthDayNano), fixed in #5135.

Lingua principale
Scala
Stelle
1.3k
Fork
391
Merge medio
2g 12h
PR unite (30g)
432

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.