Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

DataFusion: date_part over a Dictionary(Timestamp) returns a type that violates its declared Int32 (crashes GROUP BY/JOIN)

Abierto
  • ¿Algo no está bien?
#221 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
python
Área
backend, databases

Línea de trabajo

Start with the linked minimal reproduction, repros/datafusion/date_part_dictionary_panic.py, and the Rust test instructions in repros/datafusion/README.md; reproduce the schema mismatch and GROUP BY panic against DataFusion 54.0.0. The issue says this needs to be filed against apache/datafusion, so trace the temporal function's handling of dictionary inputs there. Done means the logical and physical result types agree and materialization, GROUP BY, and JOIN no longer panic.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Tracking issue for an upstream DataFusion bug (to file against apache/datafusion). The second of two blockers for dictionary-encoding coordinate columns (#217), and the one that breaks core climate SQL (diurnal climatology, anomaly).

Summary

date_part('hour', t) where t is Dictionary(Int32, Timestamp) preserves the dictionary encoding and returns a Dictionary(Int32, Int32), but the expression's declared logical type is plain Int32. The physical/logical mismatch surfaces as soon as the column is materialized:

Arrow error: Invalid argument error: column types must match schema types,
expected Int32 but found Dictionary(Int32, Int32) at column index 0

A GROUP BY / JOIN on the same expression is worse — it panics a worker thread:

thread 'tokio-rt-worker' panicked at arrow-array-58.3.0/src/cast.rs:849: primitive array

(a failed downcast of the dictionary array to a primitive array). GROUP BY date_part('hour', time) is exactly how you write a diurnal climatology, so this is not an edge case for the domain.

Minimal repro

Pure datafusion + pyarrow, no xarray-sql, no network: repros/datafusion/date_part_dictionary_panic.py (branch claude/datafusion-upstream-repros-fs1bqv; paste-in Rust test in the README).

# t is Dictionary(Int32, Timestamp)
ctx.sql("SELECT date_part('hour', t) AS h FROM x")             # -> expected Int32 but found Dictionary(Int32, Int32)
ctx.sql("SELECT date_part('hour', t), COUNT(*) FROM x GROUP BY 1")  # -> panics ("primitive array")

Expected

date_part (and temporal scalar functions generally) over a dictionary-encoded input should return a plain Int32 (decode the dictionary), or declare the Dictionary(Int32, Int32) it actually produces — logical and physical types must agree, and a downstream consumer must never panic on the result.

Environment

datafusion 54.0.0 / pyarrow 23.0.0 (arrow-rs 58.3.0).

Impact on xarray-sql

Blocks dictionary-encoding the time coordinate (#217). Because the time axis is where climate SQL applies date_part/date_trunc/extract most, encoding it breaks climatology and anomaly queries (benchmarks 02_climatology and 04_anomaly). This is the bug that made #217 unworkable without an upstream fix.

Lenguaje dominante
Python
Estrellas
142
Forks
19
Merge medio
1 d 9 h
PR fusionados (30 d)
9

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de xqlsystems/xarray-sql

Todos los issues de xqlsystems/xarray-sql

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.