DataFusion: date_part over a Dictionary(Timestamp) returns a type that violates its declared Int32 (crashes GROUP BY/JOIN)

Ouverte
  • Quelque chose ne va pas ?
#221 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 2 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
45/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Calme
Stack technique
python
Domaine
backend, databases

Piste de recherche

Start with the linked minimal reproduction, repros/datafusion/date_part_dictionary_panic.py, and the Rust test instructions in repros/datafusion/README.md; reproduce the schema mismatch and GROUP BY panic against DataFusion 54.0.0. The issue says this needs to be filed against apache/datafusion, so trace the temporal function's handling of dictionary inputs there. Done means the logical and physical result types agree and materialization, GROUP BY, and JOIN no longer panic.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Tracking issue for an upstream DataFusion bug (to file against apache/datafusion). The second of two blockers for dictionary-encoding coordinate columns (#217), and the one that breaks core climate SQL (diurnal climatology, anomaly).

Summary

date_part('hour', t) where t is Dictionary(Int32, Timestamp) preserves the dictionary encoding and returns a Dictionary(Int32, Int32), but the expression's declared logical type is plain Int32. The physical/logical mismatch surfaces as soon as the column is materialized:

Arrow error: Invalid argument error: column types must match schema types,
expected Int32 but found Dictionary(Int32, Int32) at column index 0

A GROUP BY / JOIN on the same expression is worse — it panics a worker thread:

thread 'tokio-rt-worker' panicked at arrow-array-58.3.0/src/cast.rs:849: primitive array

(a failed downcast of the dictionary array to a primitive array). GROUP BY date_part('hour', time) is exactly how you write a diurnal climatology, so this is not an edge case for the domain.

Minimal repro

Pure datafusion + pyarrow, no xarray-sql, no network: repros/datafusion/date_part_dictionary_panic.py (branch claude/datafusion-upstream-repros-fs1bqv; paste-in Rust test in the README).

# t is Dictionary(Int32, Timestamp)
ctx.sql("SELECT date_part('hour', t) AS h FROM x")             # -> expected Int32 but found Dictionary(Int32, Int32)
ctx.sql("SELECT date_part('hour', t), COUNT(*) FROM x GROUP BY 1")  # -> panics ("primitive array")

Expected

date_part (and temporal scalar functions generally) over a dictionary-encoded input should return a plain Int32 (decode the dictionary), or declare the Dictionary(Int32, Int32) it actually produces — logical and physical types must agree, and a downstream consumer must never panic on the result.

Environment

datafusion 54.0.0 / pyarrow 23.0.0 (arrow-rs 58.3.0).

Impact on xarray-sql

Blocks dictionary-encoding the time coordinate (#217). Because the time axis is where climate SQL applies date_part/date_trunc/extract most, encoding it breaks climatology and anomaly queries (benchmarks 02_climatology and 04_anomaly). This is the bug that made #217 unworkable without an upstream fix.

Langage dominant
Python
Étoiles
142
Forks
19
Merge moyen
3 j 47 min
PR mergées (30 j)
8

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de xqlsystems/xarray-sql

Toutes les issues de xqlsystems/xarray-sql

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.