DataFusion: date_part over a Dictionary(Timestamp) returns a type that violates its declared Int32 (crashes GROUP BY/JOIN)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start with the linked minimal reproduction, repros/datafusion/date_part_dictionary_panic.py, and the Rust test instructions in repros/datafusion/README.md; reproduce the schema mismatch and GROUP BY panic against DataFusion 54.0.0. The issue says this needs to be filed against apache/datafusion, so trace the temporal function's handling of dictionary inputs there. Done means the logical and physical result types agree and materialization, GROUP BY, and JOIN no longer panic.
由索引模型根据 Issue 内容生成。
描述
Tracking issue for an upstream DataFusion bug (to file against apache/datafusion). The second of two blockers for dictionary-encoding coordinate columns (#217), and the one that breaks core climate SQL (diurnal climatology, anomaly).
Summary
date_part('hour', t) where t is Dictionary(Int32, Timestamp) preserves the dictionary encoding and returns a Dictionary(Int32, Int32), but the expression's declared logical type is plain Int32. The physical/logical mismatch surfaces as soon as the column is materialized:
Arrow error: Invalid argument error: column types must match schema types,
expected Int32 but found Dictionary(Int32, Int32) at column index 0
A GROUP BY / JOIN on the same expression is worse — it panics a worker thread:
thread 'tokio-rt-worker' panicked at arrow-array-58.3.0/src/cast.rs:849: primitive array
(a failed downcast of the dictionary array to a primitive array). GROUP BY date_part('hour', time) is exactly how you write a diurnal climatology, so this is not an edge case for the domain.
Minimal repro
Pure datafusion + pyarrow, no xarray-sql, no network: repros/datafusion/date_part_dictionary_panic.py (branch claude/datafusion-upstream-repros-fs1bqv; paste-in Rust test in the README).
# t is Dictionary(Int32, Timestamp)
ctx.sql("SELECT date_part('hour', t) AS h FROM x") # -> expected Int32 but found Dictionary(Int32, Int32)
ctx.sql("SELECT date_part('hour', t), COUNT(*) FROM x GROUP BY 1") # -> panics ("primitive array")
Expected
date_part (and temporal scalar functions generally) over a dictionary-encoded input should return a plain Int32 (decode the dictionary), or declare the Dictionary(Int32, Int32) it actually produces — logical and physical types must agree, and a downstream consumer must never panic on the result.
Environment
datafusion 54.0.0 / pyarrow 23.0.0 (arrow-rs 58.3.0).
Impact on xarray-sql
Blocks dictionary-encoding the time coordinate (#217). Because the time axis is where climate SQL applies date_part/date_trunc/extract most, encoding it breaks climatology and anomaly queries (benchmarks 02_climatology and 04_anomaly). This is the bug that made #217 unworkable without an upstream fix.
- 主要语言
- Python
- 星标
- 142
- 派生
- 19
- 平均合并
- 1 天 9 小时
- 30 天内合并 PR
- 9
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
xqlsystems/xarray-sql 的其他 Issue
-
documentation
难度 1/5 1-3 小时 新手友好度 85/100
xqlsystems/xarray-sql#268 ·
维护者通常 1 天内回复
-
enhancement
难度 2/5 1-3 小时 新手友好度 64/100
xqlsystems/xarray-sql#267 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 74/100
xqlsystems/xarray-sql#243 ·
维护者通常 1 天内回复
-
publish.yml: the TestPyPI verification step is unreachable (and installs the wrong package name)未关闭
难度 2/5 1-3 小时 新手友好度 62/100
xqlsystems/xarray-sql#242 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
xqlsystems/xarray-sql#237 ·
维护者通常 1 天内回复
查看 xqlsystems/xarray-sql 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 66/100
-
[Bug]: Queen's own Ollama model loses num_ctx when a running session switches to it可能已有人在做 @anasamin26 今天认领。 未关闭bug enhancement
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
llm translation
难度 2/5 1-3 小时 新手友好度 78/100
BerriAI/litellm#47884 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
bug 🐛
难度 2/5 1-3 小时 新手友好度 72/100
netalertx/NetAlertX#1856 · 1 个 reaction ·
维护者通常 1 天内回复