Simplified filter expression has a Null type instead of Int64 type across the FFI layer
維護者通常 1 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 新手友好度
- 55/100
- Issue 類型
- 缺陷
- 描述清晰度
- 基本清楚
- 活躍度
- 冷清
研究方向
先從連結的 MRE 開始，尤其是 repro.py 和 repro_provider 目錄，接著比較 main 和 datafusion52 分支。執行 maturin develop 和 python repro.py 步驟以重現 LIMIT 5 panic，並執行針對 SELECT * FROM dummy_table WHERE a < 5 的對應 cargo test，以比較有無 FFI 層時的行為。完成的標準是 FFI 查詢能夠成功執行，且不出現 Null-versus-Int64 assertion。
由索引模型根據 Issue 內容生成。
描述
Describe the bug
A custom table provider for a ParquetSource with a trivial catalog and an Int64 column yields some errors when a SQL query has a filter with a literal limit on that column of the form
assertion `left == right` failed: Simplified expression should have the same data type as the original
left: Null
right: Int64
The error does not occur when using datafusion-python 52; it also does not occur when running the query purely in a Rust SessionContext; the backtrace for the above error shows it coming from datafusion-ffi code as well.
This may of course not be a bug, but instead some bad practice that the version 52 set of crates tolerates but which is now invalid. A MRE of this custom table provider can be found in the public repo https://github.com/jwimberl/datafusion_python_53_int64filter_repro, which contains
- a non-working datafusion53 version (in branch main)
- a baseline working version (in branch datafusion52)
and a canned dummy dataset. The README.md of this repo has more details.
To Reproduce
In the main branch, build the py_repro_provider crate with maturin develop and run python repro.py. This loads the dummy dataset as a table dummy_table and runs two queries
SELECT * FROM dummy_table LIMIT 1, which is successfulSELECT * FROM dummy_table LIMIT 5, which panics
Itss output should be something like
Successful query:
a b
0 0 42
Unsuccesful query:
thread '<unnamed>' panicked at /home/jwimberley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-expr-53.1.0/src/simplifier/mod.rs:76:17:
assertion `left == right` failed: Simplified expression should have the same data type as the original
left: Null
right: Int64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
followed by backtrace information.
Expected behavior
In the datafusion52 branch, build thepy_repro_provider with maturin develop and run python repro.py. It runs the same two queries, and its output should be
Successful query:
a b
0 0 42
Also successful query:
a b
0 0 42
1 1 42
2 2 42
3 3 42
4 4 42
Additional context
In either the main branch or datafusion52 branch, the Rust code for the table provider is in the directory repro_provider, and there is a corresponding cargo test that runs SELECT * FROM dummy_table WHERE a < 5. Without the FFI layer, this is successful with both datafusion 52 and 53.
- 主要語言
- Python
- 星號
- 606
- 分支
- 176
- 平均合併
- 1 天 7 小時
- 30 天內合併 PR
- 14
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 沒有貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
apache/datafusion-python 的其他 Issue
-
bug
難度 2/5 1-3 小時 新手友好度 75/100
apache/datafusion-python#1765 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 78/100
apache/datafusion-python#1760 ·
維護者通常 1 天內回覆
-
enhancement
難度 2/5 1-3 小時 新手友好度 70/100
apache/datafusion-python#1757 ·
維護者通常 1 天內回覆
-
documentation
難度 2/5 1-3 小時 新手友好度 72/100
apache/datafusion-python#1726 ·
維護者通常 1 天內回覆
-
難度 2/5 半天 新手友好度 88/100
apache/datafusion-python#1691 ·
維護者通常 1 天內回覆
查看 apache/datafusion-python 的全部 Issue
相似的 Issue
-
P4: low tooling
難度 2/5 1-3 小時 新手友好度 88/100
jeffknupp/association#318 ·
-
難度 2/5 1-3 小時 新手友好度 68/100
petercorke/robotics-toolbox-python#709 ·
維護者通常 2 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 88/100
MakerYuichi/Aegis-pro#114 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 85/100
維護者通常 2 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
mpfaffenberger/code_puppy#985 · 2 則留言 ·
維護者通常 1 天內回覆