DataFusion 55: eliminate_cross_join drops a join key that reads two tables of a cross join, and returns wrong rows — draft for upstream

未關閉
  • 哪裡不對？
#299 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 3 天內回覆

還沒有人認領這個 Issue。

評估

難度
4/5
預估耗時
3-5 天
新手友好度
15/100
Issue 類型
缺陷
描述清晰度
基本清楚
活躍度
活躍
技術堆疊
python, sql
領域
databases

研究方向

The repro is the Python snippet in the issue body, using datafusion-python 55.0.0 and a two-table cross join with two equality keys. Run it, then compare with DuckDB's output; the issue says the correct answer is the rows (0, 0, 10.0) and (0, 1, 20.0). The suspect is the optimizer rule eliminate_cross_join in apache/datafusion, so the fix belongs upstream rather than in this repo. Check whether PR apache/datafusion#26786 already fixes it on main before filing. Done means a clean upstream report with this repro, or a confirmed fix there.

由索引模型根據 Issue 內容生成。

描述

bug ml

🌀 Claude: A wrong answer in DataFusion 55. An inner join has two equality keys, and one of them reads two tables of an earlier cross join: b.i = a.i + t.d. The optimizer rule eliminate_cross_join reorders the joins and keeps the other key, b.j = a.j, but drops this one, so the query returns every row that matches on j.

import datafusion
import pyarrow as pa

ctx = datafusion.SessionContext()
ctx.register_record_batches("x", [pa.table({"i": [0, 0, 1, 1], "j": [0, 1, 0, 1], "v": [1.0, 2.0, 10.0, 20.0]}).to_batches()])
ctx.register_record_batches("t", [pa.table({"d": [1]}).to_batches()])
q = """SELECT a.i, a.j, b.v FROM x a CROSS JOIN t
       JOIN x b ON b.i = a.i + t.d AND b.j = a.j ORDER BY 1, 2, 3"""
print(ctx.sql(q).to_pylist())
  • Expected, as DuckDB 1.5.6 returns: (0, 0, 10.0) and (0, 1, 20.0). Only a.i = 0 has a row at i + 1.
  • Got, on DataFusion 55.0.0 (datafusion-python 55.0.0, Python 3.12, Linux x86-64): all 8 combinations of a and b with the same j, including (1, 0, 1.0), which no row satisfies.

The bug needs all three of:

  • the cross join, written CROSS JOIN t or JOIN t ON true; the comma-join form, FROM x a, t, x b WHERE …, is correct;
  • a key that reads both inputs of the cross join;
  • a second equality key. With ON b.i = a.i + t.d alone, the answer is right.

EXPLAIN VERBOSE shows where it happens. After extract_equijoin_predicate, the join is Inner Join: a.i + t.d = b.i, a.j = b.j over Cross Join(a, t). After eliminate_cross_join, it is Cross Join(Inner Join: a.j = b.j (a, b), t), and the key a.i + t.d = b.i is gone from the plan.

How it was found: writing a bilinear upsampling as SQL, for an ML demo in benchmarks/ml. The upsampling cross-joins each coarse cell with a small table of taps, then joins the cell each tap reads on its latitude and longitude. DataFusion's answer was off by a factor of about 20, and DuckDB's matched PyTorch to 2e-7.

No existing issue in apache/datafusion matches. apache/datafusion#26786 (merged 2026-10-31) refactors this rule; whether it fixes this needs checking on main.

Upstream: report to apache/datafusion with the repro above. The fix is in eliminate_cross_join, where an equality key whose side reads several inputs of the flattened cross join has to stay a join filter, or be placed on the join that brings those inputs together.

主要語言
Python
星號
142
分支
19
平均合併
3 天 47 分鐘
30 天內合併 PR
8

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 沒有 Pull Request 範本
  • 閱讀貢獻指南

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

xqlsystems/xarray-sql 的其他 Issue

查看 xqlsystems/xarray-sql 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。