DataFusion 55: eliminate_cross_join drops a join key that reads two tables of a cross join, and returns wrong rows — draft for upstream

オープン
  • 何か問題がありますか？
#299 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
15/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python, sql
領域
databases

調査の方向性

The repro is the Python snippet in the issue body, using datafusion-python 55.0.0 and a two-table cross join with two equality keys. Run it, then compare with DuckDB's output; the issue says the correct answer is the rows (0, 0, 10.0) and (0, 1, 20.0). The suspect is the optimizer rule eliminate_cross_join in apache/datafusion, so the fix belongs upstream rather than in this repo. Check whether PR apache/datafusion#26786 already fixes it on main before filing. Done means a clean upstream report with this repro, or a confirmed fix there.

索引モデルが issue の本文から書いたものです。

説明

bug ml

🌀 Claude: A wrong answer in DataFusion 55. An inner join has two equality keys, and one of them reads two tables of an earlier cross join: b.i = a.i + t.d. The optimizer rule eliminate_cross_join reorders the joins and keeps the other key, b.j = a.j, but drops this one, so the query returns every row that matches on j.

import datafusion
import pyarrow as pa

ctx = datafusion.SessionContext()
ctx.register_record_batches("x", [pa.table({"i": [0, 0, 1, 1], "j": [0, 1, 0, 1], "v": [1.0, 2.0, 10.0, 20.0]}).to_batches()])
ctx.register_record_batches("t", [pa.table({"d": [1]}).to_batches()])
q = """SELECT a.i, a.j, b.v FROM x a CROSS JOIN t
       JOIN x b ON b.i = a.i + t.d AND b.j = a.j ORDER BY 1, 2, 3"""
print(ctx.sql(q).to_pylist())
  • Expected, as DuckDB 1.5.6 returns: (0, 0, 10.0) and (0, 1, 20.0). Only a.i = 0 has a row at i + 1.
  • Got, on DataFusion 55.0.0 (datafusion-python 55.0.0, Python 3.12, Linux x86-64): all 8 combinations of a and b with the same j, including (1, 0, 1.0), which no row satisfies.

The bug needs all three of:

  • the cross join, written CROSS JOIN t or JOIN t ON true; the comma-join form, FROM x a, t, x b WHERE …, is correct;
  • a key that reads both inputs of the cross join;
  • a second equality key. With ON b.i = a.i + t.d alone, the answer is right.

EXPLAIN VERBOSE shows where it happens. After extract_equijoin_predicate, the join is Inner Join: a.i + t.d = b.i, a.j = b.j over Cross Join(a, t). After eliminate_cross_join, it is Cross Join(Inner Join: a.j = b.j (a, b), t), and the key a.i + t.d = b.i is gone from the plan.

How it was found: writing a bilinear upsampling as SQL, for an ML demo in benchmarks/ml. The upsampling cross-joins each coarse cell with a small table of taps, then joins the cell each tap reads on its latitude and longitude. DataFusion's answer was off by a factor of about 20, and DuckDB's matched PyTorch to 2e-7.

No existing issue in apache/datafusion matches. apache/datafusion#26786 (merged 2026-10-31) refactors this rule; whether it fixes this needs checking on main.

Upstream: report to apache/datafusion with the repro above. The fix is in eliminate_cross_join, where an equality key whose side reads several inputs of the flattened cross join has to stay a join filter, or be placed on the join that brings those inputs together.

主要言語
Python
スター
142
フォーク
19
平均マージ
3日 47分
マージ済み PR（30日）
8

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

xqlsystems/xarray-sql のほかの issue

xqlsystems/xarray-sql の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。