DataFusion 55: eliminate_cross_join drops a join key that reads two tables of a cross join, and returns wrong rows — draft for upstream

Offen
  • Stimmt etwas nicht?
#299 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 3 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
15/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python, sql
Bereich
databases

Rechercherichtung

The repro is the Python snippet in the issue body, using datafusion-python 55.0.0 and a two-table cross join with two equality keys. Run it, then compare with DuckDB's output; the issue says the correct answer is the rows (0, 0, 10.0) and (0, 1, 20.0). The suspect is the optimizer rule eliminate_cross_join in apache/datafusion, so the fix belongs upstream rather than in this repo. Check whether PR apache/datafusion#26786 already fixes it on main before filing. Done means a clean upstream report with this repro, or a confirmed fix there.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

bug ml

🌀 Claude: A wrong answer in DataFusion 55. An inner join has two equality keys, and one of them reads two tables of an earlier cross join: b.i = a.i + t.d. The optimizer rule eliminate_cross_join reorders the joins and keeps the other key, b.j = a.j, but drops this one, so the query returns every row that matches on j.

import datafusion
import pyarrow as pa

ctx = datafusion.SessionContext()
ctx.register_record_batches("x", [pa.table({"i": [0, 0, 1, 1], "j": [0, 1, 0, 1], "v": [1.0, 2.0, 10.0, 20.0]}).to_batches()])
ctx.register_record_batches("t", [pa.table({"d": [1]}).to_batches()])
q = """SELECT a.i, a.j, b.v FROM x a CROSS JOIN t
       JOIN x b ON b.i = a.i + t.d AND b.j = a.j ORDER BY 1, 2, 3"""
print(ctx.sql(q).to_pylist())
  • Expected, as DuckDB 1.5.6 returns: (0, 0, 10.0) and (0, 1, 20.0). Only a.i = 0 has a row at i + 1.
  • Got, on DataFusion 55.0.0 (datafusion-python 55.0.0, Python 3.12, Linux x86-64): all 8 combinations of a and b with the same j, including (1, 0, 1.0), which no row satisfies.

The bug needs all three of:

  • the cross join, written CROSS JOIN t or JOIN t ON true; the comma-join form, FROM x a, t, x b WHERE …, is correct;
  • a key that reads both inputs of the cross join;
  • a second equality key. With ON b.i = a.i + t.d alone, the answer is right.

EXPLAIN VERBOSE shows where it happens. After extract_equijoin_predicate, the join is Inner Join: a.i + t.d = b.i, a.j = b.j over Cross Join(a, t). After eliminate_cross_join, it is Cross Join(Inner Join: a.j = b.j (a, b), t), and the key a.i + t.d = b.i is gone from the plan.

How it was found: writing a bilinear upsampling as SQL, for an ML demo in benchmarks/ml. The upsampling cross-joins each coarse cell with a small table of taps, then joins the cell each tap reads on its latitude and longitude. DataFusion's answer was off by a factor of about 20, and DuckDB's matched PyTorch to 2e-7.

No existing issue in apache/datafusion matches. apache/datafusion#26786 (merged 2026-10-31) refactors this rule; whether it fixes this needs checking on main.

Upstream: report to apache/datafusion with the repro above. The fix is in eliminate_cross_join, where an equality key whose side reads several inputs of the flattened cross join has to stay a join filter, or be placed on the join that brings those inputs together.

Vorherrschende Sprache
Python
Sterne
142
Forks
19
Ø Merge
3 T. 47 Min.
Gemergte PRs (30 T.)
8

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus xqlsystems/xarray-sql

Alle Issues in xqlsystems/xarray-sql

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.