DataFusion 55: eliminate_cross_join drops a join key that reads two tables of a cross join, and returns wrong rows — draft for upstream

Ouverte
  • Quelque chose ne va pas ?
#299 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 3 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
15/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
python, sql
Domaine
databases

Piste de recherche

The repro is the Python snippet in the issue body, using datafusion-python 55.0.0 and a two-table cross join with two equality keys. Run it, then compare with DuckDB's output; the issue says the correct answer is the rows (0, 0, 10.0) and (0, 1, 20.0). The suspect is the optimizer rule eliminate_cross_join in apache/datafusion, so the fix belongs upstream rather than in this repo. Check whether PR apache/datafusion#26786 already fixes it on main before filing. Done means a clean upstream report with this repro, or a confirmed fix there.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

bug ml

🌀 Claude: A wrong answer in DataFusion 55. An inner join has two equality keys, and one of them reads two tables of an earlier cross join: b.i = a.i + t.d. The optimizer rule eliminate_cross_join reorders the joins and keeps the other key, b.j = a.j, but drops this one, so the query returns every row that matches on j.

import datafusion
import pyarrow as pa

ctx = datafusion.SessionContext()
ctx.register_record_batches("x", [pa.table({"i": [0, 0, 1, 1], "j": [0, 1, 0, 1], "v": [1.0, 2.0, 10.0, 20.0]}).to_batches()])
ctx.register_record_batches("t", [pa.table({"d": [1]}).to_batches()])
q = """SELECT a.i, a.j, b.v FROM x a CROSS JOIN t
       JOIN x b ON b.i = a.i + t.d AND b.j = a.j ORDER BY 1, 2, 3"""
print(ctx.sql(q).to_pylist())
  • Expected, as DuckDB 1.5.6 returns: (0, 0, 10.0) and (0, 1, 20.0). Only a.i = 0 has a row at i + 1.
  • Got, on DataFusion 55.0.0 (datafusion-python 55.0.0, Python 3.12, Linux x86-64): all 8 combinations of a and b with the same j, including (1, 0, 1.0), which no row satisfies.

The bug needs all three of:

  • the cross join, written CROSS JOIN t or JOIN t ON true; the comma-join form, FROM x a, t, x b WHERE …, is correct;
  • a key that reads both inputs of the cross join;
  • a second equality key. With ON b.i = a.i + t.d alone, the answer is right.

EXPLAIN VERBOSE shows where it happens. After extract_equijoin_predicate, the join is Inner Join: a.i + t.d = b.i, a.j = b.j over Cross Join(a, t). After eliminate_cross_join, it is Cross Join(Inner Join: a.j = b.j (a, b), t), and the key a.i + t.d = b.i is gone from the plan.

How it was found: writing a bilinear upsampling as SQL, for an ML demo in benchmarks/ml. The upsampling cross-joins each coarse cell with a small table of taps, then joins the cell each tap reads on its latitude and longitude. DataFusion's answer was off by a factor of about 20, and DuckDB's matched PyTorch to 2e-7.

No existing issue in apache/datafusion matches. apache/datafusion#26786 (merged 2026-10-31) refactors this rule; whether it fixes this needs checking on main.

Upstream: report to apache/datafusion with the repro above. The fix is in eliminate_cross_join, where an equality key whose side reads several inputs of the flattened cross join has to stay a join filter, or be placed on the join that brings those inputs together.

Langage dominant
Python
Étoiles
142
Forks
19
Merge moyen
3 j 47 min
PR mergées (30 j)
8

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de xqlsystems/xarray-sql

Toutes les issues de xqlsystems/xarray-sql

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.