DataFusion 55: eliminate_cross_join drops a join key that reads two tables of a cross join, and returns wrong rows — draft for upstream

Aberta
  • Algo errado?
#299 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 3 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
15/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
python, sql
Domínio
databases

Direção de pesquisa

The repro is the Python snippet in the issue body, using datafusion-python 55.0.0 and a two-table cross join with two equality keys. Run it, then compare with DuckDB's output; the issue says the correct answer is the rows (0, 0, 10.0) and (0, 1, 20.0). The suspect is the optimizer rule eliminate_cross_join in apache/datafusion, so the fix belongs upstream rather than in this repo. Check whether PR apache/datafusion#26786 already fixes it on main before filing. Done means a clean upstream report with this repro, or a confirmed fix there.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug ml

🌀 Claude: A wrong answer in DataFusion 55. An inner join has two equality keys, and one of them reads two tables of an earlier cross join: b.i = a.i + t.d. The optimizer rule eliminate_cross_join reorders the joins and keeps the other key, b.j = a.j, but drops this one, so the query returns every row that matches on j.

import datafusion
import pyarrow as pa

ctx = datafusion.SessionContext()
ctx.register_record_batches("x", [pa.table({"i": [0, 0, 1, 1], "j": [0, 1, 0, 1], "v": [1.0, 2.0, 10.0, 20.0]}).to_batches()])
ctx.register_record_batches("t", [pa.table({"d": [1]}).to_batches()])
q = """SELECT a.i, a.j, b.v FROM x a CROSS JOIN t
       JOIN x b ON b.i = a.i + t.d AND b.j = a.j ORDER BY 1, 2, 3"""
print(ctx.sql(q).to_pylist())
  • Expected, as DuckDB 1.5.6 returns: (0, 0, 10.0) and (0, 1, 20.0). Only a.i = 0 has a row at i + 1.
  • Got, on DataFusion 55.0.0 (datafusion-python 55.0.0, Python 3.12, Linux x86-64): all 8 combinations of a and b with the same j, including (1, 0, 1.0), which no row satisfies.

The bug needs all three of:

  • the cross join, written CROSS JOIN t or JOIN t ON true; the comma-join form, FROM x a, t, x b WHERE …, is correct;
  • a key that reads both inputs of the cross join;
  • a second equality key. With ON b.i = a.i + t.d alone, the answer is right.

EXPLAIN VERBOSE shows where it happens. After extract_equijoin_predicate, the join is Inner Join: a.i + t.d = b.i, a.j = b.j over Cross Join(a, t). After eliminate_cross_join, it is Cross Join(Inner Join: a.j = b.j (a, b), t), and the key a.i + t.d = b.i is gone from the plan.

How it was found: writing a bilinear upsampling as SQL, for an ML demo in benchmarks/ml. The upsampling cross-joins each coarse cell with a small table of taps, then joins the cell each tap reads on its latitude and longitude. DataFusion's answer was off by a factor of about 20, and DuckDB's matched PyTorch to 2e-7.

No existing issue in apache/datafusion matches. apache/datafusion#26786 (merged 2026-10-31) refactors this rule; whether it fixes this needs checking on main.

Upstream: report to apache/datafusion with the repro above. The fix is in eliminate_cross_join, where an equality key whose side reads several inputs of the flattened cross join has to stay a join filter, or be placed on the join that brings those inputs together.

Linguagem predominante
Python
Estrelas
142
Forks
19
Merge médio
3d 47min
PRs com merge (30d)
8

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de xqlsystems/xarray-sql

Todas as issues de xqlsystems/xarray-sql

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.