simplify_expressions misoptimizes +0.0 vs -0.0 literal equality/inequality conjunctions via ScalarValue::PartialEq

Aberta
  • Algo errado?
#27,365 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@AnuragRaut08 já está trabalhando nisso.

Desde 12/11/2026.

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
69/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
python, rust, sql
Domínio
backend, databases, testing

Direção de pesquisa

Start in datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs, around the literal comparison conjunction logic at lines 1151–1166, and trace how it compares ScalarValues for SQL equality. Check the existing optimizer tests and the equality semantics used by DataFusion's eq kernel, especially signed zero and NaN. Add regression coverage for the reported conjunctions; done means optimized and unoptimized queries return matching results.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Describe the bug

In datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs (verified on DataFusion 55.0.0 and main HEAD e1c625fdf5, lines 1151–1166), literal comparison conjunction simplification rewrites:

  • f = 0.0 AND f != -0.0 $\rightarrow$ f = 0.0
  • f = 0.0 AND f = -0.0 $\rightarrow$ false
  • f IN (0.0) AND f NOT IN (-0.0) $\rightarrow$ f = 0.0

because distinctness between two literal ScalarValues (l1 != l2) is checked using Rust's ScalarValue::PartialEq (which uses totalOrder / bit-aware comparison where Float64(0.0) != Float64(-0.0) is true), whereas DataFusion/Arrow's SQL = and != operators follow IEEE-754 numeric equality where 0.0 = -0.0 is true and 0.0 != -0.0 is false.

As a result, disabling the logical optimizer (datafusion.optimizer.max_passes = 0) vs. running with default optimizations produces opposite query results.

To Reproduce
import datafusion
import pyarrow as pa

tbl = pa.table({
    "id": pa.array([1, 2, 3], type=pa.int64()),
    "f": pa.array([0.0, -0.0, 1.5], type=pa.float64()),
})

ctx_unopt = datafusion.SessionContext(
    datafusion.SessionConfig().set("datafusion.optimizer.max_passes", "0")
)
ctx_opt = datafusion.SessionContext()
ctx_unopt.from_arrow(tbl, name="t")
ctx_opt.from_arrow(tbl, name="t")

for q in [
    "SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id",
    "SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id",
]:
    r_unopt = [r["id"] for b in ctx_unopt.sql(q).collect() for r in b.to_pylist()]
    r_opt = [r["id"] for b in ctx_opt.sql(q).collect() for r in b.to_pylist()]
    print(q)
    print("  max_passes=0:", r_unopt)
    print("  default opt :", r_opt)

Output on DataFusion 55.0.0:

SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id
  max_passes=0: []
  default opt : [1, 2]
SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id
  max_passes=0: [1, 2]
  default opt : []
Expected behavior

Under IEEE-754 floating-point equality (0.0 = -0.0 is true):

  • SELECT id FROM t WHERE f = 0.0 AND f != -0.0 should return [] (matching max_passes=0, PostgreSQL, DuckDB, and SQLite).
  • SELECT id FROM t WHERE f = 0.0 AND f = -0.0 should return [1, 2].
Additional context

In expr_simplifier.rs (around lines 1151–1166, introduced in #19731, and similarly in A = L1 AND A = L2 -> false), comparing floating-point ScalarValues (Float16, Float32, Float64) for SQL = / != simplification needs numeric zero-sign-insensitive equality (+0.0 == -0.0 and NaN handling matching DataFusion's eq kernel) rather than ScalarValue::PartialEq.

Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

Linguagem predominante
Rust
Estrelas
9.6k
Forks
2.6k
Merge médio
3d 4h
PRs com merge (30d)
345

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/datafusion

Todas as issues de apache/datafusion

Issues semelhantes

Mais issues de Rust

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.