simplify_expressions misoptimizes +0.0 vs -0.0 literal equality/inequality conjunctions via ScalarValue::PartialEq

Abierto
  • ¿Algo no está bien?
#27,365 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@AnuragRaut08 ya está trabajando en esto.

Desde el 12/11/2026.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
69/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, rust, sql
Área
backend, databases, testing

Línea de trabajo

Start in datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs, around the literal comparison conjunction logic at lines 1151–1166, and trace how it compares ScalarValues for SQL equality. Check the existing optimizer tests and the equality semantics used by DataFusion's eq kernel, especially signed zero and NaN. Add regression coverage for the reported conjunctions; done means optimized and unoptimized queries return matching results.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

In datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs (verified on DataFusion 55.0.0 and main HEAD e1c625fdf5, lines 1151–1166), literal comparison conjunction simplification rewrites:

  • f = 0.0 AND f != -0.0 $\rightarrow$ f = 0.0
  • f = 0.0 AND f = -0.0 $\rightarrow$ false
  • f IN (0.0) AND f NOT IN (-0.0) $\rightarrow$ f = 0.0

because distinctness between two literal ScalarValues (l1 != l2) is checked using Rust's ScalarValue::PartialEq (which uses totalOrder / bit-aware comparison where Float64(0.0) != Float64(-0.0) is true), whereas DataFusion/Arrow's SQL = and != operators follow IEEE-754 numeric equality where 0.0 = -0.0 is true and 0.0 != -0.0 is false.

As a result, disabling the logical optimizer (datafusion.optimizer.max_passes = 0) vs. running with default optimizations produces opposite query results.

To Reproduce
import datafusion
import pyarrow as pa

tbl = pa.table({
    "id": pa.array([1, 2, 3], type=pa.int64()),
    "f": pa.array([0.0, -0.0, 1.5], type=pa.float64()),
})

ctx_unopt = datafusion.SessionContext(
    datafusion.SessionConfig().set("datafusion.optimizer.max_passes", "0")
)
ctx_opt = datafusion.SessionContext()
ctx_unopt.from_arrow(tbl, name="t")
ctx_opt.from_arrow(tbl, name="t")

for q in [
    "SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id",
    "SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id",
]:
    r_unopt = [r["id"] for b in ctx_unopt.sql(q).collect() for r in b.to_pylist()]
    r_opt = [r["id"] for b in ctx_opt.sql(q).collect() for r in b.to_pylist()]
    print(q)
    print("  max_passes=0:", r_unopt)
    print("  default opt :", r_opt)

Output on DataFusion 55.0.0:

SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id
  max_passes=0: []
  default opt : [1, 2]
SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id
  max_passes=0: [1, 2]
  default opt : []
Expected behavior

Under IEEE-754 floating-point equality (0.0 = -0.0 is true):

  • SELECT id FROM t WHERE f = 0.0 AND f != -0.0 should return [] (matching max_passes=0, PostgreSQL, DuckDB, and SQLite).
  • SELECT id FROM t WHERE f = 0.0 AND f = -0.0 should return [1, 2].
Additional context

In expr_simplifier.rs (around lines 1151–1166, introduced in #19731, and similarly in A = L1 AND A = L2 -> false), comparing floating-point ScalarValues (Float16, Float32, Float64) for SQL = / != simplification needs numeric zero-sign-insensitive equality (+0.0 == -0.0 and NaN handling matching DataFusion's eq kernel) rather than ScalarValue::PartialEq.

Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

Lenguaje dominante
Rust
Estrellas
9.6k
Forks
2.6k
Merge medio
3 d 4 h
PR fusionados (30 d)
345

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion

Todos los issues de apache/datafusion

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.