simplify_expressions misoptimizes +0.0 vs -0.0 literal equality/inequality conjunctions via ScalarValue::PartialEq

Aperta
  • Qualcosa non va?
#27,365 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@AnuragRaut08 ci sta già lavorando.

Dal 12/11/2026.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
69/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, rust, sql
Ambito
backend, databases, testing

Direzione di ricerca

Start in datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs, around the literal comparison conjunction logic at lines 1151–1166, and trace how it compares ScalarValues for SQL equality. Check the existing optimizer tests and the equality semantics used by DataFusion's eq kernel, especially signed zero and NaN. Add regression coverage for the reported conjunctions; done means optimized and unoptimized queries return matching results.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

In datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs (verified on DataFusion 55.0.0 and main HEAD e1c625fdf5, lines 1151–1166), literal comparison conjunction simplification rewrites:

  • f = 0.0 AND f != -0.0 $\rightarrow$ f = 0.0
  • f = 0.0 AND f = -0.0 $\rightarrow$ false
  • f IN (0.0) AND f NOT IN (-0.0) $\rightarrow$ f = 0.0

because distinctness between two literal ScalarValues (l1 != l2) is checked using Rust's ScalarValue::PartialEq (which uses totalOrder / bit-aware comparison where Float64(0.0) != Float64(-0.0) is true), whereas DataFusion/Arrow's SQL = and != operators follow IEEE-754 numeric equality where 0.0 = -0.0 is true and 0.0 != -0.0 is false.

As a result, disabling the logical optimizer (datafusion.optimizer.max_passes = 0) vs. running with default optimizations produces opposite query results.

To Reproduce
import datafusion
import pyarrow as pa

tbl = pa.table({
    "id": pa.array([1, 2, 3], type=pa.int64()),
    "f": pa.array([0.0, -0.0, 1.5], type=pa.float64()),
})

ctx_unopt = datafusion.SessionContext(
    datafusion.SessionConfig().set("datafusion.optimizer.max_passes", "0")
)
ctx_opt = datafusion.SessionContext()
ctx_unopt.from_arrow(tbl, name="t")
ctx_opt.from_arrow(tbl, name="t")

for q in [
    "SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id",
    "SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id",
]:
    r_unopt = [r["id"] for b in ctx_unopt.sql(q).collect() for r in b.to_pylist()]
    r_opt = [r["id"] for b in ctx_opt.sql(q).collect() for r in b.to_pylist()]
    print(q)
    print("  max_passes=0:", r_unopt)
    print("  default opt :", r_opt)

Output on DataFusion 55.0.0:

SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id
  max_passes=0: []
  default opt : [1, 2]
SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id
  max_passes=0: [1, 2]
  default opt : []
Expected behavior

Under IEEE-754 floating-point equality (0.0 = -0.0 is true):

  • SELECT id FROM t WHERE f = 0.0 AND f != -0.0 should return [] (matching max_passes=0, PostgreSQL, DuckDB, and SQLite).
  • SELECT id FROM t WHERE f = 0.0 AND f = -0.0 should return [1, 2].
Additional context

In expr_simplifier.rs (around lines 1151–1166, introduced in #19731, and similarly in A = L1 AND A = L2 -> false), comparing floating-point ScalarValues (Float16, Float32, Float64) for SQL = / != simplification needs numeric zero-sign-insensitive equality (+0.0 == -0.0 and NaN handling matching DataFusion's eq kernel) rather than ScalarValue::PartialEq.

Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

Lingua principale
Rust
Stelle
9.6k
Fork
2.6k
Merge medio
3g 4h
PR unite (30g)
345

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion

Tutte le issue di apache/datafusion

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.