simplify_expressions misoptimizes +0.0 vs -0.0 literal equality/inequality conjunctions via ScalarValue::PartialEq

Offen
  • Stimmt etwas nicht?
#27,365 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@AnuragRaut08 arbeitet bereits daran.

Seit 12.11.2026.

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
69/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python, rust, sql
Bereich
backend, databases, testing

Rechercherichtung

Start in datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs, around the literal comparison conjunction logic at lines 1151–1166, and trace how it compares ScalarValues for SQL equality. Check the existing optimizer tests and the equality semantics used by DataFusion's eq kernel, especially signed zero and NaN. Add regression coverage for the reported conjunctions; done means optimized and unoptimized queries return matching results.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Describe the bug

In datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs (verified on DataFusion 55.0.0 and main HEAD e1c625fdf5, lines 1151–1166), literal comparison conjunction simplification rewrites:

  • f = 0.0 AND f != -0.0 $\rightarrow$ f = 0.0
  • f = 0.0 AND f = -0.0 $\rightarrow$ false
  • f IN (0.0) AND f NOT IN (-0.0) $\rightarrow$ f = 0.0

because distinctness between two literal ScalarValues (l1 != l2) is checked using Rust's ScalarValue::PartialEq (which uses totalOrder / bit-aware comparison where Float64(0.0) != Float64(-0.0) is true), whereas DataFusion/Arrow's SQL = and != operators follow IEEE-754 numeric equality where 0.0 = -0.0 is true and 0.0 != -0.0 is false.

As a result, disabling the logical optimizer (datafusion.optimizer.max_passes = 0) vs. running with default optimizations produces opposite query results.

To Reproduce
import datafusion
import pyarrow as pa

tbl = pa.table({
    "id": pa.array([1, 2, 3], type=pa.int64()),
    "f": pa.array([0.0, -0.0, 1.5], type=pa.float64()),
})

ctx_unopt = datafusion.SessionContext(
    datafusion.SessionConfig().set("datafusion.optimizer.max_passes", "0")
)
ctx_opt = datafusion.SessionContext()
ctx_unopt.from_arrow(tbl, name="t")
ctx_opt.from_arrow(tbl, name="t")

for q in [
    "SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id",
    "SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id",
]:
    r_unopt = [r["id"] for b in ctx_unopt.sql(q).collect() for r in b.to_pylist()]
    r_opt = [r["id"] for b in ctx_opt.sql(q).collect() for r in b.to_pylist()]
    print(q)
    print("  max_passes=0:", r_unopt)
    print("  default opt :", r_opt)

Output on DataFusion 55.0.0:

SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id
  max_passes=0: []
  default opt : [1, 2]
SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id
  max_passes=0: [1, 2]
  default opt : []
Expected behavior

Under IEEE-754 floating-point equality (0.0 = -0.0 is true):

  • SELECT id FROM t WHERE f = 0.0 AND f != -0.0 should return [] (matching max_passes=0, PostgreSQL, DuckDB, and SQLite).
  • SELECT id FROM t WHERE f = 0.0 AND f = -0.0 should return [1, 2].
Additional context

In expr_simplifier.rs (around lines 1151–1166, introduced in #19731, and similarly in A = L1 AND A = L2 -> false), comparing floating-point ScalarValues (Float16, Float32, Float64) for SQL = / != simplification needs numeric zero-sign-insensitive equality (+0.0 == -0.0 and NaN handling matching DataFusion's eq kernel) rather than ScalarValue::PartialEq.

Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

Vorherrschende Sprache
Rust
Sterne
9.6k
Forks
2.6k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
345

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/datafusion

Alle Issues in apache/datafusion

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.