simplify_expressions misoptimizes +0.0 vs -0.0 literal equality/inequality conjunctions via ScalarValue::PartialEq

未关闭
  • 哪里不对？
#27,365 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@AnuragRaut08 已经在做这个了。

开始于 2026年11月12日。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
69/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python, rust, sql
领域
backend, databases, testing

调研方向

Start in datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs, around the literal comparison conjunction logic at lines 1151–1166, and trace how it compares ScalarValues for SQL equality. Check the existing optimizer tests and the equality semantics used by DataFusion's eq kernel, especially signed zero and NaN. Add regression coverage for the reported conjunctions; done means optimized and unoptimized queries return matching results.

由索引模型根据 Issue 内容生成。

描述

Describe the bug

In datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs (verified on DataFusion 55.0.0 and main HEAD e1c625fdf5, lines 1151–1166), literal comparison conjunction simplification rewrites:

  • f = 0.0 AND f != -0.0 $\rightarrow$ f = 0.0
  • f = 0.0 AND f = -0.0 $\rightarrow$ false
  • f IN (0.0) AND f NOT IN (-0.0) $\rightarrow$ f = 0.0

because distinctness between two literal ScalarValues (l1 != l2) is checked using Rust's ScalarValue::PartialEq (which uses totalOrder / bit-aware comparison where Float64(0.0) != Float64(-0.0) is true), whereas DataFusion/Arrow's SQL = and != operators follow IEEE-754 numeric equality where 0.0 = -0.0 is true and 0.0 != -0.0 is false.

As a result, disabling the logical optimizer (datafusion.optimizer.max_passes = 0) vs. running with default optimizations produces opposite query results.

To Reproduce
import datafusion
import pyarrow as pa

tbl = pa.table({
    "id": pa.array([1, 2, 3], type=pa.int64()),
    "f": pa.array([0.0, -0.0, 1.5], type=pa.float64()),
})

ctx_unopt = datafusion.SessionContext(
    datafusion.SessionConfig().set("datafusion.optimizer.max_passes", "0")
)
ctx_opt = datafusion.SessionContext()
ctx_unopt.from_arrow(tbl, name="t")
ctx_opt.from_arrow(tbl, name="t")

for q in [
    "SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id",
    "SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id",
]:
    r_unopt = [r["id"] for b in ctx_unopt.sql(q).collect() for r in b.to_pylist()]
    r_opt = [r["id"] for b in ctx_opt.sql(q).collect() for r in b.to_pylist()]
    print(q)
    print("  max_passes=0:", r_unopt)
    print("  default opt :", r_opt)

Output on DataFusion 55.0.0:

SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id
  max_passes=0: []
  default opt : [1, 2]
SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id
  max_passes=0: [1, 2]
  default opt : []
Expected behavior

Under IEEE-754 floating-point equality (0.0 = -0.0 is true):

  • SELECT id FROM t WHERE f = 0.0 AND f != -0.0 should return [] (matching max_passes=0, PostgreSQL, DuckDB, and SQLite).
  • SELECT id FROM t WHERE f = 0.0 AND f = -0.0 should return [1, 2].
Additional context

In expr_simplifier.rs (around lines 1151–1166, introduced in #19731, and similarly in A = L1 AND A = L2 -> false), comparing floating-point ScalarValues (Float16, Float32, Float64) for SQL = / != simplification needs numeric zero-sign-insensitive equality (+0.0 == -0.0 and NaN handling matching DataFusion's eq kernel) rather than ScalarValue::PartialEq.

Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

主要语言
Rust
星标
9.6k
派生
2.6k
平均合并
3 天 4 小时
30 天内合并 PR
345

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/datafusion 的其他 Issue

查看 apache/datafusion 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。