Filter pushdown on float columns drops rows with NaN or -0.0

オープン
  • 何か問題がありますか？
#101 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
55/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
rust
領域
backend, databases

調査の方向性

Start by tracing scalar_value_to_datum and the scan pushdown path described in the issue, then inspect how InclusiveMetricsEvaluator handles float bounds and predicates. Compare pushed-down results with the MemTable results in the reproduction, including NaN and signed zero cases. Done means the Iceberg table returns the same rows as the MemTable for these filters while retaining safe file pruning where possible.

索引モデルが issue の本文から書いたものです。

説明

Describe the bug

Some filters on double and float columns return fewer rows from an Iceberg table than DataFusion returns for the same data. The missing rows are the ones that hold NaN or -0.0. The query succeeds, but those rows aren't in the result.

The two sides compare these values differently:

  • DataFusion treats NaN as equal to itself and greater than any other value, and treats -0.0 as equal to 0.0. SELECT CAST('NaN' AS DOUBLE) > 1.0, CAST('NaN' AS DOUBLE) = CAST('NaN' AS DOUBLE), CAST('-0.0' AS DOUBLE) = 0.0 returns true for all three.
  • iceberg-rust's InclusiveMetricsEvaluator decides that a data file holding only NaN can't match >, >=, = or IN, and NaN is left out of a file's upper bound. iceberg-rust also orders floats with total_cmp, so to it -0.0 and 0.0 are different values.

scalar_value_to_datum passes float literals through as Datum::double, and the comparison reaches the scan unchanged. So the scan skips files that contain rows DataFusion would match. Those files are never read, and filtering again in DataFusion can't bring the rows back.

To Reproduce

Using the sqllogictest engine, with one INSERT per data file:

CREATE TABLE default.default.t (id INT NOT NULL, x DOUBLE);
INSERT INTO default.default.t VALUES (1, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (2, 0.5), (3, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (4, 2.0);
INSERT INTO default.default.t VALUES (5, CAST('-0.0' AS DOUBLE));
INSERT INTO default.default.t VALUES (6, 0.0);
CREATE TABLE m AS SELECT * FROM default.default.t;

SELECT id FROM <table> WHERE <filter> returns:

Filter Iceberg table t MemTable copy m
x > 1.0 4 1, 3, 4
x = CAST('NaN' AS DOUBLE) none 1, 3
x = 0.0 6 5, 6
x >= 0.0 2, 3, 4, 6 1, 2, 3, 4, 5, 6

The same table returns 1, 3, 4 for x + 0 > 1.0, which isn't pushed down.

Which rows go missing also depends on how they're laid out in files. Row 3 is skipped because its file's upper bound is 0.5. The same NaN in a file with 2.0 comes back.

Expected behavior

t returns the same rows as m.

I can think of two ways to get there:

  1. Stop pushing down comparisons and IN lists that have a float literal. That's the simplest, but float columns lose file pruning.
  2. Push a looser predicate that still covers every row DataFusion could match, and leave the exact check to DataFusion. For example, x > c becomes x > c OR x IS NAN and x = NaN becomes x IS NAN. Zero literals would cover both signs: x = 0.0 becomes x = -0.0 OR x = 0.0, x >= 0.0 becomes x >= -0.0, and x <= -0.0 becomes x <= 0.0.

Let me know which you'd prefer.

Additional context
  • <, !=, NOT IN, isnan(x) and NOT isnan(x) return the same rows on both sides.
  • NOT (x <= 1.0) hits this too, since it's simplified to x > 1.0 before pushdown.
  • REAL columns are affected when the literal is also REAL, for example x > CAST(1 AS REAL). With a DOUBLE literal the column gets a cast, and since #91 that comparison isn't pushed down.
  • iceberg-rust main has the same NaN handling and float ordering, so updating the iceberg-rust dependency won't fix this.
  • This is the same kind of problem as #36 and #44: the pushed predicate rejects rows that the filter keeps.
主要言語
Rust
スター
33
フォーク
34
平均マージ
4日 14時間
マージ済み PR（30日）
10

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/datafusion-iceberg のほかの issue

apache/datafusion-iceberg の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。