Filter pushdown on float columns drops rows with NaN or -0.0

Abierto
  • ¿Algo no está bien?
#101 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust
Área
backend, databases

Línea de trabajo

Start by tracing scalar_value_to_datum and the scan pushdown path described in the issue, then inspect how InclusiveMetricsEvaluator handles float bounds and predicates. Compare pushed-down results with the MemTable results in the reproduction, including NaN and signed zero cases. Done means the Iceberg table returns the same rows as the MemTable for these filters while retaining safe file pruning where possible.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

Some filters on double and float columns return fewer rows from an Iceberg table than DataFusion returns for the same data. The missing rows are the ones that hold NaN or -0.0. The query succeeds, but those rows aren't in the result.

The two sides compare these values differently:

  • DataFusion treats NaN as equal to itself and greater than any other value, and treats -0.0 as equal to 0.0. SELECT CAST('NaN' AS DOUBLE) > 1.0, CAST('NaN' AS DOUBLE) = CAST('NaN' AS DOUBLE), CAST('-0.0' AS DOUBLE) = 0.0 returns true for all three.
  • iceberg-rust's InclusiveMetricsEvaluator decides that a data file holding only NaN can't match >, >=, = or IN, and NaN is left out of a file's upper bound. iceberg-rust also orders floats with total_cmp, so to it -0.0 and 0.0 are different values.

scalar_value_to_datum passes float literals through as Datum::double, and the comparison reaches the scan unchanged. So the scan skips files that contain rows DataFusion would match. Those files are never read, and filtering again in DataFusion can't bring the rows back.

To Reproduce

Using the sqllogictest engine, with one INSERT per data file:

CREATE TABLE default.default.t (id INT NOT NULL, x DOUBLE);
INSERT INTO default.default.t VALUES (1, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (2, 0.5), (3, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (4, 2.0);
INSERT INTO default.default.t VALUES (5, CAST('-0.0' AS DOUBLE));
INSERT INTO default.default.t VALUES (6, 0.0);
CREATE TABLE m AS SELECT * FROM default.default.t;

SELECT id FROM <table> WHERE <filter> returns:

Filter Iceberg table t MemTable copy m
x > 1.0 4 1, 3, 4
x = CAST('NaN' AS DOUBLE) none 1, 3
x = 0.0 6 5, 6
x >= 0.0 2, 3, 4, 6 1, 2, 3, 4, 5, 6

The same table returns 1, 3, 4 for x + 0 > 1.0, which isn't pushed down.

Which rows go missing also depends on how they're laid out in files. Row 3 is skipped because its file's upper bound is 0.5. The same NaN in a file with 2.0 comes back.

Expected behavior

t returns the same rows as m.

I can think of two ways to get there:

  1. Stop pushing down comparisons and IN lists that have a float literal. That's the simplest, but float columns lose file pruning.
  2. Push a looser predicate that still covers every row DataFusion could match, and leave the exact check to DataFusion. For example, x > c becomes x > c OR x IS NAN and x = NaN becomes x IS NAN. Zero literals would cover both signs: x = 0.0 becomes x = -0.0 OR x = 0.0, x >= 0.0 becomes x >= -0.0, and x <= -0.0 becomes x <= 0.0.

Let me know which you'd prefer.

Additional context
  • <, !=, NOT IN, isnan(x) and NOT isnan(x) return the same rows on both sides.
  • NOT (x <= 1.0) hits this too, since it's simplified to x > 1.0 before pushdown.
  • REAL columns are affected when the literal is also REAL, for example x > CAST(1 AS REAL). With a DOUBLE literal the column gets a cast, and since #91 that comparison isn't pushed down.
  • iceberg-rust main has the same NaN handling and float ordering, so updating the iceberg-rust dependency won't fix this.
  • This is the same kind of problem as #36 and #44: the pushed predicate rejects rows that the filter keeps.
Lenguaje dominante
Rust
Estrellas
33
Forks
34
Merge medio
4 d 14 h
PR fusionados (30 d)
10

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-iceberg

Todos los issues de apache/datafusion-iceberg

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.