In-memory tables: an in-place write that moves a NaN reads wrongly (the validity bitmap is computed at registration)

Abierto
  • ¿Algo no está bien?
#284 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, sql
Área
backend, databases

Línea de trabajo

Start by reading the from_dataset docstring and the in-memory table registration path described in the issue, then compare how registration builds the validity bitmap with how shared values change after an in-place write. Reproduce the two NaN-to-number and number-to-NaN cases, including with Context.prepare. Done means the behavior or its documented limitation is consistent and both cases are covered.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

🌀 Claude (xarray-sql)

Summary

A small in-memory Dataset is registered as a native in-memory table that shares its data variables' memory (#275). from_dataset's docstring says that, as with a lazy table, "changing the Dataset's values in place changes what queries return". That holds for numbers, but not for NaN. xarray-sql reads a NaN as SQL NULL, and the in-memory table works out which values are NULL once, when it is registered, as a validity bitmap beside the shared values. After an in-place write:

  • a NaN written where there was a number reads as NaN, not NULL;
  • a number written where there was a NaN reads as NULL.

A lazy table, or the same Dataset registered again, reads the new values as expected.

Reproduction

On #275's branch (in-memory-tables), DataFusion 55.0.0:

import numpy as np
import xarray as xr
import xarray_sql as xql

x = np.array([1.0, np.nan, 3.0])
ctx = xql.Context()
ctx.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
query = "SELECT i, x, x IS NULL AS null FROM t ORDER BY i"
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [1.0, None, 3.0], 'null': [False, True, False]}

x[0] = np.nan
x[1] = 2.0
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [nan, None, 3.0], 'null': [False, True, False]}

again = xql.Context()
again.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
print(again.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [None, 2.0, 3.0], 'null': [True, False, False]}

Options

  • Say so in the docstring: an in-place write must keep its NaNs where they were, or the Dataset must be registered again.
  • Keep NaN as a value in in-memory tables, with no bitmap (#251 asks for that mode in general). That would differ from lazy tables until #251 lands for both.

The same limit applies to Context.prepare, which runs a planned query again on values written in place. It checks that a write keeps its NaNs where they were.

Lenguaje dominante
Python
Estrellas
142
Forks
19
Merge medio
3 d 47 min
PR fusionados (30 d)
8

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de xqlsystems/xarray-sql

Todos los issues de xqlsystems/xarray-sql

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.