In-memory tables: an in-place write that moves a NaN reads wrongly (the validity bitmap is computed at registration)

Ouverte
  • Quelque chose ne va pas ?
#284 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 3 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
48/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
python, sql
Domaine
backend, databases

Piste de recherche

Start by reading the from_dataset docstring and the in-memory table registration path described in the issue, then compare how registration builds the validity bitmap with how shared values change after an in-place write. Reproduce the two NaN-to-number and number-to-NaN cases, including with Context.prepare. Done means the behavior or its documented limitation is consistent and both cases are covered.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

🌀 Claude (xarray-sql)

Summary

A small in-memory Dataset is registered as a native in-memory table that shares its data variables' memory (#275). from_dataset's docstring says that, as with a lazy table, "changing the Dataset's values in place changes what queries return". That holds for numbers, but not for NaN. xarray-sql reads a NaN as SQL NULL, and the in-memory table works out which values are NULL once, when it is registered, as a validity bitmap beside the shared values. After an in-place write:

  • a NaN written where there was a number reads as NaN, not NULL;
  • a number written where there was a NaN reads as NULL.

A lazy table, or the same Dataset registered again, reads the new values as expected.

Reproduction

On #275's branch (in-memory-tables), DataFusion 55.0.0:

import numpy as np
import xarray as xr
import xarray_sql as xql

x = np.array([1.0, np.nan, 3.0])
ctx = xql.Context()
ctx.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
query = "SELECT i, x, x IS NULL AS null FROM t ORDER BY i"
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [1.0, None, 3.0], 'null': [False, True, False]}

x[0] = np.nan
x[1] = 2.0
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [nan, None, 3.0], 'null': [False, True, False]}

again = xql.Context()
again.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
print(again.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [None, 2.0, 3.0], 'null': [True, False, False]}

Options

  • Say so in the docstring: an in-place write must keep its NaNs where they were, or the Dataset must be registered again.
  • Keep NaN as a value in in-memory tables, with no bitmap (#251 asks for that mode in general). That would differ from lazy tables until #251 lands for both.

The same limit applies to Context.prepare, which runs a planned query again on values written in place. It checks that a write keeps its NaNs where they were.

Langage dominant
Python
Étoiles
142
Forks
19
Merge moyen
3 j 47 min
PR mergées (30 j)
8

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de xqlsystems/xarray-sql

Toutes les issues de xqlsystems/xarray-sql

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.