In-memory tables: an in-place write that moves a NaN reads wrongly (the validity bitmap is computed at registration)

Offen
  • Stimmt etwas nicht?
#284 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 3 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python, sql
Bereich
backend, databases

Rechercherichtung

Start by reading the from_dataset docstring and the in-memory table registration path described in the issue, then compare how registration builds the validity bitmap with how shared values change after an in-place write. Reproduce the two NaN-to-number and number-to-NaN cases, including with Context.prepare. Done means the behavior or its documented limitation is consistent and both cases are covered.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

🌀 Claude (xarray-sql)

Summary

A small in-memory Dataset is registered as a native in-memory table that shares its data variables' memory (#275). from_dataset's docstring says that, as with a lazy table, "changing the Dataset's values in place changes what queries return". That holds for numbers, but not for NaN. xarray-sql reads a NaN as SQL NULL, and the in-memory table works out which values are NULL once, when it is registered, as a validity bitmap beside the shared values. After an in-place write:

  • a NaN written where there was a number reads as NaN, not NULL;
  • a number written where there was a NaN reads as NULL.

A lazy table, or the same Dataset registered again, reads the new values as expected.

Reproduction

On #275's branch (in-memory-tables), DataFusion 55.0.0:

import numpy as np
import xarray as xr
import xarray_sql as xql

x = np.array([1.0, np.nan, 3.0])
ctx = xql.Context()
ctx.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
query = "SELECT i, x, x IS NULL AS null FROM t ORDER BY i"
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [1.0, None, 3.0], 'null': [False, True, False]}

x[0] = np.nan
x[1] = 2.0
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [nan, None, 3.0], 'null': [False, True, False]}

again = xql.Context()
again.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
print(again.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [None, 2.0, 3.0], 'null': [True, False, False]}

Options

  • Say so in the docstring: an in-place write must keep its NaNs where they were, or the Dataset must be registered again.
  • Keep NaN as a value in in-memory tables, with no bitmap (#251 asks for that mode in general). That would differ from lazy tables until #251 lands for both.

The same limit applies to Context.prepare, which runs a planned query again on values written in place. It checks that a write keeps its NaNs where they were.

Vorherrschende Sprache
Python
Sterne
142
Forks
19
Ø Merge
3 T. 47 Min.
Gemergte PRs (30 T.)
8

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus xqlsystems/xarray-sql

Alle Issues in xqlsystems/xarray-sql

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.