In-memory tables: an in-place write that moves a NaN reads wrongly (the validity bitmap is computed at registration)

未关闭
  • 哪里不对？
#284 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python, sql
领域
backend, databases

调研方向

Start by reading the from_dataset docstring and the in-memory table registration path described in the issue, then compare how registration builds the validity bitmap with how shared values change after an in-place write. Reproduce the two NaN-to-number and number-to-NaN cases, including with Context.prepare. Done means the behavior or its documented limitation is consistent and both cases are covered.

由索引模型根据 Issue 内容生成。

描述

🌀 Claude (xarray-sql)

Summary

A small in-memory Dataset is registered as a native in-memory table that shares its data variables' memory (#275). from_dataset's docstring says that, as with a lazy table, "changing the Dataset's values in place changes what queries return". That holds for numbers, but not for NaN. xarray-sql reads a NaN as SQL NULL, and the in-memory table works out which values are NULL once, when it is registered, as a validity bitmap beside the shared values. After an in-place write:

  • a NaN written where there was a number reads as NaN, not NULL;
  • a number written where there was a NaN reads as NULL.

A lazy table, or the same Dataset registered again, reads the new values as expected.

Reproduction

On #275's branch (in-memory-tables), DataFusion 55.0.0:

import numpy as np
import xarray as xr
import xarray_sql as xql

x = np.array([1.0, np.nan, 3.0])
ctx = xql.Context()
ctx.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
query = "SELECT i, x, x IS NULL AS null FROM t ORDER BY i"
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [1.0, None, 3.0], 'null': [False, True, False]}

x[0] = np.nan
x[1] = 2.0
print(ctx.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [nan, None, 3.0], 'null': [False, True, False]}

again = xql.Context()
again.from_dataset("t", xr.Dataset({"x": ("i", x)}, coords={"i": np.arange(3)}))
print(again.sql(query).to_pydict())
# {'i': [0, 1, 2], 'x': [None, 2.0, 3.0], 'null': [True, False, False]}

Options

  • Say so in the docstring: an in-place write must keep its NaNs where they were, or the Dataset must be registered again.
  • Keep NaN as a value in in-memory tables, with no bitmap (#251 asks for that mode in general). That would differ from lazy tables until #251 lands for both.

The same limit applies to Context.prepare, which runs a planned query again on values written in place. It checks that a write keeps its NaNs where they were.

主要语言
Python
星标
142
派生
19
平均合并
3 天 47 分钟
30 天内合并 PR
8

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

xqlsystems/xarray-sql 的其他 Issue

查看 xqlsystems/xarray-sql 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。