`count()` and `to_arrow()` disagree for `!=` on a nullable identity partition

未关闭
  • 哪里不对？
#4,276 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@hcrosse 已经在做这个了。

开始于 2026年11月3日。

  • #4278 来自 @hcrosse —— 未关闭

评估

难度
4/5
预计耗时
3-5 天
新手友好度
25/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python, sqlite
领域
data

调研方向

The mismatch comes from the residual evaluator in pyiceberg/expressions/visitors.py, which count() in pyiceberg/table/__init__.py relies on, while to_arrow() re-applies the row filter in pyiceberg/io/pyarrow.py. Start by reproducing the snippet and comparing both paths. Choosing whether NULL matches != is an open design question, and PR #4278 is already open against this issue, so check it first. Done means count() equals len(to_arrow()) for the same scan, with tests covering NULL partitions.

由索引模型根据 Issue 内容生成。

描述

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

For a table identity-partitioned on a nullable string column p, filtering with p != 'a' gives scan.count() == 9 but len(scan.to_arrow()) == 4. The table has 3 rows with p = 'a', 4 with p = 'b', and 5 with p = NULL. count() includes the NULL rows, while to_arrow() excludes them. to_arrow_batch_reader() also returns 4 rows.

I reproduced this on PyIceberg 0.12.0 and main at e3564a54c10960307147377c578680276865ed18, with PyArrow 26.0.1 on Linux/amd64.

Install pyiceberg[pyarrow,sql-sqlite]==0.12.0 and pyarrow==26.0.1. The reproduction uses an in-process SQLite catalog and a temporary warehouse:

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import NotEqualTo
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

warehouse = tempfile.mkdtemp()
catalog = SqlCatalog(
    "default", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}"
)
catalog.create_namespace("ns")
schema = Schema(
    NestedField(1, "id", LongType(), required=False),
    NestedField(2, "p", StringType(), required=False),
    NestedField(3, "v", LongType(), required=False),
)
spec = PartitionSpec(
    PartitionField(source_id=2, field_id=1000, transform=IdentityTransform(), name="p")
)
table = catalog.create_table("ns.t", schema=schema, partition_spec=spec)
table.append(
    pa.table(
        {
            "id": list(range(1, 13)),
            "p": ["a"] * 3 + ["b"] * 4 + [None] * 5,
            "v": [1, 2, None, 1, 2, None, 3, 1, None, 2, 3, None],
        },
        schema=pa.schema([("id", pa.int64()), ("p", pa.string()), ("v", pa.int64())]),
    )
)
for field, value in [("p", "a"), ("v", 1)]:
    scan = table.scan(row_filter=NotEqualTo(field, value))
    print(
        f"{field}: count()={scan.count()} to_arrow()={len(scan.to_arrow())} batch_reader={len(scan.to_arrow_batch_reader().read_all())}"
    )

Output on both 0.12.0 and main at e3564a54:

p: count()=9 to_arrow()=4 batch_reader=4
v: count()=5 to_arrow()=5 batch_reader=5

The NULL-partition task has residual=AlwaysTrue() because the residual evaluator compares None != 'a', so count() adds the file's record_count of 5 without reading the file. to_arrow() does not use the residual: it passes the scan's full row filter to ArrowScan and re-applies that filter to each file. Arrow evaluates NULL != 'a' to null, and the scanner drops those rows.

Two null conventions are involved. PyIceberg's row evaluator, Java's Evaluator.notEq, and Java's ResidualEvaluator.notEq treat NULL as matching !=, agreeing with count() here. Java uses a nulls-first string comparator. Arrow treats NULL as not matching, agreeing with to_arrow(). When count() must read a file, it follows Arrow too: for the non-partition column v, both methods return 5 for NotEqualTo("v", 1), while the row evaluator matches 9.

Not(EqualTo("p", "a")) and NotStartsWith("p", "a") also give 9 and 4. A two-value NotIn("p", {"a", "c"}) returns 9 from both methods, because Arrow's ~isin keeps NULL rows. The table in https://github.com/apache/iceberg-python/pull/3918 shows the same difference.

Expected: count() returns len(to_arrow()) for the same scan. Whether NULL rows match != is the choice left open in https://github.com/apache/iceberg-python/pull/3918. That draft PR described the two conventions and their implications for delete() and overwrite(), and it was closed as stale.

Related:

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
主要语言
Python
星标
1.2k
派生
654
平均合并
2 天 11 小时
30 天内合并 PR
122

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/iceberg-python 的其他 Issue

查看 apache/iceberg-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。