pola-rs/polars

scan_delta file skip predicate not working for bool dtype

Open

#26,290 opened on 2026年1月26日

GitHub で見る
 (7 comments) (1 reaction) (0 assignees)Rust (38,496 stars) (2,826 forks)batch import
A-io-deltaA-io-parquetP-lowenhancementgood first issueperformancepythonupstream issue

説明

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

tmp_path = "./tmp"
df = pl.DataFrame(
    {
        "p": [10, 10, 20, 20],
        "a": [1, 2, 3, None],
        "b": [False, False, True, None]
    }
)

df.write_delta(
    tmp_path,
    delta_write_options={"partition_by": "p"},
)

expr = pl.col.a.is_null()
out = pl.scan_delta(tmp_path).filter(expr).collect()

# filter works
expr = pl.col.b.is_null()
out = pl.scan_delta(tmp_path).filter(expr).collect()

# filter does not work
expr = pl.col.b == pl.lit(False)
out = pl.scan_delta(tmp_path).filter(expr).collect()

Log output

$ POLARS_VERBOSE=1 pp issue_bool_mre.py 2>&1  | grep skipping
initialize_scan_predicate: Predicate pushdown allows skipping 1 / 2 files
initialize_scan_predicate: Predicate pushdown allows skipping 1 / 2 files
initialize_scan_predicate: Predicate pushdown allows skipping 0 / 2 files

Issue description

Equality on booleans is not supported in predicate pushdown of delta files.

Expected behavior

No clear reason why it would not be supported. Snapshot of the delta json file with statistics:

{"add":{"path":"p=10/part-00000-059a7e15-37b4-450c-acf9-c52cb10d4c59-c000.snappy.parquet","partitionValues":{"p":"10"},"size":688,"modificationTime":1769434619583,"dataChange":true,"stats":"{\"numRecords\":2,\"minValues\":{\"a\":1,\"b\":false},\"maxValues\":{\"b\":false,\"a\":2},\"nullCount\":{\"a\":0,\"b\":0}}","tags":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}

{"add":{"path":"p=20/part-00000-d0d5a29b-6f86-44d5-ae3a-794d33c73da2-c000.snappy.parquet","partitionValues":{"p":"20"},"size":677,"modificationTime":1769434619583,"dataChange":true,"stats":"{\"numRecords\":2,\"minValues\":{\"b\":true,\"a\":3},\"maxValues\":{\"b\":true,\"a\":3},\"nullCount\":{\"b\":1,\"a\":1}}","tags":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}

To be confirmed.

Installed versions

Latest main (803b8e4cb1), post 1.37.1

コントリビューターガイド

技術スタック
pythonrust
領域
backenddata
Issue 種別
bug
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
3
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
half day
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
active
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
clear
前提条件
Polars Python APIRust programmingDelta lake format
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
35
調査方針
Investigate the predicate pushdown logic in Polars' Delta scan implementation. Look for the part that handles comparison operators for different data types when determining file skipping. The issue indicates that boolean equality is not implemented. Check the Rust source in the delta related modules (e.g., polars sql/src/delta.rs or polars core/src/optimizations/predicate pushdown/delta.rs). Consider adding support for Bool type in the predicate evaluation. Also ensure that min/max statistics from the Delta log are correctly interpreted for boolean columns.
scan_delta file skip predicate not working for bool dtype · pola-rs/polars#26290 | Good First Issue