Pushdown Filter Type BLOOM_FILTER is not currently supported in PyArrow Scans

Open
#463 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Quiet
Tech stack
python
Domain
databases

Research direction

Start at the PyArrow scan path that emits the BLOOM_FILTER NotImplementedException and investigate how join filter pushdown reaches it. The report has no reproducible dataset or code, so first establish a minimal case using DuckDB 1.5.2 with Python 3.11. Done means the supported PyArrow scan no longer raises this error for the reported pushdown scenario, with behavior validated against an appropriate regression case.

Written by the indexing model from the issue text.

Description

needs more info
What happens?

Hi,

I am running into this error message and am unsure what to do.

_duckdb.NotImplementedException: Not implemented Error: Pushdown Filter Type BLOOM_FILTER is not currently supported in PyArrow Scans

It seems I can either disable pushdown filtering through joins, or materialize the query before doing my join. I would rather not do either, since it is a lot of data that I am processing.

Reading the source files with a non-pyarrow reader is unfortunately not an option at this time.

To Reproduce

Unfortunately, I don't really have a reproducer, but I think the maintainers will know what the issue is anyways from the error message. It seems to me that Pyarrow do support bloom filter, but the pushdown is just not implemented?

OS:

Ubuntu 22

DuckDB Package Version:

1.5.2

Python Version:

3.11

Full Name:

Jakob Stigenberg

Affiliation:

Qubos Systematic

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

No - I cannot share the data sets because they are confidential

Did you include all code required to reproduce the issue?
  • Yes, I have
Did you include all relevant configuration to reproduce the issue?
  • Yes, I have
Dominant language
Python
Stars
186
Forks
113
Avg merge
20h 58m
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from duckdb/duckdb-python

All issues in duckdb/duckdb-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.