Filter manifest entries on pyiceberg-core Arrow output in plan_files
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 43/100
Research direction
Read _open_manifest in table/__init__.py and the referenced issues #3658, #4007, and apache/iceberg-rust#3649 to understand the current filtering and the dependency on Rust-side decisions. Then check how the optional pyiceberg-core reader exposes Arrow columns and how the Cython reader remains the fallback. Done means filtering Arrow output before constructing DataFile objects, with the existing fallback preserved.
Written by the indexing model from the issue text.
Description
Feature Request / Improvement
_open_manifest (table/__init__.py#L2321-L2334) filters each manifest entry with Python partition and metrics evaluators while decoding (#3658), but every entry still becomes a DataFile before it is filtered. #4007 shows that building those objects costs about as much as the whole Cython decode. To beat Cython, filtering has to happen on the Arrow columns from pyiceberg-core, so that only the matching entries become Python objects.
This needs apache/iceberg-rust#3649, which settles whether the filtering runs vectorized in PyIceberg or in Rust. pyiceberg-core is an optional extra, so the Cython reader stays as the fallback.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 664
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 131
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
StringLiteral to IntegerType/LongType silently truncates fractional strings, changing predicate meaningPossibly taken @Shahzeb-DataEngineer claimed this today. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/iceberg-python#4382 · 2 comments ·
Maintainers usually reply within 1 day
-
`inspect.files()` and the other file metadata tables raise `ArrowInvalid` when a snapshot has no manifestsPossibly taken @YuangGao claimed this 12 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#4269 ·
Maintainers usually reply within 1 day
-
`inspect.entries()` raises TypeError when a data file has no column_sizesPossibly taken @Rahul-cs50 claimed this 16 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#4186 · 2 comments ·
Maintainers usually reply within 1 day
-
`older_than` reads a naive datetime as UTC and expires snapshots it should keepPossibly taken @Rahul-cs50 claimed this 15 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#4182 · 1 comment ·
Maintainers usually reply within 1 day
-
REST: Send empty first-page pageToken when rest-page-size is setPossibly taken @ehddnr301 claimed this 33 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/iceberg-python#4096 ·
Maintainers usually reply within 1 day
All issues in apache/iceberg-python
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
peach-mitao/peach#693 ·
Maintainers usually reply within 1 day
-
FTP reads inside a transaction use a nonexistent staging pathPossibly taken @Harsh23Kashyap claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
fsspec/filesystem_spec#2463 ·
Maintainers usually reply within 1 day
-
comp/cli P2 tool/code-exec type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#155989 ·
Maintainers usually reply within 1 day
-
PipesFileMessageReader parses a partially written message line: tail_file yields a line before its newline arrivesPossibly taken @PCTEJA claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
dagster-io/dagster#34442 ·
Maintainers usually reply within 1 day