Filter manifest lists on pyiceberg-core Arrow output in plan_files
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- data-engineering, performance
Research direction
Start with table/__init__.py around the plan_files manifest filtering at lines 2805–2809, then read issue #4298 for the approach and check the prerequisite apache/iceberg-rust#3649. Compare the Arrow columns returned by pyiceberg-core with the current ManifestFile filtering path; done means matching manifests are filtered before Python objects are built, with Cython still used when the optional extra is unavailable.
Written by the indexing model from the issue text.
Description
Feature Request / Improvement
Scan planning reads the manifest list into ManifestFile objects and then filters them by partition summaries (table/__init__.py#L2805-L2809). #4007 shows that pyiceberg-core parses a manifest list faster than Cython, but building the ManifestFile objects makes the end-to-end path slower.
Apply the approach from #4298 to read_manifest_list: read the list from pyiceberg-core as Arrow, and filter it on the Arrow columns so that only matching manifests become Python objects. This needs apache/iceberg-rust#3649. pyiceberg-core is an optional extra, so the Cython reader stays as the fallback.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 664
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 131
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg-python
-
StringLiteral to IntegerType/LongType silently truncates fractional strings, changing predicate meaningPossibly taken @Shahzeb-DataEngineer claimed this today. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/iceberg-python#4382 · 2 comments ·
Maintainers usually reply within 1 day
-
`inspect.files()` and the other file metadata tables raise `ArrowInvalid` when a snapshot has no manifestsPossibly taken @YuangGao claimed this 12 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#4269 ·
Maintainers usually reply within 1 day
-
`inspect.entries()` raises TypeError when a data file has no column_sizesPossibly taken @Rahul-cs50 claimed this 16 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg-python#4186 · 2 comments ·
Maintainers usually reply within 1 day
-
`older_than` reads a naive datetime as UTC and expires snapshots it should keepPossibly taken @Rahul-cs50 claimed this 15 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/iceberg-python#4182 · 1 comment ·
Maintainers usually reply within 1 day
-
REST: Send empty first-page pageToken when rest-page-size is setPossibly taken @ehddnr301 claimed this 33 days ago. Openkind:bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/iceberg-python#4096 ·
Maintainers usually reply within 1 day
All issues in apache/iceberg-python
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
peach-mitao/peach#693 ·
Maintainers usually reply within 1 day
-
FTP reads inside a transaction use a nonexistent staging pathPossibly taken @Harsh23Kashyap claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
fsspec/filesystem_spec#2463 ·
Maintainers usually reply within 1 day
-
comp/cli P2 tool/code-exec type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#155989 ·
Maintainers usually reply within 1 day
-
PipesFileMessageReader parses a partially written message line: tail_file yields a line before its newline arrivesPossibly taken @PCTEJA claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
dagster-io/dagster#34442 ·
Maintainers usually reply within 1 day