Replace manual `RowSelection` bitmap conversion with `force_mask()` after arrow-rs release
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Refactor
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- data-engineering
Research direction
After the parquet dependency includes RowSelection::force_mask(), start in datafusion/datasource-parquet/src/access_plan.rs and find the manual selector-to-bitmap conversion in ParquetAccessPlan::scan_and_select. Replace that conversion with the upstream API, remove imports made unused, and run the existing row-selection intersection tests to confirm bitmap backing and selected rows are preserved.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem or challenge?
PR #25731 preserves bitmap-backed Parquet row selections. When intersecting an existing bitmap-backed selection with a selector-backed selection, ParquetAccessPlan::scan_and_select currently builds a bitmap manually using BooleanBufferBuilder.
The upstream API proposed in apache/arrow-rs#11337, RowSelection::force_mask(), encapsulates this conversion. This issue tracks adopting it once DataFusion depends on a released parquet version that includes the API.
Describe the solution you'd like
After upgrading to a parquet version that provides RowSelection::force_mask():
- Replace the manual selector-to-bitmap conversion in
datafusion/datasource-parquet/src/access_plan.rswithselection.force_mask(). - Remove any imports made unused by the change.
- Verify that the existing row-selection intersection tests continue to pass, preserving bitmap backing and selected rows.
Describe alternatives you've considered
Keep the current manual conversion. It preserves the required behavior, but duplicates functionality provided by parquet once the new API is available.
Additional context
- Follow-up to the review discussion in #25731.
- Upstream API: apache/arrow-rs#11337.
- Dominant language
- Rust
- Stars
- 9.4k
- Forks
- 2.5k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 381
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion
-
Lambda examples in the SQL function docs fail with the default dialectPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
apache/datafusion#25960 ·
Maintainers usually reply within 1 day
-
Spark `xxhash64` hashes the raw bits of a NaN instead of the canonical NaNPossibly taken @mrhard9090 claimed this 3 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion#25913 ·
Maintainers usually reply within 1 day
-
trunc(x) on a column turns -0.0 into 0.0Possibly taken @1fanwang claimed this 10 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/datafusion#25702 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
`percentile_cont` truncates Float64 interpolation weights to six decimal placesPossibly taken @aoto-tech claimed this 21 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/datafusion#25266 · 2 comments ·
Maintainers usually reply within 1 day
-
CSV null_regex is applied to schema inference but never to the reader, so matching values are not nullPossibly taken @Developer1010x claimed this 22 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion#25213 · 1 comment ·
Maintainers usually reply within 1 day
All issues in apache/datafusion
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
install: root SSH tmpfiles.d drop-in is labeled etc_runtime_t instead of etc_tPossibly taken @andrewdunndev claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
[Misdetection] `text/tab-separated-values` file misdetected as `text/tsv`Possibly taken @bact claimed this today. Openmisdetection needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
C-bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days