Extract shared list-array construction from the split_* functions in split.rs
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
- Issue type
- Refactor
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- data-engineering
Research direction
Start in native/spark-expr/src/string_funcs/split.rs and compare the tail of the four generic split functions the issue links (regex scalar, split_sql_generic_scalar, split_sql_generic_scalar_array, split_sql_generic_array). Move the shared block that finishes the values builder, picks Utf8 or LargeUtf8 from O::IS_LARGE, builds the item field and creates the GenericListArray into a new finish_split_list helper with the signature the issue proposes. Done when all four functions call that helper and their existing behaviour and tests are unchanged.
Written by the indexing model from the issue text.
Description
What is the problem the feature request solves?
In native/spark-expr/src/string_funcs/split.rs, the four generic split functions end with the same block of code: finish the values builder, pick Utf8/LargeUtf8 from O::IS_LARGE, build the item field, and construct a GenericListArray with the offsets.
- L305-L327 (regex scalar)
- L338-L360 (
split_sql_generic_scalar) - L374-L397 (
split_sql_generic_scalar_array) - L415-L441 (
split_sql_generic_array)
They differ only in the per-row call and in how the null buffer is produced (cloned input nulls vs NullBufferBuilder). A fix to the list construction, such as the item field's nullability, has to be made in four places.
Describe the potential solution
Extract fn finish_split_list<O: OffsetSizeTrait>(offsets: Vec<O>, values: GenericStringBuilder<O>, nulls: Option<NullBuffer>) -> ColumnarValue and call it from all four functions. Optionally, also factor the per-row loop into a generic driver so each variant only supplies its null predicate and push call.
Additional context
This is small and a good first issue. Line references are pinned to commit 7d8d9b1544b7.
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 391
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 421
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion-comet
-
area:shuffle enhancement good first issue performance requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-comet#7072 ·
Maintainers usually reply within 1 day
-
Native hashing of NullType and ANSI interval values fails with "Unsupported data type in hasher"Possibly taken A pull request linked to this issue is open or already merged. Openarea:expressions area:shuffle bug priority:high requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apache/datafusion-comet#7063 ·
Maintainers usually reply within 1 day
-
area:shuffle bug priority:high requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/datafusion-comet#7062 ·
Maintainers usually reply within 1 day
-
requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/datafusion-comet#7001 ·
Maintainers usually reply within 1 day
-
Deduplicate spark_ceil and spark_floor and port floor's decimal boundary tests to ceilPossibly taken @med9110 claimed this 2 days ago. Openarea:expressions enhancement help wanted requires-triage rust tech debt
Difficulty 2/5 Half a day Newbie friendliness 68/100
apache/datafusion-comet#6974 · 1 comment ·
Maintainers usually reply within 1 day
All issues in apache/datafusion-comet
Similar issues
-
Bound wrong-magic P2P diagnostics to parsed header bytesPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
ergoplatform/ergo#2736 ·
Maintainers usually reply within 1 day
-
bug 🐞 CI/CD ⚒️
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
valkey-io/valkey-glide#7370 ·
Maintainers usually reply within 1 day
-
Improve UI text on confirmation page (at registration)Possibly taken @Arghyann claimed this today. Opengood first issue improvement
Difficulty 1/5 Under an hour Newbie friendliness 72/100
lichess-org/lila#22126 · 2 comments ·
Maintainers usually reply within 1 day
-
[VL] Remove the unused ShuffleSchema.hPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 2 days
-
agent-ready area/dev-tooling kind/docs-only layer/docs size/XS
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
marola-dev/marola#855 ·
Maintainers usually reply within 1 day