UDF fails when subexpression elimination is used in interpreted mode
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- scala, spark
- Domain
- data-engineering
Research direction
Start with the UdfTests."multiple two argument udf" case and trace how interpreted mode handles an identical child subexpression that cannot compile, such as ExpressionProxy. Review the targeted fix mentioned in #800 and the related Spark 3.4 change in SPARK-41991. Done means the interpreted UDF path no longer fails and the regression test passes.
Written by the indexing model from the issue text.
Description
As UDF expects compilation always this fails when a child cannot compile, such as ExpressionProxy.
If, in interpreted mode, a child is deemed an identical subexpression (as occurs in the UdfTests."multiple two argument udf" test) spark is able to swap the contents as an optimisation to reduce re-calculation.
In the case where the analyser deems compilation the correct approach this isn't an issue. Targeted fix as part of #800.
Known to occur on Databricks 11.3 and other lower runtimes, fixed as part of 3.4 in SPARK-41991 .
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 7h 38m
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from typelevel/frameless
-
Scala 3 buildsOpenenhancement feature help wanted
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
enhancement feature
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
All issues in typelevel/frameless
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)"Openbug requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-comet#6251 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
ergoplatform/ergo#2579 ·
Maintainers usually reply within 2 days
-
x:action/improve x:size/tiny x:type/content
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day