[VL] Consistent Exception Handling With Spark
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- backend, distributed-systems
Research direction
Start by reading the translateException hook in gluten-arrow/src/main/java/org/apache/gluten/vectorized/ColumnarBatchOutIterator.java and the error propagation in cpp/core/jni/JniError.h. Compare the Spark and Velox exception behavior described here, then review the alternatives and user experience tradeoff with maintainers. Done means the project has decided whether and how Gluten should translate Velox exceptions to Spark exceptions.
Written by the indexing model from the issue text.
Description
Description
Hey folks
I was hoping to get feedback on a proposal.
Proposal
Translate Velox generated exception to Spark exceptions to maintain consistent exception semantics with Spark. The necessary hooks have already been provided in translateException
Current State
- Today when we enable new Velox operators in gluten we propagate the Velox exception message via a gluten exception
- The Velox exception can be different than that of Spark. For example:
Spark JVM:
"Exceeds char/varchar type length limitation: 5"
Velox:
"Exceeds allowed length limitation: 5"
Why It's an Issue
Let me explain using an example. In #12976 a new Velox operator was enabled in Gluten:
- Enabling the operator was 1 line of change (remove if from the
kBlackListarray:
const std::unordered_set<std::string> kBlackList = {"split_part", "sequence", "approx_percentile", "map_from_arrays"};
- However after enabling the operator it starts to raise a different type of exception, causing the spark tests that assert the exception to fail. This caused 20+ sparks tests that relied on the operator to fail which required exclusion such as:
...
.excludeGlutenTest("NOT NULL checks for nullable map with required values (byName)")
.excludeGlutenTest("NOT NULL checks for nullable map with required values (byPosition)")
.excludeGlutenTest("NOT NULL checks for fields inside nullable maps (byPosition)")
...
To avoid losing test coverage spark tests were copied over adding 800 lines of test code (duplicate code with spark tests with a few lines different during exception assertions).
User Experience
This would change user experience though which needs a decision on the repo. User would no longer be receiving a gluten specific error message and the experience would be transparent regardless of which execution engine is running spark. A decision that the repo would need to make if its the correct experience.
Alternative Solution, Option 2
Regarding reason2 we could rely on Apache spark to enable the exception assertions in their tests to be extensible such as assertLengthCheckFailure.
The challenge with this is:
- The spark team seems reluctant of adding these extensions (please see comment)
- The extensions would only help with spark versions going forward and previous spark versions would still have the code duplication
Alternative Solution, Option 3
We rely on Velox to throw exceptions consistent with Spark. My understanding is that Velox does strive to be agnostic to the specific data processing system that uses it so incorporating these changes there might prove challenging.
Gluten version
main branch
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day