[VL] Consistent Exception Handling With Spark

オープン
#13,647 コメント 7 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
java, scala, spark

調査の方向性

Start by reading the translateException hook in gluten-arrow/src/main/java/org/apache/gluten/vectorized/ColumnarBatchOutIterator.java and the error propagation in cpp/core/jni/JniError.h. Compare the Spark and Velox exception behavior described here, then review the alternatives and user experience tradeoff with maintainers. Done means the project has decided whether and how Gluten should translate Velox exceptions to Spark exceptions.

索引モデルが issue の本文から書いたものです。

説明

enhancement
Description

Hey folks

I was hoping to get feedback on a proposal.

Proposal

Translate Velox generated exception to Spark exceptions to maintain consistent exception semantics with Spark. The necessary hooks have already been provided in translateException


Current State
  • Today when we enable new Velox operators in gluten we propagate the Velox exception message via a gluten exception
  • The Velox exception can be different than that of Spark. For example:

Spark JVM:

"Exceeds char/varchar type length limitation: 5"

Velox:

"Exceeds allowed length limitation: 5"

Why It's an Issue

Let me explain using an example. In #12976 a new Velox operator was enabled in Gluten:

  • Enabling the operator was 1 line of change (remove if from the kBlackList array:
const std::unordered_set<std::string> kBlackList = {"split_part", "sequence", "approx_percentile", "map_from_arrays"};
  • However after enabling the operator it starts to raise a different type of exception, causing the spark tests that assert the exception to fail. This caused 20+ sparks tests that relied on the operator to fail which required exclusion such as:
...
.excludeGlutenTest("NOT NULL checks for nullable map with required values (byName)")
.excludeGlutenTest("NOT NULL checks for nullable map with required values (byPosition)")
.excludeGlutenTest("NOT NULL checks for fields inside nullable maps (byPosition)")
...

To avoid losing test coverage spark tests were copied over adding 800 lines of test code (duplicate code with spark tests with a few lines different during exception assertions).


User Experience

This would change user experience though which needs a decision on the repo. User would no longer be receiving a gluten specific error message and the experience would be transparent regardless of which execution engine is running spark. A decision that the repo would need to make if its the correct experience.


Alternative Solution, Option 2

Regarding reason2 we could rely on Apache spark to enable the exception assertions in their tests to be extensible such as assertLengthCheckFailure.

The challenge with this is:

  • The spark team seems reluctant of adding these extensions (please see comment)
  • The extensions would only help with spark versions going forward and previous spark versions would still have the code duplication

Alternative Solution, Option 3

We rely on Velox to throw exceptions consistent with Spark. My understanding is that Velox does strive to be agnostic to the specific data processing system that uses it so incorporating these changes there might prove challenging.

Gluten version

main branch

主要言語
Scala
スター
1.6k
フォーク
685
平均マージ
4日 16時間
マージ済み PR(30日)
84

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/gluten のほかの issue

apache/gluten の issue をすべて見る

似ている issue

Scala の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。