apache/kyuubi

[Umbrella] Spark Configuration Validation

Open

#832 opened on 2021年7月19日

GitHub で見る
 (1 comment) (1 reaction) (0 assignees)Scala (996 forks)batch import
good first issuehelp wantedkind:feature

Repository metrics

Stars
 (2,332 stars)
PR merge metrics
 (30d に merged PR はありません)

説明

Describe the proposal

When a SparkConf is invalid for launching our engines, it is better to fail fast instead of waiting for Spark to check while engine bootstrapping.

Motivation

  1. Save time to create an invalid engine
  2. Avoid possible hang
  3. better error message

Here is an example I did for Spark to validate its own configs when talking to Kubernetes. https://github.com/apache/spark/pull/32610

Task list

Generally speaking, we can have 1) a tool in kyuubi-ctl to check server/engine conf in configuration files are valid, 2) The logic of the tool can also be called when engine bootstrapping for the configurations from the client side

  • #833
  • #1018

コントリビューターガイド