apache/kyuubi

[Umbrella] Spark Configuration Validation

Open

#832 创建于 2021年7月19日

在 GitHub 查看
 (1 评论) (1 反应) (0 负责人)Scala (996 fork)batch import
good first issuehelp wantedkind:feature

仓库指标

Star
 (2,332 star)
PR 合并指标
 (30 天内没有已合并 PR)

描述

Describe the proposal

When a SparkConf is invalid for launching our engines, it is better to fail fast instead of waiting for Spark to check while engine bootstrapping.

Motivation

  1. Save time to create an invalid engine
  2. Avoid possible hang
  3. better error message

Here is an example I did for Spark to validate its own configs when talking to Kubernetes. https://github.com/apache/spark/pull/32610

Task list

Generally speaking, we can have 1) a tool in kyuubi-ctl to check server/engine conf in configuration files are valid, 2) The logic of the tool can also be called when engine bootstrapping for the configurations from the client side

  • #833
  • #1018

贡献者指南