salesforce/TransmogrifAI

Factor out all common spark/hadoop properties

オープン

#87 opened on 2018/08/24

 (8 件のコメント) (0 件のリアクション) (0 人の担当者)Scala (399 件のフォーク)batch import
good first issue

Repository metrics

Stars
 (2,274 個のスター)
PR merge metrics
 (30d に merged PR はありません)

説明

Problem We currently largely overlapping spark.gradle files especially in terms of spark properties.

$ git ls-files | grep spark.gradle
gradle/spark.gradle
helloworld/gradle/spark.gradle
templates/simple/spark.gradle

Solution Provide a way to have a single spark.gradle or at least a single spark-transmogrifai.conf file with common properties that is passed via --properties-file to Spark.

Alternatives

  • common properties file
  • refactored spark.gradle

Additional context DRY

コントリビューターガイド