salesforce/TransmogrifAI

Factor out all common spark/hadoop properties

Open

#87 opened on Aug 24, 2018

View on GitHub
 (8 comments) (0 reactions) (0 assignees)Scala (2,274 stars) (399 forks)batch import
good first issue

Description

Problem We currently largely overlapping spark.gradle files especially in terms of spark properties.

$ git ls-files | grep spark.gradle
gradle/spark.gradle
helloworld/gradle/spark.gradle
templates/simple/spark.gradle

Solution Provide a way to have a single spark.gradle or at least a single spark-transmogrifai.conf file with common properties that is passed via --properties-file to Spark.

Alternatives

  • common properties file
  • refactored spark.gradle

Additional context DRY

Contributor guide

Tech stack
scala
Domain
build system
Issue type
refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
basic knowledge of Gradlefamiliarity with Apache Spark configuration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
Review the three spark.gradle files listed in the issue (gradle/spark.gradle, helloworld/gradle/spark.gradle, templates/simple/spark.gradle) to identify common properties. Propose a consolidated file (e.g., gradle/common spark.gradle) and update the other files to inherit or reference it. Ensure compatibility with the existing build system and test with `./gradlew build`.