Add on-prem Spark as a launcher · feathr-ai/feathr#449

(0 留言) (0 反應) (0 負責人)Scala (1,929 star) (244 fork)batch import

featurehelp wanted

描述

Currently Feathr is using cloud based spark service (Azure synapse/Databricks) as runtime, and adding on-prem spark should be straightforward by adding an additional launcher here: https://github.com/linkedin/feathr/tree/main/feathr_project/feathr/spark_provider

技術棧: pythonspark
領域: backendinfrastructuredata
議題類型: feature
難度: 3
預計時間: 1-2 days
活動狀態: stale
清晰度: clear
前置要求: PythonApache Spark basicsFamiliarity with Feathr launchers
新手友善度: 60
研究方向: Review existing launchers in https://github.com/linkedin/feathr/tree/main/feathr project/feathr/spark provider to understand the interface. Implement a new launcher class (e.g., 'OnPremSparkLauncher') that initializes a local Spark session using spark submit or programmatic SparkContext. Ensure compatibility with Feathr's job submission flow, and add tests. Check if any existing PRs or comments provide additional guidance.