feathr-ai/feathr

Add on-prem Spark as a launcher

Open

#449 创建于 2022年7月9日

在 GitHub 查看
 (0 评论) (0 反应) (0 负责人)Scala (1,929 star) (244 fork)batch import
featurehelp wanted

描述

Currently Feathr is using cloud based spark service (Azure synapse/Databricks) as runtime, and adding on-prem spark should be straightforward by adding an additional launcher here: https://github.com/linkedin/feathr/tree/main/feathr_project/feathr/spark_provider

贡献者指南

技术栈
pythonspark
领域
backendinfrastructuredata
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
1-2 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
clear
前置要求
PythonApache Spark basicsFamiliarity with Feathr launchers
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
60
研究方向
Review existing launchers in https://github.com/linkedin/feathr/tree/main/feathr project/feathr/spark provider to understand the interface. Implement a new launcher class (e.g., 'OnPremSparkLauncher') that initializes a local Spark session using spark submit or programmatic SparkContext. Ensure compatibility with Feathr's job submission flow, and add tests. Check if any existing PRs or comments provide additional guidance.