apache/airflow

SparkKubernetesOperator to support arguments

Open

#16,728 opened on Jun 30, 2021

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Python (16,781 forks)batch import
area:providersgood first issuekind:documentationkind:featureprovider:cncf-kubernetes

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Need an option ,when triggering a dag via airflow web ui, to pass JSON arguments to SparkKubernetesOperator.

Currently I don't see a way to do it as, SparkKubernetesOperator accepts application.yaml only.

Contributor guide

Research direction
Investigate how SparkKubernetesOperator currently parses the application.yaml and explore ways to accept additional JSON arguments. Look at existing operators that accept parameters via UI like the KubernetesPodOperator. Consider extending the operator to accept a 'arguments' parameter and rendering it in the YAML template.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
AirflowPython
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60