apache/airflow
Vedi su GitHubAllow jinja templating connection ids for all third party operators
Open
#35.259 aperta il 30 ott 2023
good first issuekind:feature
Metriche repository
- Star
- (44.809 star)
- Metriche merge PR
- (Merge medio 7g 18h) (834 PR mergiate in 30 g)
Descrizione
Description
We use private staging and prod S3s(Ceph clusters for example) in our office. So there are often cases where DAGs are running with only connection ids changed. We prefer to use Param rather than to use hardcoded connection ids to make our code reusable. I only gave an example for Amazon operator, but templating connection ids is required for other operators too.
Why is it needed? Code reusability
Use case/motivation
with DAG(
dag_id="example_s3",
params={
"aws_conn_id": Param("", type="string"),
},
...
) as dag:
create_object = S3CreateObjectOperator(
task_id="create_object",
s3_bucket=bucket_name,
s3_key=key,
data=DATA,
replace=True,
aws_conn_id="{{ params.aws_conn_id }}", # Params enable us to provide runtime configuration
)
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct