apache/airflow

Allow jinja templating connection ids for all third party operators

Aperta

#35.259 aperta il 30 ott 2023

 (4 commenti) (0 reazioni) (0 assegnatari)Python (16.781 fork)batch import
good first issuekind:feature

Metriche repository

Star
 (44.809 stelle)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Description

We use private staging and prod S3s(Ceph clusters for example) in our office. So there are often cases where DAGs are running with only connection ids changed. We prefer to use Param rather than to use hardcoded connection ids to make our code reusable. I only gave an example for Amazon operator, but templating connection ids is required for other operators too.

Why is it needed? Code reusability

Use case/motivation

with DAG(
    dag_id="example_s3",
    params={
        "aws_conn_id": Param("", type="string"),
    },
    ...
) as dag:

    create_object = S3CreateObjectOperator(
        task_id="create_object",
        s3_bucket=bucket_name,
        s3_key=key,
        data=DATA,
        replace=True,
        aws_conn_id="{{ params.aws_conn_id }}", # Params enable us to provide runtime configuration
    )

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Guida contributor