apache/airflow

Using .output on non-templated fields

Open

#27,285 建立於 2022年10月26日

在 GitHub 查看
 (6 留言) (0 反應) (0 負責人)Python (16,781 fork)batch import
good first issuekind:feature

倉庫指標

Star
 (44,809 star)
PR 合併指標
 (平均合併 7天 18小時) (30 天內合併 834 個 PR)

描述

Discussed in https://github.com/apache/airflow/discussions/26938

Originally posted by stephenonethree October 7, 2022 I just discovered the .output property functionality that apparently was released in Airflow 2 for classic operators, as a simple way of accessing their output XComs. I think that this is a super useful feature because it would allow simpler connections between tasks than what I have been doing until now.

Until now, I've been explicitly giving a downstream task the task_ids and XCom names that it needs to pull from its upstreams (as hardcoded string parameters). Something like:

# pushes XCom named myvar
upstream_task=SomeOperator(task_id='upstream_task')

# this is a custom operator and within execute() I have a line roughly like:
# actual_input = context['task_instance'].xcom_pull(
#     dag_id=context['dag'].dag_id, task_ids=upstream_task_id, key=upstream_xcom_name)
this_task = OtherOperator(
    task_id='this_task',
    upstream_task_id='upstream_task',
    upstream_xcom_name='myvar'
)
upstream_task >> this_task

With .output I could simplify this to:

# pushes XCom named myvar
upstream_task=SomeOperator(task_id='upstream_task')

this_task = OtherOperator(
    task_id='this_task',
    actual_input=upstream_task.output['myvar']
)
upstream_task >> this_task

Unfortunately it seems that there is one limitation. On the TaskFlow documentation page (https://airflow.apache.org/docs/apache-airflow/2.4.1/tutorial/taskflow.html#consuming-xcoms-between-decorated-and-traditional-tasks) it says: Using the .output property as an input to another task is supported only for operator parameters listed as a template_field.

I don't use Jinja templating for very many of my parameters as it's mostly irrelevant for me. So my questions are, is there a technical reason for this limitation? If not, is this limitation something that you are considering dropping in a future Airflow version? I suppose I could just make these fields templated to get around it, but I don't really want to turn on templating if I don't expect to need it, since I suppose it introduces the possibility of incorrect interpolation (though perhaps that's a remote possibility because I don't think most of my variables will include {{ or }}.

Let me know if I should file this as a feature request instead, for now I guess the Ideas category works.

貢獻者指南