apache/airflow

Improve ExternalTaskSensor execution date pattern

Open

#10,883 创建于 2020年9月11日

在 GitHub 查看
 (7 评论) (4 反应) (0 负责人)Python (16,781 fork)batch import
good first issuekind:featureprovider:standard

仓库指标

Star
 (44,809 star)
PR 合并指标
 (平均合并 7天 18小时) (30 天内合并 834 个 PR)

描述

Currently in the ExternalTaskSensor there are two ways two set the execution date of a DAG/task the sensor is waiting on:

  • Provide the execution_delta which simply adds/subtracts a time value to/from the execution_date of the sensor itself
  • Provide the execution_date_fn which is a function that takes in just the execution_date and returns a manipulated one.

I believe that the correct way to go about handling the execution_date for the ExternalTaskSensor should be analogous to the way we handle it for other operators, such as the ExternalTaskMarker, or more relevantly, the TriggerDagRunOperator, which is an operator usually used in conjunction with the ExternalTaskSensor.

Specifically, we should remove the execution_delta parameter and replace it with a templated execution_date parameter. This not only covers the existing execution_delta use case through e.g. {{ execution_date - macros.timedelta(days=1) }}, but also provides more flexibility by enabling the benefits of templated parameters in general, such as pulling from xcoms, reading from DAG/operator params, using macros, etc. It also establishes the pattern that we see in the previously mentioned operators.

The proposed solution is also sufficient enough to remove the execution_date_fn because we can register custom macros to manipulate the execution date the same way as an execution_date_fn would. But I do realize we may want to keep it for backwards compatibility reasons.

First time contributor here so happy to implement/help with this to get some airflow OSS chops.

贡献者指南