apache/airflow

Add parameter `deferrable` to `AzureBatchOperator`

Open

#59,779 建立於 2025年12月24日

在 GitHub 查看
 (6 留言) (0 反應) (0 負責人)Python (44,809 star) (16,781 fork)batch import
area:async-operatorsgood first issuekind:feature

描述

Description

The AzureBatchOperator currently has no option to set it to deferrable, which would allow the task to be run by an async low-resource worker ("triggerer"). Therefore Azure Batch tasks block a regular worker, even though they only ping the Azure platform from time to time.

I propose to add this parameter and changing the operator to use asynchronous calls.

As a reference, the AWS Batch operator has a parameter deferrable.

Use case/motivation

Adding this parameter and changing the operator to use asynchronous calls would reduce resource consumption when using Azure Batch.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

貢獻者指南

技術棧
python
領域
backendcloud
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-2 days
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
active
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
Python programmingAirflow basics
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
65
研究方向
Review the current implementation of `AzureBatchOperator` in the Airflow source code, likely located in the `airflow/providers/azure/transfers/` directory. Study the AWS Batch operator's implementation of the `deferrable` parameter and its use of async calls. Examine the triggerer architecture in Airflow documentation. Ensure that adding deferrable support does not break existing synchronous usage.