apache/airflow

Deferrable HttpSensor does not move to Triggerer when using `response_check`

Open

#40,209 创建于 2024年6月13日

在 GitHub 查看
 (5 评论) (0 反应) (1 负责人)Python (16,781 fork)batch import
contributors-workshopgood first issuekind:bugprovider:http

仓库指标

Star
 (44,809 star)
PR 合并指标
 (平均合并 7天 18小时) (30 天内合并 834 个 PR)

描述

Discussed in https://github.com/apache/airflow/discussions/39597

Originally posted by bejota May 13, 2024 Hello, I'm using HttpSensor to poll SLURM's REST API for job status. I'm using deferrable mode with response_check. The task only goes to the triggerer if I remove the response_check.

Airflow Docker Image apache/airflow:latest-python3.9 Airflow Version 2.9.0 apache-airflow-providers-http 4.10.0

Does not move to triggerer:

check_slurm_job = HttpSensor(
    task_id="check_slurm_job",
    http_conn_id="slurm_conn_id",
    method="get",
    endpoint="slurm/v0.0.40/job/999",
    response_check=lambda response: "COMPLETED" in response.text,
    poke_interval=10,
    timeout=60 * 60 * 5,
    deferrable=True,  # not deferring to triggerer ?
    dag=dag,
)

Moves to triggerer:

check_slurm_job = HttpSensor(
    task_id="check_slurm_job",
    http_conn_id="slurm_conn_id",
    method="get",
    endpoint="slurm/v0.0.40/job/999",
    # response_check=lambda response: "COMPLETED" in response.text,
    poke_interval=10,
    timeout=60 * 60 * 5,
    deferrable=True,  # not deferring to triggerer ?
    dag=dag,
)

In digging further, I see that HttpTrigger has no response_check method. I will attempt to add one, unless there is a better approach to this problem?

贡献者指南