apache/airflow

Deferrable HttpSensor does not move to Triggerer when using `response_check`

オープン

#40,209 opened on 2024/06/13

 (5 件のコメント) (0 件のリアクション) (1 人の担当者)Python (16,781 件のフォーク)batch import
contributors-workshopgood first issuekind:bugprovider:http

Repository metrics

Stars
 (44,809 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Discussed in https://github.com/apache/airflow/discussions/39597

Originally posted by bejota May 13, 2024 Hello, I'm using HttpSensor to poll SLURM's REST API for job status. I'm using deferrable mode with response_check. The task only goes to the triggerer if I remove the response_check.

Airflow Docker Image apache/airflow:latest-python3.9 Airflow Version 2.9.0 apache-airflow-providers-http 4.10.0

Does not move to triggerer:

check_slurm_job = HttpSensor(
    task_id="check_slurm_job",
    http_conn_id="slurm_conn_id",
    method="get",
    endpoint="slurm/v0.0.40/job/999",
    response_check=lambda response: "COMPLETED" in response.text,
    poke_interval=10,
    timeout=60 * 60 * 5,
    deferrable=True,  # not deferring to triggerer ?
    dag=dag,
)

Moves to triggerer:

check_slurm_job = HttpSensor(
    task_id="check_slurm_job",
    http_conn_id="slurm_conn_id",
    method="get",
    endpoint="slurm/v0.0.40/job/999",
    # response_check=lambda response: "COMPLETED" in response.text,
    poke_interval=10,
    timeout=60 * 60 * 5,
    deferrable=True,  # not deferring to triggerer ?
    dag=dag,
)

In digging further, I see that HttpTrigger has no response_check method. I will attempt to add one, unless there is a better approach to this problem?

コントリビューターガイド