apache/airflow
View on GitHubDeferrable HttpSensor does not move to Triggerer when using `response_check`
Open
#40,209 opened on Jun 13, 2024
contributors-workshopgood first issuekind:bugprovider:http
Repository metrics
- Stars
- (44,809 stars)
- PR merge metrics
- (Avg merge 7d 18h) (834 merged PRs in 30d)
Description
Discussed in https://github.com/apache/airflow/discussions/39597
Originally posted by bejota May 13, 2024
Hello,
I'm using HttpSensor to poll SLURM's REST API for job status. I'm using deferrable mode with response_check. The task only goes to the triggerer if I remove the response_check.
Airflow Docker Image apache/airflow:latest-python3.9
Airflow Version 2.9.0
apache-airflow-providers-http 4.10.0
Does not move to triggerer:
check_slurm_job = HttpSensor(
task_id="check_slurm_job",
http_conn_id="slurm_conn_id",
method="get",
endpoint="slurm/v0.0.40/job/999",
response_check=lambda response: "COMPLETED" in response.text,
poke_interval=10,
timeout=60 * 60 * 5,
deferrable=True, # not deferring to triggerer ?
dag=dag,
)
Moves to triggerer:
check_slurm_job = HttpSensor(
task_id="check_slurm_job",
http_conn_id="slurm_conn_id",
method="get",
endpoint="slurm/v0.0.40/job/999",
# response_check=lambda response: "COMPLETED" in response.text,
poke_interval=10,
timeout=60 * 60 * 5,
deferrable=True, # not deferring to triggerer ?
dag=dag,
)
In digging further, I see that HttpTrigger has no response_check method. I will attempt to add one, unless there is a better approach to this problem?