apache/airflow

Fix handling AirflowFailException `on_retry_callback`

Open

#60172 opened on Jan 6, 2026

View on GitHub
 (11 comments) (0 reactions) (0 assignees)Python (44,809 stars) (16,781 forks)batch import
affected_version:3.1area:coregood first issuekind:bug

Description

Description

This was previously possible in Airflow 2 using this solution. However, in Airflow 3 RuntimeTaskInstanceProtocol does not expose a method for modifying the task state directly.

My suggestion would be to create a new exception type that can be raised from on_retry_callback to request a state change to either success or failure.

Update: While this is not specified, it seema a bug that it does not work that way: AirflowFailException raised in on_retry_callback or any other state-change callback should immediately fail the task.

Use case/motivation

We have some long-running tasks (>24hr) that sometimes fail fast due to transient issues. We want to be able to retry whenever these fast fails occur. However, if a failure occurs hours into the task, we would prefer not to retry, as the task must be completed by a certain time and failures of this sort typically represent an actual issue we want to look into.

Related issues

Just the discussion item here from years ago as far as I can tell.

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
python
Domain
backenddata
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Airflow internalsPythonException handling
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Examine the `RuntimeTaskInstanceProtocol` class in `airflow/models/taskinstance.py` and how `on retry callback` is called. Review the linked discussion for prior solutions. Determine if a new exception type or a change to `AirflowFailException` handling in callbacks is needed. Check existing callback tests to avoid regressions.