apache/airflow

Allow task to depend on specific tasks from previous dag_run

Open

#60328 opened on Jan 9, 2026

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Python (44,809 stars) (16,781 forks)batch import
area:coregood first issuekind:featureneeds-triage

Description

Description

depends_on_past=True currently allows a task to depend only on the same task in the previous dag_run.

There is no native way for a task to depend on the successful completion of multiple specific tasks from the previous dag_run (within the same DAG). Achieving this today requires custom sensors that directly query Airflow metadata tables.

Use case/motivation

In stateful or partially retryable pipelines, it is common to require that a group of tasks in the previous dag_run has completed successfully before continuing execution in the next run.

Example:

A → B → C → D → E

If: dag_run_1 fails at task D Then for dag_run_2: Task C should not start until C, D, and E from dag_run_1 have all succeeded Today: depends_on_past is insufficient (checks only the same task) trigger_rule applies only within the same dag_run Users must implement custom PythonSensors that query DagRun / TaskInstance This adds boilerplate, DB coupling, and operational complexity for a fairly common dependency pattern.

What would you like to happen? Provide a declarative, first-class way for a task to depend on specific tasks from the previous dag_run.

Illustrative example (API shape open for discussion):

C = PythonOperator(
    task_id="C",
    depends_on_past=True,
    depends_on_previous_task_ids=["C", "D", "E"],
)

Semantics:

In dag_run(N), task C is scheduled only if the listed tasks in dag_run(N-1) all finished with success First dag_run is not blocked Backward compatible and fully opt-in This would remove the need for custom sensors while keeping cross-run dependencies explicit and readable.

Related issues

No known existing issue covering this specific capability.

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
python
Domain
backenddata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Airflow scheduler internalsDAG run and TaskInstance modelsPythonSQLAlchemy or Airflow metadata DB
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Study the current implementation of `depends on past` in the scheduler, particularly in `airflow/scheduler/job runner.py` and the `TaskInstance` model. Understand how trigger rules are resolved within a single DAG run and how cross run dependencies are handled. Then design a new parameter `depends on previous task ids` to be stored on the task instance and checked during scheduling. Consider edge cases like first dag run, backfill, and the interaction with existing trigger rules. Look at existing tests in `tests/models/` to understand the testing patterns used for similar features.