apache/airflow

Airflow Databricks Operator Task Group Launch Task Not Properly Waiting for Upstream Tasks

Open

#51,598 opened on Jun 10, 2025

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (44,809 stars) (16,781 forks)batch import
area:providersgood first issuekind:bugprovider:databricks

Description

Apache Airflow Provider(s)

databricks

Versions of Apache Airflow Providers

7.3.2

Apache Airflow version

2.10.3

Operating System

managed airflow

Deployment

Docker-Compose

Deployment details

No response

What happened

When using DatabricksWorkflowTaskGroups, a launch task is created. This launch task can be set to have upstream dependencies, however no matter what the launch task will not wait on any dependencies to execute, executing as soon as the dag run starts. This keeps us from using any parameters defined as part of a task that is intended to run before the task group initializes, for example setting parameters in the task group definition.

What you think should happen instead

This task should instead wait on prior tasks to finish as aligned with the trigger_rule assigned to that specific task. As the trigger_rule by default for the launch task is set to all_success, the task should NOT run until all upstream dependencies have finished.

How to reproduce

from airflow.operators.dummy import DummyOperator
from airflow.providers.databricks.operators.databricks_workflow import (
    DatabricksWorkflowTaskGroup,
)
with DAG(
    dag_id="example_dag",
    schedule_interval=f"0 0 * * *",
) as dag:
    start_task = DummyOperator(task_id="end", trigger_rule="none_failed")

    task_group = DatabricksWorkflowTaskGroup(
        group_id="tg",
        databricks_conn_id="FILL IN DBX CONNECTION ID",
        job_clusters={
            "job_cluster_key": "test_cluster",
            "new_cluster": {
                "cluster_name": "",
                "spark_version": "15.4.x-scala2.12",
                "node_type_id": "r4.2xlarge",
                "num_workers": 1,
                "autoscale": {"min_workers": 2, "max_workers": 8},
                "aws_attributes": {
                    "availability": "SPOT_WITH_FALLBACK",
                    "ebs_volume_count": 1,
                    "ebs_volume_size": 100,
                    "ebs_volume_type": "GENERAL_PURPOSE_SSD",
                    "first_on_demand": 1,
                    "spot_bid_price_percent": 100,
                    "zone_id": "auto",
                },
            },
        },
    )

    with task_group:
        pass

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
python
Domain
backend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Basic understanding of Airflow DAGsDatabricks provider basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Investigate the DatabricksWorkflowTaskGroup implementation in airflow/providers/databricks/operators/databricks workflow.py. The launch task's trigger rule should be respected but isn't. Look for how upstream tasks are set and how the launch task's execution is controlled. Check if the issue is in the TaskGroup's internal task creation logic or in how dependencies are translated to Databricks jobs.