apache/airflow

AIP-82: Add common messaging system providers

Open

#52,712 opened on Jul 2, 2025

View on GitHub
 (25 comments) (2 reactions) (0 assignees)Python (44,809 stars) (16,781 forks)batch import
AIP-82good first issuekind:featurekind:metaprovider:amazonprovider:googleprovider:microsoft-azureprovider:redis

Description

Body

related: #49938

Why

Several users have pointed out that the Event-Driven Scheduling feature introduced in Airflow 3 currently supports only Amazon SQS.

So I add Kafka common message queue provider in #49938, but there are still quite a few common providers that should be useful for airflow users to implement Event-Driven workflows.

What

Here are some common messaging systems:

reference for implementation: https://github.com/apache/airflow/pull/49938/files

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Tech stack
python
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Familiarity with Apache AirflowPython programmingUnderstanding of message queues
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
This issue is a meta issue listing common messaging system providers to implement for the Event Driven Scheduling feature. Several providers already have PRs: Apache Kafka (#49938), Azure Message Bus (#53356), Google PubSub (#54494), Redis (#52917). To add a new provider like Amazon Kinesis or Amazon MSK, follow the pattern in the reference PR #49938. Examine the provider structure in `airflow/providers/` and implement the `CommonMessageQueue` interface. Ensure to add tests and documentation. Comment on the issue to indicate which provider you are working on to avoid duplication.
AIP-82: Add common messaging system providers · apache/airflow#52712 | Good First Issue