apache/airflow

OpenLineage provider emits warning about use of fork() on Python 3.12

Open

#47,160 opened on Feb 27, 2025

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (44,809 stars) (16,781 forks)batch import
area:providersgood first issuekind:bugprovider:openlineage

Description

Apache Airflow Provider(s)

openlineage

Versions of Apache Airflow Providers

main

Apache Airflow version

latest

Operating System

null

Deployment

Virtualenv installation

Deployment details

No response

What happened

There is a warning being emitted. Need to investigate alternatives and adjust provider code.

WARNING - /usr/local/lib/python3.12/site-packages/airflow/providers/openlineage/plugins/listener.py:400 DeprecationWarning: This process (pid=682) is multi-threaded, use of fork() may lead to deadlocks in the child.

What you think should happen instead

No response

How to reproduce

Run dag with OpenLineage enabled on python 3.12

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Tech stack
python
Domain
backend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Python 3.12multiprocessing conceptsAirflow provider plugin structure
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
The warning occurs at /usr/local/lib/python3.12/site packages/airflow/providers/openlineage/plugins/listener.py:400. Investigate the use of fork() in that file and consider using multiprocessing spawn or other methods to avoid deadlocks. Check the OpenLineage provider code for compatibility with Python 3.12. The reporter is willing to submit a PR, so any maintainer guidance on preferred approach would be helpful.