apache/airflow

Add `execute_tasks_new_python_interpreter` argument to `PythonOperator`

Open

#41,093 opened on Jul 29, 2024

View on GitHubContributor guide
 (5 comments) (3 reactions) (0 assignees)Python (16,781 forks)batch import
area:coregood first issuekind:featureprovider:standard

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (PR metrics pending)

Description

Description

Currently from my knowledge it is only possible to force all python tasks to execute in a new interpreter for every execution globally via this setting:

Would it be possible to add this argument to the PythonOperator so it is possible to control this behavior at the task level?

Use case/motivation

Related issues

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Research direction
Examine how `execute tasks new python interpreter` is implemented globally and trace the parameter through PythonOperator's execution path to add task level override.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGitApache Airflow basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60