apache/airflow

cannot use `get_current_context()` in a `@task.virtualenv` task

Open

#34.158 aperta il 7 set 2023

Vedi su GitHub
 (14 commenti) (1 reazione) (0 assegnatari)Python (16.781 fork)batch import
area:providersgood first issuekind:featureprovider:standard

Metriche repository

Star
 (44.809 star)
Metriche merge PR
 (Merge medio 7g 18h) (834 PR mergiate in 30 g)

Descrizione

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

On Airflow 2.5.1 (on AWS MWAA), I ran this DAG:

from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.operators.python import get_current_context

@task.virtualenv(system_site_packages=True)
def test():
    data_interval_end = get_current_context()['data_interval_end']
    print(data_interval_end)

@dag(
    start_date=datetime(2023, 9, 6),
    schedule="10 * * * *",
)
def bug_test():
    test()

the_dag = bug_test()

And I got airflow.exceptions.AirflowException: Current context was requested but no context was found! Are you running within an airflow task?

I know that I can do it like this:

@task.virtualenv(system_site_packages=True)
def test(data_interval_end=None):
    print(data_interval_end)

That works fine if I only need the context directly inside that function, but where this actually popped up in practice was a DAG that used some shared lib functions that used get_current_context, which of course works fine when called from normal tasks but blew up when called from a virtualenv task.

What you think should happen instead

ideally, get_current_context() should work even if it's called from a virtualenv task.

How to reproduce

described above

Operating System

Linux? it's AWS MWAA

Versions of Apache Airflow Providers

No response

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else

this was also mentioned in this closed issue: https://github.com/apache/airflow/issues/20974#issuecomment-1674514640

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Guida contributor