Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

`@task.pyspark` does not template `config_kwargs` and `conn_id`

Open Beginner friendly
  • Is something wrong?
#75,762 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@sdaberdaku is already working on this.

Since Oct 27, 2026.

  • #75764 by @sdaberdaku — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
79/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, data-engineering

Research direction

Start with _PySparkDecoratedOperator and its template_fields, then compare them with PySparkOperator.template_fields and the Kubernetes decorated operator pattern mentioned in the issue. Reproduce the example with render_template_fields and confirm that conn_id and config_kwargs render for the decorated operator as they do for the classic operator.

Written by the indexing model from the issue text.

Description

Under which category would you file this issue?

Providers

Apache Airflow version

3.3.2

What happened and how to reproduce it?

@task.pyspark does not render Jinja in config_kwargs (or conn_id). The same arguments passed to PySparkOperator are rendered.

PySparkOperator declares both as template fields:

PySparkOperator.template_fields == ("conn_id", "config_kwargs", "templates_dict", "op_args", "op_kwargs")

_PySparkDecoratedOperator does not define template_fields. DecoratedOperator comes before PySparkOperator in its MRO, so it inherits DecoratedOperator's fields and drops the two Spark-specific ones:

_PySparkDecoratedOperator.template_fields == ("op_args", "op_kwargs")

Templated values in config_kwargs therefore reach Spark as literal text when the decorator is used.

Reproduction:

from airflow.providers.apache.spark.operators.spark_pyspark import PySparkOperator
from airflow.sdk import DAG, task

conf = {"spark.app.name": "app-{{ run_id }}"}

with DAG("repro", schedule=None):

    @task.pyspark(config_kwargs=conf)
    def decorated(spark=None):
        pass

    decorated_op = decorated().operator
    classic_op = PySparkOperator(task_id="classic", python_callable=lambda spark=None: None, config_kwargs=conf)

context = {"run_id": "manual__1"}
for op in (decorated_op, classic_op):
    op.render_template_fields(context)
    print(type(op).__name__, op.config_kwargs["spark.app.name"])

Output:

_PySparkDecoratedOperator app-{{ run_id }}
PySparkOperator app-manual__1
What you think should happen instead?

@task.pyspark should template conn_id and config_kwargs exactly like PySparkOperator. The decorated operator can declare the union of both sets of fields, the way _KubernetesDecoratedOperator keeps the fields of KubernetesPodOperator:

template_fields: Sequence[str] = (*DecoratedOperator.template_fields, "conn_id", "config_kwargs")
Operating System

Linux

Deployment

Virtualenv installation

Apache Airflow Provider(s)

apache-spark

Versions of Apache Airflow Providers

apache-airflow-providers-apache-spark==6.3.4
apache-airflow-providers-common-compat==1.23.0

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

The decorator never templated these fields: before PySparkOperator was introduced it declared template_fields = ("op_args", "op_kwargs") explicitly. The operator refactor made the inconsistency visible, since the same arguments now behave differently on the two entry points.

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Dominant language
Python
Stars
47.3k
Forks
18.1k
Avg merge
18h 16m
Merged PRs (30d)
647

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/airflow

All issues in apache/airflow

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.