Python images can't run default Python UDFs since Spark 4.2 (PyArrow and pandas not installed)
还没有人认领这个 Issue。
评估
调研方向
编辑 r-python.template，在 HAVE_PY 块中添加安装 pyarrow 和 pandas 的 pip 命令行。构建修改后的镜像，并确认普通的 @udf 不再引发 ModuleNotFoundError。
由索引模型根据 Issue 内容生成。
描述
Since Spark 4.2.0, regular Python UDFs are Arrow-optimized by default (SPARK-54555; spark.sql.execution.pythonUDF.arrow.enabled now defaults to true). The python3 images only install python3 and python3-pip (r-python.template), so the Python worker has neither PyArrow nor pandas, and a plain @udf with no options fails on the unmodified image:
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
@F.udf("int")
def plus_one(x): return x + 1
spark.range(3).select(plus_one("id")).collect()
File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 3004, in read_udfs
import pyarrow as pa
ModuleNotFoundError: No module named 'pyarrow'
Server: apache/spark:4.2.0-scala2.13-java21-python3-ubuntu running start-connect-server.sh --wait; client: pyspark-client==4.2.0 on Python 3.10. Results:
| UDF | apache/spark:4.2.0-scala2.13-java21-python3-ubuntu |
same image + pyarrow and pandas |
|---|---|---|
@udf (default) |
ModuleNotFoundError: No module named 'pyarrow' |
OK |
@udf(useArrow=False) |
OK | OK |
@pandas_udf |
ModuleNotFoundError: No module named 'pandas' |
OK |
Before 4.2 a plain @udf didn't need either library, so this only started breaking with the new default. (SPARK-37554 added them to the release-build image, not these.)
Proposal: install both in the HAVE_PY block of r-python.template so 4.2.1 and 4.3.0 pick them up:
PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --no-cache-dir "pyarrow>=18.0.0" "pandas>=2.2.0,<3"; \
The floors are PySpark 4.2's own requirements. PIP_BREAK_SYSTEM_PACKAGES=1 is there because the Ubuntu 26.04 base in #133 refuses a plain system pip install, and <3 because PySpark warns about pandas 3.x; tested on both jammy and resolute. It adds roughly 320 MB to the image.
That said, it's possible my understanding is off, or that there's a better way to handle this (e.g. documenting it instead of growing the image). Open to either. Thank you!
- 主要语言
- Dockerfile
- 星标
- 174
- 派生
- 62
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/spark-docker 的其他 Issue
-
Upgrade base images to Ubuntu 24.04可能已有人在做 @RobbertDM 于 25 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 68/100
apache/spark-docker#132 · 7 条评论 ·
查看 apache/spark-docker 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 85/100
维护者通常 3 天内回复
-
backend dev experience good first issue
难度 2/5 1-3 小时 新手友好度 78/100
cuttle-cards/cuttle#1401 ·
维护者通常 1 天内回复
-
good first issue
难度 2/5 1-3 小时 新手友好度 83/100
btclib-org/btclib-wallet#267 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
langchain-ai/langmem#194 ·