Decode Python UDFs opaquely so a scheduler needs no Python interpreter
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
调研方向
首先阅读 crates/core/src/codec.rs 以及 #1678 中的扩展 codec 工作,以了解 DFPYUDF wire format 和 codec 组合方式。围绕现有 payload 设计 scheduler 侧的 ScalarUDFImpl 和 codec,在重新编码时保留 cloudpickle 字节,同时暴露可恢复的元数据,并让 invoke 返回错误。检查 #1703 中的依赖上下文。
由索引模型根据 Issue 内容生成。
描述
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In a distributed setup the scheduler plans a query and hands stages to executors; only the executors ever call a Python UDF. Decoding an inlined Python UDF unpickles the function, which requires a Python interpreter and every module the function closes over to be importable. Doing that on the scheduler costs work nobody needs and forces the scheduler image to carry Python and the full dependency set of user code it will never run. Raised in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-5100366976.
Describe the solution you'd like
An opaque decode path: a ScalarUDFImpl that holds the still-pickled blob rather than a live Python object, and a codec that produces it. A scheduler installs that codec, decodes a plan into something it can inspect, route, and re-encode, and never touches cloudpickle. The executor installs the ordinary codec and unpickles as it does now.
The wire format already allows this. An inlined UDF payload is DFPYUDF followed by a version byte and the cloudpickle blob (crates/core/src/codec.rs), so an opaque holder can carry those bytes verbatim and no format change is needed.
The part that needs design is re-encoding. A scheduler that forwards a stage has to emit the blob byte-identically, so the executor sees exactly what the client wrote. That also raises what such a UDF should report for the things DataFusion asks of a ScalarUDFImpl during planning — name, signature, and return type are all recoverable from the payload without unpickling, since they are stored alongside the function, but invoke has to be an error rather than a surprise.
Describe alternatives you've considered
Encoding Python UDFs by name only and registering them on every node. Already supported and appropriate when the function is available everywhere; it does not cover the case inlining exists for, which is a function the receiving process does not have.
Having the scheduler unpickle and immediately drop the object. Keeps the code simple, and still requires Python plus all user dependencies on the scheduler, which is the actual cost being avoided.
Additional context
Follow-up from #1678, which made extension codecs compose so a setup like this can install a scheduler-side codec alongside others. Likely also depends on #1703, gating pyo3/extension-module, if the consumer is a Rust crate rather than a Python process.
- 主要语言
- Python
- 星标
- 605
- 派生
- 176
- 平均合并
- 1 天 23 小时
- 30 天内合并 PR
- 8
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/datafusion-python 的其他 Issue
-
documentation
难度 2/5 1-3 小时 新手友好度 72/100
apache/datafusion-python#1726 ·
-
难度 2/5 半天 新手友好度 88/100
apache/datafusion-python#1691 ·
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-python#1644 ·
-
enhancement
难度 5/5 一周以上 新手友好度 30/100
apache/datafusion-python#1737 ·
-
难度 3/5 1-2 天 新手友好度 76/100
apache/datafusion-python#1735 · 1 条评论 ·
查看 apache/datafusion-python 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 90/100
learningequality/ricecooker#747 ·
-
难度 2/5 1-3 小时 新手友好度 68/100
BSData/horus-heresy-3rd-edition#3171 ·
-
enhancement
难度 2/5 1-3 小时 新手友好度 72/100
-
难度 2/5 1-3 小时 新手友好度 76/100
run-llama/llama_index#23199 ·
-
难度 2/5 1-3 小时 新手友好度 84/100
KhronosGroup/glTF-Blender-IO#2769 ·