Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Feature request: cloudpickle.patch_multiprocessing() utility for ForkingPickler replacement

未关闭
  • 哪里不对？
#589 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 16 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
停滞
技术栈
python
领域
backend

调研方向

首先检查 issue 中描述的 multiprocessing.reduction 和 multiprocessing.connection 入口点，以及 cloudpickle 现有的 Pickler、dumps 和 loads API。在三个绑定点以及使用 lambda 的 Pool.map 中验证所提议的 helper，包括幂等应用和预期的序列化结果。

由索引模型根据 Issue 内容生成。

描述

Summary

Propose adding a cloudpickle.patch_multiprocessing() helper that replaces multiprocessing.reduction.ForkingPickler with a cloudpickle-based pickler, enabling Pool.map(lambda x: x**2, range(10)) to work out of the box.

Motivation: ecosystem fragmentation

Every project that needs cloudpickle + multiprocessing.Pool independently reinvents this patching. At least 6 projects maintain their own version:

Project Approach
loky/joblib Full custom _LokyPickler subsystem in loky/backend/reduction.py
PySpark Own CloudPickleSerializer wrapping cloudpickle.dumps/loads
Ray Bundled fork as ray.cloudpickle with custom object store
Dask Custom serialization protocol in distributed scheduler
multiprocess Complete fork of CPython's multiprocessing with dill substituted
trading-strategy/exec-sandbox/pypeln/pyrocko Ad-hoc monkey patches of varying correctness

Most ad-hoc implementations are incomplete because of a non-obvious CPython pitfall (see below).

The _ForkingPickler double-binding pitfall

CPython has two separate name bindings for ForkingPickler:

# multiprocessing/reduction.py
class ForkingPickler(pickle.Pickler):
    ...

# multiprocessing/connection.py
from .context import reduction
_ForkingPickler = reduction.ForkingPickler   # captured at import time

class Connection:
    def send(self, obj):
        self._send_bytes(_ForkingPickler.dumps(obj))  # uses the captured reference

Patching reduction.ForkingPickler alone is insufficient — Connection.send() still uses the stale _ForkingPickler reference captured at import time. You must also patch multiprocessing.connection._ForkingPickler. Most ad-hoc implementations miss this.

Additionally, reduction.dump() is a module-level function that also needs replacing for completeness.

Proposed API

import cloudpickle

cloudpickle.patch_multiprocessing()

One call, idempotent, patches all three binding sites:

  1. multiprocessing.reduction.ForkingPickler — the class
  2. multiprocessing.reduction.dump — the module-level helper
  3. multiprocessing.connection._ForkingPickler — the import-time captured reference

Reference implementation

Here's a minimal working implementation (tested on Python 3.14):

import copyreg
import io
import multiprocessing.connection
import multiprocessing.reduction

import cloudpickle


class CloudForkingPickler(cloudpickle.Pickler):
    """ForkingPickler replacement backed by cloudpickle."""
    _extra_reducers = {}
    _copyreg_dispatch_table = copyreg.dispatch_table

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.dispatch_table = self._copyreg_dispatch_table.copy()
        self.dispatch_table.update(self._extra_reducers)

    @classmethod
    def register(cls, type, reduce):
        cls._extra_reducers[type] = reduce

    @classmethod
    def dumps(cls, obj, protocol=None):
        buf = io.BytesIO()
        cls(buf, protocol).dump(obj)
        return buf.getbuffer()

    loads = staticmethod(cloudpickle.loads)


def patch_multiprocessing():
    """Replace multiprocessing's ForkingPickler with cloudpickle-based version."""
    # 1. The class itself
    multiprocessing.reduction.ForkingPickler = CloudForkingPickler
    # 2. The module-level dump() helper
    multiprocessing.reduction.dump = lambda obj, file, protocol=None: \
        CloudForkingPickler(file, protocol).dump(obj)
    # 3. The import-time captured reference in connection.py
    multiprocessing.connection._ForkingPickler = CloudForkingPickler

After patch_multiprocessing():

from multiprocessing import Pool
with Pool(4) as p:
    print(p.map(lambda x: x**2, range(10)))
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Why cloudpickle (not CPython)

There's an open discussion on discuss.python.org about adding a pluggable pickler API to multiprocessing, but no PEP has materialized. cloudpickle is the pragmatic place for this — it already provides Pickler/dumps/loads, and adding a one-shot integration helper is a small, natural extension.

Alternatives considered

  • "Just use loky/joblib" — Valid for many users, but loky replaces the entire process management layer. Many projects only need cloudpickle serialization with stdlib multiprocessing.Pool.
  • "Just use multiprocess (dill)" — Requires replacing all multiprocessing imports. dill is heavier than cloudpickle and has different serialization semantics.
  • "Document the pattern instead" — The _ForkingPickler double-binding makes documentation insufficient; people will keep getting it wrong.

Happy to submit a PR if there's interest.

主要语言
Python
星标
1.9k
派生
206
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

cloudpipe/cloudpickle 的其他 Issue

查看 cloudpipe/cloudpickle 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。