cloudpipe/cloudpickle

Feature request: support for pickling/unpickling interactively defined Cython functions and classes

Open

#502 opened on Feb 16, 2023

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (1,407 stars) (152 forks)batch import
help wanted

Description

It would be nice to be able to run interactively defined Cython functions in parallel on dask/ray/spark clusters or joblib/loky process pools.

Such interactively defined functions can typically be created using the %%cython magic in a Jupyter session or by a pyximport call in a Python script (e.g. from the __main__ module).

In both cases, it quite easy to interactively inspect the callable from Python to identify that their definition come from such an interactively defined Cython module.

A first step would be to add a new assume_shared_dynamic_cython_module_folder=False constructor parameter to the cloudpickle.Pickler class, in which case we could not need to ship the backing .so file over the wire (nor rebuild it from a source code sent over the wire).

This assumption would hold for the single host parallelization calls (e.g. joblib/loky or single host ray/dask setups) or for cluster computing setups with shared home folders between the driver node and the worker nodes (e.g. NFS home folder in jupyter hub + HPC cluster).

Contributor guide

Tech stack
python
Domain
backenddata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
CythonPython picklingdistributed computing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The issue suggests adding an `assume shared dynamic cython module folder` parameter to the cloudpickle.Pickler class. Research should focus on understanding how cloudpickle currently handles Cython generated .so files and the existing logic for serializing interactively defined Cython functions. Key files to examine are the cloudpickle source code, particularly the `cloudpickle.py` file where the Pickler class is defined. Additionally, look at Cython's `pyximport` and Jupyter `%%cython` magic to understand how modules are named and created. The single comment on the issue may provide further hints. The goal is to design a minimal implementation that avoids shipping .so files when modules are accessible via shared filesystem.