mlc.SuperPool().map() can not recognize global objects in the function · mxbi/mlcrate#2

(2 comments) (0 reactions) (0 assignees)Python (328 stars) (43 forks)batch import

bughelp wanted

Description

When I passing a lambda function to fill in additional parameters, the map function can not find the f function that is called inside lambda function:

import mlcrate as mlc
pool = mlc.SuperPool() 

def f(x,y):
    return x ** (2/y)

res = pool.map(lambda x: f(x, 2), range(1000))

Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\multiprocess\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "C:\Anaconda3\lib\site-packages\mlcrate\__init__.py", line 125, in func_tracked
    return func(x), i
  File "<ipython-input-3-c53f7e0849d6>", line 8, in <lambda>
NameError: name 'f' is not defined

Also when I have a global variable in the function, the map function also can not find the global variable

import mlcrate as mlc
pool = mlc.SuperPool() 

y = 2

def f(x):
    return x ** (2/y)

res = pool.map(f, range(1000))

Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\multiprocess\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "C:\Anaconda3\lib\site-packages\mlcrate\__init__.py", line 125, in func_tracked
    return func(x), i
  File "<ipython-input-4-474bcea6b16a>", line 7, in f
NameError: name 'y' is not defined

Is there a way to pass these objects to the process pool to let the pool know which global function and variable we want to use?

Contributor guide

Tech stack: python
Domain: backend
Issue type: bug
Difficulty: 3
Estimated time: 1-2 days
Activity status: stale
Clarity: mostly clear
Prerequisites: Python basicsunderstanding of pickling
Newbie friendliness: 40
Research direction: Investigate the `func tracked` wrapper in mlcrate's ` init .py` (line 125). Understand how the `SuperPool.map` method processes lambda functions and global variables. Look into using `dill` or `cloudpickle` to serialize the function along with its globals, similar to the `pathos` library's approach. Check if there are any comments or linked PRs that discuss workarounds or fixes. Consider modifying the `map` method to pass the globals dictionary or use a custom serialization strategy.