mxbi/mlcrate

mlc.SuperPool().map() can not recognize global objects in the function

Open

#2 建立於 2018年2月17日

在 GitHub 查看
 (2 留言) (0 反應) (0 負責人)Python (328 star) (43 fork)batch import
bughelp wanted

描述

When I passing a lambda function to fill in additional parameters, the map function can not find the f function that is called inside lambda function:

import mlcrate as mlc
pool = mlc.SuperPool() 

def f(x,y):
    return x ** (2/y)

res = pool.map(lambda x: f(x, 2), range(1000))
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\multiprocess\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "C:\Anaconda3\lib\site-packages\mlcrate\__init__.py", line 125, in func_tracked
    return func(x), i
  File "<ipython-input-3-c53f7e0849d6>", line 8, in <lambda>
NameError: name 'f' is not defined

Also when I have a global variable in the function, the map function also can not find the global variable

import mlcrate as mlc
pool = mlc.SuperPool() 

y = 2

def f(x):
    return x ** (2/y)

res = pool.map(f, range(1000))
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\multiprocess\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "C:\Anaconda3\lib\site-packages\mlcrate\__init__.py", line 125, in func_tracked
    return func(x), i
  File "<ipython-input-4-474bcea6b16a>", line 7, in f
NameError: name 'y' is not defined

Is there a way to pass these objects to the process pool to let the pool know which global function and variable we want to use?

貢獻者指南