cloudpickle cannot pickle '_jpype._JField' objects
还没有人认领这个 Issue。
评估
调研方向
首先，使用提供的 JPype 环境，重现 supersuit 的 cloudpickle.dumps() 和 cloudpickle.loads() 调用失败。阅读 cloudpickle_fast.py 以及 JPype 在 jpype/pickle.py 中的实现，并跟踪 JField 对象如何被序列化和恢复。支持的方法已记录或实现，并且环境往返不再引发 memo 错误，即视为完成。
由索引模型根据 Issue 内容生成。
描述
So, I've been working on a project which involves implementing reinforcement learning in a server-client app. The server is written in Java and the client is in Python, which is why I use JPype to import some server classes.
After importing the necessary packages and creating the environment using PettingZoo, it is time to create the model and train it using Stable-Baselines3, but the problem is that when I use Supersuit, it needs to pickle and unpickle the environment, and because the environment contains many Java objects, an error is thrown: TypeError: cannot pickle '_jpype._JField' object.
The normal Pickle package does not support JField objects, but in the JPype library, there is a JPickle version that supports JField objects. I tried to modify the cloudpickle_fast.py to add the JPickle package but I end up having a problem with the cloudpickle.loads()
Here is what I modified in cloudpickle_fast.py:
from jpype.pickle import JPickler, JUnpickler
def dump(self, obj):
try:
return Pickler.dump(self, obj)
except RuntimeError as e:
if "recursion" in e.args[0]:
msg = (
"Could not pickle object as excessively deep recursion "
"required."
)
raise pickle.PicklingError(msg) from e
else:
raise
except TypeError as e:
return JPickler.dump(self, obj)
And here is the full stacktrace I get:
---------------------------------------------------------------------------
UnpicklingError Traceback (most recent call last)
Input In [11], in <cell line: 6>()
1 env = MARL_Env_Parallel(4)
5 env = ss.pettingzoo_env_to_vec_env_v1(env)
----> 6 env = ss.concat_vec_envs_v1(env, 1, num_cpus=1, base_class='stable_baselines3')
File ~\anaconda3\envs\gym\lib\site-packages\supersuit\vector\vector_constructors.py:61, in concat_vec_envs_v1(vec_env, num_vec_envs, num_cpus, base_class)
59 def concat_vec_envs_v1(vec_env, num_vec_envs, num_cpus=0, base_class="gymnasium"):
60 num_cpus = min(num_cpus, num_vec_envs)
---> 61 vec_env = MakeCPUAsyncConstructor(num_cpus)(*vec_env_args(vec_env, num_vec_envs))
63 if base_class == "gymnasium":
64 return vec_env
File ~\anaconda3\envs\gym\lib\site-packages\supersuit\vector\concat_vec_env.py:22, in ConcatVecEnv.__init__(self, vec_env_fns, obs_space, act_space)
21 def __init__(self, vec_env_fns, obs_space=None, act_space=None):
---> 22 self.vec_envs = vec_envs = [vec_env_fn() for vec_env_fn in vec_env_fns]
23 for i in range(len(vec_envs)):
24 if not hasattr(vec_envs[i], "num_envs"):
File ~\anaconda3\envs\gym\lib\site-packages\supersuit\vector\concat_vec_env.py:22, in <listcomp>(.0)
21 def __init__(self, vec_env_fns, obs_space=None, act_space=None):
---> 22 self.vec_envs = vec_envs = [vec_env_fn() for vec_env_fn in vec_env_fns]
23 for i in range(len(vec_envs)):
24 if not hasattr(vec_envs[i], "num_envs"):
File ~\anaconda3\envs\gym\lib\site-packages\supersuit\vector\vector_constructors.py:11, in vec_env_args.<locals>.env_fn()
10 def env_fn():
---> 11 env_copy = cloudpickle.loads(cloudpickle.dumps(env))
12 return env_copy
UnpicklingError: Memo value not found at index 3
I don't have a lot of experience with Pickle, so any advice would be welcome, thanks.
- 主要语言
- Python
- 星标
- 1.9k
- 派生
- 197
- 平均合并
- 1 天 10 小时
- 30 天内合并 PR
- 1
环境准备
我们还没有检查这个项目的环境配置文件。先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
cloudpipe/cloudpickle 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 78/100
cloudpipe/cloudpickle#593 · 1 条评论 ·
-
难度 4/5 3-5 天 新手友好度 42/100
cloudpipe/cloudpickle#595 ·
-
难度 3/5 1-2 天 新手友好度 65/100
cloudpipe/cloudpickle#592 · 2 条评论 ·
-
难度 5/5 一周以上 新手友好度 35/100
cloudpipe/cloudpickle#589 ·
-
难度 5/5 一周以上 新手友好度 20/100
cloudpipe/cloudpickle#587 ·
查看 cloudpipe/cloudpickle 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 74/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
gradio-app/gradio#13895 ·
维护者通常 1 天内回复
-
build-error
难度 2/5 1-3 小时 新手友好度 76/100
spack/spack-packages#6713 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
ActivityWatch/activitywatch#1464 · 1 个 reaction ·
维护者通常 1 天内回复
-
[Bug]: The ckg tool drops the return type of every decorated Python method in class search results未关闭
难度 2/5 1-3 小时 新手友好度 78/100
bytedance/trae-agent#483 ·
维护者通常 1 天内回复