DeepLabCut-live fails to run exported superanimal_humanbody model from DeepLabCut
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 25/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
调研方向
Start in dlclive/pose_estimation_pytorch/runner.py around the detector handling near line 192 and PyTorchRunner.get_pose(). Reproduce a frame where no detector prediction exceeds cutoff_confidence, then trace the model input shape before the call shown in the traceback. Done means such frames no longer cause the reported tensor-shape error; issue 164 is already an open linked pull request.
由索引模型根据 Issue 内容生成。
描述
Concerns the DLC3-pytorch branch maxim/dlclive3
When trying to run rtmpose models or other models with a detector on top, i saw that DLCLive crashes at certain frames.
Looking more into the issue: this happens whenever the detector doesnt have any predictions > cutoff_confidence. here then it will try to pass the raw image to the model which will results in array mismatch.
`---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[11], line 1
----> 1 dlc.runner.get_pose(frame)
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\utils_contextlib.py:120, in context_decorator..decorate_context(*args, **kwargs)
117 @functools.wraps(func)
118 def decorate_context(*args, **kwargs):
119 with ctx_factory():
--> 120 return func(*args, **kwargs)
File ~\Repos\dlc_live3\DeepLabCut-live\dlclive\pose_estimation_pytorch\runner.py:211, in PyTorchRunner.get_pose(self, frame)
208 if self.precision == "FP16":
209 model_input = model_input.half()
--> 211 outputs = self.model(model_input)
212 batch_pose = self.model.get_predictions(outputs)["bodypart"]["poses"]
214 if self.dynamic is not None:
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
1773 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1774 else:
-> 1775 return self._call_impl(*args, **kwargs)
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1786, in Module._call_impl(self, *args, **kwargs)
1781 # If we don't have any hooks, we want to skip the rest of the logic in
1782 # this function, and just call forward.
1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1784 or _global_backward_pre_hooks or _global_backward_hooks
1785 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1786 return forward_call(*args, **kwargs)
1788 result = None
1789 called_always_called_hooks = set()
File ~\Repos\dlc_live3\DeepLabCut-live\dlclive\pose_estimation_pytorch\models\model.py:74, in PoseModel.forward(self, x)
72 outputs = {}
73 for head_name, head in self.heads.items():
---> 74 outputs[head_name] = head(features)
75 return outputs
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
1773 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1774 else:
-> 1775 return self._call_impl(*args, **kwargs)
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1786, in Module._call_impl(self, *args, **kwargs)
1781 # If we don't have any hooks, we want to skip the rest of the logic in
1782 # this function, and just call forward.
1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1784 or _global_backward_pre_hooks or _global_backward_hooks
1785 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1786 return forward_call(*args, **kwargs)
1788 result = None
1789 called_always_called_hooks = set()
File ~\Repos\dlc_live3\DeepLabCut-live\dlclive\pose_estimation_pytorch\models\heads\rtmcc_head.py:110, in RTMCCHead.forward(self, x)
108 feats = self.final_layer(x) # -> B, K, H, W
109 feats = torch.flatten(feats, start_dim=2) # -> B, K, hidden=HxW
--> 110 feats = self.mlp(feats) # -> B, K, hidden
111 feats = self.gau(feats)
112 x, y = self.cls_x(feats), self.cls_y(feats)
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
1773 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1774 else:
-> 1775 return self._call_impl(*args, **kwargs)
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1786, in Module._call_impl(self, *args, **kwargs)
1781 # If we don't have any hooks, we want to skip the rest of the logic in
1782 # this function, and just call forward.
1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1784 or _global_backward_pre_hooks or _global_backward_hooks
1785 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1786 return forward_call(*args, **kwargs)
1788 result = None
1789 called_always_called_hooks = set()
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\container.py:250, in Sequential.forward(self, input)
246 """
247 Runs the forward pass.
248 """
249 for module in self:
--> 250 input = module(input)
251 return input
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
1773 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1774 else:
-> 1775 return self._call_impl(*args, **kwargs)
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\module.py:1786, in Module._call_impl(self, *args, **kwargs)
1781 # If we don't have any hooks, we want to skip the rest of the logic in
1782 # this function, and just call forward.
1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1784 or _global_backward_pre_hooks or _global_backward_hooks
1785 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1786 return forward_call(*args, **kwargs)
1788 result = None
1789 called_always_called_hooks = set()
File c:\Users\User.conda\envs\dlclive3\Lib\site-packages\torch\nn\modules\linear.py:134, in Linear.forward(self, input)
130 def forward(self, input: Tensor) -> Tensor:
131 """
132 Runs the forward pass.
133 """
--> 134 return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (12x391 and 144x256)`
- 主要语言
- Python
- 星标
- 239
- 派生
- 57
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
DeepLabCut/DeepLabCut-live 的其他 Issue
-
RGB inputs have red and blue swapped可能已有人在做 @C-Achard 于 3 天前认领。 未关闭bug pytorch tensorflow
DeepLabCut/DeepLabCut-live#175 · 已指派 1 人 ·
-
Deeplabcut Live for Multi-Animal可能重新可做 @C-Achard 于 242 天前认领,目前没有进行中的 PR。 未关闭bug tensorflow
DeepLabCut/DeepLabCut-live#130 · 已指派 1 人 ·
-
[meta] docs可能重新可做 @C-Achard 于 242 天前认领,目前没有进行中的 PR。 未关闭documentation
DeepLabCut/DeepLabCut-live#78 · 2 条评论 · 1 个 reaction · 已指派 2 人 ·
查看 DeepLabCut/DeepLabCut-live 的全部 Issue
相似的 Issue
-
Zero-token evaluations are treated as missing cost in selection可能已有人在做 @sylvesterkaczmarek 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
google-research/rrsi#6 ·
-
bug
难度 2/5 1-3 小时 新手友好度 88/100
aws-samples/sample-aws-genai-db-modernizer#294 ·
维护者通常 1 天内回复
-
feedback simulation workshop
难度 2/5 1-3 小时 新手友好度 84/100
githubnext/gh-aw-workshop#4174 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 86/100
WingedGuardian/GENesis-AGI#2852 ·
维护者通常 1 天内回复
-
camlight produces degenerate target camera and light frames可能已有人在做 @kavyabhand 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
google-deepmind/mujoco_warp#1743 ·
维护者通常 1 天内回复