Movinet fails in GPU mode
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 25/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 停滞
- 技术栈
- javascript, nodejs, tensorflow
调研方向
从发生所报告故障的 src/movinet/MovinetModel.js 开始,并将其与能够成功检测 GPU 的 bin/node src/test_gputensorflow.js 进行比较。检查 TensorFlow.js GPU 后端的堆栈跟踪,并确定 MoViNet 推理无法找到已注册平台的原因。当 GPU 模式下的视频分类能够在不出现所报告错误的情况下继续进行时,即视为完成。
由索引模型根据 Issue 内容生成。
描述
Which version of recognize are you using?
6.11
Enabled Modes
Object recognition, Face recognition, Video recognition, Music recognition
TensorFlow mode
GPU mode
Downstream App
Memories App
Which Nextcloud version do you have installed?
28.0.4.1
Which Operating system do you have installed?
Ubuntu 20.0.4.4
Which database are you running Nextcloud on?
Postgres
Which Docker container are you using to run Nextcloud? (if applicable)
28.0.4.1
How much RAM does your server have?
32
What processor Architecture does your CPU have?
x86_64
Describe the Bug
Seems like after upgrade to NC 28.0.4.1 & Recognize 6.1.1 it started to report below.
It seems like it launches process, based on looking at nvidia-smi and it stays there though doesn't seem like it puts a load on GPU.
Classifier process output: Error: Session fail to run with error: 2 root error(s) found.
(0) NOT_FOUND: could not find registered platform with id: 0x7fd379c7fae4
\t [[{{node movinet_classifier/movinet/stem/stem/conv3d/StatefulPartitionedCall}}]]
\t [[StatefulPartitionedCall/_1555]]
(1) NOT_FOUND: could not find registered platform with id: 0x7fd379c7fae4
\t [[{{node movinet_classifier/movinet/stem/stem/conv3d/StatefulPartitionedCall}}]]
0 successful operations.
0 derived errors ignored.
at NodeJSKernelBackend.runSavedModel (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node-gpu/dist/nodejs_kernel_backend.js:461:43)
at TFSavedModel.predict (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node-gpu/dist/saved_model.js:341:43)
at MovinetModel.predict (/var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:46:21)
at /var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:95:24
at /var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4559:22
at Engine.scopedRun (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4569:23)
at Engine.tidy (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4558:21)
at Object.tidy (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:8291:19)
at MovinetModel.inference (/var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:92:21)
at runMicrotasks (<anonymous>)
At the same time it seems to have all what's needed (btw. it didn't raise this error prior upgrades):
./bin/node src/test_gputensorflow.js
2024-04-07 21:37:06.584377: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-07 21:37:06.593729: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:06.636405: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:06.636674: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.109813: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.110064: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.110196: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.110354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3393 MB memory: -> device: 0, name: Quadro M1200, pci bus id: 0000:01:00.0, compute capability: 5.0
And models seem to be there too:
du -shx ./models/*
50M ./models/efficientnet_lite4
794M ./models/efficientnetv2
22M ./models/landmarks_africa
41M ./models/landmarks_asia
41M ./models/landmarks_europe
41M ./models/landmarks_north_america
31M ./models/landmarks_oceania
41M ./models/landmarks_south_america
47M ./models/movinet-a3
31M ./models/musicnn
Unsure where to look for an issue.
It is after ffmpeg finishes its job.
Thanks!
Expected Behavior
Would just proceed to classify.
To Reproduce
Unsure.
Debug log
No response
- 主要语言
- PHP
- 星标
- 699
- 派生
- 68
- 平均合并
- 18 小时 34 分钟
- 30 天内合并 PR
- 2
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
nextcloud/recognize 的其他 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 72/100
-
insertDeletion looks up recognize_fs_deletions by node_id alone, which no index covers — bulk removals scan the whole table per file可能已有人在做 @marcelklehr 于 7 天前认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 82/100
-
enhancement
难度 3/5 1-2 天 新手友好度 65/100
-
enhancement
难度 4/5 3-5 天 新手友好度 45/100
-
Recognize maintenance routine unnecessarily redownloads Tensorflow CPU and GPU models every time可能重新可做 @marcelklehr 于 188 天前认领,目前没有进行中的 PR。 未关闭bug
查看 nextcloud/recognize 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
-
Awaiting Triage bug
难度 2/5 1-3 小时 新手友好度 65/100
维护者通常 1 天内回复
-
product / databases
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
enhancement
难度 2/5 1-3 小时 新手友好度 75/100
VilnaCRM-Org/user-service#525 ·
维护者通常 21 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
shukiv/jabali-panel#2029 ·
维护者通常 1 天内回复