facebookresearch/maskrcnn-benchmark

why only one gpu be used when I launch mutil gpu train?

Open

#325 建立於 2019年1月8日

在 GitHub 查看
 (10 留言) (0 反應) (0 負責人)Python (9,161 star) (2,574 fork)batch import
awaiting responsehelp wanted

描述

❓ Questions and Help

when I launch multi gpu train use this command:

python -m torch.distributed.launch --nproc_per_node=2 /path_to_maskrcnn_benchmark/tools/train_net.py --config-file "path/to/config/file.yaml"

and it's run in docker container env, there are have two p40 in docker, but it seems that only one GPU is being used. image

and there are actually two processes running image

What could be the reason?

貢獻者指南

技術棧
pythonpytorchdocker
領域
machine learningperformance
議題類型
research
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
half day
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
PyTorch distributed trainingDocker GPU passthroughMulti GPU setup
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
15
研究方向
Investigate whether the Docker container has access to both GPUs by running `nvidia smi` inside the container. Check the environment variable `CUDA VISIBLE DEVICES` and ensure it includes both GPU IDs. Additionally, review the PyTorch distributed training initialization in the repository to confirm it correctly uses the world size and rank from the launcher. Look at the training script for any hardcoded GPU indices.