facebookresearch/maskrcnn-benchmark

why only one gpu be used when I launch mutil gpu train?

Open

#325 opened on Jan 8, 2019

View on GitHub
 (10 comments) (0 reactions) (0 assignees)Python (9,161 stars) (2,574 forks)batch import
awaiting responsehelp wanted

Description

❓ Questions and Help

when I launch multi gpu train use this command:

python -m torch.distributed.launch --nproc_per_node=2 /path_to_maskrcnn_benchmark/tools/train_net.py --config-file "path/to/config/file.yaml"

and it's run in docker container env, there are have two p40 in docker, but it seems that only one GPU is being used. image

and there are actually two processes running image

What could be the reason?

Contributor guide

Tech stack
pythonpytorchdocker
Domain
machine learningperformance
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PyTorch distributed trainingDocker GPU passthroughMulti GPU setup
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Investigate whether the Docker container has access to both GPUs by running `nvidia smi` inside the container. Check the environment variable `CUDA VISIBLE DEVICES` and ensure it includes both GPU IDs. Additionally, review the PyTorch distributed training initialization in the repository to confirm it correctly uses the world size and rank from the launcher. Look at the training script for any hardcoded GPU indices.