Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

DistributedManager initialization fails on CPU-only hosts

Open
  • Is something wrong?
#1,936 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python, pytorch
Domain
distributed-systems, testing-qa

Research direction

Start in physicsnemo/distributed/manager.py around setup() near line 590 and cleanup() near line 823. Review the CPU-skipping distributed-manager test fixture and the provided torchrun reproduction, then exercise the SLURM/OpenMPI paths with the gloo backend. Done means CPU-only initialization works without division-by-zero or invalid device arguments, failed setup does not leave misleading cleanup errors, and tests cover the path.

Written by the indexing model from the issue text.

Description

Version

2.1.1 (also reproduces on current main)

On which installation method(s) does this occur?

Pip

Describe the issue

DistributedManager cannot initialize on CPU-only hosts (e.g. laptop-scale gloo runs, CPU CI):

  1. DistributedManager.setup() passes manager.device as the device_id argument of torch.distributed.init_process_group. On CPU-only hosts that is torch.device("cpu"), which recent PyTorch versions reject with ValueError: init_process_group device_id parameter must be an accelerator with an index.
  2. When local_rank is None, setup() computes rank % torch.cuda.device_count(), a division by zero when no accelerator is present (hit via the SLURM/OpenMPI initialization paths, which pass local_rank=None).
  3. As a follow-on, the failed initialization leaves _shared_state["_distributed"] = True, so the atexit cleanup() calls dist.destroy_process_group() on a process group that was never created and dies with AssertionError: Process group cannot be None, obscuring the original error.

Note the distributed manager tests are skipped on CPU by an autouse fixture, so CI does not catch this path.

Minimum reproducible example
# repro.py — run on a CPU-only machine:
#   torchrun --standalone --nproc_per_node=2 repro.py
import torch
from physicsnemo.distributed import DistributedManager

DistributedManager.initialize()
dm = DistributedManager()
t = torch.tensor([dm.rank + 1.0])
torch.distributed.broadcast(t, src=0)
DistributedManager.cleanup()
Relevant log output
  File ".../physicsnemo/distributed/manager.py", line 590, in setup
    dist.init_process_group(
  File ".../torch/distributed/distributed_c10d.py", line 1763, in init_process_group
    raise ValueError(
ValueError: init_process_group device_id parameter must be an accelerator with an index

Exception ignored in atexit callback: <staticmethod(<function DistributedManager.cleanup ...)>
  File ".../physicsnemo/distributed/manager.py", line 823, in cleanup
    dist.destroy_process_group()
AssertionError: Process group cannot be None
Environment details
nvidia-physicsnemo 2.1.1, torch 2.13.0, Python 3.12, macOS (arm64) and Linux CPU nodes; gloo backend.
Dominant language
Python
Stars
3.3k
Forks
787
Avg merge
3d 4h
Merged PRs (30d)
28

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/physicsnemo

All issues in NVIDIA/physicsnemo

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.