facebookresearch/fairseq

Add support for OpenMPI Rank Assignment

Open

#4,645 opened on Aug 11, 2022

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (6,224 forks)batch import
enhancementhelp wantedneeds triage

Repository metrics

Stars
 (29,107 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

🚀 Feature Request

Currently, there is no way to read the variables OMPI_COMM_WORLD_SIZE, OMPI_COMM_WORLD_RANK, and OMPI_COMM_WORLD_LOCAL_RANK to assign ranks to any individual processes launched with OpenMPI

Motivation

I'd like to launch jobs with OpenMPI without using multiple-data mode, which can cause issues with command length if I have to specify all of the ranks manually.

Pitch

Add _infer_ompi_init function in fairseq/distributed/utils.py which reads the environment variables:

MASTER_ADDR
MASTER_PORT
OMPI_COMM_WORLD_SIZE
OMPI_COMM_WORLD_RANK
OMPI_COMM_WORLD_LOCAL_RANK

and updates the distributed config accordingly.

Alternatives

N/A

Additional context

N/A

Contributor guide

Research direction
Examine the existing infer slurm init function in fairseq/distributed/utils.py for pattern reference and implement infer ompi init by reading the specified OpenMPI environment variables and updating the distributed configuration accordingly.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75