[RFC] Explore deprecation of DDP in favor of FSDP2
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Refactor
- Clarity
- Needs clarification
- Activity status
- Active
- Domain
- distributed-systems, machine-learning
Research direction
Start by locating the DDP and FSDP2 integration points, then compare their behavior and performance on small models. Done means establishing whether DDP can be deprecated without regressions and documenting a clear deprecation plan or the evidence for retaining both paths.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
Supporting both fsdp2 and ddp complicates the code significantly; for example, fsdp2 works as a mixin where ddp holds the nn.module as a child. Therefore, any code traversing those nn.module trees needs to handle both cases.
Describe the solution you'd like
We want to simplify the code, therefore, want to explore deprecating ddp in favor of fsdp2. However, we need to ensure no perf regressions, especially for the small model cases, where ddp is faster.
Describe alternatives you've considered
Supporting both, but results in maintenance overhead
Additional context
N/A
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 366
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 281
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/Automodel
-
community-request waiting-on-maintainers
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
NVIDIA-NeMo/Automodel#3767 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 5/100
NVIDIA-NeMo/Automodel#4627 ·
Maintainers usually reply within 1 day
-
Add sound type annotations: training lifecycle and optimizersPossibly taken @akoumpa claimed this 2 days ago. Opengood first issue help wanted PoR
NVIDIA-NeMo/Automodel#4626 · 1 assignee ·
Maintainers usually reply within 1 day
-
Add sound type annotations: model execution and adaptersPossibly taken A pull request linked to this issue is open or already merged. Opengood first issue help wanted PoR
Difficulty 4/5 3-5 days Newbie friendliness 35/100
NVIDIA-NeMo/Automodel#4622 ·
Maintainers usually reply within 1 day
-
Add sound type annotations: data, evaluation, and objectivesPossibly taken @Alan-donk claimed this 2 days ago. Opengood first issue help wanted PoR
Difficulty 3/5 Half a day Newbie friendliness 55/100
NVIDIA-NeMo/Automodel#4614 · 1 comment ·
Maintainers usually reply within 1 day
All issues in NVIDIA-NeMo/Automodel
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
jejjohnson/gaussx#752 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
521xueweihan/HelloGitHub#4270 ·
-
enhancement owner: den-haag
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 6 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ok-oldking/ok-wuthering-waves#1828 ·
Maintainers usually reply within 1 day
-
lesson-submission pending-review
Difficulty 1/5 Under an hour Newbie friendliness 65/100
Ikalus1988/MisakaNet#4512 ·
Maintainers usually reply within 1 day