Track AutoModel HybridEP OOM and metric regressions in excluded recipes
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
Research direction
Start with the PR #4037 config guard and the listed AutoModel MoE recipe configurations, then reproduce each case using pinned code, container, data, and GPU layout with DeepEP and HybridEP. Capture peak memory and failing allocation sites for OOMs, and repeat metric runs while checking validation thresholds and loss or reward trajectories. Done means re-enabling HybridEP only for recipes that pass functional and nightly checks and removing their exceptions.
Written by the indexing model from the issue text.
Description
Context
PR #4037 temporarily keeps these AutoModel MoE recipes on dispatcher: deepep after HybridEP regressions were reported:
| Recipe | Reported result after HybridEP switch |
|---|---|
sft-nanov3-30BA3B-2n8g-fsdp2 |
OOM on PR branch; passed on main |
vlm_grpo-nemotron-omni-30ba3b-clevr-1n8g-automodel-ep8.v2 |
OOM on PR branch; passed on main |
vlm_grpo-nemotron-omni-30ba3b-mmpr-4n8g-automodel-ep8.v1 |
OOM on PR branch; passed on main |
sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel |
Metric failure on both PR branch and main; may be flaky |
sft-gpt-oss-20b-1n4g-fsdp4ep4-automodel |
Metric failure only on PR branch; may be flaky |
sft-nanov3-30BA3B-2n4g-fsdp2 inherits the 2n8g Nano backend and is also kept on DeepEP. These are reported observations, not independently reproduced root causes; in particular the GPT-OSS 1n8g failure is not yet attributable to HybridEP.
Follow-up
- Reproduce each case with pinned code, container, data, and GPU layout, comparing DeepEP and HybridEP on otherwise identical configurations.
- For OOM cases, capture peak allocated/reserved memory and the failing allocation site.
- For metric failures, repeat runs and compare the relevant validation threshold and loss/reward trajectory to distinguish regression from flakiness.
- Re-enable HybridEP only for recipes that pass their functional/nightly checks, then remove their exception from the PR #4037 config guard.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 623
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 292
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/RL
-
bbox_giou_reward is unbounded: reversed-corner or NaN boxes score far above 1 (RefCOCO env)Possibly taken @jimmy-holdout claimed this 6 days ago. Openbug community-request
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
NVIDIA-NeMo/RL#4871 ·
Maintainers usually reply within 1 day
-
Use Gym’s exported server-type keysPossibly taken @kwistzzqq-byte claimed this 4 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA-NeMo/RL#4800 ·
Maintainers usually reply within 1 day
-
r0.8.0: pyrefly rejects captured optional config in make_policy_factory, blocking CIPossibly taken A pull request linked to this issue is open or already merged. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NVIDIA-NeMo/RL#4599 ·
Maintainers usually reply within 1 day
-
Feature
Difficulty 2/5 Half a day Newbie friendliness 78/100
NVIDIA-NeMo/RL#4016 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA-NeMo/RL#2948 ·
Maintainers usually reply within 1 day
Similar issues
-
enhancement help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Git-Hospital/githealthcheck#61 ·
Maintainers usually reply within 1 day
-
changelog investigate
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
ramnes/notion-sdk-py#433 ·
-
Partial method descriptors raise on class accessPossibly taken @Gambit-Checkmate claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 19 days
-
docs pkg:core
Difficulty 1/5 Under an hour Newbie friendliness 84/100
pydantic/pydantic-ai#11929 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
intake/intake-esm#789 ·