Track AutoModel HybridEP OOM and metric regressions in excluded recipes
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
Research direction
Start with the PR #4037 config guard and the listed AutoModel MoE recipe configurations, then reproduce each case using pinned code, container, data, and GPU layout with DeepEP and HybridEP. Capture peak memory and failing allocation sites for OOMs, and repeat metric runs while checking validation thresholds and loss or reward trajectories. Done means re-enabling HybridEP only for recipes that pass functional and nightly checks and removing their exceptions.
Written by the indexing model from the issue text.
Description
Context
PR #4037 temporarily keeps these AutoModel MoE recipes on dispatcher: deepep after HybridEP regressions were reported:
| Recipe | Reported result after HybridEP switch |
|---|---|
sft-nanov3-30BA3B-2n8g-fsdp2 |
OOM on PR branch; passed on main |
vlm_grpo-nemotron-omni-30ba3b-clevr-1n8g-automodel-ep8.v2 |
OOM on PR branch; passed on main |
vlm_grpo-nemotron-omni-30ba3b-mmpr-4n8g-automodel-ep8.v1 |
OOM on PR branch; passed on main |
sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel |
Metric failure on both PR branch and main; may be flaky |
sft-gpt-oss-20b-1n4g-fsdp4ep4-automodel |
Metric failure only on PR branch; may be flaky |
sft-nanov3-30BA3B-2n4g-fsdp2 inherits the 2n8g Nano backend and is also kept on DeepEP. These are reported observations, not independently reproduced root causes; in particular the GPT-OSS 1n8g failure is not yet attributable to HybridEP.
Follow-up
- Reproduce each case with pinned code, container, data, and GPU layout, comparing DeepEP and HybridEP on otherwise identical configurations.
- For OOM cases, capture peak allocated/reserved memory and the failing allocation site.
- For metric failures, repeat runs and compare the relevant validation threshold and loss/reward trajectory to distinguish regression from flakiness.
- Re-enable HybridEP only for recipes that pass their functional/nightly checks, then remove their exception from the PR #4037 config guard.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 623
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 287
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/RL
-
bbox_giou_reward is unbounded: reversed-corner or NaN boxes score far above 1 (RefCOCO env)Possibly taken @jimmy-holdout claimed this 6 days ago. Openbug community-request
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
NVIDIA-NeMo/RL#4871 ·
Maintainers usually reply within 1 day
-
Use Gym’s exported server-type keysPossibly taken @kwistzzqq-byte claimed this 3 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA-NeMo/RL#4800 ·
Maintainers usually reply within 1 day
-
r0.8.0: pyrefly rejects captured optional config in make_policy_factory, blocking CIPossibly taken A pull request linked to this issue is open or already merged. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NVIDIA-NeMo/RL#4599 ·
Maintainers usually reply within 1 day
-
Feature
Difficulty 2/5 Half a day Newbie friendliness 78/100
NVIDIA-NeMo/RL#4016 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA-NeMo/RL#2948 ·
Maintainers usually reply within 1 day
Similar issues
-
Context/namespace resources fail in-cluster: "Invalid kube-config file. No configuration found."Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug documentation
Difficulty 1/5 Under an hour Newbie friendliness 88/100
awslabs/visual-asset-management-system#880 ·
Maintainers usually reply within 1 day
-
netbox status: needs owner type: documentation
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
netbox-community/netbox#23533 ·
Maintainers usually reply within 1 day
-
app: scripts status: backlog type: documentation
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
netbox-community/netbox-scripts#68 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 91/100
marin-community/marin#10765 ·
Maintainers usually reply within 1 day