feat: generalize run_rollout_only.py into a single-controller simulator
Maintainers usually reply within 2 days
Assessment
This issue has not been assessed yet.
Description
Background
#4814 adds examples/run_rollout_only.py, which runs a GRPO recipe's rollouts exactly as training would, without creating a trainer. A training stand-in holds each step where policy.train() would run, and the weight version advances through the refit calls without moving any weights. This covers the rollout-only case on both the sync and async paths.
Proposal
Build on that driver and evolve it into examples/run_single_controller_simulator.py. In the simulator, each side of the training loop is a pluggable backend, and any combination of the two can run:
- Train backend
automodelmcorecpu: a mock that stands in for the train step and refit and uses no GPU resources
- Rollout backend
- any of the existing generation backends
cpu: a mock that returns synthetic rollouts and uses no GPU resources
- Modes
rollout_only: what #4814 supports today, i.e. a real rollout backend with the trainer stubbed outtrain_only: a real train backend fed by mock rolloutstrain + rollout: both backends real, the full loop
Because either side can be swapped for the cpu mock, one side can be run, profiled and tuned on its own, with the same config and step loop as the full run, and without allocating GPUs for the other side.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 612
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 267
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/RL
-
bbox_giou_reward is unbounded: reversed-corner or NaN boxes score far above 1 (RefCOCO env)Possibly taken @jimmy-holdout claimed this 4 days ago. Openbug community-request
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
NVIDIA-NeMo/RL#4871 ·
Maintainers usually reply within 2 days
-
Use Gym’s exported server-type keysPossibly taken @kwistzzqq-byte claimed this 1 day ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA-NeMo/RL#4800 ·
Maintainers usually reply within 2 days
-
r0.8.0: pyrefly rejects captured optional config in make_policy_factory, blocking CIPossibly taken A pull request linked to this issue is open or already merged. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NVIDIA-NeMo/RL#4599 ·
Maintainers usually reply within 2 days
-
Feature
Difficulty 2/5 Half a day Newbie friendliness 78/100
NVIDIA-NeMo/RL#4016 ·
Maintainers usually reply within 2 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA-NeMo/RL#2948 ·
Maintainers usually reply within 2 days
Similar issues
-
ci enhancement testing
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area/config comp/tools P2 platform/telegram type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
NousResearch/hermes-agent#145412 ·
Maintainers usually reply within 1 day
-
[oblt-aw][autodoc] Update docs for security org scaffolding and Docs AI menu permission changesPossibly taken @github-actions claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
curriculum documentation quality
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
githubnext/gh-aw-workshop#5039 ·
Maintainers usually reply within 1 day
-
Community project: hosted MOSS-Image studio with a visual JSON layout builder (no ComfyUI needed)Open
Difficulty 1/5 1-3 hours Newbie friendliness 65/100
OpenMOSS/MOSS-Image#27 · 1 comment ·
Maintainers usually reply within 1 day