Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Support shared-policy multi-agent RL training on the context-compaction capture path

Open
  • Is something wrong?
#4,400 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
python
Domain
machine-learning

Research direction

Start by reading the linked NeMo-RL and NeMo-Gym context-compaction work in PRs #4124 and #3368, then compare the ancestry and shared-ancestor ideas in RL #4285. Use the acceptance criteria as the completion target: deterministic multi-context reconstruction, complete call accounting, once-only eligible-action loss, correct GRPO ownership, and a learner update containing worker actions.

Written by the indexing model from the issue text.

Description

Feature

Need

Agent harnesses can delegate work to subagents with independent contexts. Training only the parent omits the workers’ generated actions. Training both requires reconstructing each agent’s actual conditioning, assigning loss to the intended actions, and keeping every training row associated with the task attempt that produced its reward.

Support both parent-only and parent-plus-worker training with one shared policy. Each agent may compact its context independently.

Foundation

Build on:

These changes already separate one logical rollout from its physical training rows, capture actual generation tokens and model inputs, verify continuation versus context rewrite, identify accepted generations, and preserve rollout-level reward/advantage semantics.

Multi-agent support should generalize that path. Harnesses retain responsibility for delegation, communication, scheduling, and compaction; NeMo-RL owns training reconstruction.

Proposed approach

  1. Identify accepted and training-eligible calls across agents. Extend shared ingestion to handle multiple sessions through ordinary outputs and stable call references. Distinguish accepted actions from retries, discarded branches, and auxiliary calls.
  2. Reconstruct a forest of contexts. Replace sequential segment grouping with traversal of verified parent links. Find endpoints among selected calls and reconstruct their ancestry. Support independent roots, interleaved continuations, compaction, and shared-prefix branches.
  3. Preserve unique action ownership. Shared ancestors may appear as conditioning in several rows, but each eligible generation receives loss exactly once. Worker answers inserted into the parent’s context remain masked observations.
  4. Enable concurrent capture. Replace the attempt-wide sequential admission restriction with independent context reservations and per-call accounting. Preserve checks for ambiguous ancestry, stale continuations, and unresolved acknowledgements.
  5. Close the whole episode safely. Reconcile admitted calls before publishing rows or cleaning up staging. A parent response finishing must not silently discard outstanding worker calls.
  6. Reuse logical-owner training. All physical rows retain their task’s ownership. Compute one reward contribution per logical rollout in GRPO and distribute its advantage to eligible actions.

Initial scope and milestones

  • Establish a parent-only baseline using an existing harness.
  • Add deterministic reconstruction tests for multiple contexts and shared ancestors.
  • Qualify concurrent capture, delayed results, cancellation, and episode closure.
  • Demonstrate parent-plus-worker training with OpenCode.
  • Qualify a second harness, such as LangGraph, to test the generality of the integration.
  • Exercise compaction within a worker and preserve existing single-agent/CC behavior.

Initial scope is one shared trainable policy and one task-level reward. Separate policies, agent-specific credit assignment, and decoder prefix-sharing optimizations can follow independently.

Acceptance criteria

A controlled episode should include a parent, two overlapping workers, parent resumption, reversed worker completion order, and compaction in one worker. Verify:

  • Exact conditioning, captured tokens, masks, and behavior logprobs.
  • Complete accounting of admitted calls.
  • Once-only loss for eligible actions.
  • One logical-rollout contribution to the GRPO baseline.
  • Loss/gradient agreement with an independently constructed per-call reference.
  • A real learner update containing worker actions.

Related work

RL #4285 provides relevant ancestry traversal and shared-ancestor masking. Adapt those ideas to CC’s accepted-call selection and logical-owner machinery.

RL #4286 addresses shared-prefix computation and can remain a separate optimization.

Dominant language
Python
Stars
2.1k
Forks
612
Avg merge
3d 14h
Merged PRs (30d)
267

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/RL

All issues in NVIDIA-NeMo/RL

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.