[Question] Multiple critic updates per actor update (SAO-style faster value update)?

Open
  • Is something wrong?
#2,329 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
machine-learning

Research direction

Start in slime/backends/megatron_utils/actor.py, comparing train_critic and train_actor and tracing the shared build_dp_schedule result. Read the handling of --num-critic-only-steps and the PPO path described in the issue. Done would require a maintainer decision on whether the 1:1 ratio is intentional or a scoped design for independent critic updates.

Written by the indexing model from the issue text.

Description

question
Your Question

I'm trying to understand how the critic is trained relative to the actor in the PPO path, and I'm confused about why there doesn't seem to be a way to update the critic more times than the actor within one rollout. From what I can tell, train_critic and train_actor both call the same train(...) with the same num_microbatches / global_batch_sizes, so they run 1:1, and the only critic-related knob is --num-critic-only-steps, which is just a cross-rollout warmup.

I was reading the SAO paper (Single-Rollout Asynchronous Optimization, https://arxiv.org/abs/2607.07508, the method behind GLM-5.2's agentic RL) and noticed it uses a "faster value update" where the value network is updated K times per policy update (K > 1, paper uses K = 2) to reduce advantage-estimation variance when the critic lags behind the policy, with an ablation showing it helps. Since slime is the framework behind those runs, I was a bit surprised this isn't exposed in the open-source code.

So I'm wondering: is the current 1:1 behavior intentional, or would it make sense to add a config (something like --critic-update-steps / a critic-side K) to allow multiple critic updates per actor update? Raising this as a question first per CONTRIBUTING.md — thanks!

slime commit: 5b449de, PPO path (advantage_estimator=ppo, use_critic=True)

What I've Tried

I searched the codebase for any critic-specific update-count config (things like critic_iters, critic_epoch, critic_update, num_critic_steps) and couldn't find one — the only critic-related argument I found is --num-critic-only-steps, which only controls a cross-rollout warmup, not a per-rollout critic:actor update ratio.

I also traced the training path: both train_critic and train_actor in slime/backends/megatron_utils/actor.py call the same module-level train(...), and the schedule (num_microbatches / global_batch_sizes) comes from a single build_dp_schedule result shared by both, so as far as I can tell the critic and actor are structurally locked to the same number of updates per rollout.

Finally I read the SAO paper (arXiv:2607.07508) to confirm the "faster value update" (K=2) is an actual part of the recipe rather than a generic PPO trick, which is what prompted this question.

Environment (if relevant)
  • slime version:
  • Python version:
  • PyTorch version:
  • CUDA/ROCm version:
  • GPU type and count:
  • OS:
Additional Context

No response

Pre-submission Checklist
Dominant language
Python
Stars
8.8k
Forks
1.3k
Avg merge
19h
Merged PRs (30d)
26

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from THUDM/slime

All issues in THUDM/slime

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.