[Question] Would an adaptive in-reward KL controller fit Slime's PPO/RLHF scope?

Open
  • Is something wrong?
#2,387 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
python
Domain
ai, machine-learning

Research direction

Start by reviewing Slime's current PPO token-level reward construction and existing --kl-coef path described in the issue, then compare the requested behavior with verl's AdaptiveKLController. A completed implementation would need bounded cross-batch updates, checkpointed state, metrics, unit tests, and a reproducible PPO benchmark while leaving defaults and GRPO semantics unchanged.

Written by the indexing model from the issue text.

Description

question
Your Question

Question

Would an opt-in adaptive in-reward KL controller for PPO-style reference-policy training fit Slime's current scope for general-purpose RL optimizations?

Proposal

The proposal is deliberately limited to the existing PPO/RLHF-style reward-side KL path:

observed reference KL -> update beta -> next-batch reward shaping
r'_t = task_reward_t - beta * KL_t

The controller would adapt the existing --kl-coef between batches toward a target KL, with bounded updates, a configurable horizon, checkpointed state, and metrics for observed KL, target KL, coefficient, and update status. The default behavior would remain fixed-coefficient and unchanged.

This would run alongside PPO clipping; it would not modify eps_clip or the PPO ratio objective.

Scope boundary

This is not a proposal to change the current GRPO reward semantics. I understand the maintainer direction in #399: GRPO should prefer kl_loss unless experiments show that reward-side KL is beneficial. The first implementation would target PPO-style training only, where reward-side KL is already part of the existing path.

Would a focused implementation with unit tests and a reproducible training benchmark be considered an in-scope general-purpose RL optimization? If so, I would be happy to prepare a small PR rather than a broad abstraction proposal.

What I've Tried

I checked the current Slime implementation and related discussions:

  • #397 identified that GRPO's non-zero --kl-coef was effectively ignored.
  • #399 proposed applying KL to GRPO rewards, but the maintainer discussion preferred kl_loss for GRPO unless supported by experiments.
  • #1247 made a similar GRPO reward-side change and was closed without being merged.
  • In the current PPO path, --kl-coef is applied during token-level reward construction before GAE.
  • In verl's implementation, AdaptiveKLController updates the coefficient from the batch KL toward a target KL. The implementation cites Ziegler et al.'s RLHF work (arXiv:1909.08593).

I have intentionally not included GRPO, TTTD, OPD, standalone kl_loss_coef, or evaluation/rollback orchestration in this proposal. Those have different semantics and should be considered separately.

Environment (if relevant)

This is an algorithm/API-scope question rather than a runtime bug. No specific hardware or environment is required at this stage.

Additional Context

Related discussion: Human-Agent-Society/reef#466.

The goal is to clarify ownership and scope before writing code. Slime would own the training-side controller; any held-out evaluation or publish/pause/rollback policy would remain outside this proposed Slime change.

Pre-submission Checklist
Dominant language
Python
Stars
8.8k
Forks
1.3k
Avg merge
20h 16m
Merged PRs (30d)
24

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from THUDM/slime

All issues in THUDM/slime

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.