OpenRLHF

Issues somebody may already be working on are listed last. List everything by date

• EvalPort: portable interchange for compute_eval_metrics results

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 40/100

  OpenRLHF/OpenRLHF#1367 · Opened Sep 27, 2026

  Maintainers usually reply within 1 day

• [Roadmap] OpenRLHF on Intel XPU

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1303 · 1 comment · Opened Aug 12, 2026

  Maintainers usually reply within 1 day

• 🔥 Molt brings an Automodel-powered backend (AutoTP/EP/CP) to OpenRLHF

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1273 · Opened Jul 27, 2026

  Maintainers usually reply within 1 day

• Feature request: reward-hacking onset monitoring hooks during PPO/GRPO training

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1263 · 1 comment · Opened Jul 10, 2026

  Maintainers usually reply within 1 day

• Consider using pre-built Flash Attention kernels via `kernels`

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 22/100

  OpenRLHF/OpenRLHF#1244 · 1 reaction · Opened May 28, 2026

  Maintainers usually reply within 1 day

• ray error for multi-nodes training

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1236 · 1 comment · Opened May 7, 2026

  Maintainers usually reply within 1 day

• Torch 2.10's `LRScheduler._update_lr` breaks LORA

  May be free again A pull request for this issue was closed without being merged. Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 Under an hour Newbie friendliness 50/100

  OpenRLHF/OpenRLHF#1225 · 1 comment · Opened Apr 21, 2026

  Maintainers usually reply within 1 day

• multimodal reward models

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1224 · 1 comment · Opened Apr 20, 2026

  Maintainers usually reply within 1 day

• Sandboxed execution for multi-turn agent rollouts

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1210 · 1 comment · Opened Apr 1, 2026

  Maintainers usually reply within 1 day

• 额外的奖励模型如何提升显卡利用率

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 25/100

  OpenRLHF/OpenRLHF#1196 · 2 comments · Opened Mar 19, 2026

  Maintainers usually reply within 1 day

• For asynchronous training with vLLM 0.16.0+, please update to OpenRLHF 0.9.5

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 Under an hour Newbie friendliness 18/100

  OpenRLHF/OpenRLHF#1193 · Opened Mar 5, 2026

  Maintainers usually reply within 1 day

• DeepSpeed Off-loading Weights (not just Optimizer States)

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1187 · Opened Feb 12, 2026

  Maintainers usually reply within 1 day

• PPO "mini batches"

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 25/100

  OpenRLHF/OpenRLHF#1185 · Opened Feb 10, 2026

  Maintainers usually reply within 1 day

• Support for tool-calling

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 8/100

  OpenRLHF/OpenRLHF#1183 · 3 comments · Opened Feb 6, 2026

  Maintainers usually reply within 1 day

• Support Qwen3vl GRPO？

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1175 · 1 comment · 2 reactions · Opened Jan 15, 2026

  Maintainers usually reply within 1 day

• Reinforcement learning fine-tuning, continuously outputting token 0.

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1165 · 2 comments · Opened Jan 4, 2026

  Maintainers usually reply within 1 day

• 知识蒸馏在zero stage 3时 偶现通信超时，通常运行一半或几个step出现

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 8/100

  OpenRLHF/OpenRLHF#1157 · Opened Dec 10, 2025

  Maintainers usually reply within 1 day

• Don't Use bf16 for RL, but fp16

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1155 · 1 comment · Opened Nov 27, 2025

  Maintainers usually reply within 1 day

• Ray+Vllm+Lora

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 Over a week Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1153 · 1 reaction · Opened Nov 24, 2025

  Maintainers usually reply within 1 day

• Significant Performance Gap between `zero_stage=1` and `zero_stage=2`

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 20/100

  OpenRLHF/OpenRLHF#1151 · 5 comments · 1 reaction · Opened Nov 12, 2025

  Maintainers usually reply within 1 day

• GRPO load_in_4bit error

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 18/100

  OpenRLHF/OpenRLHF#1150 · 2 comments · Opened Nov 10, 2025

  Maintainers usually reply within 1 day

• ref_model可以替换为其他的模型吗，当前是跟actor一致

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 25/100

  OpenRLHF/OpenRLHF#1149 · 1 comment · Opened Nov 9, 2025

  Maintainers usually reply within 1 day

• knowledge distillation train

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1146 · Opened Oct 30, 2025

  Maintainers usually reply within 1 day

• ppo eval

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 22/100

  OpenRLHF/OpenRLHF#1145 · 1 comment · Opened Oct 29, 2025

  Maintainers usually reply within 1 day

• ValueError: not enough values to unpack (expected 4, got 3)

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 35/100

  OpenRLHF/OpenRLHF#1143 · 2 reactions · Opened Oct 24, 2025

  Maintainers usually reply within 1 day

• Add experiment tracking via `trackio`

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 52/100

  OpenRLHF/OpenRLHF#1136 · Opened Oct 2, 2025

  Maintainers usually reply within 1 day

• SFT deepspeed tp bug !

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 18/100

  OpenRLHF/OpenRLHF#1134 · 9 comments · Opened Sep 24, 2025

  Maintainers usually reply within 1 day

• [Bug]: Training suddenly stopped while using hybrid engine

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1133 · 4 comments · Opened Sep 19, 2025

  Maintainers usually reply within 1 day

• Token and text concatenation issue in the process of agentic training.

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 28/100

  OpenRLHF/OpenRLHF#1128 · 3 comments · Opened Sep 4, 2025

  Maintainers usually reply within 1 day

• Feature Request: Support for Multimodal Model SFT and RL Training

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 8/100

  OpenRLHF/OpenRLHF#1127 · 1 reaction · Opened Sep 2, 2025

  Maintainers usually reply within 1 day

• save model nccl timeout

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 Over a week Newbie friendliness 12/100

  OpenRLHF/OpenRLHF#1126 · 3 comments · Opened Sep 1, 2025

  Maintainers usually reply within 1 day

• Does OpenRLHF currently support PPO training based on Process Reward Model (PRM)?

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1125 · 1 reaction · Opened Aug 28, 2025

  Maintainers usually reply within 1 day

• 使用Remote RM 会卡住

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1123 · 2 comments · Opened Aug 28, 2025

  Maintainers usually reply within 1 day

• Proposal: Fully Asynchronous RL Pipeline Support in OpenRLHF

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1116 · 1 comment · Opened Aug 24, 2025

  Maintainers usually reply within 1 day

• DeepSpeed AutoTP with Hybrid Engine

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1110 · 2 comments · Opened Aug 21, 2025

  Maintainers usually reply within 1 day

• RL Reward Instability and Reproducibility Issues Tied to vLLM Version

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1108 · 3 comments · 1 reaction · Opened Aug 18, 2025

  Maintainers usually reply within 1 day

• wandb中记录的train/generated_samples，sample和reward不是对应的吗

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 20/100

  OpenRLHF/OpenRLHF#1107 · 2 comments · Opened Aug 17, 2025

  Maintainers usually reply within 1 day

• agent_func、reward_func时--normalization_reward参数不起作用，对吗

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 28/100

  OpenRLHF/OpenRLHF#1106 · 1 comment · Opened Aug 16, 2025

  Maintainers usually reply within 1 day

• [BUG] Cannot reload the dataloader state from checkpoints in ppo training

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 28/100

  OpenRLHF/OpenRLHF#1103 · 2 comments · Opened Aug 11, 2025

  Maintainers usually reply within 1 day

• Support for NVIDIA Nsight Systems

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 12/100

  OpenRLHF/OpenRLHF#1099 · Opened Aug 6, 2025

  Maintainers usually reply within 1 day

• Are you considering supporting the FR3E algorithm？

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 5/5 Over a week Newbie friendliness 8/100

  OpenRLHF/OpenRLHF#1093 · Opened Jul 20, 2025

  Maintainers usually reply within 1 day

• Release artifacts (models, dataset) on Hugging Face

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 25/100

  OpenRLHF/OpenRLHF#1087 · Opened Jul 16, 2025

  Maintainers usually reply within 1 day

• [Memory Leak] GPU Memory Accumulates After Each Epoch in Training

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 10/100

  OpenRLHF/OpenRLHF#1086 · 1 comment · Opened Jul 16, 2025

  Maintainers usually reply within 1 day

• 疑似某次重构导致不再支持无vllm后端的rollout？

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 25/100

  OpenRLHF/OpenRLHF#1082 · 2 comments · Opened Jul 14, 2025

  Maintainers usually reply within 1 day

• 目前的Lora不支持带vllm的后端吗？

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 20/100

  OpenRLHF/OpenRLHF#1081 · 3 comments · Opened Jul 14, 2025

  Maintainers usually reply within 1 day

• Need some help for setting up a usable runtime

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 20/100

  OpenRLHF/OpenRLHF#1074 · Opened Jul 7, 2025

  Maintainers usually reply within 1 day

• How to turn off thinking mode when doing RL on Qwen3

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 4/5 3-5 days Newbie friendliness 20/100

  OpenRLHF/OpenRLHF#1073 · 5 comments · Opened Jul 7, 2025

  Maintainers usually reply within 1 day

• Does OpenRLHF support multi-node SFT training?

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 1/5 Under an hour Newbie friendliness 25/100

  OpenRLHF/OpenRLHF#1072 · 1 comment · Opened Jul 2, 2025

  Maintainers usually reply within 1 day

• SLURM 18.08.3 doesn’t support the --overlap option

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 2/5 1-3 hours Newbie friendliness 48/100

  OpenRLHF/OpenRLHF#1071 · Opened Jun 27, 2025

  Maintainers usually reply within 1 day

• For VLM training, do we have a parameter that can freeze vision encoder?

  Open

  ⋯Report a problem with this issue

  • Is something wrong?

  • Already taken
  • Not beginner-friendly
  • Stale or invalid
  • Something else…
  • Send Cancel

  Difficulty 3/5 1-2 days Newbie friendliness 15/100

  OpenRLHF/OpenRLHF#1070 · Opened Jun 27, 2025

  Maintainers usually reply within 1 day