DPO manual and CLI wizard reverse dpo_beta: they say a lower beta keeps the model closer to the reference model, but in TRL a higher beta does (F-W1-DOCS-TRAIN-07)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 68/100
- Issue 类型
- 文档
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- markdown, python
- 领域
- cli, documentation
调研方向
先查看 docs/usermanuals/en/training/dpo.md 中的 DPO 参数和调优表格（约第 111 行以及第 136-145 行），然后查看土耳其语对应文件 docs/usermanuals/tr/training/dpo.md（第 102 行以及第 125-134 行）。另外，编辑 forgelm/wizard/_collectors.py 中约第 380 行的向导提示字符串。forgelm/trainer.py 第 732 行只负责传递该值，应保持不变。完成标准是：两种语言中每处 beta 的说明以及向导提示都说明较高的 beta 会使策略更接近参考模型，并且提示建议在评估指标下降时提高 beta。
由索引模型根据 Issue 内容生成。
描述
Wave 3 · group W3 Docs · highest severity Medium · 1 finding(s) · effort docs fix
Summary
The DPO page tells users that a lower dpo_beta keeps the policy closer to the reference model and a higher one makes a more aggressive preference shift, and advises lowering beta when evaluation regresses. TRL, which receives the value unchanged, defines the opposite: a higher β means less deviation from the reference model. A user who follows the tip to fix over-fitting weakens the KL regularisation and moves the model further from the reference.
Primary finding
F-W1-DOCS-TRAIN-07 — DPO beta guidance reverses the regularisation direction
severity Medium · confidence high · checked against the installed TRL source and its reference docs
What happens. The parameter table (docs/usermanuals/en/training/dpo.md:111), the tuning table and the tip (docs/usermanuals/en/training/dpo.md:136-145) describe a lower β as staying close to the reference and a higher β as more aggressive; the Turkish mirror says the same (docs/usermanuals/tr/training/dpo.md:102, docs/usermanuals/tr/training/dpo.md:125-134). The CLI wizard prompt repeats it: dpo_beta — temperature / KL strength (lower → closer to reference model) (forgelm/wizard/_collectors.py:380). ForgeLM passes training.dpo_beta to TRL's DPOConfig unchanged (forgelm/trainer.py:732). The site's own wizard text already states the correct direction (site/js/translations.js:1651: "higher beta keeps it close to the reference"), so the site and the manual contradict each other.
Where.
docs/usermanuals/en/training/dpo.md:111(parameter table)docs/usermanuals/en/training/dpo.md:136-145(tuning table and tip)docs/usermanuals/tr/training/dpo.md:102docs/usermanuals/tr/training/dpo.md:125-134forgelm/wizard/_collectors.py:380(CLI wizard prompt)forgelm/trainer.py:732(value passed to TRL unchanged)
Evidence (re-verified on 2026-10-09).
- Installed TRL 1.7.1,
trl/trainer/dpo_config.py: "Parameter controlling the deviation from the reference model. Higher β means less deviation from the reference model." The official reference says the same: https://huggingface.co/docs/trl/dpo_trainer#trl.DPOConfig forgelm/trainer.py:732setskwargs["beta"] = self.config.training.dpo_betawith no transformation.
Recommended fix. Reverse the direction in the parameter table, the tuning table and the tip, in English and Turkish: a lower β weakens the KL penalty and lets the policy move further from the reference (a more aggressive preference shift); a higher β keeps it closer. When evaluation regresses while training loss keeps falling, advise a higher β (or fewer steps, or a lower learning rate) checked against held-out evaluation. Change the CLI wizard prompt to (higher → closer to reference model). Linking TRL's DPOConfig reference from the parameter row keeps the two from drifting again.
Acceptance criteria
- The English and Turkish DPO pages describe β the way TRL's
DPOConfigdoes, in the parameter table, the tuning table and the tip. - The CLI wizard prompt and the site wizard describe the same direction.
- Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
- The finding is resolved or explicitly closed with a reason.
Found by the September 2026 independent review of ForgeLM against commit cab2947 and re-verified against development at f94595f on 2026-10-09. Code links point at cab2947; line numbers may have moved since. Finding IDs (F-W1-…) are stable references for this backlog.
- 主要语言
- Python
- 星标
- 9
- 派生
- 1
- 平均合并
- 17 小时 14 分钟
- 30 天内合并 PR
- 17
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
HodeTech/ForgeLM 的其他 Issue
-
area: site bug good first issue severity: low wave: 4
难度 2/5 1-3 小时 新手友好度 75/100
HodeTech/ForgeLM#961 · 1 条评论 ·
维护者通常 1 天内回复
-
area: dev-tooling bug severity: low source: roadmap wave: 4
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
documentation good first issue severity: low source: review-2026-09 wave: 4
难度 1/5 1-3 小时 新手友好度 80/100
维护者通常 1 天内回复
-
documentation severity: medium source: review-2026-09 wave: 3
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
documentation i18n severity: low source: review-2026-10 wave: 4
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复
相似的 Issue
-
`lvm_compact` has no self-hosted twin, and `docs/158` §5 says every language-face component must未关闭bug sev-3 vm
难度 2/5 1-3 小时 新手友好度 65/100
-
难度 2/5 1-3 小时 新手友好度 82/100
sierra-research/tau2-bench#635 ·
维护者通常 1 天内回复
-
The add-ssm-model skill requires a Cython simulator; the contributing guide allows Python ones可能已有人在做 @fmuia 今天认领。 未关闭documentation skill
难度 2/5 1-3 小时 新手友好度 83/100
lnccbrown/ssm-simulators#406 ·
维护者通常 4 天内回复
-
难度 1/5 1 小时以内 新手友好度 95/100
btclib-org/btclib-ecc#309 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 74/100
autowarefoundation/autoware_carla_scenario#281 ·
维护者通常 1 天内回复