RFC: Deprecate allow_training_without_logprobs option

首先审阅提案和相关的 PR #527，然后在 Python 代码库中定位 allow_training_without_logprobs 选项的所有使用位置。确认是否有任何用户依赖该选项，以及是否接受移除该选项；完成的标准是维护者作出决定，随后彻底移除该选项及其替代训练路径。

Summary

We're considering removing the allow_training_without_logprobs option from ART. This RFC is to gather community feedback before making this change.

Background

The allow_training_without_logprobs option allows training without requiring generation logprobs from the model. However, this approach has several drawbacks:

1. Importance sampling requires logprobs for stable training: In our experiments and in the wider RL community, having generation logprobs is essential for importance sampling, which is critical for stable training results. Training without them leads to less reliable outcomes.

2. Code complexity: Maintaining this alternative path adds complexity to the codebase and makes it harder to reason about the training flow.

3. Subtle bugs: The additional code path creates opportunities for subtle bugs. For example, in PR #527 we discovered tool-call tokenization issues that were partially enabled by this mode's complexity.

Proposal

Remove the allow_training_without_logprobs option entirely, simplifying the codebase and ensuring all users benefit from the more robust training path that uses logprobs.

Request for Feedback

Is anyone in the community actively using allow_training_without_logprobs with good results?

If you're using this option and it's working well for your use case, please let us know:

• What is your use case?
• Why do you need to train without logprobs?
• What results are you seeing?

If we don't hear from users who depend on this feature, we plan to remove it in an upcoming release.

Related: #527