额外的奖励模型如何提升显卡利用率
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning, performance
Research direction
The issue is a configuration/architecture question about splitting a node's 8 GPUs between an external vLLM reward-model server and the RL training step; no file or test is named in the payload. The reporter notes the current reward path seems to use a value head and a single forward pass rather than vLLM, so the relevant entry points are the training script's resource-allocation arguments and the reward-model setup. Done means a clear answer or config that lets both roles share all GPUs, which requires maintainer design input rather than a newcomer edit.
Written by the indexing model from the issue text.
Description
奖励模型采用让大模型输出文本的形式，然后对评判结果解析，根据解析结果给定离散得分。
有没有什么设置可以在rollout、reward、train actor等步骤中，可以让对应模型占据节点的所有显卡。
当前有一个节点8张卡，4张卡额外启动了vllm奖励模型服务，另外4张用于rollout、train actor等剩下的步骤，但这样每个阶段只有4张卡工作，另外4张的利用率为0，想问下有没有提升显卡利用率的方法或配置。
我看现在的代码，如果使用奖励模型时，好像是使用一个value head，经过一次前向传播得到了reward，好像没有使用vllm。
- Dominant language
- Python
- Stars
- 10.1k
- Forks
- 1k
- Avg merge
- 5h 19m
- Merged PRs (30d)
- 4
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenRLHF/OpenRLHF
-
\fbox{} answers never extract — math reward is always 0 for fbox responsesPossibly taken @dhvani7014 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
PPO critic averages LR and grad-norm telemetry across training stepsPossibly taken @Madhustat claimed this 13 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 30/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 40/100
Maintainers usually reply within 1 day
-
math_utils.grade_answer ignores units when both reference and prediction carry one, so the right number with the wrong unit (or wrong rate) is graded correctPossibly taken @MaybeIcanShow claimed this 20 days ago. Open
Difficulty 3/5 Half a day Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Suboptimal forward dispatch order in make_experience causes unnecessary serializationPossibly taken @RainieLLM claimed this 26 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 28/100
OpenRLHF/OpenRLHF#1358 · 1 comment ·
Maintainers usually reply within 1 day
All issues in OpenRLHF/OpenRLHF
Similar issues
-
Path subtraction removes repeated suffix characters instead of one suffixPossibly taken @Gwerdonatus claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
joke2k/django-environ#671 ·
-
good first issue help wanted windows
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
test_pgbouncer deploys the server under test from Charmhub instead of the local build (track/1.31)Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
canonical/temporal-k8s-operator#195 · 1 comment ·
Maintainers usually reply within 4 days
-
`burr` fails to start when the Python path has a space in itPossibly taken @enekoreto claimed this today. Openstatus/needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 5 days
-
device support request
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
zigpy/zha-device-handlers#5461 ·
Maintainers usually reply within 4 days