Stale To-Do in vLLM/app.py for supporting prompt + generation token IDs
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 65/100
Research direction
Start in responses_api_models/vllm_model/app.py at line 251 and compare the current vLLM and NeMo RL versions with the commented token-ID options. Check how return_token_id_information is handled and run the relevant model tests if available. Done means the stale TODO is addressed and the requested prompt and generation token ID support is wired through.
Written by the indexing model from the issue text.
Description
In responses_api_models/vllm_model/app.py, on line 251, there is a block of code that has a TODO regarding adding support for getting generation and prompt token IDs using the flag return_tokens_ids=True. The TODO mentions the upgrade should be done once NeMo RL upgrades to vLLM 0.10.2. NeMo RL is already on 0.17.1.
if self.config.return_token_id_information:
body_dict |= dict(
logprobs=True,
# Typically passed via OpenAI client extra_body.
return_tokens_as_token_ids=True,
# TODO add this when NeMo RL upgrades to vLLM 0.10.2 support for prompt token ids
# For prompt and generation token IDs
# return_token_ids=True,
# For prompt token IDs
# prompt_logprobs=0,
)
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 2d 40m
- Merged PRs (30d)
- 320
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/Gym
-
community-request waiting-on-maintainers
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
NVIDIA-NeMo/Gym#3500 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 74/100
NVIDIA-NeMo/Gym#3328 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
NVIDIA-NeMo/Gym#3312 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
NVIDIA-NeMo/Gym#2994 ·
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
NVIDIA-NeMo/Gym#2937 · 1 comment ·
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100