Verify evals on Papers with Code
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 45/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- huggingface
- Domain
- documentation, machine-learning
Research direction
Start with the linked Papers with Code pages for ProRL Agent and Polar and compare their listed evaluations with the paper results. Verify the scores, model names, benchmark protocols, and openness metadata, then correct any discrepancies or report that the entries are accurate.
Written by the indexing model from the issue text.
Description
Hi,
Niels here from the open-source team at Hugging Face.
I've made the following papers and their evaluation results available on Papers with Code:
- ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents — 1 paper-native evaluation.
- Polar: Agentic RL on Any Harness at Scale — 1 paper-native evaluation.
Would it be possible to verify these results and let me know if any score, model name, benchmark protocol, or openness metadata should be corrected?
You can also edit the task, methods, project page, and GitHub URL directly from each paper page using your Hugging Face account.
Kind regards,
Niels
- Dominant language
- Python
- Stars
- 857
- Forks
- 96
- Avg merge
- 6h 52m
- Merged PRs (30d)
- 1
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-NeMo/ProRL-Agent-Server
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
opencode harness captures zero completions — config doesn't include gateway baseURLMay be free again @billxbf claimed this 137 days ago, and no pull request is open. Openbug
NVIDIA-NeMo/ProRL-Agent-Server#39 · 1 comment · 1 assignee ·
All issues in NVIDIA-NeMo/ProRL-Agent-Server
Similar issues
-
Use mhctools predictor_required=False for saved-result rescoringPossibly taken @iskandr claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
521xueweihan/HelloGitHub#4051 · 1 comment ·
-
bug documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
awslabs/visual-asset-management-system#514 ·
Maintainers usually reply within 1 day
-
backend bug spec:REQ-053
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
nolte/kamerplanter#2547 ·
Maintainers usually reply within 1 day