Bug report in otb project
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning
Research direction
Read the underthinking-bench branch in projects/otb/eval.py around lines 39-40, then inspect the failing metadata access in projects/otb/evals/underthink_eval.py around lines 33-34. Confirm that entries with source values aime or hmmt use the math evaluator while other UnderThinkingBench entries retain the existing evaluator behavior.
Written by the indexing model from the issue text.
Description
In UnderThinkingBench, there exists some items without "source_dataset", but "source" instead with values "aime" and "hmmt", which seems to be in the UnderThinkingBench-Math subset.
UnderThinkingBench entry:
https://github.com/facebookresearch/RAM/blob/264c0479aa2df0ea697a0fadbb10356d1e4ba41f/projects/otb/eval.py#L39-L40
Error encountered here:
https://github.com/facebookresearch/RAM/blob/264c0479aa2df0ea697a0fadbb10356d1e4ba41f/projects/otb/evals/underthink_eval.py#L33-L34
Possible solution in eval.py:
import json
# ...
elif subset == "underthinking-bench" or "underthinking" in subset:
metadata = json.loads(row["metadata"])
if "source" in metadata and metadata["source"] in ["aime", "hmmt"]:
acc = eval_math(row, tokenizer, model_name)
else:
acc = eval_underthink(row)
- Dominant language
- Python
- Stars
- 391
- Forks
- 51
- Avg merge
- 1h 38m
- Merged PRs (30d)
- 6
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from facebookresearch/RAM
-
Inquiry about the Self-Taught EvaluatorMay be free again @xianxl claimed this 635 days ago, and no pull request is open. Open
facebookresearch/RAM#22 · 2 assignees ·
-
Could you please let us know when the evaluation of the LI command feature will be opened?May be free again @jxmsML claimed this 725 days ago, and no pull request is open. Open
facebookresearch/RAM#13 · 1 comment · 1 assignee ·
All issues in facebookresearch/RAM
Similar issues
-
bug needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
debpalash/VoiceStudio#2624 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Make Catch2 optional when `RDK_BUILD_CPP_TESTS=OFF`Possibly taken @pechersky claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 2 days
-
There are a few redundant calls to `fdesc._setCloseOnExec()`Possibly taken @gudnimg claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day