Add QASPER to Full Benchmark
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start by reviewing how the Full Benchmark currently incorporates datasets, then determine the requirements for adding QASPER and whether FLEX is relevant. Done means the intended dataset is integrated into the Full Benchmark with its evaluation behavior verified.
Written by the indexing model from the issue text.
Description
use to test generalization to unseen domain; maybe use FLEX?
- Dominant language
- Python
- Stars
- 42
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from bigscience-workshop/evaluation
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Refactor task template to merge multilingual.json and english.jsonMay be free again @wilsonyhlee claimed this 1855 days ago, and no pull request is open. Open
bigscience-workshop/evaluation#64 · 1 assignee ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
bigscience-workshop/evaluation#61 · 2 comments · 3 reactions ·
-
Setup testingMay be free again @tttyuntian claimed this 1871 days ago, and no pull request is open. Openengineering
bigscience-workshop/evaluation#57 · 2 comments · 3 assignees ·
-
Start overleaf for benchmark tech reportMay be free again @arunraja-hub claimed this 1866 days ago, and no pull request is open. Opendocumentation
bigscience-workshop/evaluation#54 · 1 comment · 3 reactions · 2 assignees ·
All issues in bigscience-workshop/evaluation
Similar issues
-
good first issue hacktoberfest
Difficulty 1/5 Under an hour Newbie friendliness 92/100
RogueAlg0/taken#387 · 4 comments ·
Maintainers usually reply within 1 day
-
tool-calling
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
vllm-project/vllm#59838 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
raullenchai/Rapid-MLX#4042 ·
Maintainers usually reply within 1 day
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 92/100
transitmatters/mbta-slow-zone-bot#70 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
open-webui/open-webui#31871 · 1 comment ·
Maintainers usually reply within 1 day