bigscience-workshop/evaluation

View on GitHub

Code and Data for Evaluation WG

Stars: 42
Forks: 24
Open beginner issues: 0
Indexed issues: 41

Dominant language: Python
License: No license data
Last GitHub push: Apr 28, 2022
Latest indexed: Sep 14, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: help wanted
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

41 open issues indexed

Add MKQA to Full Benchmark Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#73 · Opened Feb 7, 2022
Refactor task template to merge multilingual.json and english.json Open

bigscience-workshop/evaluation#64 · 1 assignee · Opened Sep 2, 2021
Wrap evaluation benchmark using HF-trainer Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

bigscience-workshop/evaluation#61 · 2 comments · 3 reactions · Opened Aug 25, 2021
Setup testing Open

engineering

bigscience-workshop/evaluation#57 · 2 comments · 3 assignees · Opened Aug 19, 2021
Start overleaf for benchmark tech report Open

documentation

bigscience-workshop/evaluation#54 · 1 comment · 3 reactions · 2 assignees · Opened Aug 16, 2021
translate validation prompts into all training languages Open

multilingual simple_benchmark

bigscience-workshop/evaluation#50 · 9 comments · 2 assignees · Opened Aug 12, 2021
benchmark mt5 on tydiqa prompting setup Open

simple_benchmark

bigscience-workshop/evaluation#49 · 1 comment · 2 assignees · Opened Aug 12, 2021
Convert validation code to work with Megatron as well as huggingface Open

engineering

Difficulty 5/5 Over a week Newbie friendliness 25/100

bigscience-workshop/evaluation#48 · Opened Aug 12, 2021
Create toy tasks/dummy code for fine-tuning evals Open

engineering

bigscience-workshop/evaluation#40 · 1 comment · 4 assignees · Opened Aug 10, 2021
Create Targeted Minimal Pair "Stress-Tests" for Sensitivity to Social Groups Open

social_impact

Difficulty 5/5 Over a week Newbie friendliness 15/100

bigscience-workshop/evaluation#38 · 2 comments · Opened Aug 10, 2021
Add CrowS-Pairs to Full Benchmark Open

In-Progress social_impact

bigscience-workshop/evaluation#37 · 5 comments · 2 assignees · Opened Aug 10, 2021
Add Jigsaw Toxicity Classification to Full Benchmark Open

social_impact

Difficulty 5/5 Over a week Newbie friendliness 20/100

bigscience-workshop/evaluation#36 · 4 comments · Opened Aug 10, 2021
Add WinoMT to Full Benchmark Open

social_impact

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#35 · 2 comments · Opened Aug 10, 2021
Add HANS to Full Benchmark Open

few_shot

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#34 · 3 comments · Opened Aug 10, 2021
Add MNLI to Full Benchmark Open

few_shot

Difficulty 5/5 Over a week Newbie friendliness 20/100

bigscience-workshop/evaluation#33 · 3 comments · Opened Aug 10, 2021
Add ANLI to Full Benchmark Open

few_shot

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#32 · 2 comments · Opened Aug 10, 2021
Add HuffPo Text Classification to Full Benchmark Open

few_shot

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#31 · 3 comments · Opened Aug 10, 2021
Add TyDiQA for non-training languages to Full Benchmark Open

few_shot multilingual

Difficulty 4/5 3-5 days Newbie friendliness 20/100

bigscience-workshop/evaluation#30 · Opened Aug 10, 2021
Add BioASQ to Full Benchmark Open

few_shot

bigscience-workshop/evaluation#29 · 2 comments · 1 assignee · Opened Aug 10, 2021
Add QASPER to Full Benchmark Open

few_shot

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#28 · 2 comments · Opened Aug 10, 2021
Add Edge Probing Suite to Full Benchmark Open

linguistic_structure

Difficulty 5/5 Over a week Newbie friendliness 15/100

bigscience-workshop/evaluation#27 · Opened Aug 10, 2021
Add LAMA to Full Benchmark Open

In-Progress linguistic_structure

bigscience-workshop/evaluation#26 · 6 comments · 1 assignee · Opened Aug 10, 2021
Add LinCE Testbed to Full Benchmark Open

In-Progress linguistic_structure multilingual

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#25 · 2 comments · Opened Aug 10, 2021
Add POS Tagging with UD to Full Benchmark Open

linguistic_structure multilingual

Difficulty 5/5 Over a week Newbie friendliness 20/100

bigscience-workshop/evaluation#24 · 1 comment · Opened Aug 10, 2021
Add QA-SRL to Full Benchmark Open

linguistic_structure

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#23 · 2 comments · Opened Aug 10, 2021
Add BLiMP to Full Benchmark Open

In-Progress linguistic_structure

bigscience-workshop/evaluation#22 · 5 comments · 1 assignee · Opened Aug 10, 2021
Add WikiANN to Full Benchmark Open

multilingual NER

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#21 · 7 comments · Opened Aug 10, 2021
Add MasaskhaNER to Full Benchmark Open

multilingual NER

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#20 · 5 comments · Opened Aug 10, 2021
Add CRD3 to Full Benchmark Open

NLG

bigscience-workshop/evaluation#19 · 2 comments · 1 assignee · Opened Aug 10, 2021
Add GEM E2E to Full Benchmark Open

NLG

Difficulty 3/5 1-2 days Newbie friendliness 25/100

bigscience-workshop/evaluation#18 · 2 comments · Opened Aug 10, 2021
Add GEM WebNLG to Full Benchmark Open

In-Progress NLG

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#17 · 2 comments · Opened Aug 10, 2021
Add GEM ToTTo to Full Benchmark Open

In-Progress NLG

bigscience-workshop/evaluation#16 · 3 comments · 1 assignee · Opened Aug 10, 2021
Add GEM Response Generation to Full Benchmark Open

NLG

Difficulty 5/5 Over a week Newbie friendliness 15/100

bigscience-workshop/evaluation#15 · Opened Aug 10, 2021
Add GEM WikiAuto to Full Benchmark Open

In-Progress NLG

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#14 · 1 comment · Opened Aug 10, 2021
Add GEM Wikilingua to Full Benchmark Open

NLG

Difficulty 3/5 1-2 days Newbie friendliness 25/100

bigscience-workshop/evaluation#13 · 1 comment · Opened Aug 10, 2021
Add GEM MLSum to Full Benchmark Open

help wanted In-Progress NLG

bigscience-workshop/evaluation#11 · 2 comments · 1 assignee · Opened Aug 10, 2021
Add Flores 101 to to Full Benchmark (as LM tasks, not MT tasks) Open

multilingual NLG

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#10 · 1 comment · Opened Aug 10, 2021
Add TyDiQA to Full Benchmark Open

multilingual NLU

Difficulty 5/5 Over a week Newbie friendliness 20/100

bigscience-workshop/evaluation#7 · 3 comments · Opened Aug 10, 2021
Add SuperGLUE Tasks to Full Benchmark Open

NLU

bigscience-workshop/evaluation#6 · 2 comments · 1 assignee · Opened Aug 10, 2021
Add DiaBLa to Full Benchmark Open

MT

bigscience-workshop/evaluation#5 · 5 comments · 1 assignee · Opened Aug 10, 2021
Add WMT to Full Benchmark Open

MT

Difficulty 4/5 3-5 days Newbie friendliness 25/100

bigscience-workshop/evaluation#4 · 2 comments · Opened Aug 10, 2021

bigscience-workshop/evaluation

Get new issues in your inbox