brain-score/language

View on GitHub

Evaluate computational models on their alignment to behavioral and neural measurements in the domain of language

Stars: 45
Forks: 21
Open beginner issues: 0
Indexed issues: 41
Avg merge: 1h 11m
Merged PRs (30d): 3

Dominant language: Jupyter Notebook
License: MIT
Last GitHub push: Sep 17, 2026
Latest indexed: Sep 14, 2026
Contributing guide: No contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

41 open issues indexed

build status not showing Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

brain-score/language#324 · Opened Jan 30, 2026
Severe slowdown in `scipy.stats.pearsonr` calls when using Dask (via result_caching) Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

brain-score/language#241 · Opened Jan 26, 2025
add EWOK benchmark Open

benchmark-request

Difficulty 5/5 Over a week Newbie friendliness 20/100

brain-score/language#239 · Opened Jun 23, 2024
bug in Futrell2018 corpus? Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#237 · Opened May 13, 2024
add generic unit tests for all models Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#234 · Opened Mar 7, 2024
add word length baseline model Open

model-request

Difficulty 5/5 Over a week Newbie friendliness 25/100

brain-score/language#233 · Opened Mar 3, 2024
BS Core utility hooks refactor Open

brain-score/language#231 · 2 comments · 2 assignees · Opened Jan 16, 2024
add Mistral and Mixtral Open

model-request

Difficulty 3/5 1-2 days Newbie friendliness 35/100

brain-score/language#228 · Opened Dec 11, 2023
add "Narratives" fMRI recording Open

benchmark-request

Difficulty 4/5 3-5 days Newbie friendliness 30/100

brain-score/language#223 · Opened Nov 24, 2023
syntax gym single benchmark parent should be syntax gym Open

Difficulty 3/5 1-2 days Newbie friendliness 15/100

brain-score/language#185 · Opened May 23, 2023
SyntaxGym redundant suites? Open

question

brain-score/language#184 · 8 comments · 1 assignee · Opened May 23, 2023
add mosaic's MPT models Open

model-request

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#180 · Opened May 13, 2023
add generic unit tests for all benchmarks Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

brain-score/language#177 · Opened May 1, 2023
Ensure that new model & new benchmark get scored together when added simultaneously Open

bug

brain-score/language#175 · 1 assignee · Opened Apr 29, 2023
Run memory-intense / slow / GPU-only tests on Jenkins Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#172 · Opened Apr 28, 2023
add python 3.7 to build matrix again Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

brain-score/language#171 · Opened Apr 28, 2023
add behavioral benchmarks from Huang et al. 2023 Open

benchmark-request

Difficulty 4/5 3-5 days Newbie friendliness 35/100

brain-score/language#169 · Opened Apr 22, 2023
add StableLM Open

model-request

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#168 · Opened Apr 20, 2023
add Pythia Open

model-request

Difficulty 4/5 3-5 days Newbie friendliness 35/100

brain-score/language#167 · Opened Apr 13, 2023
add ScLM Open

model-request

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#150 · 1 comment · Opened Apr 3, 2023
Routine cleanup for `automerge-web` PRs Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

brain-score/language#149 · Opened Apr 2, 2023
separate test status display for library and plugins Open

enhancement

brain-score/language#141 · 1 assignee · Opened Mar 26, 2023
add Alpaca Open

model-request

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#138 · Opened Mar 17, 2023
add LLAMA Open

model-request

Difficulty 5/5 Over a week Newbie friendliness 20/100

brain-score/language#137 · Opened Mar 7, 2023
add CLIP Open

model-request

Difficulty 5/5 Over a week Newbie friendliness 25/100

brain-score/language#135 · Opened Mar 3, 2023
Travis-CI Builds in Serial? Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

brain-score/language#126 · Opened Dec 15, 2022
Dead code hunk Open

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

brain-score/language#124 · Opened Dec 15, 2022
automatically trigger benchmark runs for metric/data changes Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

brain-score/language#104 · Opened Nov 2, 2022
protect against duplicate identifiers Open

brain-score/language#103 · 1 assignee · Opened Nov 2, 2022
make clear that plugins are independent of one another Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

brain-score/language#93 · Opened Oct 26, 2022
better specify recording target Open

Difficulty 2/5 1-3 hours Newbie friendliness 30/100

brain-score/language#92 · Opened Oct 26, 2022
plugin tests summary only shows 1 plugin even though 2 failed Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

brain-score/language#86 · Opened Oct 18, 2022
is there a function for loading local files similar to load_from_s3 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

brain-score/language#67 · 1 comment · Opened Sep 9, 2022
display available {data,metrics,benchmarks,models} in documentation Open

brain-score/language#57 · 2 comments · 1 reaction · 1 assignee · Opened Aug 24, 2022
`digest_text` Huggingface tokenization logic may insert extraneous BOS/EOS tokens Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

brain-score/language#56 · Opened Aug 22, 2022
get libraries into pypi Open

brain-score/language#44 · 1 assignee · Opened Aug 20, 2022
rename 'atlas' to 'network' Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

brain-score/language#43 · Opened Aug 20, 2022
use more descriptive attrs keys than 'raw' Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

brain-score/language#42 · Opened Aug 20, 2022
mapping metric identifier Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

brain-score/language#41 · Opened Aug 20, 2022
change assembly S3 prefix Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

brain-score/language#40 · 3 comments · Opened Aug 20, 2022
Update Futrell2018 benchmark Open

enhancement

brain-score/language#28 · 1 comment · 1 assignee · Opened Aug 3, 2022

brain-score/language

Get new issues in your inbox