facebookresearch/ProgramBench

View on GitHub

Can Language Models Rebuild Programs From Scratch?

Stars: 928
Forks: 67
Open beginner issues: 0
Indexed issues: 12

Dominant language: Python
License: MIT
Last GitHub push: Sep 8, 2026
Latest indexed: Sep 15, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

11 open issues indexed

pytest-rerunfailures 16.6.1 duplicates JUnit records and inflates scores Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

facebookresearch/ProgramBench#64 · Opened Sep 4, 2026
Reference oracle and required build output share ./executable, causing agents to overwrite the oracle Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

facebookresearch/ProgramBench#60 · Opened Aug 15, 2026
csview: evaluator compatibility issues resolved — unchanged submission reaches official SOLVED (✅ 335 tests) Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

facebookresearch/ProgramBench#59 · Opened Aug 8, 2026
ProgramBench csview: false not_run records and missing results.xml Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

facebookresearch/ProgramBench#56 · Opened Jul 31, 2026
`eval_clean_hashes` is missing or stale for some `task_cleanroom_v6` images Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

facebookresearch/ProgramBench#54 · Opened Jul 17, 2026
Right of Reply: Construct-validity Audit Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

facebookresearch/ProgramBench#50 · 1 comment · Opened Jun 30, 2026
Clarify and harden cleanroom rules around reference executable instrumentation Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

facebookresearch/ProgramBench#44 · 1 comment · Opened Jun 18, 2026
Request for model trajectories Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

facebookresearch/ProgramBench#38 · 1 comment · Opened Jun 12, 2026
Can the agent distinguish stdout and stderr? Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 64/100

facebookresearch/ProgramBench#37 · 2 comments · Opened Jun 10, 2026
Request gold compile.sh scripts Open

question

Difficulty 4/5 3-5 days Newbie friendliness 45/100

facebookresearch/ProgramBench#23 · 3 comments · 2 reactions · Opened May 14, 2026
Some task_cleanroom Docker images do not contain all documentation Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

facebookresearch/ProgramBench#7 · 5 comments · Opened May 6, 2026

facebookresearch/ProgramBench

Get new issues in your inbox