Owner
benchflow-ai
2 indexed repositories · View on GitHub
-
skillsbench
SkillsBench evaluates how well skills work and how effective agents are at using them.
PDDL · 1830 stars
-
awesome-evals
A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintain...
1024 stars
Issues somebody may already be working on are listed last. List everything by date
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
benchflow-ai/skillsbench#1090 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
benchflow-ai/skillsbench#1086 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
benchflow-ai/skillsbench#1066 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
benchflow-ai/skillsbench#1065 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 35/100
benchflow-ai/awesome-evals#142 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 46/100
benchflow-ai/skillsbench#1062 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
benchflow-ai/skillsbench#1057 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 55/100
benchflow-ai/awesome-evals#90 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 20/100
benchflow-ai/skillsbench#1049 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
benchflow-ai/skillsbench#1046 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
benchflow-ai/skillsbench#1045 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 25/100
benchflow-ai/skillsbench#1041 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#1019 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
benchflow-ai/skillsbench#1020 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
benchflow-ai/skillsbench#1023 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
benchflow-ai/skillsbench#1004 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
benchflow-ai/skillsbench#999 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
benchflow-ai/skillsbench#1000 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 43/100
benchflow-ai/skillsbench#1001 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#995 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
benchflow-ai/skillsbench#987 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
benchflow-ai/skillsbench#988 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
benchflow-ai/skillsbench#981 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
benchflow-ai/skillsbench#963 · 1 comment · 2 reactions ·
-
Difficulty 2/5 Half a day Newbie friendliness 48/100
benchflow-ai/skillsbench#945 · 2 reactions ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
benchflow-ai/skillsbench#879 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#865 · 1 comment · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 50/100
benchflow-ai/skillsbench#856 · 3 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
benchflow-ai/skillsbench#836 ·
-
task
Difficulty 4/5 3-5 days Newbie friendliness 42/100
benchflow-ai/skillsbench#830 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#788 · 3 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
benchflow-ai/skillsbench#783 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#750 · 6 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
benchflow-ai/skillsbench#708 · 21 comments · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
benchflow-ai/skillsbench#704 · 1 comment · 1 reaction ·
-
18 task verifiers grade against files the agent can overwritePossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 15/100
benchflow-ai/skillsbench#1064 ·
-
[xlsx-recover-data] Ground truth and verifier disagree on Growth Analysis!B8 expected valuePossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
benchflow-ai/skillsbench#1050 · 1 comment ·
-
`weighted-gdp-calc`: instruction says years are in row 10, they're in row 9Possibly taken @linhongyu510 claimed this 68 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 35/100
benchflow-ai/skillsbench#1048 ·
-
`weighted-gdp-calc`: verifier CSV fallback is off by 8 rows, formula-only submissions always score 0Possibly taken @linhongyu510 claimed this 68 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
benchflow-ai/skillsbench#1047 ·
-
Remove answer leakage from dapt-intrusion-detection skills and clarify detection contractPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#1018 ·
-
Remove task-specific answer values from edit-pdf skillsPossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 55/100
benchflow-ai/skillsbench#1022 ·
-
Remove target-output leakage from video-silence-remover skillsPossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 47/100
benchflow-ai/skillsbench#1021 ·
-
python-scala-translation exposes a complete implementation in the agent imagePossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 48/100
benchflow-ai/skillsbench#1007 ·
-
Accept valid http.request_header in suricata-custom-exfil verifierPossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 55/100
benchflow-ai/skillsbench#1003 ·
-
Reset protected tests before grading spring-boot-jakarta-migrationPossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 5/5 Over a week Newbie friendliness 35/100
benchflow-ai/skillsbench#1002 ·
-
Package shock-analysis-demand data and specify verifier layout rulesPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
benchflow-ai/skillsbench#998 · 1 comment ·
-
Make simpo-code-reproduction verifier dependencies installablePossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#994 ·
-
Clarify join semantics for sales-pivot-analysisPossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 45/100
benchflow-ai/skillsbench#996 ·
-
Align pddl-airport-planning example syntax with verifier parserPossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 53/100
benchflow-ai/skillsbench#993 ·
-
Fix seismic-phase-picking SeisBench image build on DaytonaPossibly taken @xdotli claimed this 12 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 43/100
benchflow-ai/skillsbench#990 ·