zapier/AutomationBench

A benchmark for evaluating AI agents on realistic business workflows

Stars: 290
Forks: 45
Open beginner issues: 0
Indexed issues: 13

Dominant language: Python
License: No license data
Last GitHub push: Aug 4, 2026
Latest indexed: Sep 20, 2026
Contributing guide: No contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

13 open issues indexed

assertion_results: an excluded assertion carries no reason, and authored vs runtime exclusions are indistinguishable Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

zapier/AutomationBench#25 · Opened Sep 8, 2026
Bugs with benchmark tasks Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

zapier/AutomationBench#24 · Opened Sep 8, 2026
jira_projects_search ignores projects seeded in JiraState.projects Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

zapier/AutomationBench#23 · Opened Sep 3, 2026
Mapping AutomationBench's Task/AssertionRegistry/partial_credit onto EvalPort's open TestCase/Grader/Result interchange Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

zapier/AutomationBench#22 · Opened Aug 24, 2026
Several rubric handlers don't honor list-valued `*_contains`, contrary to the documented behavior Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

zapier/AutomationBench#20 · Opened Aug 20, 2026
`partial_credit` rebuilds the free-assertion baseline from the raw task dict, re-running every `default_factory` Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

zapier/AutomationBench#19 · Opened Aug 20, 2026
Question: private evaluation submission and external agent eligibility Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

zapier/AutomationBench#18 · Opened Aug 20, 2026
Question about the significant performance boost in the latest update Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

zapier/AutomationBench#17 · 2 reactions · Opened Aug 20, 2026
About the parameter configurations you used for the results on the public benchmark dataset Open

Difficulty 1/5 Under an hour Newbie friendliness 35/100

zapier/AutomationBench#15 · Opened Aug 13, 2026
schema definition mismatch between api_search and api_fetch Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

zapier/AutomationBench#14 · Opened Aug 9, 2026
`api_fetch`: the JSON string `"null"` in `params`/`body` crashes any handler that unpacks the argument, surfacing a raw `TypeError` to the agent Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

zapier/AutomationBench#12 · 2 comments · 2 reactions · Opened Aug 7, 2026
EvalScope integration: one-command evaluation support for AutomationBench Open

Difficulty 1/5 1-3 hours Newbie friendliness 70/100

zapier/AutomationBench#9 · Opened Jul 24, 2026
Discrepancy of gemini-3.5-flash public-set pass rate: independent rerun measures 29.9% vs README's 14.83% Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

zapier/AutomationBench#7 · 5 comments · 1 reaction · Opened Jul 20, 2026

A short digest of beginner-friendly GitHub issues.