[Feature]: Add webarena_verified_hard benchmark subset configuration
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 83/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start with browsergym/experiments/src/browsergym/experiments/benchmark/configs.py at the TODO around line 163 and read how webarena_verified is configured. Use the linked upstream subset file as the source for the 258 hard task IDs. Done means DEFAULT_BENCHMARKS["webarena_verified_hard"]() returns the hard subset while the existing full benchmark remains unchanged.
Written by the indexing model from the issue text.
Description
Feature Request: webarena_verified_hard Benchmark Preset
Add a preset for the official 258-task WebArena-Verified hard subset to BrowserGym.
File: browsergym/experiments/src/browsergym/experiments/benchmark/configs.py (line 163, resolves existing TODO)
Breaking changes: None
Summary
DEFAULT_BENCHMARKS only exposes the full 812-task webarena_verified. This adds webarena_verified_hard, filtered to the 258 hard tasks from the upstream subset file.
| Full | Hard | |
|---|---|---|
| Tasks | 812 | 258 |
| Eval time / API cost | baseline | ~68% lower |
Why
- Faster, cheaper iteration on complex multi-step tasks
- One-line standardized access, no ad-hoc filter scripts:
DEFAULT_BENCHMARKS["webarena_verified_hard"]()
- Existing
webarena_verifiedstays untouched
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 202
- Avg merge
- 12d 2h
- Merged PRs (30d)
- 1
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ServiceNow/BrowserGym
-
WebArena-Verified reward averages evaluator scores instead of using the benchmark's task score (partial credit for failed tasks)Possibly taken @salma2vec claimed this 3 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 32/100
ServiceNow/BrowserGym#412 ·
-
full_reset() gives no signal whether the environment was reset; skipped reset is indistinguishable from a successful onePossibly taken @Tharun-Varshan-S claimed this 5 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 38/100
ServiceNow/BrowserGym#406 · 3 comments ·
-
AssistantBench scorer treats a thousands separator as a decimal point: "3,080,000" scores 0 against gold 3080000, while "1,000" scores 1.0 against gold 1Possibly taken @Andiii208 claimed this 25 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 25/100
ServiceNow/BrowserGym#404 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
ServiceNow/BrowserGym#401 ·
-
`miniwob.find-greatest` can be marked successful after selecting a non-maximum cardPossibly taken @jstar0 claimed this 101 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
ServiceNow/BrowserGym#392 ·
All issues in ServiceNow/BrowserGym
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
scikit-learn-contrib/category_encoders#553 ·
Maintainers usually reply within 1 day
-
tests/strategies_test.py:174 goes uncovered on about 1 run in 100, failing the coverage ratchetOpenflaky-watch
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
btclib-org/btclib#2839 ·
Maintainers usually reply within 1 day
-
BarrelList constructor drops false-valued iterable contentsPossibly taken @Gambit-Checkmate claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 19 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
vllm-project/vllm#63863 ·
Maintainers usually reply within 1 day