WebArena-Verified reward averages evaluator scores instead of using the benchmark's task score (partial credit for failed tasks)
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 32/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- playwright, python
- 领域
- backend, testing-qa
调研方向
Read WebArenaVerifiedEvaluator.__call__ in browsergym/webarena_verified/src/browsergym/webarena_verified/evaluators.py around lines 124–131, then inspect the harness TaskEvalResult.create behavior described in the issue. Reproduce task 44 with one passing and one failing evaluator, and add a test for the task-level score. Done means BrowserGym returns the harness score of 0.0 for failed or errored tasks instead of averaging evaluator scores; the issue author says they plan to open a PR.
由索引模型根据 Issue 内容生成。
描述
WebArenaVerifiedEvaluator.__call__ returns the mean of the per-evaluator scores when a task has several evaluators:
webarena-verified scores a task differently. In TaskEvalResult.create (webarena-verified 1.2.3, types/eval.py), the task score is 1.0 only if every evaluator scores 1.0, and 0.0 if any evaluator fails or errors. A task whose answer is correct but whose required backend change was never made is a failure in the benchmark. In BrowserGym it gets a reward of 0.5.
This happens on a lot of tasks: 488 of the 812 tasks in webarena-verified.json have two or more evaluators (391 have 2, 97 have 3 to 11). AgentLab reports avg_reward, the mean of cum_reward, so these partial rewards go straight into the reported success rate. That number is then higher than what the official harness would report for the same runs.
Reproduced on main at 4ccc723 with webarena-verified 1.2.3. I called the evaluator on task 44 (AgentResponseEvaluator + NetworkEventEvaluator), with the harness mocked to return its own TaskEvalResult for one passing and one failing evaluator. The browser trace, WebArena instance and Playwright were stubbed.
per-evaluator scores=[1.0, 0.0] webarena-verified task score=0.0 (failure) BrowserGym reward=0.5
The same happens when one evaluator passes and the other errors: the harness reports status error, score 0.0, and BrowserGym returns 0.5.
The docstring of __call__ already says it returns "1.0 or 0.0". I'll open a PR that returns results.score, the harness's task score, and adds a test.
- 主要语言
- Python
- 星标
- 1.4k
- 派生
- 202
- 平均合并
- 6 天 15 小时
- 30 天内合并 PR
- 3
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
ServiceNow/BrowserGym 的其他 Issue
-
[Feature Proposal]: Add experiment lifecycle callbacks to `ExpArgs` in `loop.py`可能已有人在做 @sriram687 于 3 天前认领。 未关闭
ServiceNow/BrowserGym#417 · 已指派 1 人 ·
-
AssistantBench scorer treats a thousands separator as a decimal point: "3,080,000" scores 0 against gold 3080000, while "1,000" scores 1.0 against gold 1可能已有人在做 @Andiii208 于 29 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 25/100
ServiceNow/BrowserGym#404 · 1 条评论 ·
-
难度 5/5 一周以上 新手友好度 28/100
ServiceNow/BrowserGym#401 ·
-
`miniwob.find-greatest` can be marked successful after selecting a non-maximum card可能已有人在做 @jstar0 于 105 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 25/100
ServiceNow/BrowserGym#392 ·
-
Issues with the coordinates action space可能已有人在做 @FazeelUsmani 于 187 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 25/100
ServiceNow/BrowserGym#375 · 2 条评论 ·
查看 ServiceNow/BrowserGym 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
namespace operations
难度 1/5 1 小时以内 新手友好度 65/100
EclipseFdn/open-vsx.org#15057 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
FinanceFlash/unvibecode#349 ·
-
Mission statement: why this is free, in the README, Why This Exists and the wiki Home可能已有人在做 @ChiefGyk3D 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
ChiefGyk3D/Clouds-Against-Humanity#348 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 74/100
vllm-project/vime#508 ·
维护者通常 4 天内回复