Fail to Reproduce WebArena Results with GenericAgent-GPT-4o

未关闭
  • 哪里不对？
#249 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
12/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
停滞
技术栈
python
领域
ai-infra-agents

调研方向

This is a reproduction question rather than a code change: the reporter ran make_study with benchmark "webarena" and AGENT_4o from agentlab.agents.generic_agent and got 25 against a leaderboard value of 31.4. Start by reading the study and agent configuration that make_study builds, then check whether the reporter's run matches the leaderboard setup. Done means a documented reproduction recipe or a confirmed explanation for the gap, and the issue has been unanswered for about a year, so expect to chase the maintainers first.

由索引模型根据 Issue 内容生成。

描述

bug

Dear AgentLab Authors,

Thank you for the great work! I'm trying to reproduce the WebArena Results with GenericAgent-GPT-4o. In particular, I used the following code. Everything should just follow AgentLab's default. However the number I got is 25 which is significantly lower than 31.4 as shown on the BrowserGym Leaderboard. Do you have any suggestions for the reproduction? Any code available to reproduce the performance ~31?

Thanks again for you great contribution to the community!

from agentlab.agents.generic_agent import AGENT_4o 

from agentlab.experiments.study import make_study
from agentlab.experiments.study import Study

study = make_study(
    benchmark="webarena", 
    agent_args=[AGENT_4o],
    comment="repo 4o agent",
)



study.run(n_jobs=5)
主要语言
Python
星标
644
派生
134
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

ServiceNow/AgentLab 的其他 Issue

查看 ServiceNow/AgentLab 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。