Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Proposal: EvalPort adapter to export ExpResult data as portable eval results

Open
  • Is something wrong?
#401 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
python
Domain
ai

Research direction

Start with browsergym/experiments/src/browsergym/experiments/loop.py and read how ExpResult exposes summary_info and exp_args. The issue proposes a standalone adapter but does not establish whether maintainers want it in this repository or define an accepted interface, so clarify scope before starting. If approved, the proposal describes a standalone package with tests and a README; done would include converting results and validating the result set.

Written by the indexing model from the issue text.

Description

EvalPort (https://github.com/adhabnr-ux/evalport) is a small open interchange spec + SDK (evalport-sdk on PyPI) for portable LLM/agent eval datasets and results, so results produced by one framework can be validated and read by tooling that doesn't know that framework's internals.

I read through browsergym/experiments/src/browsergym/experiments/loop.py and think BrowserGym's ExpResult is a genuinely good fit, not just eval-adjacent:

  • get_exp_result(exp_dir) / yield_all_exp_results(savedir_base) already give you a per-task-run iterator over a benchmark run.
  • ExpResult.summary_info is a real structured per-episode record: n_steps, cum_reward, cum_raw_reward, terminated, truncated, err_msg, stack_trace, plus flattened stats.* (aggregated StepInfo.stats, itself built from AbstractBrowserTask.validate()'s (reward, done, message, info) return).
  • ExpResult.exp_args (ExpArgs → agent_args, env_args.task_name, env_args.task_seed) gives a stable per-run identifier.

That maps cleanly onto EvalPort's per-test-case result model: one ExpResult -> one result record (id from task_name/task_seed, pass/fail from terminated + absence of err_msg, score from cum_reward, everything else as metadata).

Sketch of what a zero-footprint browsergym-openeval-adapter package would let you do:

# today
from browsergym.experiments import yield_all_exp_results

for exp_result in yield_all_exp_results("./results/my_run"):
    summary = exp_result.summary_info   # n_steps, cum_reward, terminated, truncated, err_msg, stats.*
    args = exp_result.exp_args          # env_args.task_name, env_args.task_seed
    print(args.env_args.task_name, summary["cum_reward"], summary.get("terminated"))

# with an adapter
from browsergym_openeval_adapter import results_to_openeval
from openeval.validate import validate_result_set

result_set = results_to_openeval(yield_all_exp_results("./results/my_run"))
validate_result_set(result_set)
# each ExpResult becomes one OpenEval result:
#   id       = f"{args.env_args.task_name}_{args.env_args.task_seed}"
#   passed   = bool(summary.get("terminated")) and not summary.get("err_msg")
#   score    = summary.get("cum_reward")
#   metadata = {"n_steps": ..., "cum_raw_reward": ..., "stack_trace": ..., **stats}

This would let anyone who runs BrowserGym benchmarks (MiniWoB, WebArena, WorkArena, etc.) drop their savedir_base results into any OpenEval-consuming tool without writing custom parsing against the pickle/json layout.

Happy to build this myself and open it as a PR (adapters/browsergym-openeval-adapter/, standalone installable package, depends on evalport-sdk, includes tests + README) if that's of interest — just wanted to check first given BrowserGym has no CONTRIBUTING.md spelling out a preferred process for this kind of thing.

— Sahi, independent contributor, maintainer of EvalPort

Dominant language
Python
Stars
1.4k
Forks
202
Avg merge
6d 12h
Merged PRs (30d)
2

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ServiceNow/BrowserGym

All issues in ServiceNow/BrowserGym

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.