Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Proposal: EvalPort adapter to export ExpResult data as portable eval results

Abierto
  • ¿Algo no está bien?
#401 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
28/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Tranquilo
Stack tecnológico
python
Área
ai

Línea de trabajo

Start with browsergym/experiments/src/browsergym/experiments/loop.py and read how ExpResult exposes summary_info and exp_args. The issue proposes a standalone adapter but does not establish whether maintainers want it in this repository or define an accepted interface, so clarify scope before starting. If approved, the proposal describes a standalone package with tests and a README; done would include converting results and validating the result set.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

EvalPort (https://github.com/adhabnr-ux/evalport) is a small open interchange spec + SDK (evalport-sdk on PyPI) for portable LLM/agent eval datasets and results, so results produced by one framework can be validated and read by tooling that doesn't know that framework's internals.

I read through browsergym/experiments/src/browsergym/experiments/loop.py and think BrowserGym's ExpResult is a genuinely good fit, not just eval-adjacent:

  • get_exp_result(exp_dir) / yield_all_exp_results(savedir_base) already give you a per-task-run iterator over a benchmark run.
  • ExpResult.summary_info is a real structured per-episode record: n_steps, cum_reward, cum_raw_reward, terminated, truncated, err_msg, stack_trace, plus flattened stats.* (aggregated StepInfo.stats, itself built from AbstractBrowserTask.validate()'s (reward, done, message, info) return).
  • ExpResult.exp_args (ExpArgs → agent_args, env_args.task_name, env_args.task_seed) gives a stable per-run identifier.

That maps cleanly onto EvalPort's per-test-case result model: one ExpResult -> one result record (id from task_name/task_seed, pass/fail from terminated + absence of err_msg, score from cum_reward, everything else as metadata).

Sketch of what a zero-footprint browsergym-openeval-adapter package would let you do:

# today
from browsergym.experiments import yield_all_exp_results

for exp_result in yield_all_exp_results("./results/my_run"):
    summary = exp_result.summary_info   # n_steps, cum_reward, terminated, truncated, err_msg, stats.*
    args = exp_result.exp_args          # env_args.task_name, env_args.task_seed
    print(args.env_args.task_name, summary["cum_reward"], summary.get("terminated"))

# with an adapter
from browsergym_openeval_adapter import results_to_openeval
from openeval.validate import validate_result_set

result_set = results_to_openeval(yield_all_exp_results("./results/my_run"))
validate_result_set(result_set)
# each ExpResult becomes one OpenEval result:
#   id       = f"{args.env_args.task_name}_{args.env_args.task_seed}"
#   passed   = bool(summary.get("terminated")) and not summary.get("err_msg")
#   score    = summary.get("cum_reward")
#   metadata = {"n_steps": ..., "cum_raw_reward": ..., "stack_trace": ..., **stats}

This would let anyone who runs BrowserGym benchmarks (MiniWoB, WebArena, WorkArena, etc.) drop their savedir_base results into any OpenEval-consuming tool without writing custom parsing against the pickle/json layout.

Happy to build this myself and open it as a PR (adapters/browsergym-openeval-adapter/, standalone installable package, depends on evalport-sdk, includes tests + README) if that's of interest — just wanted to check first given BrowserGym has no CONTRIBUTING.md spelling out a preferred process for this kind of thing.

— Sahi, independent contributor, maintainer of EvalPort

Lenguaje dominante
Python
Estrellas
1.4k
Forks
202
Merge medio
6 d 15 h
PR fusionados (30 d)
3

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ServiceNow/BrowserGym

Todos los issues de ServiceNow/BrowserGym

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.