Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

full_reset() gives no signal whether the environment was reset; skipped reset is indistinguishable from a successful one

Cerrado
  • ¿Algo no está bien?
#406 3 comentarios 0 reacciones 0 asignados Ver en GitHub

@Tharun-Varshan-S ya está trabajando en esto.

Desde el 21/10/2026.

  • #411 de @Tharun-Varshan-S — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
tooling

Línea de trabajo

Read browsergym/webarena/src/browsergym/webarena/instance.py around WebArenaInstance.full_reset() and the three call sites in browsergym/experiments/src/browsergym/experiments/benchmark/utils.py inside prepare_backend(). Compare the skipped and successful reset paths, then decide with maintainers how the result should be surfaced. Done means callers can distinguish a skipped reset from a successful one without changing the current default behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

WebArenaInstance.full_reset() gives the caller no way to tell whether a reset happened.

prepare_backend() calls it with no arguments, so skip_if_not_set=True is always in force. With no
reset endpoint configured, the reset is skipped with a logged warning — but the function then falls
through to the warm-start block and returns None, which is exactly what it returns after a
successful reset. So the benchmark proceeds against an environment that was never reset, and the
caller has no signal to distinguish the two cases. The only evidence is one WARNING line in a long
benchmark log.

This matters because the upstream WebArena documentation treats the reset as required for correct
evaluation:

After evaluating the 812 examples, reset the environment to the initial state
— web-arena-x/webarena, README

All 812 shipped task configs also carry require_reset: false, which is consistent with the intended
protocol being one reset per sweep rather than per task. So the single reset that BrowserGym performs
is the one carrying the reproducibility guarantee — and it is the one that can silently not happen.

Where

browsergym/webarena/src/browsergym/webarena/instance.py:59

def full_reset(self, skip_if_not_set: bool = True):
    base_url = os.environ.get(self.RESET_URL_VAR, None)
    if not base_url:
        logger.error(f"Environment variable {self.RESET_URL_VAR} is missing or empty, required for a full instance reset.")
        if skip_if_not_set:
            logger.warning("Skipping automated reset. Make sure the instance has been manually reset.")
        else:
            raise RuntimeError(f"Could not reset instance, aborting.")
    else:
        ...  # performs the reset, polls /status until "Ready for duty!"

    # warm-start the instance (navigate to every domain)   <-- reached in BOTH branches
    retries_left = 3
    while retries_left:
        ...
        self._check_is_reachable(timeout=60)

Note the fall-through: the warm-start block sits outside the if/else, so the skipped branch does not
return early — and neither branch returns a value that identifies itself.

Called from browsergym/experiments/src/browsergym/experiments/benchmark/utils.py in
prepare_backend(), with no arguments, for all three backends:

line backend reset env var
147 webarena WA_FULL_RESET
176 webarena_verified WA_FULL_RESET
205 visualwebarena VWA_FULL_RESET

(VisualWebArenaInstance subclasses WebArenaInstance and overrides RESET_URL_VAR, so it inherits
the same behaviour.)

Reproduction

No WebArena instance needed — this is the control flow, not the network:

git clone https://github.com/ServiceNow/BrowserGym && cd BrowserGym   # verified at 0.14.4, c05d7f3

Every call site uses the silent default:

grep -n "full_reset()" browsergym/experiments/src/browsergym/experiments/benchmark/utils.py
# 147:            default_instance.full_reset()
# 176:            default_instance.full_reset()
# 205:            default_instance.full_reset()

And the branch behaviour, run directly (no live instance needed):

import importlib.util, logging, os
logging.basicConfig(level=logging.INFO)
spec = importlib.util.spec_from_file_location(
    "wa_instance", "browsergym/webarena/src/browsergym/webarena/instance.py")
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)

os.environ.pop("WA_FULL_RESET", None)
inst = mod.WebArenaInstance.__new__(mod.WebArenaInstance)  # bypass __init__, needs live env vars
inst.urls = {"probe": "https://example.com"}               # stand in for the six sites

print("silent branch returns:", inst.full_reset())         # -> None
inst.full_reset(skip_if_not_set=False)                     # -> RuntimeError, by contrast

Observed on 0.14.4:

ERROR    Environment variable WA_FULL_RESET is missing or empty, required for a full instance reset.
WARNING  Skipping automated reset. Make sure the instance has been manually reset.
silent branch returns: None
fail-closed path raises: Could not reset instance, aborting.

A successful reset also returns None — that is the point. The return value carries no information
about whether the environment was reset.

Why I think it is worth changing

The failure is invisible at the point where it matters. A user who has not set WA_FULL_RESET (or
VWA_FULL_RESET) gets a clean run, a normal-looking return value, and results that depend on whatever
state their instance happened to be in. Because the only evidence is one WARNING in a long benchmark
log, the most likely outcome is that nobody notices the reset was skipped — and there is currently no
way for a downstream script to check.

I am not claiming any published WebArena number is wrong. I have not run the benchmark, and the
reset is available to anyone who configures it — this is about the default being silent rather than
about the mechanism being absent.

Suggested fix (policy call, so I'd rather discuss than impose)

Failing closed — full_reset(skip_if_not_set=False) at the three call sites — would break every
current user who has no reset endpoint, so I don't think it should be done unilaterally. Two smaller
options:

  1. Fail legibly. Have full_reset() return a bool (or a small enum: reset / skipped), and
    have prepare_backend() surface "environment reset: SKIPPED (WA_FULL_RESET unset)" in the run
    header or run metadata, so a run is self-documenting and downstream tooling can assert on it.
  2. Fail closed behind an opt-in. Keep the current default, but add an explicit
    WA_REQUIRE_RESET=1 that turns the silent skip into a hard error, for anyone who wants the
    guarantee enforced.

Either way the current behaviour is preserved by default, so nothing breaks.

Happy to open a PR for whichever you prefer — including just the docs clarification that
WA_FULL_RESET / VWA_FULL_RESET are required for a reproducible run.

Lenguaje dominante
Python
Estrellas
1.4k
Forks
202
Merge medio
6 d 15 h
PR fusionados (30 d)
3

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ServiceNow/BrowserGym

Todos los issues de ServiceNow/BrowserGym

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.