full_reset() gives no signal whether the environment was reset; skipped reset is indistinguishable from a successful one

Read browsergym/webarena/src/browsergym/webarena/instance.py around WebArenaInstance.full_reset() and the three call sites in browsergym/experiments/src/browsergym/experiments/benchmark/utils.py inside prepare_backend(). Compare the skipped and successful reset paths, then decide with maintainers how the result should be surfaced. Done means callers can distinguish a skipped reset from a successful one without changing the current default behavior.

Summary

WebArenaInstance.full_reset() gives the caller no way to tell whether a reset happened.

prepare_backend() calls it with no arguments, so skip_if_not_set=True is always in force. With no
reset endpoint configured, the reset is skipped with a logged warning — but the function then falls
through to the warm-start block and returns None, which is exactly what it returns after a
successful reset. So the benchmark proceeds against an environment that was never reset, and the
caller has no signal to distinguish the two cases. The only evidence is one WARNING line in a long
benchmark log.

This matters because the upstream WebArena documentation treats the reset as required for correct
evaluation:

After evaluating the 812 examples, reset the environment to the initial state
— web-arena-x/webarena, README

All 812 shipped task configs also carry require_reset: false, which is consistent with the intended
protocol being one reset per sweep rather than per task. So the single reset that BrowserGym performs
is the one carrying the reproducibility guarantee — and it is the one that can silently not happen.

Where

browsergym/webarena/src/browsergym/webarena/instance.py:59

def full_reset(self, skip_if_not_set: bool = True):
    base_url = os.environ.get(self.RESET_URL_VAR, None)
    if not base_url:
        logger.error(f"Environment variable {self.RESET_URL_VAR} is missing or empty, required for a full instance reset.")
        if skip_if_not_set:
            logger.warning("Skipping automated reset. Make sure the instance has been manually reset.")
        else:
            raise RuntimeError(f"Could not reset instance, aborting.")
    else:
        ...  # performs the reset, polls /status until "Ready for duty!"

    # warm-start the instance (navigate to every domain)   <-- reached in BOTH branches
    retries_left = 3
    while retries_left:
        ...
        self._check_is_reachable(timeout=60)

Note the fall-through: the warm-start block sits outside the if/else, so the skipped branch does not
return early — and neither branch returns a value that identifies itself.

Called from browsergym/experiments/src/browsergym/experiments/benchmark/utils.py in
prepare_backend(), with no arguments, for all three backends:

line	backend	reset env var
147	webarena	WA_FULL_RESET
176	webarena_verified	WA_FULL_RESET
205	visualwebarena	VWA_FULL_RESET

(VisualWebArenaInstance subclasses WebArenaInstance and overrides RESET_URL_VAR, so it inherits
the same behaviour.)

Reproduction

No WebArena instance needed — this is the control flow, not the network:

git clone https://github.com/ServiceNow/BrowserGym && cd BrowserGym   # verified at 0.14.4, c05d7f3

Every call site uses the silent default:

grep -n "full_reset()" browsergym/experiments/src/browsergym/experiments/benchmark/utils.py
# 147:            default_instance.full_reset()
# 176:            default_instance.full_reset()
# 205:            default_instance.full_reset()

And the branch behaviour, run directly (no live instance needed):

import importlib.util, logging, os
logging.basicConfig(level=logging.INFO)
spec = importlib.util.spec_from_file_location(
    "wa_instance", "browsergym/webarena/src/browsergym/webarena/instance.py")
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)

os.environ.pop("WA_FULL_RESET", None)
inst = mod.WebArenaInstance.__new__(mod.WebArenaInstance)  # bypass __init__, needs live env vars
inst.urls = {"probe": "https://example.com"}               # stand in for the six sites

print("silent branch returns:", inst.full_reset())         # -> None
inst.full_reset(skip_if_not_set=False)                     # -> RuntimeError, by contrast

Observed on 0.14.4:

ERROR    Environment variable WA_FULL_RESET is missing or empty, required for a full instance reset.
WARNING  Skipping automated reset. Make sure the instance has been manually reset.
silent branch returns: None
fail-closed path raises: Could not reset instance, aborting.

A successful reset also returns None — that is the point. The return value carries no information
about whether the environment was reset.

Why I think it is worth changing

The failure is invisible at the point where it matters. A user who has not set WA_FULL_RESET (or
VWA_FULL_RESET) gets a clean run, a normal-looking return value, and results that depend on whatever
state their instance happened to be in. Because the only evidence is one WARNING in a long benchmark
log, the most likely outcome is that nobody notices the reset was skipped — and there is currently no
way for a downstream script to check.

I am not claiming any published WebArena number is wrong. I have not run the benchmark, and the
reset is available to anyone who configures it — this is about the default being silent rather than
about the mechanism being absent.

Suggested fix (policy call, so I'd rather discuss than impose)

Failing closed — full_reset(skip_if_not_set=False) at the three call sites — would break every
current user who has no reset endpoint, so I don't think it should be done unilaterally. Two smaller
options:

1. Fail legibly. Have full_reset() return a bool (or a small enum: reset / skipped), and
   have prepare_backend() surface "environment reset: SKIPPED (WA_FULL_RESET unset)" in the run
   header or run metadata, so a run is self-documenting and downstream tooling can assert on it.
2. Fail closed behind an opt-in. Keep the current default, but add an explicit
   WA_REQUIRE_RESET=1 that turns the silent skip into a hard error, for anyone who wants the
   guarantee enforced.

Either way the current behaviour is preserved by default, so nothing breaks.

Happy to open a PR for whichever you prefer — including just the docs clarification that
WA_FULL_RESET / VWA_FULL_RESET are required for a reproducible run.

---