Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Feature Proposal]: Add experiment lifecycle callbacks to `ExpArgs` in `loop.py`

Open
  • Is something wrong?
#417 0 comments 0 reactions 1 assignee View on GitHub

@sriram687 is already working on this.

Since Oct 27, 2026.

Assessment

This issue has not been assessed yet.

Description

Summary & Current Limitations

Currently, experiment execution in ExpArgs.run() operates as a closed, synchronous loop. While detailed step information (StepInfo) and summaries are recorded to disk and written to standard log files, there is no standardized event-driven mechanism to intercept what is happening while the experiment is actually running.

As a result:

  1. Blind Execution During Long Benchmarks: When evaluating an agent across hundreds of tasks (which can take hours or days), researchers only see flat, scrolling logger text. There is no native way to display clean live progress indicators, step latencies, or task completion rates in the console.
  2. No Budget & Safety Guardrails: Web agents querying commercial LLMs can occasionally encounter infinite action loops or repetitive failing actions. Without an execution hook, researchers cannot set runtime safety limits (such as aborting or truncating an episode if token count or estimated API spend exceeds a defined threshold).
  3. Disconnected from External Dashboards & UIs: Modern AI evaluation workflows rely on external tracking platforms and web dashboards. Because the execution loop does not emit lifecycle events, external tools cannot listen to agent progress in real time without monkey-patching core files.
The Proposed Concept: How Callbacks Solve This

A Callback System transforms the experiment loop from a passive black box into an active event broadcaster.

Instead of hardcoding specific logging tools into the loop, the loop simply notifies a list of listener objects at key lifecycle moments:

  • When an experiment/task begins.
  • When an observation is received.
  • When an agent selects an action (along with latency and token info).
  • When the environment steps and evaluates the action.
  • When the experiment concludes (success, failure, or error).
How it Behaves:
  • Zero Overhead & 100% Backward Compatible: By default, no callbacks are active. Existing scripts, tests, and pipelines continue running exactly as they do today with zero performance penalty.
  • Fail-Safe Design: Any error occurring inside a user's monitoring callback is safely handled so that monitoring issues never crash or corrupt a running benchmark experiment.
  • Pluggable & Extensible: Researchers choose only the callbacks they need for their specific environment.
What We Will Deliver

If the team is aligned with this direction, we propose delivering a clean, modular foundation:

  1. Terminal-Level Experience:

    • A built-in console progress callback that provides clean, single-line progress tracking (task progress, current step, step latency, and token count) without spamming the terminal with raw debug logs.

  2. Safety & Budget Guardrails:

    • A budget limit callback that monitors cumulative token consumption reported by the agent and safely stops runaway tasks if cost/token budgets are breached, protecting researchers from unexpected API bills.

  3. Cloud & Web UI Bridge (Weights & Biases, MLflow, Custom UIs):

    • A clean base callback interface that allows researchers to stream live BrowserGym events to cloud experiment trackers (like Weights & Biases or MLflow) or hook them directly into custom web dashboards (Streamlit, Gradio, or WebSocket frontends).
Maintainer Feedback

We would love to hear your thoughts on introducing this event-driven callback architecture to BrowserGym before opening a PR!

Dominant language
Python
Stars
1.4k
Forks
202
Avg merge
6d 12h
Merged PRs (30d)
2

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ServiceNow/BrowserGym

All issues in ServiceNow/BrowserGym

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.