Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Feature Proposal]: Add experiment lifecycle callbacks to `ExpArgs` in `loop.py`

オープン
  • 何か問題がありますか？
#417 コメント 0 件 リアクション 0 件 担当者 1 名 GitHub で見る

@sriram687 がすでに取り組んでいます。

2026年10月27日 から。

評価

この issue はまだ評価されていません。

説明

Summary & Current Limitations

Currently, experiment execution in ExpArgs.run() operates as a closed, synchronous loop. While detailed step information (StepInfo) and summaries are recorded to disk and written to standard log files, there is no standardized event-driven mechanism to intercept what is happening while the experiment is actually running.

As a result:

  1. Blind Execution During Long Benchmarks: When evaluating an agent across hundreds of tasks (which can take hours or days), researchers only see flat, scrolling logger text. There is no native way to display clean live progress indicators, step latencies, or task completion rates in the console.
  2. No Budget & Safety Guardrails: Web agents querying commercial LLMs can occasionally encounter infinite action loops or repetitive failing actions. Without an execution hook, researchers cannot set runtime safety limits (such as aborting or truncating an episode if token count or estimated API spend exceeds a defined threshold).
  3. Disconnected from External Dashboards & UIs: Modern AI evaluation workflows rely on external tracking platforms and web dashboards. Because the execution loop does not emit lifecycle events, external tools cannot listen to agent progress in real time without monkey-patching core files.
The Proposed Concept: How Callbacks Solve This

A Callback System transforms the experiment loop from a passive black box into an active event broadcaster.

Instead of hardcoding specific logging tools into the loop, the loop simply notifies a list of listener objects at key lifecycle moments:

  • When an experiment/task begins.
  • When an observation is received.
  • When an agent selects an action (along with latency and token info).
  • When the environment steps and evaluates the action.
  • When the experiment concludes (success, failure, or error).
How it Behaves:
  • Zero Overhead & 100% Backward Compatible: By default, no callbacks are active. Existing scripts, tests, and pipelines continue running exactly as they do today with zero performance penalty.
  • Fail-Safe Design: Any error occurring inside a user's monitoring callback is safely handled so that monitoring issues never crash or corrupt a running benchmark experiment.
  • Pluggable & Extensible: Researchers choose only the callbacks they need for their specific environment.
What We Will Deliver

If the team is aligned with this direction, we propose delivering a clean, modular foundation:

  1. Terminal-Level Experience:

    • A built-in console progress callback that provides clean, single-line progress tracking (task progress, current step, step latency, and token count) without spamming the terminal with raw debug logs.

  2. Safety & Budget Guardrails:

    • A budget limit callback that monitors cumulative token consumption reported by the agent and safely stops runaway tasks if cost/token budgets are breached, protecting researchers from unexpected API bills.

  3. Cloud & Web UI Bridge (Weights & Biases, MLflow, Custom UIs):

    • A clean base callback interface that allows researchers to stream live BrowserGym events to cloud experiment trackers (like Weights & Biases or MLflow) or hook them directly into custom web dashboards (Streamlit, Gradio, or WebSocket frontends).
Maintainer Feedback

We would love to hear your thoughts on introducing this event-driven callback architecture to BrowserGym before opening a PR!

主要言語
Python
スター
1.4k
フォーク
202
平均マージ
6日 15時間
マージ済み PR（30日）
3

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ServiceNow/BrowserGym のほかの issue

ServiceNow/BrowserGym の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。