Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

`miniwob.find-greatest` can be marked successful after selecting a non-maximum card

オープン
  • 何か問題がありますか？
#392 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

@jstar0 がすでに取り組んでいます。

2026年7月17日 から。

  • #396 @jstar0 による — オープン

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
25/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
javascript, python
領域
ai, testing-qa

調査の方向性

Start with the attached source/browsergym_miniwob_base.py and trace how RAW_REWARD_GLOBAL becomes the reported reward and success value; compare that with the wrong-card branch in source/find-greatest.html. Check the linked pull request #396 and any relevant tests before choosing a fix. Done means submitting a non-maximum card no longer reports success, while selecting the greatest card still does.

索引モデルが issue の本文から書いたものです。

説明

Summary

There appears to be a success-threshold bug in the BrowserGym MiniWoB wrapper for miniwob.find-greatest.

The task text says:

Find and pick the card with the greatest number, then press submit.

However, a run can be marked successful after selecting a card that is not the greatest card.

In the attached concrete run, the page had cards with values 4, 7, and 2. The agent selected the card with value 2 and submitted. The final DOM still shows another card with value 7, so this should not satisfy the task. The native evaluator nevertheless reports success: true.

Why this happens

In MiniWoB++ find-greatest.html, selecting the correct card ends with reward 1.0, but selecting the wrong card and submitting still ends the episode with a positive partial reward:

if(userIndex === expectedIndex.toString()) core.endEpisode(1.0, true);
else core.endEpisode(0.1, true);

In BrowserGym's MiniWoB wrapper, AbstractMiniwobTask.validate() converts any positive raw reward into a full success reward:

reward = float(info["RAW_REWARD_GLOBAL"] > 0)  # TODO: shouldn't it be 0.5?

As a result, the wrong-card partial reward 0.1 becomes reward: 1.0 and success: true in the run output.

Actual behavior in the attached run

The attached evidence/native_evaluator_output_agent_a.json shows:

"RAW_REWARD_GLOBAL": 0.1,
"reward": 1.0,
"success": true

It also shows the agent actions:

"action": "click('17')"
"action": "click('20')"

The attached final DOM snapshot evidence/final_dom_step_002_agent_a.html shows that bid 17 is the revealed selected card, with value 2:

<div class="card" data-index="2" ... bid="17">
  <span class="card-value" ...>2</span>
</div>

The same DOM line also shows another card with value 7:

<div class="card hidden" data-index="1" ... bid="15">
  <span class="card-value" ...>7</span>
</div>

So the submitted card was not the greatest card, but the run was still reported as successful.

Expected behavior

For find-greatest, BrowserGym should only report success when the selected card is actually the greatest card.

Possible fixes:

  • change the MiniWoB success threshold so partial positive rewards such as 0.1 do not count as success, for example RAW_REWARD_GLOBAL > 0.5 or RAW_REWARD_GLOBAL >= 1.0; or
  • add task-specific handling where partial rewards are not treated as benchmark success; or
  • if the intended fix belongs in MiniWoB++, make wrong-card submission produce a non-success raw reward rather than 0.1.

The issue seems most directly triggered by BrowserGym treating any RAW_REWARD_GLOBAL > 0 as success.

Attached evidence package

  • source/browsergym_miniwob_base.py: BrowserGym MiniWoB wrapper showing reward = float(info["RAW_REWARD_GLOBAL"] > 0).
  • source/find-greatest.html: MiniWoB++ task source showing the task text and wrong-card core.endEpisode(0.1, true) branch.
  • evidence/native_evaluator_output_agent_a.json: run output showing RAW_REWARD_GLOBAL=0.1, reward=1.0, and success=true.
  • evidence/final_dom_step_002_agent_a.html: final DOM showing selected card value 2 while another card value is 7.
  • evidence/final_screenshot_step_002_agent_a.png: final screenshot for the same run.
  • evidence/validation_final_agent_a.json and evidence/task_info_final_agent_a.json: final validator/task state.
  • evidence/miniwob_scores_flat.csv: score table; row 170 contains this miniwob.find-greatest / Agent A run.

find_greatest_agent_a_wrong_card_success_issue.zip

主要言語
Python
スター
1.4k
フォーク
202
平均マージ
6日 15時間
マージ済み PR（30日）
3

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ServiceNow/BrowserGym のほかの issue

ServiceNow/BrowserGym の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。