Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

`miniwob.find-greatest` can be marked successful after selecting a non-maximum card

Open
  • Is something wrong?
#392 0 comments 0 reactions 0 assignees View on GitHub

@jstar0 is already working on this.

Since Jul 17, 2026.

  • #396 by @jstar0 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
javascript, python
Domain
ai, testing-qa

Research direction

Start with the attached source/browsergym_miniwob_base.py and trace how RAW_REWARD_GLOBAL becomes the reported reward and success value; compare that with the wrong-card branch in source/find-greatest.html. Check the linked pull request #396 and any relevant tests before choosing a fix. Done means submitting a non-maximum card no longer reports success, while selecting the greatest card still does.

Written by the indexing model from the issue text.

Description

Summary

There appears to be a success-threshold bug in the BrowserGym MiniWoB wrapper for miniwob.find-greatest.

The task text says:

Find and pick the card with the greatest number, then press submit.

However, a run can be marked successful after selecting a card that is not the greatest card.

In the attached concrete run, the page had cards with values 4, 7, and 2. The agent selected the card with value 2 and submitted. The final DOM still shows another card with value 7, so this should not satisfy the task. The native evaluator nevertheless reports success: true.

Why this happens

In MiniWoB++ find-greatest.html, selecting the correct card ends with reward 1.0, but selecting the wrong card and submitting still ends the episode with a positive partial reward:

if(userIndex === expectedIndex.toString()) core.endEpisode(1.0, true);
else core.endEpisode(0.1, true);

In BrowserGym's MiniWoB wrapper, AbstractMiniwobTask.validate() converts any positive raw reward into a full success reward:

reward = float(info["RAW_REWARD_GLOBAL"] > 0)  # TODO: shouldn't it be 0.5?

As a result, the wrong-card partial reward 0.1 becomes reward: 1.0 and success: true in the run output.

Actual behavior in the attached run

The attached evidence/native_evaluator_output_agent_a.json shows:

"RAW_REWARD_GLOBAL": 0.1,
"reward": 1.0,
"success": true

It also shows the agent actions:

"action": "click('17')"
"action": "click('20')"

The attached final DOM snapshot evidence/final_dom_step_002_agent_a.html shows that bid 17 is the revealed selected card, with value 2:

<div class="card" data-index="2" ... bid="17">
  <span class="card-value" ...>2</span>
</div>

The same DOM line also shows another card with value 7:

<div class="card hidden" data-index="1" ... bid="15">
  <span class="card-value" ...>7</span>
</div>

So the submitted card was not the greatest card, but the run was still reported as successful.

Expected behavior

For find-greatest, BrowserGym should only report success when the selected card is actually the greatest card.

Possible fixes:

  • change the MiniWoB success threshold so partial positive rewards such as 0.1 do not count as success, for example RAW_REWARD_GLOBAL > 0.5 or RAW_REWARD_GLOBAL >= 1.0; or
  • add task-specific handling where partial rewards are not treated as benchmark success; or
  • if the intended fix belongs in MiniWoB++, make wrong-card submission produce a non-success raw reward rather than 0.1.

The issue seems most directly triggered by BrowserGym treating any RAW_REWARD_GLOBAL > 0 as success.

Attached evidence package

  • source/browsergym_miniwob_base.py: BrowserGym MiniWoB wrapper showing reward = float(info["RAW_REWARD_GLOBAL"] > 0).
  • source/find-greatest.html: MiniWoB++ task source showing the task text and wrong-card core.endEpisode(0.1, true) branch.
  • evidence/native_evaluator_output_agent_a.json: run output showing RAW_REWARD_GLOBAL=0.1, reward=1.0, and success=true.
  • evidence/final_dom_step_002_agent_a.html: final DOM showing selected card value 2 while another card value is 7.
  • evidence/final_screenshot_step_002_agent_a.png: final screenshot for the same run.
  • evidence/validation_final_agent_a.json and evidence/task_info_final_agent_a.json: final validator/task state.
  • evidence/miniwob_scores_flat.csv: score table; row 170 contains this miniwob.find-greatest / Agent A run.

find_greatest_agent_a_wrong_card_success_issue.zip

Dominant language
Python
Stars
1.4k
Forks
202
Avg merge
6d 15h
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ServiceNow/BrowserGym

All issues in ServiceNow/BrowserGym

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.