Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

`miniwob.find-greatest` can be marked successful after selecting a non-maximum card

Aperta
  • Qualcosa non va?
#392 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

@jstar0 ci sta già lavorando.

Dal 17/7/2026.

  • #396 di @jstar0 — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
javascript, python
Ambito
ai, testing-qa

Direzione di ricerca

Start with the attached source/browsergym_miniwob_base.py and trace how RAW_REWARD_GLOBAL becomes the reported reward and success value; compare that with the wrong-card branch in source/find-greatest.html. Check the linked pull request #396 and any relevant tests before choosing a fix. Done means submitting a non-maximum card no longer reports success, while selecting the greatest card still does.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

There appears to be a success-threshold bug in the BrowserGym MiniWoB wrapper for miniwob.find-greatest.

The task text says:

Find and pick the card with the greatest number, then press submit.

However, a run can be marked successful after selecting a card that is not the greatest card.

In the attached concrete run, the page had cards with values 4, 7, and 2. The agent selected the card with value 2 and submitted. The final DOM still shows another card with value 7, so this should not satisfy the task. The native evaluator nevertheless reports success: true.

Why this happens

In MiniWoB++ find-greatest.html, selecting the correct card ends with reward 1.0, but selecting the wrong card and submitting still ends the episode with a positive partial reward:

if(userIndex === expectedIndex.toString()) core.endEpisode(1.0, true);
else core.endEpisode(0.1, true);

In BrowserGym's MiniWoB wrapper, AbstractMiniwobTask.validate() converts any positive raw reward into a full success reward:

reward = float(info["RAW_REWARD_GLOBAL"] > 0)  # TODO: shouldn't it be 0.5?

As a result, the wrong-card partial reward 0.1 becomes reward: 1.0 and success: true in the run output.

Actual behavior in the attached run

The attached evidence/native_evaluator_output_agent_a.json shows:

"RAW_REWARD_GLOBAL": 0.1,
"reward": 1.0,
"success": true

It also shows the agent actions:

"action": "click('17')"
"action": "click('20')"

The attached final DOM snapshot evidence/final_dom_step_002_agent_a.html shows that bid 17 is the revealed selected card, with value 2:

<div class="card" data-index="2" ... bid="17">
  <span class="card-value" ...>2</span>
</div>

The same DOM line also shows another card with value 7:

<div class="card hidden" data-index="1" ... bid="15">
  <span class="card-value" ...>7</span>
</div>

So the submitted card was not the greatest card, but the run was still reported as successful.

Expected behavior

For find-greatest, BrowserGym should only report success when the selected card is actually the greatest card.

Possible fixes:

  • change the MiniWoB success threshold so partial positive rewards such as 0.1 do not count as success, for example RAW_REWARD_GLOBAL > 0.5 or RAW_REWARD_GLOBAL >= 1.0; or
  • add task-specific handling where partial rewards are not treated as benchmark success; or
  • if the intended fix belongs in MiniWoB++, make wrong-card submission produce a non-success raw reward rather than 0.1.

The issue seems most directly triggered by BrowserGym treating any RAW_REWARD_GLOBAL > 0 as success.

Attached evidence package

  • source/browsergym_miniwob_base.py: BrowserGym MiniWoB wrapper showing reward = float(info["RAW_REWARD_GLOBAL"] > 0).
  • source/find-greatest.html: MiniWoB++ task source showing the task text and wrong-card core.endEpisode(0.1, true) branch.
  • evidence/native_evaluator_output_agent_a.json: run output showing RAW_REWARD_GLOBAL=0.1, reward=1.0, and success=true.
  • evidence/final_dom_step_002_agent_a.html: final DOM showing selected card value 2 while another card value is 7.
  • evidence/final_screenshot_step_002_agent_a.png: final screenshot for the same run.
  • evidence/validation_final_agent_a.json and evidence/task_info_final_agent_a.json: final validator/task state.
  • evidence/miniwob_scores_flat.csv: score table; row 170 contains this miniwob.find-greatest / Agent A run.

find_greatest_agent_a_wrong_card_success_issue.zip

Lingua principale
Python
Stelle
1.4k
Fork
202
Merge medio
6g 15h
PR unite (30g)
3

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ServiceNow/BrowserGym

Tutte le issue di ServiceNow/BrowserGym

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.