AssistantBench scorer treats a thousands separator as a decimal point: "3,080,000" scores 0 against gold 3080000, while "1,000" scores 1.0 against gold 1

Aperta
  • Qualcosa non va?
#404 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

@Andiii208 ci sta già lavorando.

Dal 2/10/2026.

  • #405 di @Andiii208 — aperta
  • #414 di @Vijaysrimari — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
python
Ambito
testing-qa

Direzione di ricerca

Start with browsergym/assistantbench/src/browsergym/assistantbench/evaluation/evaluator.py and evaluation/evaluate_utils/evaluate_dicts.py, then read the shared numeric normalizer in evaluation/evaluate_utils/evaluate_strings.py. Run the pinned cases in tests/assistantbench/test_evaluation.py; done means comma-formatted thousands values score correctly without breaking decimal-comma cases. Pull request #405 is already open for this issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

browsergym/assistantbench/src/browsergym/assistantbench/evaluation/evaluator.py:56-67 (the same replace(",", ".") is duplicated in evaluation/evaluate_utils/evaluate_dicts.py:46):

def fix_number(number):
    if type(number) == str:
        copy_ans = number
        copy_ans = " ".join(
            " ".join(" ".join(copy_ans.split("$")).split("%")).split("sqft")
        ).strip()
        copy_ans = copy_ans.strip()
        copy_ans = copy_ans.replace(",", ".").replace(" square kilometers", "")
        try:
            return float(copy_ans), True
        except:
            return number, False

The comma-to-dot rewrite is applied to both the prediction and the gold. It is correct for a European decimal comma (14,2 -> 14.2, which is dev gold validation.2), but a US thousands separator goes through the same path: 1,000 becomes 1.0, and 3,080,000 becomes 3.080.000, which float() rejects, so the prediction is passed to evaluate_numbers as a string and scored 0.

Measured

Real evaluation/evaluator.py loaded from the repo (only the package __init__ chain bypassed) and question_scorer called directly. numpy 2.3.1, scipy 1.16.0.

input result note
question_scorer('1000', '1000') 1.0 control, pass
question_scorer('2000', '1000') 0.307 control, log-distance partial credit as designed
question_scorer('$55', '$55') 1.0 control, dev gold validation.29
question_scorer('14,2', '14.2') 1.0 control, EU decimal comma, dev gold validation.2
question_scorer('3,080,000', '3080000') 0 correct answer, dev gold validation.26
question_scorer('1,010,000', '1010000') 0 correct answer, dev gold validation.3
question_scorer('1,000', '1000') 0 correct answer
question_scorer('1,000', '1') 1.0 wrong by 1000x
question_scorer('1,010', '1.01') 1.0 wrong by 1000x
fix_number('1,000') (1.0, True)
fix_number('3,080,000') ('3,080,000', False) falls through to evaluate_numbers -> float() ValueError -> 0

The two dev-split tasks above ("What's the lowest price a Single Family house was sold in Queen Anne in January 2023?", "What's the highest price a high-rise apartment was sold for in Mission Bay, San Francisco, in 2021?") ask for dollar amounts that Zillow and most models render with thousands separators, and neither task text asks for a comma-free number.

Consequence

AssistantBenchTask.validate returns this accuracy as the episode reward. A correctly answered task written as 3,080,000 counts as a 0 in cum_reward, and an answer that is off by three orders of magnitude but happens to carry a single comma counts as a full 1.0. The dict path is affected too: a JSON value "1,000" against gold 1000 scores the pair 0.5 instead of 1.0 through the duplicate in evaluate_dicts.py:46. task.py:140 feeds the raw chat message to question_scorer with no normalisation. None of the 33 rows pinned in tests/assistantbench/test_evaluation.py contain a digit-comma, so the test suite does not cover either direction.

This code is a verbatim copy of the official leaderboard evaluator (AssistantBench/leaderboard, evaluation/evaluator.py), so the hidden test split scored on the leaderboard has the same behaviour; the point of raising it here is that dev-split rewards computed inside BrowserGym are silently wrong for comma-formatted numbers, and a fix should probably be mirrored upstream so the two scorers stay aligned.

Suggested fix

Disambiguate the comma before the float() attempt instead of rewriting it unconditionally, e.g. in fix_number (and the duplicate in evaluate_dicts.py):

  • if the string matches ^\d{1,3}(,\d{3})+(\.\d+)?$ (comma groups of exactly three digits, optional dot decimal), drop the commas;
  • otherwise, if it contains exactly one comma and no dot, treat the comma as a decimal point (the existing 14,2 case);
  • leave everything else as is.

The package already has a small numeric normalizer, evaluate_utils/evaluate_strings.py:_normalize_number / _is_number, so the rule could live there and be shared by fix_number in both files. Adding ('3,080,000', '3080000') -> 1.0, ('1,000', '1') -> 0, and ('14,2', '14.2') -> 1.0 to the pinned evaluation data would lock the behaviour in.

Happy to open the PR.

Lingua principale
Python
Stelle
1.4k
Fork
202
Merge medio
6g 15h
PR unite (30g)
3

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ServiceNow/BrowserGym

Tutte le issue di ServiceNow/BrowserGym

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.