Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

AssistantBench scorer treats a thousands separator as a decimal point: "3,080,000" scores 0 against gold 3080000, while "1,000" scores 1.0 against gold 1

Abierto
  • ¿Algo no está bien?
#404 1 comentario 0 reacciones 0 asignados Ver en GitHub

@Andiii208 ya está trabajando en esto.

Desde el 2/10/2026.

  • #405 de @Andiii208 — abierto
  • #414 de @Vijaysrimari — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
python
Área
testing-qa

Línea de trabajo

Start with browsergym/assistantbench/src/browsergym/assistantbench/evaluation/evaluator.py and evaluation/evaluate_utils/evaluate_dicts.py, then read the shared numeric normalizer in evaluation/evaluate_utils/evaluate_strings.py. Run the pinned cases in tests/assistantbench/test_evaluation.py; done means comma-formatted thousands values score correctly without breaking decimal-comma cases. Pull request #405 is already open for this issue.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

browsergym/assistantbench/src/browsergym/assistantbench/evaluation/evaluator.py:56-67 (the same replace(",", ".") is duplicated in evaluation/evaluate_utils/evaluate_dicts.py:46):

def fix_number(number):
    if type(number) == str:
        copy_ans = number
        copy_ans = " ".join(
            " ".join(" ".join(copy_ans.split("$")).split("%")).split("sqft")
        ).strip()
        copy_ans = copy_ans.strip()
        copy_ans = copy_ans.replace(",", ".").replace(" square kilometers", "")
        try:
            return float(copy_ans), True
        except:
            return number, False

The comma-to-dot rewrite is applied to both the prediction and the gold. It is correct for a European decimal comma (14,2 -> 14.2, which is dev gold validation.2), but a US thousands separator goes through the same path: 1,000 becomes 1.0, and 3,080,000 becomes 3.080.000, which float() rejects, so the prediction is passed to evaluate_numbers as a string and scored 0.

Measured

Real evaluation/evaluator.py loaded from the repo (only the package __init__ chain bypassed) and question_scorer called directly. numpy 2.3.1, scipy 1.16.0.

input result note
question_scorer('1000', '1000') 1.0 control, pass
question_scorer('2000', '1000') 0.307 control, log-distance partial credit as designed
question_scorer('$55', '$55') 1.0 control, dev gold validation.29
question_scorer('14,2', '14.2') 1.0 control, EU decimal comma, dev gold validation.2
question_scorer('3,080,000', '3080000') 0 correct answer, dev gold validation.26
question_scorer('1,010,000', '1010000') 0 correct answer, dev gold validation.3
question_scorer('1,000', '1000') 0 correct answer
question_scorer('1,000', '1') 1.0 wrong by 1000x
question_scorer('1,010', '1.01') 1.0 wrong by 1000x
fix_number('1,000') (1.0, True)
fix_number('3,080,000') ('3,080,000', False) falls through to evaluate_numbers -> float() ValueError -> 0

The two dev-split tasks above ("What's the lowest price a Single Family house was sold in Queen Anne in January 2023?", "What's the highest price a high-rise apartment was sold for in Mission Bay, San Francisco, in 2021?") ask for dollar amounts that Zillow and most models render with thousands separators, and neither task text asks for a comma-free number.

Consequence

AssistantBenchTask.validate returns this accuracy as the episode reward. A correctly answered task written as 3,080,000 counts as a 0 in cum_reward, and an answer that is off by three orders of magnitude but happens to carry a single comma counts as a full 1.0. The dict path is affected too: a JSON value "1,000" against gold 1000 scores the pair 0.5 instead of 1.0 through the duplicate in evaluate_dicts.py:46. task.py:140 feeds the raw chat message to question_scorer with no normalisation. None of the 33 rows pinned in tests/assistantbench/test_evaluation.py contain a digit-comma, so the test suite does not cover either direction.

This code is a verbatim copy of the official leaderboard evaluator (AssistantBench/leaderboard, evaluation/evaluator.py), so the hidden test split scored on the leaderboard has the same behaviour; the point of raising it here is that dev-split rewards computed inside BrowserGym are silently wrong for comma-formatted numbers, and a fix should probably be mirrored upstream so the two scorers stay aligned.

Suggested fix

Disambiguate the comma before the float() attempt instead of rewriting it unconditionally, e.g. in fix_number (and the duplicate in evaluate_dicts.py):

  • if the string matches ^\d{1,3}(,\d{3})+(\.\d+)?$ (comma groups of exactly three digits, optional dot decimal), drop the commas;
  • otherwise, if it contains exactly one comma and no dot, treat the comma as a decimal point (the existing 14,2 case);
  • leave everything else as is.

The package already has a small numeric normalizer, evaluate_utils/evaluate_strings.py:_normalize_number / _is_number, so the rule could live there and be shared by fix_number in both files. Adding ('3,080,000', '3080000') -> 1.0, ('1,000', '1') -> 0, and ('14,2', '14.2') -> 1.0 to the pinned evaluation data would lock the behaviour in.

Happy to open the PR.

Lenguaje dominante
Python
Estrellas
1.4k
Forks
202
Merge medio
6 d 15 h
PR fusionados (30 d)
3

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ServiceNow/BrowserGym

Todos los issues de ServiceNow/BrowserGym

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.