Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

evaluate_model_outputs.py: rows whose gold cannot be extracted are dropped from the accuracy denominator, so the printed accuracy disagrees with the is_correct column the script writes

Abierto Apto para principiantes
  • ¿Algo no está bien?
#83 0 comentarios 0 reacciones 0 asignados Ver en GitHub

@shaurya416 ya está trabajando en esto.

Desde el 25/9/2026.

  • #84 de @shaurya416 — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
84/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
cli, tooling

Línea de trabajo

Comienza en evaluate_model_outputs.py:74-112 y sigue cómo process_answers actualiza total_count, correct_count y las filas de salida cuando metric.py:61-64 lanza una excepción. Reproduce los casos bare-gold y error-row descritos en el issue y verifica después que el resumen de la consola y los valores CSV de is_correct usen el mismo denominador; inspecciona también la declaración de --gold_is_latex en la línea 12 si abordas el problema relacionado de CLI.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

evaluate_model_outputs.py:74-107 (the documented "Advanced Usage" script in the README):

        grade = 0
        try:
            grade, extracted_answers = verify_func([row['gold']], [row['answer']])
            ...
            total_count += 1                      # :85, only reached if verify_func returned
            if grade == 1:
                correct_count += 1
            ...
        except Exception as e:
            results.append({
                ...
                'is_correct': grade == 1,         # grade is still 0 -> False
                'error': str(e)
            })                                    # total_count is not incremented
...
    accuracy = correct_count / total_count if total_count > 0 else 0   # :112

math_metric raises ValueError("No gold targets found ...") when parse(gold) returns [] (metric.py:61-64, commented "Assert on empty gold"). That exception is caught here, the row is written to the output CSV with is_correct=False and an error value, but it is excluded from total_count. The console summary never mentions it.

Bare-number golds hit this path readily. --gold_is_latex is declared action='store_true', default=True (:12), so it is True with or without the flag and the ExprExtractionConfig() gold branch at :64 cannot be selected from the CLI. LatexExtractionConfig only matches $..$, \[..\], \(..\), \boxed{} or \frac{}{}, so a gold cell such as 42 (GSM8K/AMC style), an empty cell, or a NaN cell yields [].

Measured

Unmodified evaluate_model_outputs.py, metric.py, parser.py at ba3d3aa; process_answers called on small DataFrames. (latex2sympy2_extended is a git submodule that was not checked out, so a minimal stand-in was used for the LaTeX-to-sympy step. The failing path, no regex match giving [], never reaches it: parse('42', [LatexExtractionConfig()]) is [], parse('$42$', [LatexExtractionConfig()]) is [42, '42'].)

Case Rows Printed summary is_correct column in the output CSV
control: gold in $..$, 3 right / 3 wrong 6 Total 6, Correct 3, 50.00% 3/6 = 50.00%, 0 error rows
control: examples/model_outputs.csv from the README 6 Total 6, Correct 6, 100.00% 6/6, 0 error rows
control: blank (NaN) and '' answers, gold $42$ 2 Total 2, Correct 0, 0.00% 0/2 (answer-side failures are counted correctly)
the 6 control rows + 4 wrong answers with bare gold 42 10 Total 6, Correct 3, 50.00% 3/10 = 30.00%, 4 rows with error="No gold targets found for at least one gold. Gold: ['42'], ..."
1 formatted right row + 5 bare-gold rows 6 Total 1, Correct 1, 100.00% 1/6 = 16.67%, 5 error rows
5 wrong answers, all gold bare 5 Total 0, Correct 0, 0.00% 0/5, 5 error rows
NaN gold and '' gold + 1 right + 1 wrong 4 Total 2, Correct 1, 50.00% 1/4 = 25.00%, 2 error rows

Printed accuracy is correct / (rows - errored), which is greater than or equal to the CSV's own correct / rows, and strictly greater whenever there is at least one error row and at least one correct row. When every row errors, the script prints Accuracy: 0.00%, the same figure it prints for a model that got everything wrong.

Consequence

The number reported for a run depends on how many gold cells the extractor could not read, and nothing in the console output says so. Two runs over the same model outputs can differ by tens of points if one dataset stores golds as 42 and the other as $42$. The error column is present only in the CSV, and only on the affected rows.

Suggested fix

The same except branch already records the row as is_correct: grade == 1 (False), and metric.py:61-64 raises on purpose so that an unreadable gold is loud. The summary should follow what the row-level output already says. Smallest change: move total_count += 1 above the try: so every row is counted. A stricter variant that keeps unscorable rows out of the ratio should print how many were skipped (for example Skipped (error): N) and refuse to print an accuracy when total_count == 0, instead of the else 0 fallback. extract_answers.py shows the pattern for the failure row itself: its except branch appends a row with extraction_success: False and keeps it in the output.

Separately, --gold_is_latex can never be False as declared (store_true with default=True), so plain-number golds cannot be scored through this script. action=argparse.BooleanOptionalAction would make the flag work as its help text and the maintainer reply in #28 imply.

Happy to open the PR.

Lenguaje dominante
Python
Estrellas
1.2k
Forks
65
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de huggingface/Math-Verify

Todos los issues de huggingface/Math-Verify

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.