Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

evaluate_model_outputs.py: rows whose gold cannot be extracted are dropped from the accuracy denominator, so the printed accuracy disagrees with the is_correct column the script writes

Aperta Adatta ai principianti
  • Qualcosa non va?
#83 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

@shaurya416 ci sta già lavorando.

Dal 25/9/2026.

  • #84 di @shaurya416 — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
84/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
cli, tooling

Direzione di ricerca

Inizia da evaluate_model_outputs.py:74-112 e traccia il modo in cui process_answers aggiorna total_count, correct_count e le righe di output quando metric.py:61-64 solleva un’eccezione. Riproduci i casi bare-gold e error-row descritti nell’issue, quindi verifica che il riepilogo della console e i valori CSV di is_correct usino lo stesso denominatore; controlla anche la dichiarazione di --gold_is_latex alla riga 12 se affronti il problema CLI correlato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

evaluate_model_outputs.py:74-107 (the documented "Advanced Usage" script in the README):

        grade = 0
        try:
            grade, extracted_answers = verify_func([row['gold']], [row['answer']])
            ...
            total_count += 1                      # :85, only reached if verify_func returned
            if grade == 1:
                correct_count += 1
            ...
        except Exception as e:
            results.append({
                ...
                'is_correct': grade == 1,         # grade is still 0 -> False
                'error': str(e)
            })                                    # total_count is not incremented
...
    accuracy = correct_count / total_count if total_count > 0 else 0   # :112

math_metric raises ValueError("No gold targets found ...") when parse(gold) returns [] (metric.py:61-64, commented "Assert on empty gold"). That exception is caught here, the row is written to the output CSV with is_correct=False and an error value, but it is excluded from total_count. The console summary never mentions it.

Bare-number golds hit this path readily. --gold_is_latex is declared action='store_true', default=True (:12), so it is True with or without the flag and the ExprExtractionConfig() gold branch at :64 cannot be selected from the CLI. LatexExtractionConfig only matches $..$, \[..\], \(..\), \boxed{} or \frac{}{}, so a gold cell such as 42 (GSM8K/AMC style), an empty cell, or a NaN cell yields [].

Measured

Unmodified evaluate_model_outputs.py, metric.py, parser.py at ba3d3aa; process_answers called on small DataFrames. (latex2sympy2_extended is a git submodule that was not checked out, so a minimal stand-in was used for the LaTeX-to-sympy step. The failing path, no regex match giving [], never reaches it: parse('42', [LatexExtractionConfig()]) is [], parse('$42$', [LatexExtractionConfig()]) is [42, '42'].)

Case Rows Printed summary is_correct column in the output CSV
control: gold in $..$, 3 right / 3 wrong 6 Total 6, Correct 3, 50.00% 3/6 = 50.00%, 0 error rows
control: examples/model_outputs.csv from the README 6 Total 6, Correct 6, 100.00% 6/6, 0 error rows
control: blank (NaN) and '' answers, gold $42$ 2 Total 2, Correct 0, 0.00% 0/2 (answer-side failures are counted correctly)
the 6 control rows + 4 wrong answers with bare gold 42 10 Total 6, Correct 3, 50.00% 3/10 = 30.00%, 4 rows with error="No gold targets found for at least one gold. Gold: ['42'], ..."
1 formatted right row + 5 bare-gold rows 6 Total 1, Correct 1, 100.00% 1/6 = 16.67%, 5 error rows
5 wrong answers, all gold bare 5 Total 0, Correct 0, 0.00% 0/5, 5 error rows
NaN gold and '' gold + 1 right + 1 wrong 4 Total 2, Correct 1, 50.00% 1/4 = 25.00%, 2 error rows

Printed accuracy is correct / (rows - errored), which is greater than or equal to the CSV's own correct / rows, and strictly greater whenever there is at least one error row and at least one correct row. When every row errors, the script prints Accuracy: 0.00%, the same figure it prints for a model that got everything wrong.

Consequence

The number reported for a run depends on how many gold cells the extractor could not read, and nothing in the console output says so. Two runs over the same model outputs can differ by tens of points if one dataset stores golds as 42 and the other as $42$. The error column is present only in the CSV, and only on the affected rows.

Suggested fix

The same except branch already records the row as is_correct: grade == 1 (False), and metric.py:61-64 raises on purpose so that an unreadable gold is loud. The summary should follow what the row-level output already says. Smallest change: move total_count += 1 above the try: so every row is counted. A stricter variant that keeps unscorable rows out of the ratio should print how many were skipped (for example Skipped (error): N) and refuse to print an accuracy when total_count == 0, instead of the else 0 fallback. extract_answers.py shows the pattern for the failure row itself: its except branch appends a row with extraction_success: False and keeps it in the output.

Separately, --gold_is_latex can never be False as declared (store_true with default=True), so plain-number golds cannot be scored through this script. action=argparse.BooleanOptionalAction would make the flag work as its help text and the maintainer reply in #28 imply.

Happy to open the PR.

Lingua principale
Python
Stelle
1.2k
Fork
65
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di huggingface/Math-Verify

Tutte le issue di huggingface/Math-Verify

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.