Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

evaluate_model_outputs.py: rows whose gold cannot be extracted are dropped from the accuracy denominator, so the printed accuracy disagrees with the is_correct column the script writes

オープン 初心者向け
  • 何か問題がありますか？
#83 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

@shaurya416 がすでに取り組んでいます。

2026年9月25日 から。

  • #84 @shaurya416 による — オープン

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
84/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
cli, tooling

調査の方向性

evaluate_model_outputs.py:74-112 から始め、metric.py:61-64 が例外を発生させたときに process_answers が total_count、correct_count、出力行をどのように更新するかを追跡します。Issue に記載されている bare-gold と error-row のケースを再現し、その後、コンソールの概要と CSV の is_correct の値が同じ分母を使用していることを確認します。関連する CLI の問題に対応する場合は、12 行目の --gold_is_latex の宣言も確認します。

索引モデルが issue の本文から書いたものです。

説明

evaluate_model_outputs.py:74-107 (the documented "Advanced Usage" script in the README):

        grade = 0
        try:
            grade, extracted_answers = verify_func([row['gold']], [row['answer']])
            ...
            total_count += 1                      # :85, only reached if verify_func returned
            if grade == 1:
                correct_count += 1
            ...
        except Exception as e:
            results.append({
                ...
                'is_correct': grade == 1,         # grade is still 0 -> False
                'error': str(e)
            })                                    # total_count is not incremented
...
    accuracy = correct_count / total_count if total_count > 0 else 0   # :112

math_metric raises ValueError("No gold targets found ...") when parse(gold) returns [] (metric.py:61-64, commented "Assert on empty gold"). That exception is caught here, the row is written to the output CSV with is_correct=False and an error value, but it is excluded from total_count. The console summary never mentions it.

Bare-number golds hit this path readily. --gold_is_latex is declared action='store_true', default=True (:12), so it is True with or without the flag and the ExprExtractionConfig() gold branch at :64 cannot be selected from the CLI. LatexExtractionConfig only matches $..$, \[..\], \(..\), \boxed{} or \frac{}{}, so a gold cell such as 42 (GSM8K/AMC style), an empty cell, or a NaN cell yields [].

Measured

Unmodified evaluate_model_outputs.py, metric.py, parser.py at ba3d3aa; process_answers called on small DataFrames. (latex2sympy2_extended is a git submodule that was not checked out, so a minimal stand-in was used for the LaTeX-to-sympy step. The failing path, no regex match giving [], never reaches it: parse('42', [LatexExtractionConfig()]) is [], parse('$42$', [LatexExtractionConfig()]) is [42, '42'].)

Case Rows Printed summary is_correct column in the output CSV
control: gold in $..$, 3 right / 3 wrong 6 Total 6, Correct 3, 50.00% 3/6 = 50.00%, 0 error rows
control: examples/model_outputs.csv from the README 6 Total 6, Correct 6, 100.00% 6/6, 0 error rows
control: blank (NaN) and '' answers, gold $42$ 2 Total 2, Correct 0, 0.00% 0/2 (answer-side failures are counted correctly)
the 6 control rows + 4 wrong answers with bare gold 42 10 Total 6, Correct 3, 50.00% 3/10 = 30.00%, 4 rows with error="No gold targets found for at least one gold. Gold: ['42'], ..."
1 formatted right row + 5 bare-gold rows 6 Total 1, Correct 1, 100.00% 1/6 = 16.67%, 5 error rows
5 wrong answers, all gold bare 5 Total 0, Correct 0, 0.00% 0/5, 5 error rows
NaN gold and '' gold + 1 right + 1 wrong 4 Total 2, Correct 1, 50.00% 1/4 = 25.00%, 2 error rows

Printed accuracy is correct / (rows - errored), which is greater than or equal to the CSV's own correct / rows, and strictly greater whenever there is at least one error row and at least one correct row. When every row errors, the script prints Accuracy: 0.00%, the same figure it prints for a model that got everything wrong.

Consequence

The number reported for a run depends on how many gold cells the extractor could not read, and nothing in the console output says so. Two runs over the same model outputs can differ by tens of points if one dataset stores golds as 42 and the other as $42$. The error column is present only in the CSV, and only on the affected rows.

Suggested fix

The same except branch already records the row as is_correct: grade == 1 (False), and metric.py:61-64 raises on purpose so that an unreadable gold is loud. The summary should follow what the row-level output already says. Smallest change: move total_count += 1 above the try: so every row is counted. A stricter variant that keeps unscorable rows out of the ratio should print how many were skipped (for example Skipped (error): N) and refuse to print an accuracy when total_count == 0, instead of the else 0 fallback. extract_answers.py shows the pattern for the failure row itself: its except branch appends a row with extraction_success: False and keeps it in the output.

Separately, --gold_is_latex can never be False as declared (store_true with default=True), so plain-number golds cannot be scored through this script. action=argparse.BooleanOptionalAction would make the flag work as its help text and the maintainer reply in #28 imply.

Happy to open the PR.

主要言語
Python
スター
1.2k
フォーク
65
30日以内にマージされた PR はありません

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

huggingface/Math-Verify のほかの issue

huggingface/Math-Verify の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。