verify() returns True for any two equations with no free symbols
评估
调研方向
Start by running the two reproduction snippets in the issue: verify(gold=parse("$3 = 4$"), target=parse("$1 = 2$")) should return False but currently returns True. The bug is in the equation fallback of verify, which solves each side over an empty symbol set and treats the resulting empty solution lists as a match. Done means symbol-free equations that are not identical no longer verify equal, and the existing tests still pass.
由索引模型根据 Issue 内容生成。
描述
verify reports two equations as equivalent whenever both parse to relations with no free symbols. The equation fallback solves each side over an empty symbol set, obtains two empty solution lists, and treats the empty pairwise comparison as a match, so any symbol-free pair compares equal, including outright false equations.
Reproduce (math-verify 0.9.0, current PyPI release, Python 3.12):
from math_verify import parse, verify
gold = parse("$I_{e}(0)=\\gamma I_{i}(0)$")
target = parse("$I_{i}(0) = I_{e}(0)$")
print(verify(gold=gold, target=target)) # True
print(verify(gold=parse("$3 = 4$"), target=parse("$1 = 2$"))) # True
Both parses are sympy Equality objects whose free_symbols set is empty: the terms are undefined-function applications such as I_i(0), which do not count as free symbols. verify returns True for both pairs.
Expected: an empty solution list should not satisfy the equation comparison; symbol-free relations that are not trivially identical should not verify equal.
Impact: this inflated benchmark scores in an LLM evaluation harness that grades extracted LaTeX answers through parse/verify — a candidate equation and a structurally different reference equation, both without free symbols, always received deterministic credit. Evidence and the downstream guard:
- https://github.com/marin-community/evalchemy/issues/207 — report with an OlympiadBench regrade record (
s3://marin-us-east-02a/marin/evals/ablations/olympiadbench-minimax-m3-judge-20261009-guided-choice-thinking/results/inclusionAI--Ling-lite-1.5.json, document 4, trial 6) - https://github.com/marin-community/evalchemy/pull/208 — downstream guard that skips
verifywhen both parses hold only symbol-free relations
- 主要语言
- Python
- 星标
- 1.2k
- 派生
- 67
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
huggingface/Math-Verify 的其他 Issue
-
verify() raises ValueError comparing chained relations of different length可能已有人在做 @chetana1410 于 4 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
huggingface/Math-Verify#93 ·
-
A period inside the math delimiters (`$\frac{1}{3}.$`) makes correct answers fail and can make wrong ones pass可能已有人在做 @MohammadHijjawi97 于 27 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 90/100
huggingface/Math-Verify#85 ·
-
evaluate_model_outputs.py: rows whose gold cannot be extracted are dropped from the accuracy denominator, so the printed accuracy disagrees with the is_correct column the script writes可能已有人在做 @shaurya416 于 32 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
huggingface/Math-Verify#83 ·
-
verify() rejects a not-equal written in the other order可能已有人在做 @chetana1410 于 4 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 35/100
huggingface/Math-Verify#100 ·
-
verify() rejects an absolute-value inequality that is the same interval可能已有人在做 @chetana1410 于 4 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 35/100
huggingface/Math-Verify#98 ·
查看 huggingface/Math-Verify 的全部 Issue
相似的 Issue
-
enhancement
难度 2/5 1-3 小时 新手友好度 78/100
glossyio/traffic-monitor#264 ·
维护者通常 1 天内回复
-
Add self-host未关闭
难度 1/5 1 小时以内 新手友好度 75/100
Atarity/deploy-your-own-saas#102 ·
-
[Bug]: RDFlex silently drops the covariate adjustment when the outcome or treatment column is integer-typed可能已有人在做 @vrjdev 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
DoubleML/doubleml-for-py#424 ·
维护者通常 1 天内回复
-
agentic-dmx docs enhancement
难度 1/5 1-3 小时 新手友好度 86/100
-
难度 1/5 1 小时以内 新手友好度 72/100
521xueweihan/HelloGitHub#4150 ·