JsonMatch treats matching JSON null values as incorrect
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- testing-qa
Research direction
Start with evals.elsuite.basic.json_match.json_match() and inspect its top-level None check and recursive dictionary comparison. Add regression coverage for top-level, object, and list null matches, plus missing-key versus null mismatches. Done means real JSON null values match while missing keys remain distinct and existing exact matching semantics are unchanged.
Written by the indexing model from the issue text.
Description
Describe the bug
evals.elsuite.basic.json_match.json_match() currently returns False whenever either side is Python None:
if sampled_json is None or correct_json is None:
return False
That conflates two different cases:
- a real JSON
null, whichjson.loads()legitimately decodes toNone - a missing dictionary key, which the recursive comparison also currently represents with
.get(...) == None
As a result, valid equal JSON containing null is scored incorrectly. For example, {"value": null} does not match the identical ideal answer.
Expected behavior
- JSON
nullshould match JSONnull - a missing key should still not match a present key whose value is
null - existing exact recursive matching semantics should remain unchanged
Proposed fix
Use a private missing-value sentinel when comparing dictionary keys so real None values can be compared normally. Add regression coverage for top-level/null-in-object/null-in-list matches and for missing-key-vs-null mismatches.
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openai/evals
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100