Failed result serialization destroys a previously saved JSON result

Open Beginner friendly
  • Is something wrong?
#867 0 comments 0 reactions 0 assignees View on GitHub

@chessing234 is already working on this.

Since Nov 15, 2026.

  • #868 by @chessing234 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
tooling

Research direction

Start in src/evaluate/saving.py at the save function, where line 43 runs json.dump(data, f) after the destination is already open for writing. Run the reproducer in the issue to confirm the truncated file, then move serialization to a json.dumps call before the file is opened so a TypeError leaves existing bytes untouched and creates no new file. Done means the reproducer prints Previous result preserved: True, a fresh destination stays absent after failure, and existing saving tests still pass on both dependency stacks; note the reporter indicates a fix and test cases already exist, so check for in-flight work first.

Written by the indexing model from the issue text.

Description

evaluate.save() opens an existing result in write mode before it knows whether the new payload can be JSON-serialized. A serialization error can therefore destroy an earlier valid result and leave an incomplete JSON file.

Reproducer (no Hub downloads):

from pathlib import Path
from tempfile import TemporaryDirectory
import traceback
import numpy as np
import evaluate

with TemporaryDirectory() as directory:
    destination = Path(directory) / "results.json"
    evaluate.save(destination, accuracy=0.75)
    previous = destination.read_bytes()
    try:
        evaluate.save(destination, accuracy=1.0, details={"scores": np.array([1, 2])})
    except TypeError:
        traceback.print_exc()
    print("Previous result preserved:", destination.read_bytes() == previous)
    print(destination.read_text(encoding="utf-8"))

Actual output after the exception:

Previous result preserved: False
{"accuracy": 1.0, "details": {"scores":

Expected: still raise TypeError for unsupported JSON values, but preserve the previous file unchanged. For a new destination, a failed serialization should not leave a partial result file. This is not a request to add NumPy encoding support or promise atomic writes against operating-system/process failures.

Full traceback from the standalone reproduction (installation prefixes shortened):

Traceback (most recent call last):
  File "repro.py", line 51, in save_failure
    evaluate.save(destination, accuracy=1.0, details={"scores": np.array([1, 2])})
  File "src/evaluate/saving.py", line 43, in save
    json.dump(data, f)
  File "python3.11/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "python3.11/json/encoder.py", line 432, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "python3.11/json/encoder.py", line 439, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ndarray is not JSON serializable

Reproduced on unchanged main a5db2ad54aaeadf15c7aa1f056bc6f7bbeeb7e5f, macOS 26.6.2 / Darwin 25.6.0, Python 3.11.13, with datasets 2.21.0 / NumPy 1.26.4 and datasets 5.0.1 / NumPy 2.4.6. PyTorch 2.8.0 is installed but is not involved; TensorFlow is not installed.

Searches of open/closed reports for save, overwrite, corruption, serializable, atomic, truncate, and _setup_path found no matching fix. Related #808 handles missing Git and #809 handles dotted directories. The original #7 discusses format choices and JSONL concurrency, not this single-call serialization failure.

A focused fix serializes the payload before opening the destination. Four failure cases cover existing/new files and scalar/nested NumPy values; another case keeps intentional replacement of a valid explicit destination working. These and the existing saving tests pass on both dependency stacks (8 per stack), and repository quality checks pass. All four preservation cases fail on unchanged main.

Dominant language
Python
Stars
2.5k
Forks
380
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/evaluate

All issues in huggingface/evaluate

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.