SWE-bench Pro and ProgramBench reports leave out instances that errored during inference

Aperta
#791 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

@jahanvipandey22 ci sta già lavorando.

Dal 12/11/2026.

Valutazione

Difficoltà
3/5
Tempo stimato
Mezza giornata
Idoneità per principianti
62/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start with write_report in benchmarks/swebenchpro/eval_infer.py and aggregate_eval_results in benchmarks/programbench/eval_infer.py, and compare them with the errors-file handling in benchmarks/gaia/eval_infer.py and the Commit0 fix in #772. Run the repro script from the issue with SRC pointed at the checkout; it currently prints REPRODUCED. Done when both reports count the errored instance in total_instances and set error_instances to 1, and the script exits 0.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

When an instance errors during inference, output_errors.jsonl gets its row and output.jsonl does not. write_report in benchmarks/swebenchpro/eval_infer.py and aggregate_eval_results in benchmarks/programbench/eval_infer.py only look at output.jsonl (or the harness results built from it), so the report has one instance fewer and error_instances is 0. A resolve rate computed as resolved_instances / total_instances is then higher than with the instance counted. benchmarks/gaia/eval_infer.py already reads the errors file and reports these as error_instances.

Four instances, one errored in inference, two resolved:

  • total_instances: expected 4, got 3
  • error_instances: expected 1, got 0

Repro

script (extracts the two report functions, no dependencies)

Set SRC to a checkout of the repo.

import ast, json, logging, os, sys, tempfile
from pathlib import Path
from typing import Any

SRC = os.environ.get("SRC", "src")


def load(path, names, ns):
    tree = ast.parse(Path(path).read_text())
    body = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name in names]
    exec(compile(ast.Module(body=body, type_ignores=[]), path, "exec"), ns)
    return ns


logger = logging.getLogger("x")
spro = load(f"{SRC}/benchmarks/swebenchpro/eval_infer.py", {"write_report"}, {"json": json, "Path": Path})
pb = load(f"{SRC}/benchmarks/programbench/eval_infer.py",
          {"_read_submitted_ids", "aggregate_eval_results"},
          {"json": json, "Path": Path, "Any": Any, "logger": logger})

tmp = Path(tempfile.mkdtemp())
# 4 instances run; D errored and aggregate_results removed it from output.jsonl
out = tmp / "output.jsonl"
out.write_text("".join(json.dumps({"instance_id": i}) + "\n" for i in "ABC"))
(tmp / "output_errors.jsonl").write_text(json.dumps({"instance_id": "D", "error": "runner"}) + "\n")

er = tmp / "eval_results.json"
er.write_text(json.dumps({"A": True, "B": True, "C": False}))
r1 = spro["write_report"](er, tmp / "r.json")

run = tmp / "run"
for i, ok in zip("ABC", [1, 1, 0]):
    (run / i).mkdir(parents=True)
    (run / i / f"{i}.eval.json").write_text(json.dumps(
        {"error_code": None, "test_results": [{"status": "passed" if ok else "failed"}]}))
r2 = pb["aggregate_eval_results"](run, pb["_read_submitted_ids"](out))

bad = False
for name, r in (("swebenchpro", r1), ("programbench", r2)):
    print(name, "total_instances observed:", r["total_instances"], "expected: 4;",
          "error_instances observed:", r["error_instances"], "expected: 1")
    bad |= r["total_instances"] != 4 or r["error_instances"] != 1
if bad:
    print("REPRODUCED")
    sys.exit(1)

Output:

swebenchpro total_instances observed: 3 expected: 4; error_instances observed: 0 expected: 1
programbench total_instances observed: 3 expected: 4; error_instances observed: 0 expected: 1
REPRODUCED

Seen on 542cb71c860c, which is current main.

Related

#493 reported this for the Commit0 report, and #772 fixes it in benchmarks/commit0/eval_infer.py only. The SWE-bench Pro and ProgramBench reports have the same gap and are not touched by #772.

Lingua principale
Python
Stelle
134
Fork
97
Merge medio
8g 23h
PR unite (30g)
2

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenHands/benchmarks

Tutte le issue di OpenHands/benchmarks

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.