SWE-bench Pro and ProgramBench reports leave out instances that errored during inference
@jahanvipandey22 ci sta già lavorando.
Dal 12/11/2026.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- Mezza giornata
- Idoneità per principianti
- 62/100
Direzione di ricerca
Start with write_report in benchmarks/swebenchpro/eval_infer.py and aggregate_eval_results in benchmarks/programbench/eval_infer.py, and compare them with the errors-file handling in benchmarks/gaia/eval_infer.py and the Commit0 fix in #772. Run the repro script from the issue with SRC pointed at the checkout; it currently prints REPRODUCED. Done when both reports count the errored instance in total_instances and set error_instances to 1, and the script exits 0.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
When an instance errors during inference, output_errors.jsonl gets its row and output.jsonl does not. write_report in benchmarks/swebenchpro/eval_infer.py and aggregate_eval_results in benchmarks/programbench/eval_infer.py only look at output.jsonl (or the harness results built from it), so the report has one instance fewer and error_instances is 0. A resolve rate computed as resolved_instances / total_instances is then higher than with the instance counted. benchmarks/gaia/eval_infer.py already reads the errors file and reports these as error_instances.
Four instances, one errored in inference, two resolved:
total_instances: expected 4, got 3error_instances: expected 1, got 0
Repro
script (extracts the two report functions, no dependencies)
Set SRC to a checkout of the repo.
import ast, json, logging, os, sys, tempfile
from pathlib import Path
from typing import Any
SRC = os.environ.get("SRC", "src")
def load(path, names, ns):
tree = ast.parse(Path(path).read_text())
body = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name in names]
exec(compile(ast.Module(body=body, type_ignores=[]), path, "exec"), ns)
return ns
logger = logging.getLogger("x")
spro = load(f"{SRC}/benchmarks/swebenchpro/eval_infer.py", {"write_report"}, {"json": json, "Path": Path})
pb = load(f"{SRC}/benchmarks/programbench/eval_infer.py",
{"_read_submitted_ids", "aggregate_eval_results"},
{"json": json, "Path": Path, "Any": Any, "logger": logger})
tmp = Path(tempfile.mkdtemp())
# 4 instances run; D errored and aggregate_results removed it from output.jsonl
out = tmp / "output.jsonl"
out.write_text("".join(json.dumps({"instance_id": i}) + "\n" for i in "ABC"))
(tmp / "output_errors.jsonl").write_text(json.dumps({"instance_id": "D", "error": "runner"}) + "\n")
er = tmp / "eval_results.json"
er.write_text(json.dumps({"A": True, "B": True, "C": False}))
r1 = spro["write_report"](er, tmp / "r.json")
run = tmp / "run"
for i, ok in zip("ABC", [1, 1, 0]):
(run / i).mkdir(parents=True)
(run / i / f"{i}.eval.json").write_text(json.dumps(
{"error_code": None, "test_results": [{"status": "passed" if ok else "failed"}]}))
r2 = pb["aggregate_eval_results"](run, pb["_read_submitted_ids"](out))
bad = False
for name, r in (("swebenchpro", r1), ("programbench", r2)):
print(name, "total_instances observed:", r["total_instances"], "expected: 4;",
"error_instances observed:", r["error_instances"], "expected: 1")
bad |= r["total_instances"] != 4 or r["error_instances"] != 1
if bad:
print("REPRODUCED")
sys.exit(1)
Output:
swebenchpro total_instances observed: 3 expected: 4; error_instances observed: 0 expected: 1
programbench total_instances observed: 3 expected: 4; error_instances observed: 0 expected: 1
REPRODUCED
Seen on 542cb71c860c, which is current main.
Related
#493 reported this for the Commit0 report, and #772 fixes it in benchmarks/commit0/eval_infer.py only. The SWE-bench Pro and ProgramBench reports have the same gap and are not touched by #772.
- Lingua principale
- Python
- Stelle
- 134
- Fork
- 97
- Merge medio
- 8g 23h
- PR unite (30g)
- 2
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di OpenHands/benchmarks
-
create_docker_workspace never reuses locally-built images (tag mismatch with DockerDevWorkspace)Aperta
Difficoltà 3/5 1-2 giorni Idoneità per principianti 68/100
OpenHands/benchmarks#779 ·
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
OpenHands/benchmarks#776 ·
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 28/100
OpenHands/benchmarks#764 · 16 commenti ·
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 38/100
OpenHands/benchmarks#744 ·
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
OpenHands/benchmarks#726 ·
Tutte le issue di OpenHands/benchmarks
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
trycua/hermes-cua-spaces#17 ·
-
area:codegen backend:hip bug direction:output good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
I maintainer di solito rispondono entro 1 giorno
-
status:waiting-for-triage type:bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 67/100
microsoftgraph/msgraph-sdk-python#1628 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
nautechsystems/nautilus_trader#5718 ·
I maintainer di solito rispondono entro 1 giorno
-
[Bug]: reme CLI crashes when the running server used a relative config pathForse già presa @nurlanmnn l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
agentscope-ai/ReMe#770 ·
I maintainer di solito rispondono entro 1 giorno