pytest-rerunfailures 16.6.1 duplicates JUnit records and inflates scores
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
Research direction
Start at parse_test_results and reproduce the issue with pytest-rerunfailures==16.6.1, --reruns=2, and the always-failing test described here. Verify that duplicate JUnit records are grouped by logical test and scored using the final attempt, that retry-selection metrics count logical tests, and that the installed rerun plugin version is reproducible.
Written by the indexing model from the issue text.
Description
Problem
The evaluator installs pytest-rerunfailures without a version constraint. Version 16.6.1 changed its JUnit reporting behavior so that every rerun attempt produces a <testcase> record.
For a test that still fails after two reruns, the XML contains two empty records followed by one failure record for the same classname and name:
<testcase classname="test_rerun" name="test_always_fails" />
<testcase classname="test_rerun" name="test_always_fails" />
<testcase classname="test_rerun" name="test_always_fails">
<failure message="assert False">AssertionError</failure>
</testcase>
parse_test_results currently processes each record independently and treats a record without a result child as passed. The single failing logical test is therefore scored as two passes and one failure, producing a score of 2/3 instead of 0/1.
Reproduction
- Install
pytest-rerunfailures==16.6.1. - Run an always-failing test with
pytest --reruns=2 --junitxml=results.xml. - Pass the resulting XML to
parse_test_results.
With 16.6, the XML contains only the final failure record. With 16.6.1, it contains the three records shown above.
Expected behavior
Each logical test should contribute exactly once to the score, using the final attempt's status. Retry-selection metrics should also count logical tests rather than raw JUnit records. The installed rerun plugin version should be reproducible.
A fix is available in #63.
- Dominant language
- Python
- Stars
- 928
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from facebookresearch/ProgramBench
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Difficulty 4/5 3-5 days Newbie friendliness 58/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
facebookresearch/ProgramBench#50 · 1 comment ·
All issues in facebookresearch/ProgramBench
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
syfoud/Simulated_Scepter#172 ·
-
A cancelled tests run makes the coverage comment workflow fail and reports it as a red check on main Openarea: ci bug perceived difficulty: 3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Nitjsefnie-Harness-Commons/daedalus#921 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·