Incorrect Testing Failures Message
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 45/100
Research direction
Reproduce the ADD example using the shown Java test cases and compare the current and correct failure messages. Trace the test or report-generation entry point responsible for formatting assertion failures; done means failed tests label the intended value as expected and the returned value as actual.
Written by the indexing model from the issue text.
Description
I've observed that the test reports for some bugs seem to get the “expected“ value and “but was” value obtained reversed.
For example:
Bug_ID: ADD
Buggy_Code:
/* Add two numbers x and y
>>> add(2, 3)
5
>>> add(5, 7)
12 */
public class ADD {
public static int add(int x, int y) {
return x | y;
}
}
Test Cases:
public class TEST_ADD {
@org.junit.Test(timeout = 3000)
public void test_0() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(0, 1);
org.junit.Assert.assertEquals(
result, 1
);
}
@org.junit.Test(timeout = 3000)
public void test_1() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(1, 0);
org.junit.Assert.assertEquals(
result, 1
);
}
@org.junit.Test(timeout = 3000)
public void test_2() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(2, 3);
org.junit.Assert.assertEquals(
result, 5
);
}
@org.junit.Test(timeout = 3000)
public void test_3() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(5, 7);
org.junit.Assert.assertEquals(
result, 12
);
}
@org.junit.Test(timeout = 3000)
public void test_4() throws java.lang.Exception {
int result = humaneval.buggy.ADD.add(7, 5);
org.junit.Assert.assertEquals(
result, 12
);
}
}
Current (Incorrect) Testing Failures Message:
Tests run: 5, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in humaneval.TEST_ADD
test_2(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<5>
at humaneval.TEST_ADD.test_2(TEST_ADD.java:23)
test_3(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<7> but was:<12>
at humaneval.TEST_ADD.test_3(TEST_ADD.java:31)
test_4(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<7> but was:<12>
at humaneval.TEST_ADD.test_4(TEST_ADD.java:39)
Correct Testing Failures Message:
Tests run: 5, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in humaneval.TEST_ADD
test_2(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<3>
at humaneval.TEST_ADD.test_2(TEST_ADD.java:23)
test_3(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<12> but was:<7>
at humaneval.TEST_ADD.test_3(TEST_ADD.java:31)
test_4(humaneval.TEST_ADD) Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError: expected:<12> but was:<7>
at humaneval.TEST_ADD.test_4(TEST_ADD.java:39)
- Dominant language
- Java
- Stars
- 22
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100