Incorrect Testing Failures Message

Open
#6 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
java
Domain
testing

Research direction

Reproduce the ADD example using the shown Java test cases and compare the current and correct failure messages. Trace the test or report-generation entry point responsible for formatting assertion failures; done means failed tests label the intended value as expected and the returned value as actual.

Written by the indexing model from the issue text.

Description

I've observed that the test reports for some bugs seem to get the “expected“ value and “but was” value obtained reversed.
For example:

Bug_ID: ADD

Buggy_Code:

/* Add two numbers x and y
>>> add(2, 3)
5
>>> add(5, 7)
12 */

public class ADD {
    public static int add(int x, int y) {
        return x | y;
    }
}

Test Cases:

public class TEST_ADD {
    @org.junit.Test(timeout = 3000)
    public void test_0() throws java.lang.Exception {
        int result = humaneval.buggy.ADD.add(0, 1);
        org.junit.Assert.assertEquals(
            result, 1
        );
    }

    @org.junit.Test(timeout = 3000)
    public void test_1() throws java.lang.Exception {
        int result = humaneval.buggy.ADD.add(1, 0);
        org.junit.Assert.assertEquals(
            result, 1
        );
    }

    @org.junit.Test(timeout = 3000)
    public void test_2() throws java.lang.Exception {
        int result = humaneval.buggy.ADD.add(2, 3);
        org.junit.Assert.assertEquals(
            result, 5
        );
    }

    @org.junit.Test(timeout = 3000)
    public void test_3() throws java.lang.Exception {
        int result = humaneval.buggy.ADD.add(5, 7);
        org.junit.Assert.assertEquals(
            result, 12
        );
    }

    @org.junit.Test(timeout = 3000)
    public void test_4() throws java.lang.Exception {
        int result = humaneval.buggy.ADD.add(7, 5);
        org.junit.Assert.assertEquals(
            result, 12
        );
    }
}

Current (Incorrect) Testing Failures Message:

Tests run: 5, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in humaneval.TEST_ADD
test_2(humaneval.TEST_ADD)  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<5>
	at humaneval.TEST_ADD.test_2(TEST_ADD.java:23)

test_3(humaneval.TEST_ADD)  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError: expected:<7> but was:<12>
	at humaneval.TEST_ADD.test_3(TEST_ADD.java:31)

test_4(humaneval.TEST_ADD)  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError: expected:<7> but was:<12>
	at humaneval.TEST_ADD.test_4(TEST_ADD.java:39)

Correct Testing Failures Message:

Tests run: 5, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in humaneval.TEST_ADD
test_2(humaneval.TEST_ADD)  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<3>
	at humaneval.TEST_ADD.test_2(TEST_ADD.java:23)

test_3(humaneval.TEST_ADD)  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError: expected:<12> but was:<7>
	at humaneval.TEST_ADD.test_3(TEST_ADD.java:31)

test_4(humaneval.TEST_ADD)  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError: expected:<12> but was:<7>
	at humaneval.TEST_ADD.test_4(TEST_ADD.java:39)
Dominant language
Java
Stars
22
Forks
3
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.