webdriverio/webdriverio

Tests failed via Mocha afterEach conditional failure reported as passed *and* failed!

Aperta

#5181 aperta il 31 mar 2020

 (13 commenti) (0 reazioni) (0 assegnatari)JavaScript (1793 fork)batch import
Bug 🐛good first pickhelp wanted

Metriche repository

Star
 (6029 stelle)
Metriche merge PR
 (Merge medio 15g 15h) (48 PR mergiate in 30 g)

Descrizione

Environment (please complete the following information):

  • **WebdriverIO version: 6.0.5
  • **Mode: Testrunner
  • **If WDIO Testrunner, running sync/async: Sync
  • **Node.js version: 10.19.0
  • **NPM version: 6.13.4
  • **Browser name and version: Chrome 80.0.3987.149
  • **Platform name and version: CentOS 8
  • **Additional wdio packages used (if applicable): node_modules/@wdio contains: cli config local-runner logger mocha-framework protocols repl reporter runner spec-reporter sync utils

Config of WebdriverIO wdio.conf.js as generated, plus hooks:

    before: function() {
        const chai = require('chai');
        global.expect = chai.expect;
        chai.Should();
    },

    beforeTest: function (test, context) {
        browser.config.specErrors = [];
    },

Describe the bug The above test uses try/catch to allow ALL errors in a mocha test, rather than fail on first, and uses a conditional fail at the end as advised here: https://github.com/mochajs/mocha/wiki/HOW-TO:-Conditionally-fail-a-test-after-completion

It works fine, except that WDIO output reports the test as both failed AND passed!....

------------------------------------------------------------------
[chrome  linux #0-0] Spec: /home/user/dev/WDIO/test/specs/main.js
[chrome  linux #0-0] Running: chrome on linux
[chrome  linux #0-0] Session ID: 7602b29963a51e412dc07dab542274cd
[chrome  linux #0-0]
[chrome  linux #0-0] suite definition
[chrome  linux #0-0]    ✖ spec definition 1
[chrome  linux #0-0]
[chrome  linux #0-0] 1 passing (138ms)
[chrome  linux #0-0] 1 failing
[chrome  linux #0-0]
[chrome  linux #0-0] 1) suite definition spec definition 1
[chrome  linux #0-0] just because
[chrome  linux #0-0] Error: just because
[chrome  linux #0-0]     at Context.<anonymous> (/home/user/dev/WDIO/test/specs/main.js:7:25)
[chrome  linux #0-0]     at callFn (/home/user/dev/WDIO/node_modules/mocha/lib/runnable.js:395:21)
[chrome  linux #0-0]     at Hook.Runnable.run (/home/user/dev/WDIO/node_modules/mocha/lib/runnable.js:382:7)
[chrome  linux #0-0]     at next (/home/user/dev/WDIO/node_modules/mocha/lib/runner.js:389:10)
[chrome  linux #0-0]     at Immediate._onImmediate (/home/user/dev/WDIO/node_modules/mocha/lib/runner.js:433:5)
[chrome  linux #0-0]     at runCallback (timers.js:705:18)
[chrome  linux #0-0]     at tryOnImmediate (timers.js:676:5)
[chrome  linux #0-0]     at processImmediate (timers.js:658:5)
[chrome  linux #0-0]     at process.topLevelDomainCallback (domain.js:126:23)


Spec Files:	 0 passed, 1 failed, 1 total (100% completed) in 00:00:01 


To Reproduce Steps to reproduce the behavior:

Sample spec:


describe('suite definition', function () {

    this.afterEach(function () {
        this.test.error(new Error('just because'));
    })

    it('spec definition 1', function () {
        expect(1).to.equal(1);
    })

})

Expected behavior The tests are being failed, which is correct - and the summary report at the end of the run is correct. However, it appears the count of passed tests is erroneously being incremented. This should not happen. In the above example, 0 tests should be reported as passed.

Log If applicable, add logs to help explain your problem. N/A - above should be sufficient

Additional context None

Guida contributor