Tests failed via Mocha afterEach conditional failure reported as passed *and* failed!
#5181 aperta il 31 mar 2020
Metriche repository
- Star
- (6029 stelle)
- Metriche merge PR
- (Merge medio 15g 15h) (48 PR mergiate in 30 g)
Descrizione
Environment (please complete the following information):
- **WebdriverIO version: 6.0.5
- **Mode: Testrunner
- **If WDIO Testrunner, running sync/async: Sync
- **Node.js version: 10.19.0
- **NPM version: 6.13.4
- **Browser name and version: Chrome 80.0.3987.149
- **Platform name and version: CentOS 8
- **Additional wdio packages used (if applicable): node_modules/@wdio contains: cli config local-runner logger mocha-framework protocols repl reporter runner spec-reporter sync utils
Config of WebdriverIO wdio.conf.js as generated, plus hooks:
before: function() {
const chai = require('chai');
global.expect = chai.expect;
chai.Should();
},
beforeTest: function (test, context) {
browser.config.specErrors = [];
},
Describe the bug The above test uses try/catch to allow ALL errors in a mocha test, rather than fail on first, and uses a conditional fail at the end as advised here: https://github.com/mochajs/mocha/wiki/HOW-TO:-Conditionally-fail-a-test-after-completion
It works fine, except that WDIO output reports the test as both failed AND passed!....
------------------------------------------------------------------
[chrome linux #0-0] Spec: /home/user/dev/WDIO/test/specs/main.js
[chrome linux #0-0] Running: chrome on linux
[chrome linux #0-0] Session ID: 7602b29963a51e412dc07dab542274cd
[chrome linux #0-0]
[chrome linux #0-0] suite definition
[chrome linux #0-0] ✖ spec definition 1
[chrome linux #0-0]
[chrome linux #0-0] 1 passing (138ms)
[chrome linux #0-0] 1 failing
[chrome linux #0-0]
[chrome linux #0-0] 1) suite definition spec definition 1
[chrome linux #0-0] just because
[chrome linux #0-0] Error: just because
[chrome linux #0-0] at Context.<anonymous> (/home/user/dev/WDIO/test/specs/main.js:7:25)
[chrome linux #0-0] at callFn (/home/user/dev/WDIO/node_modules/mocha/lib/runnable.js:395:21)
[chrome linux #0-0] at Hook.Runnable.run (/home/user/dev/WDIO/node_modules/mocha/lib/runnable.js:382:7)
[chrome linux #0-0] at next (/home/user/dev/WDIO/node_modules/mocha/lib/runner.js:389:10)
[chrome linux #0-0] at Immediate._onImmediate (/home/user/dev/WDIO/node_modules/mocha/lib/runner.js:433:5)
[chrome linux #0-0] at runCallback (timers.js:705:18)
[chrome linux #0-0] at tryOnImmediate (timers.js:676:5)
[chrome linux #0-0] at processImmediate (timers.js:658:5)
[chrome linux #0-0] at process.topLevelDomainCallback (domain.js:126:23)
Spec Files: 0 passed, 1 failed, 1 total (100% completed) in 00:00:01
To Reproduce Steps to reproduce the behavior:
Sample spec:
describe('suite definition', function () {
this.afterEach(function () {
this.test.error(new Error('just because'));
})
it('spec definition 1', function () {
expect(1).to.equal(1);
})
})
Expected behavior The tests are being failed, which is correct - and the summary report at the end of the run is correct. However, it appears the count of passed tests is erroneously being incremented. This should not happen. In the above example, 0 tests should be reported as passed.
Log If applicable, add logs to help explain your problem. N/A - above should be sufficient
Additional context None