BDD skips tests in feature file if background fails in one test

首先使用 Playwright 重現提供的 feature 檔案，並檢視 step_definitions/mock.steps.js 中的步驟。追蹤 BDD runner 如何處理兩個情境之間失敗的 Background；完成標準是，失敗的 background 之後的情境會獨立嘗試，而不是被標記為跳過，並為此範例提供回歸涵蓋。

What are you trying to achieve?

Run a feature file without failing fast and skipping tests if background fails.

What do you get instead?

If you have a feature file with 5 scenarios and a Background. If a background step fails on scenario 3, then scenario 4 and 5 are marked as skipped. As we know sometimes a step fails because of timing issues or a page fails to load, this doesnt mean the background will fail each time. Isn't a Background treated as a beforeEach in mocha anyway? So it should run independently for each scenario.

Provide console output if related. Use --verbose mode for more details.

CodeceptJS v3.0.7
Helpers: Playwright, MochiavelliHelper, PlaywrightHelper, ChaiWrapper
Plugins:

Failing Background --
    [1]  Starting recording promises
 › [Session] Starting singleton browser session
  Search for something @test1
    Given I am on page "https://google.com"
      I am on page "https://google.com"
      › [Browser:Error]
      › [Browser:Error]
      › [Browser:Error]
    Then I should see "Fail test here"
      I see "Fail test here"
    [1] Error | Error
    [1] Error | Error
    [1] Starting <teardown> session
    [1] <teardown> Stopping recording promises
  ✖ FAILED in 23592ms

    [2]  Starting recording promises
 › [Session] cleaning cookies and localStorage
  S Search for something else @test2

-- FAILURES:

  1) Failing Background
       "before each" hook: Before for "Search for something @test1":

      expected web application to include "Fail test here"
      + expected - actual

      -About
      -Store
      -GmailImages
      -Sign in
      -
      -
      -
      -Celebrating the diverse stories of Asian and Pacific Islander communities
      -Advertising
      -Business
      ---( 6 lines more )---
      +Fail test here



  Scenario Steps:
  - I.see("Fail test here") at ./step_definitions/mock.steps.js:148:5
  - I.amOnPage("https://google.com") at ./step_definitions/mock.steps.js:144:5



  FAIL  | 0 passed, 1 failed, 1 skipped   // 24s

Provide test source code if related

steps:

Given('I am on page {string}', (url) => {
  I.amOnPage(url);
});

Then('I should see {string}', (text) => {
  I.see(text);
});

When('I fill in {string} in the search field', (text) => {
  I.fillField('.gLFyf.gsfi', text);
  I.pressKey('Enter');
});

feature file

Feature: Failing Background

  Background: Test
    Given I am on page "https://google.com"
    Then I should see "Fail test here"

  @test1
  Scenario: Search for something
    When I fill in "Test" in the search field

  @test2
  Scenario: Search for something else
    When I fill in "Google" in the search field

Details

• CodeceptJS version: 3.0.7
• NodeJS Version: v14.15.1
• Operating System: osx 10.15
• playwright 1.10