[bug] Retry + assert in async scenario does not work

CodeceptJS 3.0.5 または 3.0.6、Node.js 14.16.1、Playwright を使って issue の 2 つの async シナリオを再現し、retry と no-retry の実行を比較する。その再現から async シナリオの retry 処理を追跡する。完了条件は、予期しない assertion が retry 後もテスト失敗として残り、最終出力で失敗したテストが 1 件報告されること。

What are you trying to achieve?

Assertion like the doc example (but with an unexpected value in the assertion):

const assert = require('assert')

Scenario.only('test page title 1', async ({ I }) => {
  I.amOnPage('https://www.google.com')
  const title = await I.grabTitle()
  assert.strictEqual(title, 'Bad value')
}).retry(2)

Scenario('test page title 2', { retry: 2 }, async ({ I }) => {
  I.amOnPage('https://www.google.com')
  const title = await I.grabTitle()
  assert.strictEqual(title, 'Bad value')
})

What do you get instead?

With retry option, the test does not fail (ok at the end), it's unexpected:

> codeceptjs run --steps "--verbose"

CodeceptJS v3.0.5
Using test root "---"
Helpers: Playwright, Context, Network
Plugins: screenshotOnFail, retryFailedStep, tryTo

US checkout / LP0001 --
    [1]  Starting recording promises
 › [Session] Starting singleton browser session
  test page title 1
    I am on page "https://www.google.com"
    I grab title
    [1] Error | AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ 'Google'
- 'Bad value'
    [1] Starting <teardown> session
    [1] <teardown> Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /----/codecept/output/test_page_title_1.failed.png
    [2]  Starting recording promises
 › [Session] cleaning cookies and localStorage

  OK  | 0 passed   // 2s

Without the retry, it's good, the test fails as expected:

> lv-front@1.0.2 codecept
> codeceptjs run --steps

CodeceptJS v3.0.5
Using test root "-----"

US checkout / LP0001 --
  test page title 1
    I am on page "https://www.google.com"
    I grab title
  ✖ FAILED in 1059ms


-- FAILURES:

  1) US checkout / LP0001
       test page title 1:

      Expected values to be strictly equal:
+ actual - expected

+ 'Google'
- 'Bad value'
      + expected - actual

      -Google
      +Bad value

  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
  + actual - expected



  Artifacts:
  - screenshot: /----/codecept/output/test_page_title_1.failed.png


  FAIL  | 0 passed, 1 failed   // 1s

Details

• CodeceptJS version: 3.0.5 (but 3.0.6 tested too with the same bug)
• NodeJS Version: 14.16.1
• Operating System: macosX
• playwright