[bug] Retry + assert in async scenario does not work

Riprodurre i due scenari async dell’issue con CodeceptJS 3.0.5 o 3.0.6, Node.js 14.16.1 e Playwright, confrontando le esecuzioni con retry e senza retry. Tracciare la gestione dei retry dello scenario async a partire da questa riproduzione; il lavoro è completato quando l’assertion imprevista rimane un errore del test dopo i retry e l’output finale segnala un test fallito.

What are you trying to achieve?

Assertion like the doc example (but with an unexpected value in the assertion):

const assert = require('assert')

Scenario.only('test page title 1', async ({ I }) => {
  I.amOnPage('https://www.google.com')
  const title = await I.grabTitle()
  assert.strictEqual(title, 'Bad value')
}).retry(2)

Scenario('test page title 2', { retry: 2 }, async ({ I }) => {
  I.amOnPage('https://www.google.com')
  const title = await I.grabTitle()
  assert.strictEqual(title, 'Bad value')
})

What do you get instead?

With retry option, the test does not fail (ok at the end), it's unexpected:

> codeceptjs run --steps "--verbose"

CodeceptJS v3.0.5
Using test root "---"
Helpers: Playwright, Context, Network
Plugins: screenshotOnFail, retryFailedStep, tryTo

US checkout / LP0001 --
    [1]  Starting recording promises
 › [Session] Starting singleton browser session
  test page title 1
    I am on page "https://www.google.com"
    I grab title
    [1] Error | AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ 'Google'
- 'Bad value'
    [1] Starting <teardown> session
    [1] <teardown> Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /----/codecept/output/test_page_title_1.failed.png
    [2]  Starting recording promises
 › [Session] cleaning cookies and localStorage

  OK  | 0 passed   // 2s

Without the retry, it's good, the test fails as expected:

> lv-front@1.0.2 codecept
> codeceptjs run --steps

CodeceptJS v3.0.5
Using test root "-----"

US checkout / LP0001 --
  test page title 1
    I am on page "https://www.google.com"
    I grab title
  ✖ FAILED in 1059ms


-- FAILURES:

  1) US checkout / LP0001
       test page title 1:

      Expected values to be strictly equal:
+ actual - expected

+ 'Google'
- 'Bad value'
      + expected - actual

      -Google
      +Bad value

  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
  + actual - expected



  Artifacts:
  - screenshot: /----/codecept/output/test_page_title_1.failed.png


  FAIL  | 0 passed, 1 failed   // 1s

Details

• CodeceptJS version: 3.0.5 (but 3.0.6 tested too with the same bug)
• NodeJS Version: 14.16.1
• Operating System: macosX
• playwright