[bug] Retry + assert in async scenario does not work
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- javascript, node.js, playwright
- Domain
- testing-qa
Research direction
Reproduce the two async scenarios from the issue with CodeceptJS 3.0.5 or 3.0.6, Node.js 14.16.1, and Playwright, comparing retry and no-retry runs. Trace the async scenario retry handling from that reproduction; done means the unexpected assertion remains a test failure after retries and the final output reports one failed test.
Written by the indexing model from the issue text.
Description
What are you trying to achieve?
Assertion like the doc example (but with an unexpected value in the assertion):
const assert = require('assert')
Scenario.only('test page title 1', async ({ I }) => {
I.amOnPage('https://www.google.com')
const title = await I.grabTitle()
assert.strictEqual(title, 'Bad value')
}).retry(2)
Scenario('test page title 2', { retry: 2 }, async ({ I }) => {
I.amOnPage('https://www.google.com')
const title = await I.grabTitle()
assert.strictEqual(title, 'Bad value')
})
What do you get instead?
With retry option, the test does not fail (ok at the end), it's unexpected:
> codeceptjs run --steps "--verbose"
CodeceptJS v3.0.5
Using test root "---"
Helpers: Playwright, Context, Network
Plugins: screenshotOnFail, retryFailedStep, tryTo
US checkout / LP0001 --
[1] Starting recording promises
› [Session] Starting singleton browser session
test page title 1
I am on page "https://www.google.com"
I grab title
[1] Error | AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected
+ 'Google'
- 'Bad value'
[1] Starting <teardown> session
[1] <teardown> Stopping recording promises
› <screenshotOnFail> Test failed, try to save a screenshot
› Screenshot is saving to /----/codecept/output/test_page_title_1.failed.png
[2] Starting recording promises
› [Session] cleaning cookies and localStorage
OK | 0 passed // 2s
Without the retry, it's good, the test fails as expected:
> lv-front@1.0.2 codecept
> codeceptjs run --steps
CodeceptJS v3.0.5
Using test root "-----"
US checkout / LP0001 --
test page title 1
I am on page "https://www.google.com"
I grab title
✖ FAILED in 1059ms
-- FAILURES:
1) US checkout / LP0001
test page title 1:
Expected values to be strictly equal:
+ actual - expected
+ 'Google'
- 'Bad value'
+ expected - actual
-Google
+Bad value
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected
Artifacts:
- screenshot: /----/codecept/output/test_page_title_1.failed.png
FAIL | 0 passed, 1 failed // 1s
Details
- CodeceptJS version: 3.0.5 (but 3.0.6 tested too with the same bug)
- NodeJS Version: 14.16.1
- Operating System: macosX
- playwright
- Dominant language
- JavaScript
- Stars
- 4.2k
- Forks
- 756
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 16
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from codeceptjs/CodeceptJS
-
stale
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
codeceptjs/CodeceptJS#5420 · 1 comment ·
-
stale
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
codeceptjs/CodeceptJS#5358 · 1 comment ·
-
stale
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
codeceptjs/CodeceptJS#4958 · 10 comments · 2 reactions ·
-
stale
Difficulty 1/5 Under an hour Newbie friendliness 72/100
codeceptjs/CodeceptJS#4778 · 3 comments ·
-
stale
Difficulty 4/5 3-5 days Newbie friendliness 38/100
codeceptjs/CodeceptJS#5618 · 1 comment ·
All issues in codeceptjs/CodeceptJS
Similar issues
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Automattic/safe-publish#594 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
dream-num/dsh-univer-office#104 ·
-
comp/dashboard invalid P3
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
NousResearch/hermes-agent#121143 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
avniproject/avni-webapp#1811 ·
-
area/auroraboot area/webui bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100