intermittent test failures: web-forms e2e: firefox visuals
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- playwright, typescript
- Domain
- testing-qa
Research direction
The failing checks are screenshot comparisons in e2e/test-cases/visual/all-question-types.test.ts (lines 68 and 82) and image-labels.test.ts, run through the expectScreenshot helper in e2e/screenshot.ts and MapControl.expectMapScreenshot in e2e/page-objects/controls/MapControl.ts. Start by reproducing the Firefox visual project locally and comparing the diff images named in the log. Done means the Firefox visual tests pass consistently across repeated CI runs without masking real regressions.
Written by the indexing model from the issue text.
Description
https://github.com/getodk/central-frontend/actions/runs/43368646506/job/131701057077?pr=1990
Running 21 tests using 1 worker
··××F××F·········×±×±······
1) [visual-firefox] › e2e/test-cases/visual/all-question-types.test.ts:68:5 › All Question Types (Visual) › Select one from map › zooms in and out, pans the map and zooms to fit all features
Error: expect(page).toHaveScreenshot(expected) failed
30275 pixels (ratio 0.08 of all image pixels) are different.
Snapshot: select-map-zoom-out.png
Call log:
- Expect "toHaveScreenshot(select-map-zoom-out.png)" with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 30275 pixels (ratio 0.08 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 30275 pixels (ratio 0.08 of all image pixels) are different.
at screenshot.ts:102
100 | const box = await getBox(locator, screenshotName);
101 |
> 102 | await expect(page).toHaveScreenshot(screenshotName, {
| ^
103 | fullPage: isFullPage,
104 | clip: {
105 | x: Math.round(box.x + scroll.x),
at expectScreenshot (/__w/central-frontend/central-frontend/packages/web-forms/e2e/screenshot.ts:102:22)
at MapControl.expectMapScreenshot (/__w/central-frontend/central-frontend/packages/web-forms/e2e/page-objects/controls/MapControl.ts:216:5)
at /__w/central-frontend/central-frontend/packages/web-forms/e2e/test-cases/visual/all-question-types.test.ts:70:7
attachment #1: select-map-zoom-out (image/png) ─────────────────────────────────────────────────
Expected: e2e/test-cases/visual/all-question-types.test.ts-snapshots/select-map-zoom-out-firefox.png
Received: test-results/test-cases-visual-all-ques-481b5-d-zooms-to-fit-all-features-visual-firefox/select-map-zoom-out-actual.png
Diff: test-results/test-cases-visual-all-ques-481b5-d-zooms-to-fit-all-features-visual-firefox/select-map-zoom-out-diff.png
Error Context: test-results/test-cases-visual-image-la-f28e5-cked-columns-and-no-buttons-visual-firefox/error-context.md
2 failed
[visual-firefox] › e2e/test-cases/visual/all-question-types.test.ts:68:5 › All Question Types (Visual) › Select one from map › zooms in and out, pans the map and zooms to fit all features
[visual-firefox] › e2e/test-cases/visual/all-question-types.test.ts:82:5 › All Question Types (Visual) › Select one from map › opens details of saved feature and remove saved feature
2 flaky
[visual-firefox] › e2e/test-cases/visual/image-labels.test.ts:53:5 › Image Selects (Visual) › Select with images › select one with fixed column count
[visual-firefox] › e2e/test-cases/visual/image-labels.test.ts:66:5 › Image Selects (Visual) › Select with images › select one with packed columns and no buttons
17 passed (2.8m)
Error: Process completed with exit code 1.
- Dominant language
- JavaScript
- Stars
- 233
- Forks
- 241
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 13
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from getodk/central
-
sessions are not reapedPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
The `New Submission` button on the Submissions tab is now harder to access on mobilePossibly taken @sadiqkhoja claimed this 1 day ago. Openfrontend
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
getodk/central#2334 · 1 assignee ·
Maintainers usually reply within 1 day
-
Copying text from table clicks on the rowPossibly taken @sadiqkhoja claimed this 1 day ago. Openfrontend needs testing
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
getodk/central#2331 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
Hide closed formsPossibly taken @nicoleorlowski claimed this 8 days ago. Openfrontend needs testing
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
getodk/central#2323 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
product-os/flowzone#2048 ·
Maintainers usually reply within 1 day
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
hugomrdias/playwright-test#786 · 1 comment ·
Maintainers usually reply within 1 day
-
code-review secondshift
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
piotrekno1/second-shift#419 ·
Maintainers usually reply within 1 day
-
area:compiler effort:S impact:low platform:wasm type:chore
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
PolicyEngine/cliff-watch#55 ·
Maintainers usually reply within 2 days