intermittent test failures: web-forms e2e: firefox visuals

Open
#2,327 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
1-2 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
playwright, typescript
Domain
testing-qa

Research direction

The failing checks are screenshot comparisons in e2e/test-cases/visual/all-question-types.test.ts (lines 68 and 82) and image-labels.test.ts, run through the expectScreenshot helper in e2e/screenshot.ts and MapControl.expectMapScreenshot in e2e/page-objects/controls/MapControl.ts. Start by reproducing the Firefox visual project locally and comparing the diff images named in the log. Done means the Firefox visual tests pass consistently across repeated CI runs without masking real regressions.

Written by the indexing model from the issue text.

Description

frontend

https://github.com/getodk/central-frontend/actions/runs/43368646506/job/131701057077?pr=1990

Running 21 tests using 1 worker
··××F××F·········×±×±······

  1) [visual-firefox] › e2e/test-cases/visual/all-question-types.test.ts:68:5 › All Question Types (Visual) › Select one from map › zooms in and out, pans the map and zooms to fit all features 

    Error: expect(page).toHaveScreenshot(expected) failed

      30275 pixels (ratio 0.08 of all image pixels) are different.

      Snapshot: select-map-zoom-out.png

    Call log:
      - Expect "toHaveScreenshot(select-map-zoom-out.png)" with timeout 5000ms
        - verifying given screenshot expectation
      - taking page screenshot
        - disabled all CSS animations
      - waiting for fonts to load...
      - fonts loaded
      - 30275 pixels (ratio 0.08 of all image pixels) are different.
      - waiting 100ms before taking screenshot
      - taking page screenshot
        - disabled all CSS animations
      - waiting for fonts to load...
      - fonts loaded
      - captured a stable screenshot
      - 30275 pixels (ratio 0.08 of all image pixels) are different.


       at screenshot.ts:102

      100 |   const box = await getBox(locator, screenshotName);
      101 |
    > 102 |   await expect(page).toHaveScreenshot(screenshotName, {
          |                      ^
      103 |     fullPage: isFullPage,
      104 |     clip: {
      105 |       x: Math.round(box.x + scroll.x),
        at expectScreenshot (/__w/central-frontend/central-frontend/packages/web-forms/e2e/screenshot.ts:102:22)
        at MapControl.expectMapScreenshot (/__w/central-frontend/central-frontend/packages/web-forms/e2e/page-objects/controls/MapControl.ts:216:5)
        at /__w/central-frontend/central-frontend/packages/web-forms/e2e/test-cases/visual/all-question-types.test.ts:70:7

    attachment #1: select-map-zoom-out (image/png) ─────────────────────────────────────────────────
    Expected: e2e/test-cases/visual/all-question-types.test.ts-snapshots/select-map-zoom-out-firefox.png
    Received: test-results/test-cases-visual-all-ques-481b5-d-zooms-to-fit-all-features-visual-firefox/select-map-zoom-out-actual.png
    Diff:     test-results/test-cases-visual-all-ques-481b5-d-zooms-to-fit-all-features-visual-firefox/select-map-zoom-out-diff.png
    Error Context: test-results/test-cases-visual-image-la-f28e5-cked-columns-and-no-buttons-visual-firefox/error-context.md

  2 failed
    [visual-firefox] › e2e/test-cases/visual/all-question-types.test.ts:68:5 › All Question Types (Visual) › Select one from map › zooms in and out, pans the map and zooms to fit all features 
    [visual-firefox] › e2e/test-cases/visual/all-question-types.test.ts:82:5 › All Question Types (Visual) › Select one from map › opens details of saved feature and remove saved feature 
  2 flaky
    [visual-firefox] › e2e/test-cases/visual/image-labels.test.ts:53:5 › Image Selects (Visual) › Select with images › select one with fixed column count 
    [visual-firefox] › e2e/test-cases/visual/image-labels.test.ts:66:5 › Image Selects (Visual) › Select with images › select one with packed columns and no buttons 
  17 passed (2.8m)
Error: Process completed with exit code 1.

Dominant language
JavaScript
Stars
233
Forks
241
Avg merge
1d 22h
Merged PRs (30d)
13

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from getodk/central

All issues in getodk/central

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.