BackstopJS Not Detecting Missing UI Elements Despite Zero Mismatch Threshold - False Positive Issue

Abierto
  • ¿Algo no está bien?
#1,607 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
javascript
Área
testing

Línea de trabajo

Comienza reproduciendo la comparación con misMatchThreshold establecido en 0 con ambas configuraciones de requireSameDimensions, usando una imagen de referencia que contenga el botón y una imagen de prueba que no lo contenga. Traza cómo BackstopJS compara imágenes con alturas diferentes y define una prueba de regresión que demuestre que el contenido faltante se informa como un mismatch en lugar de como un pass.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

BackstopJS Not Detecting Missing UI Elements Despite Zero Mismatch Threshold - False Positive Issue

Issue Description

BackstopJS is producing false positives by failing to detect visual regressions when UI elements are completely missing from the test image compared to the reference image, even with misMatchThreshold set to 0.

Expected Behavior

When misMatchThreshold is set to 0, BackstopJS should detect any visual differences between the reference and test images, including missing UI elements. A completely missing "Send feedback" button should result in a test failure.

Actual Behavior

BackstopJS reports the test as PASSING (false positive) when a significant UI element is completely missing from the test image. The "Send feedback" button is entirely absent in the test image but present in the reference image, yet the comparison passes.

Configuration Tested

{
  "misMatchThreshold": 0,
  "requireSameDimensions": false
}

Also tested with:

{
  "misMatchThreshold": 0,
  "requireSameDimensions": true
}

Steps to Reproduce

  1. Set up BackstopJS configuration with misMatchThreshold: 0 and requireSameDimensions: false
  2. Create a reference image with complete UI including the "Send feedback" button
  3. Generate a test scenario where the "Send feedback" button is missing/removed
  4. Run BackstopJS visual comparison
  5. Result: Test passes despite obvious missing button

Environment

  • BackstopJS Version: [6.3.25] Also tried out with 6.2.2
  • Browser: [Chromium]
  • OS: [MAC OS]

Detailed Findings

With requireSameDimensions: false (Current Issue)
  • Test passes despite missing "Send feedback" button
  • No visual regression detected
  • Images have different heights due to missing button
  • This is the main issue - false positive
With requireSameDimensions: true (Also Problematic)
  • Test fails due to dimension mismatch
  • BUT fails for wrong reason (dimension difference, not missing button)
  • Doesn't specifically identify what visual content is different
  • Catches the height change but not the semantic difference

Screenshots

Image

Core Problem

BackstopJS appears to have issues with:

  1. Pixel-by-pixel comparison when images have different dimensions
  2. Missing content detection at page boundaries

Questions for Maintainers

  1. How does BackstopJS handle pixel comparison when images have different heights?
  2. Is this expected behavior when requireSameDimensions: false?

Expected Fix/Enhancement

BackstopJS should detect missing UI elements as visual regressions regardless of dimension differences when misMatchThreshold is set to 0. The comparison algorithm should account for missing buttons and other UI elements, flagging them appropriately.

Lenguaje dominante
JavaScript
Estrellas
7.2k
Forks
617
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de garris/BackstopJS

Todos los issues de garris/BackstopJS

Issues similares

Más issues de JavaScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.