Report the retry budget in assertion failure messages

Abierto
  • ¿Algo no está bien?
#5,223 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
72/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
playwright, python
Área
testing

Línea de trabajo

Empieza en Browser/assertion_engine.py, leyendo with_assertion_polling y sus dos puntos donde se hace raise junto con las estadísticas DEBUG existentes. Revisa atest/test/08_Scope_Tests/scope_keywords.resource y las pruebas de aceptación del repositorio para evaluar la compatibilidad y la ubicación de los mensajes. Se considera terminado cuando los fallos de aserción reintentados exponen el presupuesto de reintentos, mientras que los fallos en el primer intento conservan sus mensajes existentes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

priority: medium

Is your feature request related to a problem? Please describe.

When an assertion keyword fails, the failure message says nothing about the fact
that the assertion was retried, or for how long. Get Element States id=not_here contains hidden
fails with:

Elements states '['detached']' (list) should contain '['hidden']' (list)

That message is identical whether the assertion failed instantly or was retried for a
full second before giving up. There is no hint that retrying happened, that it is
configurable, or that Set Retry Assertions For exists.

This is a discoverability problem more than a debugging one. Users who do not already
know about retry_assertions_for have no path to discovering it, because the one moment
they would care about it — an assertion that failed after quietly retrying — is exactly
the moment the library stays silent. Anecdotally, experienced Robot Framework users are
surprised the setting exists at all.

The information is already computed. Browser/assertion_engine.py logs it at DEBUG in the
finally block of with_assertion_polling:

Assertion polling statistics:
First element asserted in: 0.2534 seconds
Total tries: 4
Elapsed time in retries 0.7864 seconds

DEBUG is the wrong level for something a user needs in order to know a feature exists.

Describe the solution you'd like

Include the retry budget in the assertion failure message itself, at the two raise sites
in with_assertion_polling (Browser/assertion_engine.py).

Gate it on retries having actually happened (tries > 1). That is both the only case where
the information is useful and the thing that keeps the change small: an assertion that failed
on its first attempt, or one running with retry_assertions_for=0, keeps its message
byte-for-byte.

Describe alternatives you've considered

  • Leave it at DEBUG. Status quo. Does not solve discoverability — nobody turns on DEBUG
    before they know there is something to look for.
  • Document it harder. Does not reach the user at the moment they need it.
  • Always append, ungated. Larger compatibility surface for no extra benefit, since a
    non-retried assertion has no retry budget worth reporting.

Additional context

Compatibility. Appending to assertion messages is a breaking change for
Run Keyword And Expect Error patterns that match exactly. Measured in this repository's
own acceptance tests:

Run Keyword And Expect Error sites 225
pattern uses a glob / regexp 141
exact match 84

The tries > 1 gate narrows the affected set considerably — the exact-match sites are
mostly non-retrying errors (Error: Crash, plugin errors) rather than assertion-keyword
failures. The residual should be measured before implementing. The same consideration
applies to users' suites, which is why this needs release notes.

Design question — where the information goes. The blast radius depends on placement, and
that is worth deciding deliberately rather than defaulting to "append to the end":

  • appended as a trailing sentence — simplest, but touches the message that exact matches see;
  • a separate WARN/INFO log line beside the failure — zero message compatibility cost,
    but easier for a user to miss;
  • attached only when the assertion exhausted its budget (as opposed to any retry at all).

Secondary benefit. retry_assertions_for is currently the one scope setting whose effect
cannot be verified except with a stopwatch. atest/test/08_Scope_Tests measures wall-clock
duration to infer it, and that is the source of roughly 20 CI failures across the last ~1200
runs. Every other scope setting can be verified from a message — Playwright names the timeout
it used (Timeout 1500ms exceeded), and strict mode raises strict mode violation. Naming
the retry budget in the failure would let those tests assert on a message too, and drop the
last timing-based assertion in that suite.

Related: Browser/assertion_engine.py, atest/test/08_Scope_Tests/scope_keywords.resource.

Lenguaje dominante
Python
Estrellas
662
Forks
149
Merge medio
6 h 9 min
PR fusionados (30 d)
119

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de MarketSquare/robotframework-browser

Todos los issues de MarketSquare/robotframework-browser

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.