Tests Deadliness and Higher Order Mutations

Abierto
  • ¿Algo no está bien?
#174 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
15/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
python
Área
cli, testing-qa

Línea de trabajo

Empieza leyendo el comportamiento existente de mutmut results y el manejo de las opciones de la CLI; después, revisa la discusión relacionada en el issue 113. El issue no menciona archivos ni tests; para darlo por terminado harían falta un alcance definido y un comportamiento documentado para la letalidad de los tests y las mutaciones de orden superior acotadas, incluido --max_order.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Hey boxed and mutmut contributors,
thanks for this great library! It really helps in developing tests, which test the real corner cases!

I was wondering, whether it would be possible to implement two new features, namely test deadliness and higher order mutations.

Test Deadliness

As far as I am aware, mutmut does not report the number of tests which killed a mutant. The more tests fail after applying a mutant, the higher the deadliness for said mutant is. I think this would be interesting, as this reveals which locations are really critical for the correctness of the code base. It also reveals which locations are well tested and which are not.
As an example, a surviving mutant has a test deadliness of 0, a mutant killed by a single failed test has a test deadliness of 1 and so on.
Maybe this could be normalized by the number of tests to account for small or large test bases.

Reporting an aggregate measure (average, median) or the histogram of test deadliness over all mutants reveals whether

  • there are too few tests (aggregate test deadliness close to 0),
  • there are just a few independent tests, which address individual lines (aggregate test deadliness around 1) or
  • there are also very sensitive tests, which test corner cases of higher level functions (aggregate test deadliness above 1).

As this should be quite easy to obtain I would propose just outputting the results when calling mutmut results.

Higher Order Mutations

Mutmut currently only applies a single mutation before running tests. Combining these mutations (two mutations -> second order, three mutations -> third order, an so on) would allow to evaluate the test robustness in a more extreme setting (edit: I noticed, that you mentioned this idea already in https://github.com/boxed/mutmut/issues/113#issuecomment-484938645). I would argue that this setting (for reasonable low orders) is still quite realistic, as the example below shows:

  • for i in range(0,5): ... the reference code
  • for i in range(1,5): ... a normal first order mutant, with changed iteration start and thus the iteration length
  • for i in range(0,6): ... a normal first order mutant, with changed iteration end and thus the iteration length
  • for i in range(1,6): ... a second order mutant, with changed iteration start and end, but not the iteration length

Another example would be an if A and B or (C and not D): ... statement with multiple logical expressions A, B, C and D. A single mutant will likely change the behavior, but multiple mutations could partially restore parts of the logical behavior and thus have tests wrongly pass.

Obviously this does not scale well for high orders if the number of mutants N is large, I think there are potentially N*(N-1)/2 higher order mutants, which would need to be tested. But within all potential mutant combinations there are some, which are more interesting and more likely to survive. Those combinations are most likely those, of which the independent mutant weren't able to kill (m)any tests.
Thus, building on the test deadliness proposed above, one could order or limit the higher order mutations by the test deadliness of their individual tests. Afterall two (or more) mutations, which were not killed, might also survive in conjunction.

Note, that test deadliness should also be measured for these higher order mutants, which allows us to do the following:
For example, if mutations A and B have td=0, testing AB would be interesting. On the other hand if mutations A and B have td>>0, testing AB would not be interesting.
If if mutations A ,B and C have td=0, the second order mutations AB, AC and CB can be tested. Furthermore the third order mutation ABC could be tested, and this would be determined by the test deadliness of he second order mutations AB, AC and CB.

To use this feature, I would propose a new CLI argument --max_order, which defaults to 1 and thus defaults to the current mutmut behavior. A usage like --max_order=2 would additionally test all possible pair combinations of the mutants. Low digit values of --max_order should reveal the most interesting combinations, as more and more concurrent mutations increase the probability of a test or simply a execution failure.
Something like --td_limit_higher_order could be used to provide a test deadliness limit to skip mutant combinations, where the individuals already have a high average test deadliness.

I would be happy to hear your or any of the mutmut contributors opinion on the usefulness of these two features!

Lenguaje dominante
Python
Estrellas
1.5k
Forks
190
Merge medio
1 d 11 h
PR fusionados (30 d)
8

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de boxed/mutmut

Todos los issues de boxed/mutmut

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.