Test improvements

Abierto
  • ¿Algo no está bien?
#1,697 4 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
20/100
Tipo de issue
Refactorización
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
c, erlang
Área
testing, tooling

Línea de trabajo

Read the example tests and README.md mentioned in the draft, then compare them with the tests in erlang_tests/ and libs/estdlib/. The issue proposes broad changes to test helpers, naming, organization, summaries, OTP compatibility, Valgrind, and flaky CI steps, so first identify which part remains relevant. Done means an agreed scope and corresponding test and CI changes; the issue gives no specific files or acceptance criteria.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

I was working with tests for some time now and I have few observations:

  • Tests are split to NIFs and Erlang impl.
  • Running tests in BEAM works only from one directory (-pa .).
  • There are a lot of files in erlang_tests/. They're unorganized and few create a series that could be done in one module.
  • Some older tests use magic values for communicating intent, some new have own helpers that are copied around, some are hard to follow.

It should undergo some cleaning.

I'd propose that:

  1. tests should depend on set of common helpers. Some superset of macros in etest would be nice. Having both homebrew framework and etest is quite messy.
  2. we should decide on naming scheme, right now this is the mix of test_name and name.
    • I like prefixed names with category – e.g. nif_binary_match, gc_tree_ast, bin_mqqt, bif_pow, etc.
  3. estdlib should be treated specially and tests from erlang_tests and libs/estdlib should be merged.
  4. atom table shouldn't use atoms defined in default_atoms.def. They're always out of sync and it doesn't bring any value.
  5. TEST_CASE_EXPECTED should be removed prefering TEST_CASE.
  6. Test summary should be aggregated at the bottom, in format of "Test executed: {k}, tests skipped: {k}, tests failed: {k}\nFailed:\n{list of test modules}".
  7. clear designation when tests are available on later versions of BEAM. Would be nice to have macros that compile testing functions conditionally, like ok = ?RUN_WITH_MIN_OTP(24, binary_match_test_fun).
  8. Run Valgrind with full report (--leak-check=full). Right now it only gives a report about leaked memory and you need to run it yourself to see what leaked.
  9. Move known flaky tests (tcp, udp, etc) to separate steps or disable them completely until they're fixed. They very often fail CI and it's hard to see at glance if check failed due to flaky tests or changes to the code.

Draft of what it could look like (or any variation on that). Look in example tests and README.md.

Related: #1060

Lenguaje dominante
Erlang
Estrellas
2.3k
Forks
171
Merge medio
5 d 11 h
PR fusionados (30 d)
19

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de atomvm/AtomVM

Todos los issues de atomvm/AtomVM

Issues similares

Más issues de Testing & QA

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.