Test improvements

Aperta
  • Qualcosa non va?
#1,697 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Refactoring
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
c, erlang
Ambito
testing, tooling

Direzione di ricerca

Read the example tests and README.md mentioned in the draft, then compare them with the tests in erlang_tests/ and libs/estdlib/. The issue proposes broad changes to test helpers, naming, organization, summaries, OTP compatibility, Valgrind, and flaky CI steps, so first identify which part remains relevant. Done means an agreed scope and corresponding test and CI changes; the issue gives no specific files or acceptance criteria.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

I was working with tests for some time now and I have few observations:

  • Tests are split to NIFs and Erlang impl.
  • Running tests in BEAM works only from one directory (-pa .).
  • There are a lot of files in erlang_tests/. They're unorganized and few create a series that could be done in one module.
  • Some older tests use magic values for communicating intent, some new have own helpers that are copied around, some are hard to follow.

It should undergo some cleaning.

I'd propose that:

  1. tests should depend on set of common helpers. Some superset of macros in etest would be nice. Having both homebrew framework and etest is quite messy.
  2. we should decide on naming scheme, right now this is the mix of test_name and name.
    • I like prefixed names with category – e.g. nif_binary_match, gc_tree_ast, bin_mqqt, bif_pow, etc.
  3. estdlib should be treated specially and tests from erlang_tests and libs/estdlib should be merged.
  4. atom table shouldn't use atoms defined in default_atoms.def. They're always out of sync and it doesn't bring any value.
  5. TEST_CASE_EXPECTED should be removed prefering TEST_CASE.
  6. Test summary should be aggregated at the bottom, in format of "Test executed: {k}, tests skipped: {k}, tests failed: {k}\nFailed:\n{list of test modules}".
  7. clear designation when tests are available on later versions of BEAM. Would be nice to have macros that compile testing functions conditionally, like ok = ?RUN_WITH_MIN_OTP(24, binary_match_test_fun).
  8. Run Valgrind with full report (--leak-check=full). Right now it only gives a report about leaked memory and you need to run it yourself to see what leaked.
  9. Move known flaky tests (tcp, udp, etc) to separate steps or disable them completely until they're fixed. They very often fail CI and it's hard to see at glance if check failed due to flaky tests or changes to the code.

Draft of what it could look like (or any variation on that). Look in example tests and README.md.

Related: #1060

Lingua principale
Erlang
Stelle
2.3k
Fork
171
Merge medio
5g 11h
PR unite (30g)
19

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di atomvm/AtomVM

Tutte le issue di atomvm/AtomVM

Issue simili

Altre issue su Testing & QA

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.