avajs/ava

Randomize test runs

Open

#595 opened on Mar 3, 2016

View on GitHub
 (10 comments) (3 reactions) (0 assignees)JavaScript (20,600 stars) (1,408 forks)batch import
:dollar: Funded on Issuehuntenhancementhelp wantedscope:scheduling

Description

We should kick off tests in a random order. This will flush out any interdependencies per test. However the order should be reproducible to debug such interdependencies. Ruby's minitest has a --seed option which can be set on the CLI to control the order. A random seed is used if no value is provided and it's reported in the output.

Starting tests in a random order within a test file is the easiest place to start. Presumably .serial would run in source order.

We could consider running test files in a random order as well, though they're already somewhat random because each test file runs in its own process, and because each test file runs in its own process it'd be hard to run them in a deterministic order. So probably best not to even try.

Thoughts?

Backers (Total: $80.00)

Submitted pull Requests

Become a backer now!

Or submit a pull request to get the deposits!

Tips

Contributor guide

Tech stack
javascriptnodejs
Domain
testing
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaScriptNode.jsAVA basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
The issue proposes randomizing test order with a reproducible seed, similar to Ruby's minitest. Examine the existing test runner code in AVA to understand how tests are currently executed. The linked PR #3442 may provide an implementation approach. Review the comments in the issue for additional considerations, such as the need for deterministic ordering for `.serial` tests.