What to do about "sometimes failing" tests?

Open
#3,820 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
cmake, cpp
Domain
devtools, testing-qa

Research direction

Start by running the listed tests through ctest with multiple jobs and rerunning failures with a single job. Determine whether the failures are reproducible or intermittent, then establish whether the test handling or the wiki guidance should change.

Written by the indexing model from the issue text.

Description

In the last testruns I got the following tests failing but passing on rerun (sometimes the n-th time):

eventfd
async_signal_syscalls2-32
check_lost_interrupts
thread_stress
thread_stress-32
checkpoint_mmap_shared
condvar_stress-no-syscallbuf
mmap_adjacent_to_rr_usage
mutex_pi_stress
simple_threads_stress-32

I've got similar issues (also with other tests) before on several machines with different CPUs.
What I've recognized is that this is much more likely to happen when I run ctest with multiple jobs (more jobs, more likely).

Question: Should those failures just be ignored (and the wiki adjusted to always rerun the failed tests with a single job)?
If not: how to analyze this best?

Dominant language
C++
Stars
10.7k
Forks
662
Avg merge
2d 3h
Merged PRs (30d)
2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from rr-debugger/rr

All issues in rr-debugger/rr

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.