Repository metrics

Stars: (5,625 stars)
PR merge metrics: (Avg merge 10d 11h) (11 merged PRs in 30d)

Description

This is an umbrella issue to track progress making the Sphinx test suite reliable. That means:

No flaky tests.
Tests can run in parallel.
Tests are independent of each other.

Regarding the last item: test independence is important to ensure that each test is checking the behaviours it claims to, and does not inadvertently rely on the side-effects of some other test. Similarly, a test must not begin failing if any other test is run before it -- if it does, that could mean (but does not definitely mean) that the application's behaviour itself may vary based on the order in which code is evaluated.

It might seem like there are a large number of items here! However, please bear in mind that we have a total of more than 2000 individual pytest test cases that run.

Test flakiness

test_build_linkcheck.test_connect_to_selfsigned_fails (pr: #12166)
test_build_linkcheck.test_defaults (pr: #12166)
test_build_linkcheck.test_linkcheck_exclude_documents (pr: #12189)
test_numfig_disabled_warn (pr: #14107)
test_n_slowest_value (pr: #14110)
... report another?

Test parallelization

tests.test_builders.test_build_linkcheck (pr: #12126)
... report another?

Test independence

tests.test_builders.test_build_text.py (pr: #12868)
tests.test_domains.test_domain_cpp (pr: #12187)
tests.test_builders.test_build_html (pr: #12118)
tests.test_builders.test_build_html_download (pr: #12119)
... report another?

Contributor guide

Research direction: Pick one of the open flaky tests (e.g., test numfig disabled warn), reproduce the flakiness locally by running the test multiple times, and identify the root cause. Then propose a fix that makes the test deterministic and independent.
Tech stack: python
Domain: testing
Issue type: Test
Difficulty: 3
Estimated time: 1-2 days
Activity status: Active
Clarity: Clear
Prerequisites: Pythonpytest
Newbie friendliness: 40