How to set test timeout dynamically ?

Open
#443 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
cpp
Domain
testing-qa

Research direction

Start with the Boost.Test timeout documentation linked in the issue and compare the compile-time, runtime-variable, and command-line examples. Trace when BOOST_AUTO_TEST_CASE(timeout(get_timeout())) is evaluated versus when master_test_suite().argc and argv are initialized. Done means identifying a supported way to switch the timeout dynamically, or documenting the limitation and an appropriate alternative.

Written by the indexing model from the issue text.

Description

I am using Boost.Test version 1.87.0.
I am considering setting a uniform timeout for test cases. The purpose is to terminate a test case if it gets stuck due to some issue and proceed to the next test case.

However, I want to disable this timeout when debugging with a debugger such as gdb, as timeouts during debugging can interfere with the process.

According to the documentation:
https://www.boost.org/doc/libs/1_87_0/libs/test/doc/html/boost_test/testing_tools/timeout.html
it appears that timeouts can be specified using the described method.

First, I specified a timeout using a constant, following the example in the documentation.

This resulted in the expected timeout.
However, to disable the timeout for debugging, recompilation is required.

Next, I attempted to specify a variable instead of a compile-time constant.

This also resulted in the expected timeout.

To dynamically switch the timeout on or off and adjust its duration, I considered passing it as a command-line argument.

inline unsigned long get_timeout() {
    std::cout << "In get_timeout(), argc:" << boost::unit_test::framework::master_test_suite().argc << std::endl;
    if (boost::unit_test::framework::master_test_suite().argc > 1) {
        return
            std::stoul(
                boost::unit_test::framework::master_test_suite().argv[1]
            );
    }
    return 0;
}

However, in this case, the timeout did not take effect.

Upon analysis, it appears that:

BOOST_AUTO_TEST_CASE(t1, * boost::unit_test::timeout(get_timeout())) {

calls get_timeout(), but at this point,
boost::unit_test::framework::master_test_suite().argc and
boost::unit_test::framework::master_test_suite().argv
have not yet been initialized.

To confirm this, I added the following code inside the test case:

auto tout = get_timeout();
std::cout 
    << "In test case argc: "
    << boost::unit_test::framework::master_test_suite().argc 
    << " result of get_timeout(): "
    << tout
    << std::endl;

At this timing, argc and argv were correctly retrieved.

Is there a good way to dynamically switch the timeout?

Dominant language
C++
Stars
213
Forks
149
Avg merge
18h 17m
Merged PRs (30d)
2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from boostorg/test

All issues in boostorg/test

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.