Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Test Setup Time


#1

How can we include test setup time in our total time to run a test suite, but not include setup time in each test case.

Example: We have to run the installer first, then we can immediately do all 50 tests in a test suite in any order.

There are many approaches we have been trying but all have their drawbacks. I want to find out if there is a way to avoid these.

  1. Don’t include setup time in the estimates. This is bad because the time to complete is off when we have to run several tests that require a lot of setup.
  2. Include the setup time in the first test in the test suite. This is bad because sometimes we don’t run the 1st test 1st and sometimes we don’t run it at all. In these cases, the setup time disappears.
  3. Allocate the setup time across multiple tests. This is bad because once the setup is done, the time to complete immediately goes down, but our tests don’t reflect this.
  4. Include a single test called “Setup” and put the setup time in there. This doesn’t work because I don’t want this “test” to be included in our pass/fail metrics.

Are there any ideas I am missing?

Thanks much!


#2

Hello Danielle,

Thanks for your posting. There’s currently no dedicated feature for tracking setup times unfortunately and I would recommend going with option 4 and include this as the first test/case in your suite. Regarding the pass/fail metrics: you could look into using a separate status for this or simply consider the setup test as a regular test and then mark the test as Passed as usual.

I’m also happy to add setup times as a regular feature to our todo/feature request list.

Regards,
Tobias