How can we include test setup time in our total time to run a test suite, but not include setup time in each test case.
Example: We have to run the installer first, then we can immediately do all 50 tests in a test suite in any order.
There are many approaches we have been trying but all have their drawbacks. I want to find out if there is a way to avoid these.
- Don’t include setup time in the estimates. This is bad because the time to complete is off when we have to run several tests that require a lot of setup.
- Include the setup time in the first test in the test suite. This is bad because sometimes we don’t run the 1st test 1st and sometimes we don’t run it at all. In these cases, the setup time disappears.
- Allocate the setup time across multiple tests. This is bad because once the setup is done, the time to complete immediately goes down, but our tests don’t reflect this.
- Include a single test called “Setup” and put the setup time in there. This doesn’t work because I don’t want this “test” to be included in our pass/fail metrics.
Are there any ideas I am missing?