Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

When best to use multiple test suites on a project?


#1

Hi there, we are getting busy with creating various types of tests on a project; e.g. UI automation, service-level tests, manual tests etc.

Our thought right now is that each type of testing would suit its own test suite, as the suite ID will be a helpful way of keeping service tests separate from UI automation tests etc. But I’ve seen various threads from this community lamenting the choice of using multiple suites on a project… so when _is _ the right time to use multiple suites?!

Looking forward to the replies from everybody :slight_smile:


#2

One thing I don’t like about having the Multiple Test Suite option for a project is Test Runs are separated by the Test Suite. I can’t create a test run and select test cases across the Test Suites. I want them separated by Test Suite, but I don’t want the test runs to be separated by that. It still works, but feels like an additional thing we don’t need.


#3

One tipping point is when you find that Test Plans are more useful than mere Runs. Obviously this is impacted by the dev/test methodology used and how big the system under test is. An old school way of looking at Cases, Suites, Runs, and Plans is a sheet of paper, tab separated sheets you may wish to execute (or not), duplicated test results sheets or collection of sheets, wholly duplicated binders of test results.

When you find your testing questions focusing around “Do we need to execute these tests this time around?” or “Do we have tests that test ‘x’?” That’s the time to think about suites and plans rather than mere cases and runs. Obviously care must be taken to organize suites that make sense from the point of opting to test something in detail. If you are just doing Smoke and Acceptance… create separate suites for that. Reserve the fine grain of suites for detailed regression testing that perhaps you do only once a release. Or if you have the option, automate your regression suite testing and reporting.

Automation and types? To date I’ve not found a compelling reason to put automated test results in TestRail… usually the automation itself winds up reporting results independently of TestRail. I reserve TestRail for help in organizing those tests that don’t lend themselves readily to automation. Of course, if you find TestRail dashboard essential to making release decisions and need ALL testing results reported there, then you have a different problem to solve.