We are having a hard time using configurations for the following reasons:
Let’s say we have 50 test cases in a Suite divided into subsections of the Suite. Now we have three of the 50 test cases that need to run in about 20 different configurations. The problem here is there is no default “None” configuration which I think is the main problem. We would have to have a “None” option for every existing Configuration Group and then the permutations race out of control to make this feasible.
The only viable workaround we found was to create Suites of one test case and then apply the twenty configurations to each test case. It is counterintuitive to do this and not the way we want to organize our tests but we are forced to due to limitations on how configurations were designed.
There should be:
- More than one giant group of configurations, there should be many groups to select from
- There should be a built-in “None” option that does not cause permutation explosion
- Configurations cannot be applied at the subsection level of a Suite, this makes subsections not a very powerful organizational item
I’m happy to get on a Webex or Zoom if TestRail folks want to see the problems for themselves.