Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Configurations are not very friendly and don't meet our needs


#1

We are having a hard time using configurations for the following reasons:

Let’s say we have 50 test cases in a Suite divided into subsections of the Suite. Now we have three of the 50 test cases that need to run in about 20 different configurations. The problem here is there is no default “None” configuration which I think is the main problem. We would have to have a “None” option for every existing Configuration Group and then the permutations race out of control to make this feasible.

The only viable workaround we found was to create Suites of one test case and then apply the twenty configurations to each test case. It is counterintuitive to do this and not the way we want to organize our tests but we are forced to due to limitations on how configurations were designed.

There should be:

  1. More than one giant group of configurations, there should be many groups to select from
  2. There should be a built-in “None” option that does not cause permutation explosion
  3. Configurations cannot be applied at the subsection level of a Suite, this makes subsections not a very powerful organizational item

I’m happy to get on a Webex or Zoom if TestRail folks want to see the problems for themselves.


#2

I am having the exact same problem!


#3

Hi Michael,

Thanks for your posting. If there’s a strict separation between regular and configuration-dependent cases, then it can make sense to use separate test suites for each group. This would make it much easier to create regular runs and also set up the configurations/runs for the configuration-dependent cases.

If the configurations differ per test case, we can recommend adding a custom field on the case level that describes the configurations per case and which can be used on the test plan form as the basis for the case selection (via the filters on the Select Cases dialog). This approach can also make sense if you are using a single, combined suite and you wouldn’t necessarily need to include all cases in every configuration/run.

I hope this helps!

Cheers,
Tobias


#4

Thanks Tobias but it doesn’t make organizational sense for me to break things into separate suites simply because configurations are so limited in their applicability/design. Is there going to be any upcoming release of TestRail that allows for a better/richer usage of configurations or this is the design that is going to be carried forward? I do use plenty of custom fields but I don’t really want to use them in lieu of a proper configuration scheme built into TestRail.

I’m also disappointed that subsections are not a powerful organizational object/entity…this exacerbates the issue with configurations being so limited in the current implementation. I’ve made some suggestions on how to help correct this in my initial comment. Hopefully somebody that wants to improve this product will read those suggestions.