Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.

Configuration groupings


Hi all,

Is there a way to change the groups of configurations that are created after selecting available configurations?

For example lets say I have the following configurations available:
Config Group A: OS1, OS2, OS3, OS4, OS5
Config Group B: db1, db2, db3, db4, db5
Config Group C: Net1, Net2, Net3, Net4, Net5

I select the following configurations:
Group A: OS1, OS3, OS5
Group B: db1, db3, db5
Group C: Net1, Net3, Net5

This results in 27 runs, one for each unique combination of the 9 configuration options I selected. What I would like is only 3 runs:

Run1: OS1, db1, Net1
Run2: OS3, db3, Net3
Run3: OS5, db5, Net5

I see a few work arounds (none of which seem very good):

  1. I could simply find and execute the 3 runs I want leaving the remaining 24 runs untouched. I would think this would make reporting and status very ugly to interpret.

  2. I could create configurations in model of:
    Group A: OS1\db1\Net1, OS2\db2\Net2, OS3\db3\Net3 …
    I see this approach as getting kinda messy and hard to maintain. For example, lets say you want to mix up those combinations between RC1 and RC2 … then we either edit them all or have 6 vs 3.

Any suggestions? If not, is this a possible feature enhancement?


Hello Joel,

There is currently no way to change the configuration combinations. They are always “multiplied” in the sense that a test run is created for each possible configuration combination. This isn’t the optimal solution in all cases and changing/improving this behavior is already on our feature request and TODO list.

I would suggest going with workaround 2 in the meantime. In fact, we are using workaround 2 for testing TestRail itself. We are testing TestRail against multiple OS, database and PHP versions and not all combinations make sense (Linux + SQL Server, e.g.) or are uncommon combinations (Windows with a very old PHP version, e.g.), so we just added those combinations that we really need to test:

Ubuntu 6 / MySQL / PHP 5.1 Ubuntu 8 / MySQL / PHP 5.2 Ubuntu 10 / MySQL / PHP 5.3 Windows 2003 / SQL Server 2005 / PHP 5.2 ...

I understand that a way to select the actual configuration combinations would be better and we are looking into this for a future version. Thanks for the feedback!



Hi Tobias,
Thanks for the quick response! I think we will use option 2 for now and roll with it.

I have another suggestion along the same lines. It would be great if we could select the test cases on a per test run/configuration basis as opposed to the entire test suite. Using my example from the previous post, this is what I mean:

I select the following configurations to be applied to my tests suite 1: (which has 15 test cases)

Config 1: OS1\db1\Net1
Config 2: OS2\db2\Net2
Config 3: OS3\db3\Net3

This results in 3 test runs, I would like to be able to select which of the 15 test cases should be executed in each test run.

For example, configuration 1 & 2 (used in test run 1 & 2) are common use cases which I want to run against all test cases in the suite while configuration 3 (used in test run 3) is a fringe use case that I only want to run against the priority 5 test cases therefor excluding 10 of the 15 test cases in the suite. Does that make sense?



Hello Joel,

To implement this, you could just add the test suite multiple times to your test plan and then select the test cases individually for each test suite.

I would recommend adding the test suite twice in this case. The first test suite would have all test cases included with configuration 1 & 2. The second test suite could exclude 10 of the test cases and use configuration 3.

Hope this helps!



Hi Tobias,
I didn’t think of doing that, great idea! That will work.