Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Configuration groupings


#1

Hi all,

Is there a way to change the groups of configurations that are created after selecting available configurations?

For example lets say I have the following configurations available:
Config Group A: OS1, OS2, OS3, OS4, OS5
Config Group B: db1, db2, db3, db4, db5
Config Group C: Net1, Net2, Net3, Net4, Net5

I select the following configurations:
Group A: OS1, OS3, OS5
Group B: db1, db3, db5
Group C: Net1, Net3, Net5

This results in 27 runs, one for each unique combination of the 9 configuration options I selected. What I would like is only 3 runs:

Run1: OS1, db1, Net1
Run2: OS3, db3, Net3
Run3: OS5, db5, Net5

I see a few work arounds (none of which seem very good):

  1. I could simply find and execute the 3 runs I want leaving the remaining 24 runs untouched. I would think this would make reporting and status very ugly to interpret.

  2. I could create configurations in model of:
    Group A: OS1\db1\Net1, OS2\db2\Net2, OS3\db3\Net3 …
    I see this approach as getting kinda messy and hard to maintain. For example, lets say you want to mix up those combinations between RC1 and RC2 … then we either edit them all or have 6 vs 3.

Any suggestions? If not, is this a possible feature enhancement?


#2

Hello Joel,

There is currently no way to change the configuration combinations. They are always “multiplied” in the sense that a test run is created for each possible configuration combination. This isn’t the optimal solution in all cases and changing/improving this behavior is already on our feature request and TODO list.

I would suggest going with workaround 2 in the meantime. In fact, we are using workaround 2 for testing TestRail itself. We are testing TestRail against multiple OS, database and PHP versions and not all combinations make sense (Linux + SQL Server, e.g.) or are uncommon combinations (Windows with a very old PHP version, e.g.), so we just added those combinations that we really need to test:

Ubuntu 6 / MySQL / PHP 5.1 Ubuntu 8 / MySQL / PHP 5.2 Ubuntu 10 / MySQL / PHP 5.3 Windows 2003 / SQL Server 2005 / PHP 5.2 ...

I understand that a way to select the actual configuration combinations would be better and we are looking into this for a future version. Thanks for the feedback!

Regards,
Tobias


#3

Hi Tobias,
Thanks for the quick response! I think we will use option 2 for now and roll with it.

I have another suggestion along the same lines. It would be great if we could select the test cases on a per test run/configuration basis as opposed to the entire test suite. Using my example from the previous post, this is what I mean:

I select the following configurations to be applied to my tests suite 1: (which has 15 test cases)

Config 1: OS1\db1\Net1
Config 2: OS2\db2\Net2
Config 3: OS3\db3\Net3

This results in 3 test runs, I would like to be able to select which of the 15 test cases should be executed in each test run.

For example, configuration 1 & 2 (used in test run 1 & 2) are common use cases which I want to run against all test cases in the suite while configuration 3 (used in test run 3) is a fringe use case that I only want to run against the priority 5 test cases therefor excluding 10 of the 15 test cases in the suite. Does that make sense?

Thanks,
Joel


#4

Hello Joel,

To implement this, you could just add the test suite multiple times to your test plan and then select the test cases individually for each test suite.

I would recommend adding the test suite twice in this case. The first test suite would have all test cases included with configuration 1 & 2. The second test suite could exclude 10 of the test cases and use configuration 3.

Hope this helps!

Regards,
Tobias


#5

Hi Tobias,
I didn’t think of doing that, great idea! That will work.

Thanks,
Joel