Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Unique configurations for sections and subsections


#1

Hi, I’m new to Testrail and I’m liking it so far. My question is specific to Configurations for Test Suites/Test Cases within a Test Plan. Lets say I have a Test Suite (TS1) with two sections under it, S1 and S2 and test cases under them:
TS1

  • S1
    • TC1
    • TC2
  • S2
    • TC3
    • TC4

I then have the following Configuration Groups:
Operating Systems

  • Windows
  • OS X

Browser

  • Firefox
  • IE

I would like the configuration groups to be unique to each Section, S1 and S2. For example, I would like “Operating Systems” to apply only to S1 and “Browser” to apply only to S2. So I can have this as my test plan:
Windows, TC1, TC2
OS X, TC1, TC2
Firefox, TC3, TC4
IE, TC3, TC4

But right now, if I select all configuration options, this is what I get:
Windows, Firefox, TC1, TC2, TC3, TC4
Windows, IE, TC1, TC2, TC3, TC4
OS X, Firefox, TC1, TC2, TC3, TC4
IE, IE, TC1, TC2, TC3, TC4

The only workaround I can think of for now is to make the sections (S1 and S2) as test suites. However, that forces me to have too many Test suites. Is there any other way to do what I need?

Thanks in advance for the help.


#2

Anybody? Even a “No, it can’t be done” or asking for further clarity on my question?


#3

Hi Sriram,

Thanks for your posting! Sure, happy to help.

If you want to test against each configuration separately I would recommend storing them in the same configuration group (e.g. called “Platform” or “Environment”). Using multiple configuration groups will create a configuration matrix with all possible configurations instead (Windows + Firefox, Windows + IE, etc.).

You can configure the case selection per configuration/run once you selected your configurations via the “select” link as part of the Test Cases column.

To make the selection as easy as possible by using filters, I can recommend adding a custom field on the test case level with similar values as your configurations. For example, you can look into a custom field of type “multi-select” and then use this field as filter on the Select Cases dialog.

http://docs.gurock.com/testrail-userguide/howto-fields

I hope this helps and I look forward to your reply.

Cheers,
Tobias


#4

Thanks Tobias. I may follow your suggestion about using custom fields. But I would like to explain my situation a little bit more so you can tell me what would be the best way to use TestRail.

The real issue here is that I have a single test case that has to be run against a huge amout of permutations and combinations of configurations. In other words, a single test case has to be run under permutations of 5 configurations. So the number of times that single test case is run can be in the thousands.

For example, I have the following configuration groups:
G1 with values 1.1, 1.2, 1.3, 1.4, 1.5
G2 with values 2.1, 2.2, 2.3, 2.4
G3 with values 3.1, 3.2, 3.3
G4 with values 4.1, 4.2
G5 with values 5.1, 5.2, 5.3

And I have 1 test case TC1 that can be run as:
TC1 against 1.1, 2.1, 3.1, 4.1, 5.1
TC1 against 1.1, 2.1, 3.1, 4.1, 5.2
TC1 against 1.1, 2.1, 3.1, 4.1, 5.3
TC1 against 1.1, 2.1, 3.1, 4.2, 5.1
TC1 against 1.1, 2.1, 3.1, 4.2, 5.2

TC1 against 1.1, 2.1, 3.3, 4.2, 5.1
TC1 against 1.1, 2.1, 3.3, 4.2, 5.2
TC1 against 1.1, 2.1, 3.3, 4.2, 5.3
… and thousands more! (Thankfully they are automated tests!)

In this case, how would you suggest I organize these test cases in TestRail? Using Configurations Groups under Testruns and create thousands of test runs of a single test case or would you suggest creating a Test Suite and add the thousands of test cases using custom fields to specify the configurations?

Thanks!
Sriram


#5

Hi Sriram,

How many test cases do you roughly have that require so many configurations? From looking at the numbers, I would advise against using separate test runs & configurations in this case and you could alternatively look into using a custom field on the test result level and adding multiple results per test instead. This would scale much better and you wouldn’t need to create thousands of test runs.

Cheers,
Tobias


#6

Hi Tobias,

Thanks for your reply. I have around ten test cases which could potentially have 3000 thousand permutations of configurations EACH!

I understand your suggestion of adding a custom field at the test result level to capture the configurations. The issue with that, however, is that when I create a test plan, I can’t decide or capture ahead of time what configurations I want this test run against. The tester will have to remember to run say, 1200 configurations, run them and capture them in the test results. So the test plan would say that we have planned 1 test case but we’re actually running 1200 test cases as part of the test plan or test run.

I want to use this tool like its meant to be used (so I’m not caught off guard when it comes to running reports, reviewing results etc.). Currently, I’m leaning towards adding custom field(s) in the test case itself, then creating 3000 variations of each test case in a TestSuite (or sections). I will then choose the configurations I want to add into a particular test plan (using the filter function) and assign them to the appropriate tester(s). But I’m not sure if this is the best way to use TestRail. This is how I’m doing it in Testlink which is my current Test Management tool and one which I’m trying to migrate away from. Therefore, I much appreciate your advice on tackling this problem in the best way possible in Testrail.

Thanks,
Sriram


#7

Hi Sriram,

Thanks for the additional details, that helps a lot. Generating 3000 configuration combinations (which essentially creates 3000 test runs) is not really practical and structuring this on the test case level would likely make more sense. That is, adding a case per configuration combination and then simply creating a single test run with 3000 cases/tests (or a subset thereof). I would also recommend looking into using test suites as additional organizational unit and you could use one test suite per case (so 10 test suites in total, each with 3000 configurations/cases):

Given that these are automated tests, I would also recommend reusing the same test runs multiple times (e.g. within the same sprint or iteration) as closing test runs will copy the cases. This will result in a large database pretty quickly otherwise given your case numbers and presumably high number of test runs. How often do you run your automated tests?

Cheers,
Tobias


#8

Thanks Tobias, that makes sense. And good tip about the reusing the same test run within the same sprint!

I’m going the Test Suite route. Apart from these 10 test cases (which will be a test suite each) I have several other sets of test cases which I am going organize into 15 other test suites. But these 15 test suites don’t have the ridiculous amount of configurations like the 10 test cases. So I will have 10 “ridiculous” test suites and 15 other normal test suites for a total of 25 test suites. This will all be part of a test plan with some of the test suites only using a subset of the test cases.

How many is too many test suites in a project? Is there a number when it can start to cause issues in TestRail?

Sorry for so many questions. I want to make sure I can get the most out of TestRail, the right way.

Thanks,
Sriram


#9

Hi Sriram,

Thanks for the update, sounds good! The number of cases and tests/results over time are more important than the number of suites.

It’s worth mentioning that a high number of automated runs can naturally result in a large® database. While TestRail was designed with this in mind, for a high number of automated runs + results it can make sense to implement some kind of auto-purging for old test runs to keep the database at a manageable level. For example, you can look into automatically deleting test runs that are older than a certain threshold (e.g. 4 weeks) as the value of these old runs decreases with time anyway in our experience. To do this, you can use the get_runs/get_plans (with a created_on filter) and the delete_plan/delete_run API method.

Cheers,
Tobias