Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Configurations?


#1

In the documentation, there are references to “configurations”, that you can select to run multiple test suites against. However I can’t find any documentation on how to create a configuration, nor where to find it, as it doesn’t seem to show up in the same place as described in the documentation (on the suite screen).

Any tips on how I can create / use configurations?

Thank you


#2

Hello Chris,

Thanks for your posting. Configurations can be found on the test plan pages (Test Runs & Results > Add Test Plan). Test plans are basically a collection of test runs with its own consolidated status and can contain one or more test runs:

When you add a test suite to a test plan, you can use the Select Configurations link to add and select configurations. When you then add the test plan, TestRail automatically creates test runs for each configuration combination.

For example, if you have the configuration groups ‘Browser’ and ‘OS’ and pick the configurations IE, Chrome, Firefox as well as Windows 7, Windows 8, TestRail creates the following test runs:

IE, Windows 7 IE, Windows 8 Chrome, Windows 7 Chrome, Windows 8 Firefox, Windows 7 Firefox, Windows 8

I hope this helps. Just let me know in case you have any further questions.

Regards,
Tobias


#3

This was not how I expected configurations to work and isn’t useful at all. Imagine you don’t test only on Windows, so you create a group for Windows, Linux, Mac. Then you want to have different configurations for each group (like IE, Chrome, Firefox, Safari; or browsers with device below that like Android, iPhone, etc)

Example:
Windows Group:

  • IE
  • Chrome
  • Firefox
    Linux Group:
  • Chrome
  • Firefox
    Mac Group:
  • Safari
  • Chrome
  • Firefox

Let’s say I want to run Mac/Safari, Windows/IE, and Linux/Chrome in today’s test pass. So I check those boxes.

It creates 9 test runs for:

  • Mac for each of: Safari, Chrome (which I didn’t ask for), IE (which just doesn’t make sense)
  • Windows for each of: IE, Safari (Doesn’t make sense), Chrome (makes sense but I didn’t intend to run this)
  • Linux, similar to above.

Even if I follow your scheme, let’s make an OS group (with Windows, Linux, Mac) and a Browsers group (with IE, Chrome, Firefox, Safari). This still creates 12 total combinations many of which are just crazy.

I’d rather have a hierarchy like test cases with no automatic combinatorial matrix (and maybe just quick copy function) and be able to manage these globally and just have:
Desktop group:

  • Windows group:
  • IE11
  • IE10
  • IE9
  • Firefox
  • Chrome
  • Linux group:
  • Chrome
  • Firefox
    Phones group:
  • iPhone group:
  • Safari
  • Chrome
  • Android group:
  • Chrome
  • Stock Android browser
    Tablets group:
  • iPad group:
  • Safari

Or just give me the matrix and allow me to select what I want.

Combinatorial testing is great and all but there are way too many forms of it for you to automatically decide which combinations make sense.


#4

Hello Christopher,

Thanks for your email. TestRail generates a full matrix with all possible combinations and you can then pick the configurations you want to use:

(please see the disabled IE on OS X combination, for example)

We recommend using a Operating System and Browser configuration group. You can then simply check all configurations and limit the matrix to the combinations that make sense in your scenario.

I hope this helps!

Regards,
Tobias