Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Feedback requested: Configuration support


#1

Hello,

based on the feedback we’ve received so far on this forum, by email and from customers we visited on-site, we are looking into introducing a new feature (set) that we originally planned to add in a later version, namely configuration and test plan support. Please note that we are also looking at other new features, but we want to get more feedback about this particular feature in this thread.

This feature would basically allow you to easily start multiple test runs at once and group the resulting runs in a test plan. We would also add support for configurations, while a configuration can be any variation a project needs to be tested against (different operating systems, browser versions, databases etc).

When you select to test a suite against specific configurations, TestRail would automatically create a run for each configuration combination you specified. To make it easier to have a standard test plan for your projects, we would also add a feature to easily save and load test plan templates. From our view, adding test plan and configuration support would help with the following issues:

[list=]
[
]Making it really easy to start multiple related test runs at once[/]
[
]Specify configurations to start multiple test runs for configuration combinations. We expect this to be a very common task/situation[/]
[
]Providing a way to save/load complex test plans from templates. If you regularly want to start runs for multiple suites (e.g., your full project release test), having a way to do this with just a few clicks is important[/]
[
]Because we expect users to have many test runs active at the same time for bigger projects, the resulting runs of a test plan would be grouped together on the overview pages to make them easier to comprehend[/*]
[/list]

Before we consider implementing this feature, we want to gather a bit more feedback about this and ask for your comments and ideas. To make it easier for you to understand how we would implement such a feature, we’ve created the following detailed mockups.

One of our design goals of this feature is that it’s completely optional. If you just want to start single test runs, you can ignore this feature and don’t have to create any test plans. If, however, you have some more complex requirements, you can benefit from this in addition to the existing functionality.

Mockups
The following mockup shows the modified Test Run overview page. In addition to listing single test runs, this page would also include all active test plans and provide a way to add new ones:


Click here for full version

And the following page shows the Add Test Plan page, where you can create a new test plan and start multiple test runs. You can add test suites to your test plan, configure all run settings (assignment, name, cases to include) and assign configurations:


Click here for full version

We welcome any ideas, comments and questions about the mockups and feature idea. It would also be great if you could tell us if and how this feature would help you. Thanks!

Regards,
Dennis


#2

Yes, this is something that will definitely help the usability of the product. I look forward to trying out this feature when it becomes available. Do you plan to do this for 1.0?


#3

I like this. One question about your second bullet point:

“Specify configurations to start multiple test runs for configuration combinations. We expect this to be a very common task/situation”

Can you provide more details about what you intend here? We specifically gave feedback around this item so I’m curious about your thoughts on implementation. Here are a few more details regarding a feature like this that we could really use:

  1. When we kick off a test run, it would be nice to specify a set of configurations that test run should be executed under. For example, let’s say my test run comprised three test cases:
    a. Open File
    b. Delete File
    c. Read/Write File

I would like the ability to create a test plan under which I could specify configurations that those three should be executed:

  • Windows 2000
  • RHEL 5 Update 3
  • CENTOS 5.3
    etc. etc.

This would “explode” the number of test cases in the test run to n testcases times n configurations, or in this example, 9 instances of a test case that need execution.

Here is why I believe this feature would be valuable. Not only would it help web-based application testing because you could specify different browser configurations to run the same test cases on, but also compatibility testing where you need to test an application against many platforms or operating system configurations. Likewise, in our case, we have an application that has many, many configuration options, and we want to run the same “test case” under those different configuration options without having to duplicate the test over and over again for them.


#4

Can,

Good to hear that you would find this feature useful. Yes, we currently plan to have this in the 1.0 and also plan to introduce it in one of the next beta releases.

Regards,
Dennis


#5

Hi Andrew,

We plan to implement the feature exactly as you described and just like you we believe this to be a very useful feature to test a product against different platforms/web browsers/databases etc.

So if you specify multiple configurations for a test suite, there will be multiple resulting runs, one for each configuration combination. We are looking forward to get this feature released in one of the next beta releases so everyone can take a look at the implementation.

Regards,
Dennis