Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Difference between test plan and test run


#1

It is not clear for me what is the real difference between test plan and test run. I would expect test plan to include multiple test runs - but I could not achieve it.

I will be grateful for clarifications.


#2

Test plans are indeed basically a collection/group of related test runs. When you create a new test plan, you can select the test suites on the right sidebar to include (by clicking on the icon right to a test suite). So, for example, when you include three test suites in your test plan, this will effectively result in a test plan with three test runs.

When you view a test plan, you can see the individual test runs and the aggregated statistics of all test runs.

Test plans get especially interesting if you need support for so called configurations (for example, different operating systems, browser versions etc). You can specify multiple configurations for a suite in a plan and TestRail automatically creates a test run for each configuration combination.

I know that the documentation is currently not explaining any of this and we will have a more complete documentation when TestRail launches.

Regards,
Tobias


#3

Thank you for explanations. It works. The only missing thing I see is that the report for the test plan displays only progress bar per run, without the textual data, e.g. 123 tests passed, 234 untested, etc.


#4

Yes, the smaller overview (with the 32px icons instead of the large icons) used in a test plan currently does not include this information. However, you can hover over the bar chart/progress bar to see this details. We will consider adding this to the textual description as well.

Kind regards,
Tobias


#5

Hi there!

Does Test Plan give some benefit in case the same Test Run (or the same type Test Run) is repeated many times? For example to having one Test Plan for nightly build testing and then relate the repeated Nightly Build Test Run to the Test Plan? Or create a new test run for each nightly build and relate all them to the same test plan?

Regards
jarnoe


#6

Hi Jarnoe,

Yes, it can make sense to group related test runs under the same test plan. For example, this makes it very easy to compare test runs over time with the Comparison for Cases report from the Reports tab. You would just need to select the test plan as filter for the runs and this would generate a side-by-side comparison for all test runs of the plan. Using a single plan would also show the aggregate statistics as well as progress/projected completed date for all test runs when you view the plan.

We would recommend limiting the number of runs in a plan and retiring/purging old test plans at some point and starting a new test plan every few weeks. For example, it can make sense to group all runs of the same month in the same plan and then purge old test plans from time to time.

Cheers,
Tobias


#7

Hi, I also have a question concerning test plans in single repository mode vs multiple suite project mode.

  1. In the first case (single repository mode which we want to start with), when I add a test plan where I want to include 2 already saved test runs, when I click on “add test run” it automatically creates a new test run containing all the cases from my repository and I am forced to do the selection again - why am I not able to choose from the test runs that I have previously created?? It duplicates the test runs in my case.

  2. In multiple suite project, it’s the same, I create my test runs in the first place (test suite = test run), then I create a test plan where I want to add my recently created test runs but the same problem appears like in case 1 - I am forced to choose a test suite based on which it will create me another test run, a duplicate of what I already made.

Either I’m doing something wrong or it’s a flaw in the design. Thanks in advance!


#8

Hello,

Thank you for the post and additional questions. Test suites can be thought of single repositories in a sense. So you can have projects that have one large repository and you can have projects that have multiple suites (repositories). Keep in mind that when you use suites they are separate from each other. Meaning they have their own individual test runs.

In the case of a single repository, if you create a new test plan, you can click the ‘Rerun test run’ button to the right hand side. This will allow you to select any closed test run you previously completed and include it in this new test plan. The same would apply for test suites however you would only see test runs that were completed in relation to that specific test suite.

Does this help? Please let me know if you have additional questions regarding this.