Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Test Results History


#1

Hi, I’m new to testrails and would like to know how you advise on setting up our first project (all our projects work in similair way as mentioned below):

A project has many tests, which can be created under one suite (use groups for management) or multiple suites. This project (or more like a feature) is delivered in specific final releases which we have the dates for (these are also the milestones). So lets say this year we have 2 release milestones - one called releaseA and its date is 06/17 and one called releaseB with 12/17 date.

For releaseA, we need to run X tests and make sure we have pass coverage > 95% by the miletone date. For releaseB, we need to run all the tests and have the same pass coverage by its milestone date. In addition, we get weekly releases where we run some or all of the tests and report the results. For the final releases, we don’t run all the tests again, we rely on what we achieved (coverage and results) from the weekly releases.

My first thought was to create a test plan for releaseA, and for each weekly release we run tests (until we reach releaseA milestone date) create a test run, but then I saw the test plan statistics are for overall tests (i.e. I have the same tests cases in multiple runs, so the test plan counts them twice as if I had double the tests).

So what I did is create one test run for releaseA, and for each weekly release we can update the test case results and provide the release info in the test result (can add cusom field like release version). However, in this approach, if at one point I would like to know which tests were executed for a specific weekly release and what were the results, there’s no way to get this info (I see if I click on a test case in a test run I get the results history but I would need to go over each test and search for the relevant result based on the weekly release).

Any suggestions on how to work with testrails with the above need?

Thanks,
Michal


#2

Hi Michal,

Thanks for your post! TestRail’s reports/statistics would use the most recent result for a test, so when you keep a test run open and add new results to each test, then it may be difficult to track down your previous results. I would recommend creating the Test Plans to group your test runs as you already tried, or just create separate test runs and label them accordingly to match them to your releases. From there, you can look into using the Comparison for Cases report, as you would be able to compare the test case results of each test run side-by-side, and this report also includes the Coverage column to see the latest result of each test case. This report might work a bit better for you. Everything else seems to be structured in the recommended approach (single test case repository under a project, milestones to manage releases, and test runs/plans to manage execution). Hope this helps!

Regards,
Marco