Hi, I’m new to testrails and would like to know how you advise on setting up our first project (all our projects work in similair way as mentioned below):
A project has many tests, which can be created under one suite (use groups for management) or multiple suites. This project (or more like a feature) is delivered in specific final releases which we have the dates for (these are also the milestones). So lets say this year we have 2 release milestones - one called releaseA and its date is 06/17 and one called releaseB with 12/17 date.
For releaseA, we need to run X tests and make sure we have pass coverage > 95% by the miletone date. For releaseB, we need to run all the tests and have the same pass coverage by its milestone date. In addition, we get weekly releases where we run some or all of the tests and report the results. For the final releases, we don’t run all the tests again, we rely on what we achieved (coverage and results) from the weekly releases.
My first thought was to create a test plan for releaseA, and for each weekly release we run tests (until we reach releaseA milestone date) create a test run, but then I saw the test plan statistics are for overall tests (i.e. I have the same tests cases in multiple runs, so the test plan counts them twice as if I had double the tests).
So what I did is create one test run for releaseA, and for each weekly release we can update the test case results and provide the release info in the test result (can add cusom field like release version). However, in this approach, if at one point I would like to know which tests were executed for a specific weekly release and what were the results, there’s no way to get this info (I see if I click on a test case in a test run I get the results history but I would need to go over each test and search for the relevant result based on the weekly release).
Any suggestions on how to work with testrails with the above need?