Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Track and record tests with multiple iterations


#1

A number of our tests are made up of the same process being executed many times, over many days and different releases. Can TestRail record and report the number of iterations executed within a single test? If so are there any TestRail reports to demonstrate reliability on repeated tests ?

Thank you


#2

Hi!

Thanks for your posting. Yes, TestRail comes with a dedicated report for this and you can use the Comparison for Cases report from the Reports tab in this case:

This report makes it very easy to compare cases and their results over time and you can flexibly configure the test scope of this report (e.g., which test runs to include).

Cheers,
Tobias


#3

Thanks Tobias, I should’ve given an example to go along with my question. Some of our tests are a single test which comprise of multiple iterations. An example would be power cycle, this test requires we power down a device, verify that the device powers down as expected, wait a period of time and return power to the device and verify the device powers up as expected, which would be repeated for 100 iterations. The 100 iterations might be executed on a single release, multiple releases or spread across multiple releases. If the 100 iteration attempts were spread across 4 releases with 25 iteration attempts for each release. When the tests were executed the test failed on the 5th iteration on the 1st release, 10th iteration on the 2nd release, 18th iteration on the 3rd release and completed all 25 iterations on the final release. We would then be able to look at the reliability of the power cycle function and how it improved or regressed through the various releases. 1st release reliability metric = 20% , 2nd release reliability metric = 40 %, 3rd release reliability metric = 72% and 4th release reliability metric = 100%. Our requirement is firstly to be able to record iteration counts for each release and to be able to easily pull the iteration counts for daily status and test exit reports.

Thanks

Greg


#4

Hello Greg,

Thanks for the additional details! You could look into using the Results > Property Distribution report which might help in this scenario (grouped by status). If it doesn’t, you can always use the exports and generate your own reports based on this and this would be the most flexible option.

One option that is used by other customers as well is to use the XML export feature (e.g. on the milestone level) and then generate custom reports by analyzing/post-processing the XML files according to your requirements. The XML exports include everything you would need for that such as test runs, tests and results in a structured format. The XML format used by TestRail is very easy to follow and we are also happy to help in case any questions come up.

http://docs.gurock.com/testrail-userguide/howto-import

(this is the import format for cases but the export format for tests/results is very similar)

Cheers,
Tobias