Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Reports : Results – Comparison for cases


#1

The “Results – Comparison for cases” report is not picking up all the data from the Test Runs included int the reports. Is there a limit to number or data-rows it considers when making the calculations to get the “Latest Coverage”? . We are looking for a way to show the latest results of all the test cases we executed form the suite and this is the only report we see that can obtain that and it is showing incorrect data. I tried small sample and it works, but once I use the report on my suite that has 6,000 test cases it is showing incorrect data.


#2

Hi Viken,

Thanks for your posting. The Latest Coverage column uses the most recent result per test across all test runs in the scope of the report. Please note that the report only takes into account the latest 10 runs by default but you can change this on the report form on the Test Runs tab (Maximum number of test runs to include).

Cheers,
Tobias


#3

Hi Tobias,

Yes we did notice that variable and increased it to the highest value. We feel that the report does not account for all the rows in the run. we have organized our runs by folder/sections within the Test run, can that cause an issue?.

What can I send from here for you to look into this issue?. Thanks so much for your prompt reply!

Viken Tanielian
Pacific Life


#4

Hi Tobias,

This is a critical report for us at this moment in time. I would like to have a Webex/Go-to-Meeting to illustrate and walkthrough this issue to get resolution.

Thanks so much for your help!
Viken


#5

Hi Viken,

Thanks for your feedback. Sure, happy to help. The Latest/Coverage column uses the most recent test result per test across all runs. You can click on the result/status in the Latest column to jump to the actual test result and open the test and compare this to the generated report.

Cheers,
Tobias


#6

Thanks, Tobias.

We understand the report and like to use for its purpose. Unfortunately, we do not believe the report is picking up all the data on large sets of test runs and therefore showing incorrect results in the Latest Coverage column. When we attempt to run the report with display > 1000 to verify, the report runs indefinitely. We let it run over (14) hours and it did not finish and had to stop it.

We would like to have ½ hour Go-To-meeting to illustrate it to you. We are at PST time in USA-Southern California. This is urgent for us.

Here is a sample where the report does not go beyond 1,000 test cases even when we know there is more than 1,000 distinct test cases and pass fail numbers are therefore incorrect.

Thanks
Viken Tanielian
Pacific Life

[cid:image001.png@01D23E52.12D05810]


#7

Hi Viken,

Ah okay, thanks for that! You would need to include all cases in the report and it would then show the correct statistics. The issue you see with the indefinitely running report is likely caused by TestRail hitting PHP’s memory limit and this is very low by default. You can increase or disable the memory limit as follows and the report would run through after changing this option:

http://docs.gurock.com/testrail-faq/system-memory-limit

Cheers,
Tobias


#8

Hello Tobias.

We are including ALL the test cases in a Milestone, and selecting “All” for Maximum number of test run to include. Also, we are not specifying ANY additional Filters in the Test Cases Tab. (see below).

[cid:image001.png@01D23E86.49464F10]

Are you saying the indefinitely running report issue is also effecting how many test cases the report grabs to present statistics for?

Thanks!
Viken


#9

Hi Viken,

Reports are usually generated within a few minutes and if a report doesn’t finish, it’s very likely an issue with the memory limit of your PHP environment. If TestRail hits the configured memory limit, the report can’t be generated and this will look like the report never completes in the UI. The larger the scope of the report (in your case, the number of runs or cases), the more likely this scenario is and you can simply increase or disable the low default memory limit:

http://docs.gurock.com/testrail-faq/system-memory-limit

Can you ask your IT or TestRail administrator to look into this? We are also happy to help in case you have any questions.

Cheers,
Tobias


#10

Hello Tobias:

This still does not address our # 1 issue with report “Results - Comparison for cases” when it actually completes fast but does not include all test results in the report computation.

Thanks!
Viken


#11

Hi Viken,

You would need to include all test cases in the report to have the correct statistics as the stats are based on the case scope and number of included cases. So, please turn off the default 1000 case limit and let the report run with all cases. As mentioned, you would need to increase the memory limit if the report doesn’t complete successfully and this limit cannot be influenced by TestRail and only by your PHP environment and configuration.

I hope this helps!

Cheers,
Tobias