Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Report for initial quality - pass-rate for executed test cases only


#1

Hello,

Test Rail reporting seems to report Pass rate based on total number of test cases in test run. For a similar report, how can generate a report where the pass/fail rate is based on what has been executed thus far and not the total test case count.

thanks for any feedback.

Michael


#2

Hi Michael,

Thanks for your post! TestRail would always use the latest result per test in the statistics/reports to calculate the passed percentage and to display in the pie charts and graphs. Can you let us know which report you’re using so we can better understand the type of report you’re wanting to accomplish?

Regards,
Marco


#3

Hi Marco,

I’m doing a Summary Run report. The Pass % for all your report is based on total # of test cases. I am looking for Pass% for what has been executed thus far.

Michael


#4

Hi, very much interested in this query too. it would definitely help as it would give a “true” indication of the state of the test as “scoped out” tests are unfortunately included in the Pass percentage. If we could base it on the number of the actual executed tests, it would definitely be better.


#5

Hi all,

Thanks for your feedback! The Summary reports and statistics areas would always base the percentage on the latest result for each test. You may want to consider closing your test runs after the initial test execution, and opening a new test run using the ‘Rerun’ feature to include test cases from the previous test run that had a certain status (e.g. such as Failed, Untested, and Retest). You could then include all relevant test runs in the summary report, and get a more accurate Passed percentage based on each execution. The Rerun feature makes this easy to accomplish and you would get a per-execution pass percentage on your reports.

Hope this helps!

Regards,
Marco


#6

Hi Marco,

I think you are missing the question. Let’s say I have the following results:

16 Passed, 10% set to Passed
0 Blocked, 0% set to Blocked
0 Retest, 0% set to Retest
2 Failed, 1% set to Failed
0 Deferred, 0% set to Deferred
0 CATS Running, 0% set to CATS Running
10% passed
148 / 166 untested (89%).

Per the reports above you have 10% passed based on 16 / 166 test cases. What I am looking for is 16 / 18 = 89% passing. A passing % based on the executed test cases only (this case 18 and not the total of 166).

This is how we measure quality throughout the test cycle.

Michael


#7

Hi Michael,

Thanks for your reply! The statistics would always be based on the full scope of the test run (as Untested is considered a test status as well for reporting purposes). That said, we’re happy to review making this a configurable report option for summary reports (e.g. having the option to exclude certain statuses for the pass percentage) for a future update to TestRail.

Regards,
Marco


#8

Hi all,

I agree it is useful (essential actually) to have a report on quality; the current TestRail report is a test progress report, not a quality report. In my opinion you need both.

I will therefore manually create quality reports, i.e. divide tests passed by tests run, excluding those marked as Not applicable.


#9

Hi Marco,

+1 for “making configurable report option for summary reports” so it can show 16/18 =89% passed.

And in my case, would be ok to simply allow making the system statuses inactive then user can choose just Passed/Failed/Untest as active & current summary:
16 Passed, 10% set to Passed
2 Failed, 1% set to Failed
148 / 166 untested (89%).

Ryan