Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Report on Test Cases NOT run


#1

Hi

I simply want know which Test Cases we have not Run

That means, I want to report on which Test Cases are NOT in any Test Runs at all;

AND Test Cases that ARE in Test Runs, but which are Untested

I guess there may be an easy way to do this, but I haven’t found it yet…


#2

Hi @geoffg,

Thanks for your posting! Yes, there’s an easy way and you can use the Results > Comparison for Cases report in this case:

This is a side-by-side comparison for test runs and includes the status for each test case/run combination. If a case is not included in a run, the cell is empty. Untested (but included) cases are indicated with an Untested status. The last column (Latest/Coverage) shows the most recent results across all runs and is also empty/Untested if the case is not included/included but untested.

I hope this helps!

Cheers,
Tobias


#3

Thanks

I don’t appear to be able to to see a list of ONLY test cases that are Untested or that are NOT in Test Runs

I’d also like to be able to just see a total no. of test cases in this category

The report is very unwieldy, as we have a lot of Test Runs, so it’s hard to identify Untested test cases or test cases that aren’t in Test Runs, because that requires a lot of horizontal scrolling

It appears also that reports can’t be exported to other formats (eg Excel), to make the date easier to work with.

Also, the report filters don’t allow me to just see Untested test cases, or test cases for which all Test Runs are blank


Test Case Coverage Report
#4

Thanks for your reply! You can disable the per-run columns on the report form and focus on the overall Latest/Coverage instead. Filters for this report are only available for case-level properties and not for tests/statuses/results because a case can be part of multiple runs. You can simply disable the per-run columns (turn off The test results for each selected test run / result) if you don’t need the results per test run and this makes the report much more manageable with many test runs.

Cheers,
Tobias


#5

thanks a lot

I disabled the per-run columns, and am now only seeing the Latest (Coverage) column, which is what I want to see, so that’s good

It doesn’t look like there’s a way to save a set of report filters, so I can simply run the report with those filters again, without having to set the filters each time

Also, I only want to see test cases in the report that are for the next release of the software; I want to exclude test cases/runs that are for the subsequent release. I’ve set these releases up as Milestones, so we’re setting Milestone in the test cases to either one of these releases. But I can’t filter out test cases using the Milestone field.

I guess the only way to filter out the relevant test cases out would be to set up a Release 2 Section, put the relevant test cases in that, and then filter out this Section - but I’ve already identified the test cases by Milestone, so I thought I should be able to filter by Milestone.


#6

You are welcome! Regarding saving filters: you can simply rerun the report and this will start with the same settings by default. Just click the copy icon (Reports overview page) or use the Create Similar button when viewing a report and this will open the report form with the same settings. You can also create a recurring report that is generated automatically based on a custom schedule, e.g. daily, weekly, etc. (with email notifications when the new report is available):

Regarding the milestones: are you using the Milestone field on the test case level? This is not enabled by default (on Administration > Customizations), but if it is you can also filter your cases on the Test Cases tab on the report form when creating a report. If you use milestones on the test execution level instead (to group/organize your test runs on Test Runs & Results), you can filter the included runs and customize the run selection on Sections & Test Runs on the report form:

I hope this helps!

Cheers,
Tobias


#7

Hi, I really hope there will be an EASY report just showing all testcase that are not inlcuded in a run. It would make live so much easier.


#8

Hello Simone,

Thanks for your reply! As Tobias mentioned above, this can already be accomplished with the Comparison for Cases report with the options selected as mentioned. Once you have this report generated once, you can just use this report as a template to copy each time you need to run it again (so that you don’t have to configure the options manually each time). That said, we’re also happy to look into additional report options that could show just the test cases not included in a run for a future update as well. Hope this helps!

Regards,
Marco


#9

Hello,

I’m having similar issues to geoffg and Simone: I need an easy way to see which test cases have NOT been included in test runs.

Ultimately, I would like to see which test cases have and have not been tested in a certain time period. I can get pretty close to that using the Comparison for Cases report and including the relevant test runs for that time period.

That lets me scroll through a list of every test case and see its status (or lack of status). My only complaint is that a test case that is not in a test run only shows up with an empty status, rather than something like “untested”. That means the stats at the top of the report might show 100% Passed, when many of the tests were never run. I need to scroll through the list of all test cases to check that they were actually tested.

I’ve also tried using the Property Distribution (Results) report, but that report completely excludes test cases that have never been added to a test run.

Is there no way to generate a report that just says "show me the test cases that haven’t been run [this month/ this milestone/ in this test plan/ etc.]?

Overall, I’ve been pretty impressed with TestRail’s reports, but being able to easily track which tests are being executed seems like a very basic feature that TestRail is missing.


#10

Hi James,

Thanks for your reply and feedback! TestRail currently doesn’t have a report that would show just the test cases that haven’t been included in a test run recently. The Comparison for Cases report is the best way to see this as you can see when a test hasn’t had a result added for it in recent test runs, however you would have to find these within the report manually as well. That said, we’re happy to look into adding a report that can provide this specific functionality to a future version of TestRail.

Hope this helps!

Regards,
Marco