Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.

Report on summary over multiple Test Runs in a single milestone


So here is the situation:
As we are going through acceptance testing, we create a Milestone for the release and then we create a test plan for the specific SW build we are on. We begin testing and as we identify issues, we adjudicate the ones that require fixing. Once we have those fixes ready, we will re-run the test plan under for the new build and include all procedures that were not tested, blocked or failed. If we think we need to re-run any of these procedures (based on rIsk) we will pull them over by hand.

At this point, we typically close out the initial Test Run so we don’t get accidental updates. All of the untested/failed/blocked procedures are completed in the next Test Run. The problem now is that if I generate a summary report for the milestone, I am getting a large number of untested procedures that WERE actually tested the final Test Run. Is there a way to generate a report over the entire test plan that simply shows what passed/failed and where? I don’t want the untested procedures counting against us in the summary report, but the only way I can see to manage that is to remove the untested procedures from the initial test run AFTER I re-run it.


Hi Tom,

Thanks for your post! Currently, as these are considered separate tests and the milestone summary report generates the report based on all tests and not just across the test cases, this would be the expected behavior (in order to show the full summary over time). You can continue to use the workaround you have now in regards to removing the test cases from the run after you’ve performed the rerun test case selection if needed. Alternatively, you can also look into using the Comparison for Cases (Results) report as this would show the different test case statuses across the test runs, but it would also show the “Latest (Coverage)” column which would include the latest result added for each test case and give a more accurate representation of these metrics in terms of the most recent result. The report also provides an option to filter the test runs to include only test runs linked to any given milestone. Hope this helps!