Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Need the ability to filter out failed tests if now passing, for given milestone


#1

So we have a scenario where we may have multiple test runs against a given milestone. And of these test runs some tests are retested, from one run to the next.

So in the case where a given test failed the first couple of runs, but in the latest it passes, I’d like to be able to see the milestone pie chart summary updated to reflect this pass, but also, ignore it’s failure from before since that was an older test run.

Example:

  • We have 2 tests.
  • The first run, 1 passes, 1 fails.
  • The pie chart summary for the milestone shows 1 pass and 1 fail. We’re great so far!
  • The second run, both pass.
  • The pie chart summary for the milestone shows 3 passes and 1 fail. This is what I’d like the ability to filter…

We want the ability for anyone, mgmt, etc, to look at this milestone summary and know if, after the latest run, which tests are still failing, if any. But as it is now, it’ll always show a failure (in the summary), and it’ll force us to sift through all the test runs and see if that given test passed sometime later. So having a milestone summary that displays the net failures would be very helpful.

We’re trying to use TestRail for it’s “visibility” for fellow engineers and mgmt. But if it’s difficult to know for certain what the state of our tests are for a given version, it’s pretty much all for naught, if what mgmt will be looking at is a pie chart that will forever show all tests that failed, regardless if they now pass, that’s not helpful and doesn’t give good insight into the current state of tests…