Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Aggregate Multiple Results


#1

Is there a way to make the UI show different final status for tests with multiple results? That is, could I do something like have it show Failed if any result failed, not just the last to be reported?

Our use case is we have multiple testers executing the same case on different devices. A failure on any device should cause the overall test to Fail.

I’m guessing configurations are one method of approaching this problem, but it is not practical to manage as it makes far too many test runs and we seldom know the devices ahead of time.


One Case, multiple, parametrized test runs for test case
#2

By the way, the full behavior I would love to have is something like below:

Final status on a Test with multiple results:

  • Failed: At least one “Failed” result
  • Passed: No “Failed” results; at least one “Passed”
  • Blocked: No “Failed” or “Passed”; at least one “Blocked”
  • Retest: No “Failed”, “Passed”, or “Blocked”; at least one “Retest”
  • Skipped: No “Failed”, “Passed”, “Blocked”, or “Retest”; all “Skipped” or empty

#3

Hello Logan,

Thanks for your posting and my apology for the delayed response. The UI (e.g. when viewing a test run) always shows the latest status (test result) for each test. I would also recommend looking into using separate test runs and/or configurations for your scenario. That said, if this doesn’t work for you, it would also be possible to implement the desired behavior with a custom report and we have details on how to implement custom reports on the following website:

http://docs.gurock.com/testrail-custom/reports-introduction

We are happy to help in case you want to pursue this and have any questions.

Regards,
Tobias


#4

Hi Tobias,

I think a custom report would be acceptable for our use case. I would love to be able to generate the summary reports (Milestone, Plan, Runs) except swapping out the test status with our custom calculation mentioned above. This includes both the pie chart at the top and the individual test statuses.

Is there an easy way to re-use those reports except modify where status comes from?


#5

Hello Logan,

Thanks for the update. You would need to copy the template files of the existing summary plugins (to your “custom” directory as explained in the linked article) and can then adjust the code. That said, this would involve changing large parts of the logic of the reports (e.g. where the statuses come from) and you couldn’t use most of the built-in/shared functions that the reports currently use.

Most of the code of the reports (except for the shared functions and views) is readable/changeable. We also have details on how to access the database in reports in the linked article and we are also happy to help in case you have any questions or feedback.

Regards,
Tobias