Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Mark a test as failed if there is at least one failed result


#1

If a test has multiple results, its status will be treated by the status of last result. I want my test status to be failed if there is at least one failed result in a test, doesn’t matter if it is last or not.

Is it possible to achieve right now? If not, then are there any plans for this feature in the future?


#2

Let me understand this: assume you have a test in a test run, and have 3 results for it, say, Fail for the first result, and Pass for Results 2 and 3. For this set of test results, you want to say the test failed?

If this is so, could you explain the motivation for such behavior. I would think the Pass results of 2 and 3 are better and “out-vote” the Fail of result 1. Or perhaps these 3 results are due to running different test steps within the test?


#3

Yes, your understanding is correct.

While I see that current behavior is the most proper one for the majority of cases, let me explain why I want it to be changed in my particular case.

I have a Test Suite. I create Test Runs based on this Test Suite. I need to run this Suite simultaneously on, say, 20 different mobile devices. For every Test Case in Test Suite I have my automated tests, which are integrated with TestRail with the help of TestRail API.

First solution for me was to create a new Test Run for each device (I use cloud devices and Test Runs are creating automatically). Then it would leave me with 20 Test Runs. It creates a lot of mess in TestRail, especially having in mind that I plan to increase the amount of devices in the future or/and the run my tests more frequently.

To get rid of the mess, I decided to create a single Test Run, where each Test Result in a Test corresponds to the device where Test Case was executed (I add device name to Test Result description to differentiate them).

That’s why I want my Test to be failed if there is at least one failed Test Result in it. So I will pay my attention to this Test, open it, and see on what device it failed.

I’m open to any criticism about my solution. Maybe I’s missing something about TestRail.