Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Steps results status hierarchies


#1

Hello,

Steps results are awesome as they help creating test cases that have multiple verification points, however, they have an important miss-design/flaw: When you have multiple verification points, there is no way of setting up a hierarchy between the different statuses of said verification points within the test case and the overall test case status:

For example: I have a test case that has 3 steps results: I set the first one to pass, the second one to in progress, and the third one is still as not run.

Overall test case will be set automatically as Passed, which can be very dangerous and gives false real time statistics.

Other software solutions offer the possibility of defining the hierarchies of each sub status so the over status of the test case it’s update to the relevant status. On the example above for example, if one set result is set to Passed and another one to in progress, overall test case status should be in progress. As soon as all steps are set to pass ,the the test case is set to pass. If there is a fail ,then the test case will be set to fail, etc…

Would it be possible to add this support? It would help immensely, and in my opinion, is how this function should work.

Thanks in advance!
Edu Varela


I cannot change main test result status of a previously saved test result
#2

Hi Edu,

Thanks for your posting. That’s already partially supported and the overall status of a result is automatically set to Failed if you set a step to Failed. This is the only assumption TestRail can safely make because a different status for a step may or may not change the overall status (this depends on the context and TestRail doesn’t necessarily know the circumstances when it’s okay to change the overall status).

It’s planned to extend this to other statuses as well and this would probably require some kind of ranking or rule system when it’s okay to change the overall statuses and when it’s not. We want to make sure to get this right (because the behavior would be unexpected otherwise) but it’s already planned to look into this for a future version.

Thanks again for your feedback!

Cheers,
Tobias


#3

Thanks Tobias,

Yeah, I realised that it works for fail. I also understand that forcing the hierarchy on your end would be complicated, as different teams/projects may have different views.

A possible idea would be allowing admins to set this hierarchy the steps results, so you can configure the overall status depending on your needs. For example, I might want to have it set in a way that in progress takes precedence in the overall status over failed as long as there’s an in progress status, but other PM might want to do it the other way around. Still, I think is a much needed addition to make this field truly awesome

Thanks for the quick reply! greatly appreciated.
Edu Varela


#4

Hi Edu,

Thanks for your feedback. Yes, that’s what I meant with a ranking or rule system and this would likely need to be configurable by administrators. We are happy to look into this for a future version and will review this again for a future release.

Thanks again!

Cheers,
Tobias


#5

Of all the ways to address the current behavior (and I am ssoooo tempted to say fix) of how the Test Status field is set, the proposal here by @EduVarela accomplishes how I would want TestRail to perform.

I understand that the current design was intended, and may even be beneficial for some organizations, but it just does not work for our group- there’s not 100.000% confidence that a Test Status of “Pass” is exactly 100% of the Test Steps in a “Pass” state. Proposals such as of adding a custom field just add more work, when I want a tool that saves me work (and yes, I admit that is an ironic statement in that the current Test Status behavior is designed to save time).

It may help to know that we have tests that are 20, 30 or more steps long, and some steps can take several hours to run (because we test hardware as well as software), so it is very possible that a tester “A” would start a test, then have to pause its execution when they are done for the work day. This then opens up possibility that the tester may get re-directed to other work then next day (or be absent, etc). Another tester “B” could be assigned to finish off “A”'s tests, but may skip over the above as the Test Status may have mistakenly got set to “Pass”.