Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Pass/fail individual test steps from the test case page


#1

So, after configuring steps with individual results (thanks to Unable to add individual step failures for steps-separated which I hope is documented better somewhere). I can set results per step.

I have two requests from my quick trial of this feature:

  • Allow all results to update the overall test result based on a ranking of possible results where Not Run and Pass should be at one end of the scale (most easily over-ruled), and Fail at the other so that it (typically) over-rules all other results, however if all steps Pass bar one that is NeedsReview, then the test should have the overall result of NeedsReview
  • Make the Run test UI the same as the View test UI and just add step results buttons at the right of each step. Having to open the Add Results frame and getting a UI 1/4 the size is not a good experience for the primary task a tester is there to do (run tests).

Results per step in the wider UI format:
The View test case view is a much nicer UI to see and work through multiple steps than the stacked Add Result UI.
Also, please don’t use a drop down to record results. Use an icon/button to set a result, the same as you do for your Set all steps. Just put that to the right of every step. This way testers only need to click once per step. That would be a 50% reduction in clicks and a much better UX. Also, shading of the background of the step with the light version of the result’s colour will give the tester a visual representation of their progress in the test.

Overall results with mixed step results:
If any test step is not a Pass, this is what should be reported in the overall results. The customisation of results could be used to rank the possible options so that a test with blocked and missed steps could report the overall result as blocked or skipped, depending on the ranking set by the admin.

Also, is there a report that shows the break down of test step results for test cases?
Step Results> Pass Fail Blocked Not Run
Test 1 7 0 2 4
Test 2 8 1 3 0
etc.


#2

Hello Dave,

Thanks for your feedback, that’s really appreciated. The official documentation to set up the steps can be found here:

http://docs.gurock.com/testrail-faq/config-steps

Thanks for the suggestion with the ranking-based system, that’s definitely interesting to review. Currently, only the failed status would set the overall status to failed, but it’s already planned to make this more flexible/configurable in the future.

We are also happy to review any UX enhancements. The dropdown provides a cleaner interface and scales better but I agree that a button is a bit faster to use.

Thanks again,
Tobias


#3

Hello Tobias,

I will be happy, if you give us to choose: dropdown or icons.

Thank you!
Sincerely yours,
Oksana


#4

Hello Oksana,

That looks great, thanks for the suggestion! Would definitely save a few clicks (especially with many steps).

Cheers,
Tobias


#5

Again sorry for reply to an old post but only one i can find regards this.

When i mistakenly set one of the steps as failed the result gets updated before saving the results i change all steps to pass but the result at the top does not auto update. Is this expected or should it also auto update to reflect that all steps are marked as passed?


#6

Hi Dobryan,

Thanks for your reply! This is the expected functionality for this, as the automatic status update would only occur once when setting a step to failed. From here, the Add Result dialog would assume that any other changes to step statuses (even the one marked as failed) would require a manual update to the overall status to override anything that was set automatically. This is done so that the status isn’t changed automatically each time in the case that the actual overall status should remain the same, regardless of the step status. Hope this helps!

Regards,
Marco


#7

Hi Marco,

Thanks for the reply. This is a very touchy subject here in the office. This function not updating the main status is looked at as a deal breaker. As it is far to easy to accidentally set a step as failed. I under stand that the function is not updating for each drop down list and why. but can it not be set on the buttons when you click on the P for pass all test that the main status would update?

Is it possible for us to change this function our self using a UI Script?


#8

Hi Dobryan,

Thanks for your feedback. It wouldn’t be possible to modify this currently, and we haven’t tested any UI scripts to change this functionality. It might generally be possible to customize this using a UI script, however we don’t have any examples of this type of script and you would need some HTML, CSS, and JavaScript experience to attempt to build one for this functionality. You can learn more about UI scripts on our website here if you haven’t had the chance to look at this yet:

http://docs.gurock.com/testrail-custom/uiscripts-introduction

Regards,
Marco