So, after configuring steps with individual results (thanks to Unable to add individual step failures for steps-separated which I hope is documented better somewhere). I can set results per step.
I have two requests from my quick trial of this feature:
- Allow all results to update the overall test result based on a ranking of possible results where Not Run and Pass should be at one end of the scale (most easily over-ruled), and Fail at the other so that it (typically) over-rules all other results, however if all steps Pass bar one that is NeedsReview, then the test should have the overall result of NeedsReview
- Make the Run test UI the same as the View test UI and just add step results buttons at the right of each step. Having to open the Add Results frame and getting a UI 1/4 the size is not a good experience for the primary task a tester is there to do (run tests).
Results per step in the wider UI format:
The View test case view is a much nicer UI to see and work through multiple steps than the stacked Add Result UI.
Also, please don’t use a drop down to record results. Use an icon/button to set a result, the same as you do for your Set all steps. Just put that to the right of every step. This way testers only need to click once per step. That would be a 50% reduction in clicks and a much better UX. Also, shading of the background of the step with the light version of the result’s colour will give the tester a visual representation of their progress in the test.
Overall results with mixed step results:
If any test step is not a Pass, this is what should be reported in the overall results. The customisation of results could be used to rank the possible options so that a test with blocked and missed steps could report the overall result as blocked or skipped, depending on the ranking set by the admin.
Also, is there a report that shows the break down of test step results for test cases?
Step Results> Pass Fail Blocked Not Run
Test 1 7 0 2 4
Test 2 8 1 3 0