Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Test results:all steps failed but overall result 'passed'; can't edit


#1

Hi, I’m playing around with a test installation of TestRail and I have a question about test results.

I set up a simple test case with 4 steps. It’s part of a test suite, and in turn part of a test run.

I assigned it to a user with role ‘tester’ (permissions left at default setting the test installation came with).

As the tester, I went to my todo list, clicked on the run, then the test case, then ADD TEST RESULT.

So far so good.

I can click on the ‘F’ icon by the Test Results, which automatically sets the dropdown for each individual test step to FAILED. Still good.

At the top of the ADD TEST RESULT overlay, I have a dropdown where I can set the status (passed/blocked/retest/failed). By default this seems to be set to PASSED.

Problem:

  • If I choose the F icon to set all test results at once to FAILED, that dropdown at the top of the box doesn’t change. It still says PASSED.
  • If I manually set one test step to F, then the status at the top of the box changes to FAILED, as expected.

So say I set all steps to F, then overlook that the overall status is still PASSED, even though each step failed. I click ADD TEST RESULT.

Now I have a test result that says

  • The test status has been set to ‘passed’.
  • 0 passed, 4 failed and 0 untested steps.

That makes no sense, so I try to edit it. I can click on edit, and change the status of the individual test steps, but I can’t edit the overall status.

Is this a bug? Is this meant to be this way – and if so is there a way around it?


#2

Hello,

Thanks for your feedback. The overall status value of a test result is independent of the status of the test steps in general, but we do set the test status to Failed automatically when a test step is marked as failed as you mentioned. It appears that this is not done when you click the F icon and I’ve added this as a bug report so we can change this behavior in the next update. I suspect that we haven’t seen a report for this issue before as most testers usually only set a single step to failed (and then stop executing the test steps as executing all test steps usually doesn’t make sense when a previous step failed). Thanks for reporting the issue!

Regarding editing test results: while you can change some details of a test result by editing it (e.g. to fix typos or to clarify some of the details), changing the test result is not yet supported (as the result is used for various calculations and statistics). However, you can simply ‘override’ the result by adding another result to the test. Only the latest test result is used for the status reports and you can thus override the status by adding another result.

Thanks,
Dennis


#3

Depending on the nature of a test and the severity of the defect found it may be better to continue past a failure and complete the remaining steps to exercise the rest of code. This increases the opportunity for early defect discovery, rather than stopping and waiting until 1 defect is fixed only to find another in the next run a few steps further through the test.


#4

Hi Dave,

Yep, that makes sense. It’s also always possible to mark as test as Failed later or add additional defects (by adding an additional result), even if you haven’t set the original result to Failed.

Cheers,
Tobias


#5

I think the overall result of the test should still be auto-set to failed if there’s a failure in one or more test steps.
If no steps fail, it doesn’t make sense to me to fail the test overall. I’m not keen on being able to override the overall result separately to the steps results.

IMO, the overall result should be read only and automatically set to the worst step result as I mention here - Improve test case result setting for results per step test cases


#6

Hi Dave,

Thanks for your feedback. I believe a combination would be best, i.e. setting the overall status based on a rule/weight system but still let users override the status if they want to (optionally). We will definitely make sure to look into this along with your other step control suggestions, thanks again :slight_smile:

Cheers,
Tobias


#7

Is there any fix for this yet from TestRail prespective ? It is kind of majorly affecting our automation…If any test step in-mid gets failed and the last step gets passed we want the whole test case to show as Failed …currently this is showing as Passed.