Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.

Automated Test Results Obscured


My apologies if someone has written about this before, but I couldn’t find anything…

We are using Specflow (C#) for our automated tests. I have everything set up to write the pass/fail results to TestRail. This works just as expected. For my TestRail code I am…

  1. Grabbing the list of active test runs for a given project
  2. Grabbing a list of all of the test cases for those runs
  3. I then write the results to the test when the Scenario title == TestRail test case name

This works well and all of my results are being logged. My issue is that each of my scenarios has at least 2 unique tests. This means that when I log a result in TestRail, I enter 2 results.

This is fine too. However, when I just do a quick glance at the results, I only see the most recent result. This could easily (and does) obscure failures if they came first.

Is there a setting, value, config that I can set that would allow me to get beyond this? Ideally, when I look at the results for a given run, I would see not just the latest entry but the entire body of results.

If my test is configured to put a string into something. I pump in A, B, and C. The tests run in alphabetical order. A fails, B passes, and C passes. When I look at the run results, I see green but hidden in the test details is the failure for A.

I realize I can change my tests to run as single tests but that defeats part of the functionality & strengths of Specflow.



I think I may have found my answer…

Does anyone know if multiple iterations may be handled in the future?


Hi David,

Thanks for your posting. You have multiple options here. You could either look into using a single case with multiple steps, or use separate tests or possibly runs/configurations.

Multiple steps look as follows:

You can assign a separate status per step and also choose an overall status. For tests that have both failed and passed results, you could use a custom status that indicates this instead of setting the full test to passed.

I hope this helps!