Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Automated Test Results Obscured


#1

My apologies if someone has written about this before, but I couldn’t find anything…

We are using Specflow (C#) for our automated tests. I have everything set up to write the pass/fail results to TestRail. This works just as expected. For my TestRail code I am…

  1. Grabbing the list of active test runs for a given project
  2. Grabbing a list of all of the test cases for those runs
  3. I then write the results to the test when the Scenario title == TestRail test case name

This works well and all of my results are being logged. My issue is that each of my scenarios has at least 2 unique tests. This means that when I log a result in TestRail, I enter 2 results.

This is fine too. However, when I just do a quick glance at the results, I only see the most recent result. This could easily (and does) obscure failures if they came first.

Is there a setting, value, config that I can set that would allow me to get beyond this? Ideally, when I look at the results for a given run, I would see not just the latest entry but the entire body of results.

Example…
If my test is configured to put a string into something. I pump in A, B, and C. The tests run in alphabetical order. A fails, B passes, and C passes. When I look at the run results, I see green but hidden in the test details is the failure for A.

I realize I can change my tests to run as single tests but that defeats part of the functionality & strengths of Specflow.

Thanks!


#2

I think I may have found my answer…

Does anyone know if multiple iterations may be handled in the future?


#3

Hi David,

Thanks for your posting. You have multiple options here. You could either look into using a single case with multiple steps, or use separate tests or possibly runs/configurations.

Multiple steps look as follows:

http://docs.gurock.com/testrail-faq/config-steps

You can assign a separate status per step and also choose an overall status. For tests that have both failed and passed results, you could use a custom status that indicates this instead of setting the full test to passed.

I hope this helps!

Cheers,
Tobias