Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 
  This website uses cookies to ensure you get the best experience on our website.
Yes, I consent No, Thanks

Parsing test results with Cucumber

#1

Hello,

I’m currently building scripts to use the API to call/report our QA automation tests. So far I was successful, except for one part - automating the process in reporting the test results.

Recently, I’m blocked in figuring out a solution to scan both Gherkin style scenarios and RSpec steps, match it with their Test Case ID, and send the report to the server. What is the best design approach to achieve this for both Cucumber/Capybara and RSpec test runs? Also, is having the tests saved as Test Cases or under Test Suites make a difference when using the API to call them?

The framework we use supports Selenium 3.0.5 with Cucumber/Capybara, RSpec, and Appium for our internal Admin UI portal and mobile applications. The language used is Ruby 2.4.0.

Also, my background is in QA testing and ui automation; so my programing knowledge is intermediate with Ruby and limited to this field using black-box testing.

Any help or guidance is appreciated, thanks.

#2

Hi Ken,

Thanks for your posting. We usually recommend storing the test case IDs together with your automated tests and this makes it very easy to submit test results. The case IDs are static and don’t change over time and you can simply submit results by adding a new test run (or reusing a previous one) and then calling add_result_for_case (or the bulk version add_results_for_cases):

http://docs.gurock.com/testrail-api2/reference-results

I hope this helps!

Cheers,
Tobias

#3

Thanks Tobias, but that didn’t answer my question unfortunately.

I already read the reference doc for ‘results’ and was able to use the add_result_for_case / add_result_for_cases API as a manual call. So I was seeking if anyone has any experience in what the best practice will be to use it with the Cucumber BDD framework.

Our projects consist of mobile games, which I set up our framework using the Appium tool for automation. And, internal admin web UI tools that both our customer service and marketing staff use, which that is setup using the Ruby gem Capybara that works with Cucumber.

But the problem I’m stuck with is when the tests are completed and the results are saved in a JSON output file, by finding the best method to scan the data of the results. From there, determine what the status of the test outcome was and match each test scenario with their Test Case in Testrail. So I was thinking to make it easier, if it was possible to save the Test Case ID using the TAG feature that Cucumber BDD supports.

Feature: Login

@516555 <-- example of tag
  Scenario: Unable to login with a blank email and password
    Given   I load the domain url
    When    I submit with a blank email and password
    Then    I should see the invalid account error message

I found similar results searching on Github from repositories that use RSpec like so:

describe "some test" do
  id "some spec", testrail: 1234 do # 1234 is the id of a test case
    # ...
  end
end

Last question, our Test Cases are saved using the current admin default settings. I notice there’s an option to activate and create Test Suites to better organize the Test Cases. My question is if the API supports finding Test Cases within a Test Suite, or is it better to not use them and just organize Test Cases with only sub-categories?

And thanks for the assistance by the way.

#4

Hi Ken,

We don’t have much experience with Cucumber ourselves unfortunately but storing the case ID with your tests and test results is definitely the easiest way to handle this. As you’ve already seen, this allows you to submit test results easily with add_result_for_case (or add_results_for_cases) if you have an existing run, or create a new test run with add_run:

http://docs.gurock.com/testrail-api2/reference-runs

So, if possible, storing the case IDs as attribute definitely makes sense and this approach is often used with other automation tools and frameworks like xUnit etc.

Cheers,
Tobias

#5

Hi Ken ,

Did you managed to parse test result for a test Id with Cucumber ? I’m stuck with the same problem too :frowning: .

Thank you