Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

Recommendations for integration with automation framework


#1

Hi,

So, here’s my use-case.

  • We’re using Robot Framework to drive the execution of automated tests.
  • We have a long backlog of test cases that we have as potential candidates for automated testing.
  • I have pared that down (let’s say to 500 cases) that are high-value and we want to get done first.
  • These cases are currently spread across several Test Suites.
  • I want it so that when an automated test case is executed, it pushes the result into TestRail so that we can document what the overall project’s test progress/quality is looking like.
  • We need to be able to run these tests against two versions of our software.

Given this, here is what I’m thinking.

  1. Create a Test Plan and include all 500 test cases. Let’s assume this creates 10 test runs, all assigned to the “Next Version” milestone.
  2. The copies of the test cases that are now present in these 10 Test Runs now all have a new test case ID. I can use this ID and the TestRail API to push the test result into TestRail.
  3. However, the instances of these test cases in the “Current Version” milestone will have Different test case ID’s.

How can I run an automated test against one version of our software and update the copy contained in “Current Version”, and then re-run the test against a different version and have it update the copy contained in “Next Version”?

Is it possible to reference a test case contained within a Milestone using the original test case ID (i.e. the one you see if you navigate to it through the Test Suites tab)?

Thanks! Looking forward to having solid integration between the automates stuff and TestRail’s dashboards and reporting capabilities.


#2

So, I just learned that the API shows that you POST with the Test Run number and the base Test Case ID (i.e. the one from the Test Suite, NOT the one from the Test Run). So that answers the biggest hurdle. We’ll have to use custom variable files and import them at runtime. Not a big deal.

But still, I guess we can have a discussion about how other people have integration their test automation frameworks with TestRail. Maybe share some tips/tricks/stumbling points?


#3

I developed my own framework with Java / TestNG / Selenium / Cucumber and run my tests every day via Jenkins jobs.

All tests are in 1 big test suite. I create test runs and add all test cases that need to be tested in these test runs.
Then I start my Jenkins job that matches the test run by test run ID, each test cases consits of the correct test ID of the test case in TestRail so the results are matches bei “Test run ID -> Test case ID”.


#4

Cool. So you have a Jenkins Job per Test Run. So it sounds like each run does the following:

  1. Query the specified Test Run for a list of Test Cases that you want to run
  2. For each test case, you query TestRail to obtain execution steps (i.e. your Gherkin-style keywords)
  3. Then your framework translates those keywords into Selenium actions

That’s an interesting way to do it. Robot Framework uses keywords as well, but I haven’t thought about how we might try to query for keywords as part of the test setup, and then run tests based on the keywords that are obtained at runtime. Is there any special configuration for Cucumber that allows you to do that?


Configuring Test Cases to create BDD
#5

Mhhh not really…

  • I have a test case in TestRail with the title “Scenario: Open login
    page” and the ID of that test case could be 123 which is in the
    section “Login”
  • That test case contains seperated steps e.g.:
  1. Given I am on the homepage
  2. When I navigate to “Login”
  3. Then the login page appears
  • List item
  • For that test case in my framework I have “Login.feature” file (Name of the section, but not really important) containing that test case as following:
  • Scenario: Open login page - TestCaseId[123]
    Given I am on the homepage
    When I navigate to "Login"
    Then the login page appears
  • I put that test case into my test run “Smoketest” which has the ID 12
  • Now I run my Jenkins job, which containts the “Login.feature” file and the results should be matched against the test run with the ID 12

Of course I had to write my own classes and methods that collect all the results for each test case, test step etc. and report it via the TestRail API to TestRail.

I hope this makes it a bit mor clear.

I also have written methods that create .feature files based on the test cases in TestRail and I can update the stuff I wrote in the .feature files to TestRail.
The last method is pretty important for me, because I can write my test steps with syntax highlighting / auto complete etc. much faster in Eclipse than in TestRail and simply update the changes to TestRail.


TestRail, Cucumber and BDD
#6

Hi all!

There’s one important set of API functions which makes adding test results via the (static) case IDs very easy:

  • add_result_for_case
  • add_results_for_cases

http://docs.gurock.com/testrail-api2/reference-results#add_result_for_case

These functions were specifically designed to submit test results using the static case IDs (C###), specifically to avoid the need to manage the dynamic test IDs (T###) as part of your automated tests (which change with every test run). We usually recommend storing the case IDs as part of your automated tests (e.g. as code attribute) and you can then simply submit test results via the API functions above + the ID of a newly created test run.

I hope this helps and I look forward to your reply!

Regards,
Tobias


#7

Yeah I’m already using this. Test run ID + test case ID.


#8

Sounds good and that’s great to hear!

Regards,
Tobias


#9

My project is using TestNG, Selenium webdriver, Java. My regression suite is running on jenkins. Can my automation test results be linked from Jenkins to TestRail? Can i integrate jenkins with TestRail using TestNg framework.


#10

You probably can. TestRail has a pretty easy-to-use Rest API. If your TestNg framework has test case teardown concepts, you can create a POST request to send the test case results into TestRail, passing both the Test Run ID and the Test Case Number (Note: This is the one you see when you view it from the Suite, NOT from the Test Run).

I also found that if I create a TestRail user for my test framework, it makes it clear to see which cases are automated compared to which are being manually executed by the rest of the team. If you do this make sure that you also POST the user ID as well, otherwise the case will become unassigned


#11

Hello Alice,

The tips @ben_fariello provided are spot on and we would also usually recommend directly integrating TestRail’s API with your automated tests:

http://docs.gurock.com/testrail-api2/

Many of our customers use this and also integrate their automated tests with Jenkins. This way whenever your automated tests are triggered e.g. by Jenkins, they would be submitted to TestRail.


#12

Just use the open API from TestRail an write your own method to set results from your automated tests in TestRail.
I match the test cases by test case id and test run id.


#13

Thanks everyone for your comments. I shall test this.


#14

Hi There

I got the testrail api, its easy to use. I am not getting how to integrate it with Testng. So that it will be ran, after the test finished.


#15

Hello Rocky,

Thanks for your posting! One approach that’s used often is to post-process the result files generated by testng or other testing frameworks and then submit the test results to TestRail via TestRail’s API. You can also directly hook into the test execution (via listeners), but post-processing the result files is usually the recommended and most flexible approach. It’s also planned to offer direct integration scripts in the future for various tools but you can also directly use the API in the meantime.

I hope this helps!

Cheers,
Tobias


#16

Hi ,
Actually my concern here is , for example
I have 500 Test cases in test Rail and i need to execute all through my code ( say in Eclipse ),
It should collect the number of test cases (count) dynamically to run the loop instead of specifying manually.
Please help me how to get the total no of test cases that i need to run from eclipse. Here is my sample code…

/*for(int i=1;i<=2;i++)
{

JSONObject c = (JSONObject) client.sendGet("get_case/"+**i**);

System.out.println(c.get(“title”));

        Map data = new HashMap();
        data.put("status_id", new Integer(1));
        data.put("comment", "This test worked fine!");
        `JSONObject r = (JSONObject) client.sendPost("add_result_for_case/1/"+**i**, data);`
    }*/

Thanks,
Gouri


#17

Hello Gouri,

Thanks for your posting! You would usually map the case IDs in TestRail to your automated tests in Java and then use these APIs when submitting test results.

An alternative is to use the get_cases API method which returns a full list of test cases for case repository/project:

http://docs.gurock.com/testrail-api2/reference-cases

Cheers,
Tobias


#18

Hi Tobias,
Back in February you mentioned offering direct integration scripts in the future. Are you planning direct integration with TestNG tests run in Jenkins? We are considering investing some effort in updating the existing open-source Jenkins plug-in that supports JUnit test results so that it supports TestNG, however it would not be worth the effort if you plan on providing TestNG+Jenkins integration in the next quarter or two.


#19

Hi Barbara,

Thanks for your posting. Yes, it’s still our plan to add specific integrations for the most popular tools. But we don’t have an estimate yet (even a rough one), or a list of supported tools and plans can still change, so we currently cannot tell if or when this will be available. I know this is a bit vague but plans and scope for this version are not yet finalized, I hope you understand.

Cheers,
Tobias


#20

I do it in the same way :slight_smile: