Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

How can I map my automation test scripts (robot framework) with testrail test cases?


#1

I am looking to map all my automation scripts to the test cases in test rail .
The goal is to run a job in jenkins which run all the test scripts and updates the test rail test case results appropriately, for this I will need to find a way to map my test cases to the automation scripts?

Any ideas as to how this can be implemented is very much helpful.
Thanks in Advance.


#2

Hi @gauthami.pingili,

I’ve just done this myself. I’ll explain how I do it with my automation suite and see if that helps you at all?

I don’t use Jenkins, this is all done within a C# project and triggered in our internal build process.

For our QA process in testrail, we have X amount of tests, some are manually tested, some are automated.

So when we run our automation suite, I call into test rail to get all the test cases of type Automation (Type_id =3) for that project.

JArray response = (JArray) api.SendGet("get_cases/" + projectId + "/&type_id=3");

I then create an array with all the test case ids and then pass that to the create test run API call. (“add_run/”).

        var data = new Dictionary<string, object>
        {
            { "suite_id", 1 },
            { "name", "Test Run From Framework" },
            { "include_all", false },
            { "case_ids", testCaseIds }
        };

        JObject testRailResponse = (JObject)api.SendPost("add_run/" + projectId, data);

This will then create a test run with only automated tests in the run. Then you can just go through and update each test with pass or fail as needed.

Hope that helps and let me know if you need anything further.

Regards,

Kyle.


#3

Hey @Kyle,
Thanks for explaining you approach! Really appreciate it.

I am actually looking to map the automation suite with test cases so I need not manually run the automated tests myself.
Do you know a better way to do that?


#4

Hey @gauthami.pingili,

The only other way I can see you doing what you want (if I’m reading it correctly) is to have the test case id’s hardcoded into you framework, so that when they run, they know which testcases to report against.

Where you say:

so I need not manually run the automated tests myself.

This seems like more of an internal issue within your QA/Dev team and not a testrail issue? As I don’t run/kickoff the automation tests either, they are triggered whenever a dev pushes new code to our build servers.

Does that help at all?


#5

Hi all,

One popular approach is also to store the case IDs together with your automated tests and then use these IDs to add results via add_result_for_case (or the bulk version add_results_for_cases). The case IDs are static and do not change over time. The basic idea is to start test run dynamically, remember the returned run ID and then add results based on the (dynamic) run and (static) case IDs.

I hope this helps!

Cheers,
Tobias


#6

There are also many libraries out there to help you out communicating with the TestRail API. For example, on NPM https://www.npmjs.com/browse/keyword/testrail

Good luck!


#7

Thanks, Kevin! And another useful link:

Cheers,
Tobias


#8

Hi Tobias,
Is this the recommended way to integrate an existing automation framework (with it’s test scripts) into TestRail ?

Keep in mind that we are targeting an end-to-end integration like this:

If so, I have a couple of more questions:

  • what do you put in the testrail test cases since the automation test scripts contain the info

    • if you do put steps, expected results in testrail, how do you keep them in-sync with the actual test scripts
    • we were thinking of leaving testrail test cases essentially empty with a small description
  • in an automation-driven workflow where you have both devs/QA writing automation scripts, does Testrail become solely a repository of results ?

    • currently, for our automation tests, the “test run” is constructed in Jenkins by inspection of the automation folder in source code, not Testrail

any advice would be appreciated.

thanks,

-Rami


#9

Hi Rami,

Yes, typically the cases for automated tests are very lightweight in TestRail and usually just contain the title and maybe a small description. For automated tests, TestRail primary use case is in fact to store results and create reports based on the results (and the automated tests/cases). We found that it’s great to have both automated and manual/functional test results in the same tool, even if the automated tests are executed elsewhere and TestRail is “just” used for recording the results and reports.

As mentioned above, we recommend adding a test case in TestRail for each automated test and then adding results using this (static) case ID via add_result_for_case and add_result_for_cases (bulk version). We have ready to use API bindings for TestRail which you can find here:

http://docs.gurock.com/testrail-api2/start

For example, here’s the API binding for Java which is likely a good fit for the Jenkins integration:

http://docs.gurock.com/testrail-api2/bindings-java

You can also find a very good overview of how other teams implement the test automation integration on our blog here:

https://blog.gurock.com/testrail-test-automation-test-management/

I hope this helps!

Cheers,
Tobias


#10

Hi folks,

We’ve just started evaluating TestRail and this thread addresses some mental blocks I’ve been struggling with - thanks! We currently have several hundred tests which are run (NUnit) as part of or CI process. After each suite is completed we are using ReportUnit, which parses TestResult.xml to archive the results. Are there any thoughts regarding adding test results in bulk vs. after each test is run?

What do folks generally do for data-driven tests? We have a couple of test fixtures (each containing about 15 tests) which take scenarios from a spreadsheet (which has a variable number of rows). Create a new test run for each iteration?

Also, hard-coding the test name to the ID seems somewhat fragile to me (not to mention a lot of initial data entry for our existing tests). Adding a lookup by test name to the API would be great (and I see someone has already done something like this: http://blog.vevo.com/cucumber-and-testrail-salad-with-ruby-dressing/). I’d be interested in hearing from existing users how they’ve handled this. And as long as I’m requesting features: Ability to post back screen shots of failing tests would be fantastic.

Thanks!

  • John

#11

Hi John,

Thanks for your posting, great to hear this thread is useful :slight_smile:

We usually recommend starting a test run for each independent collection of tests (i.e. that can be and is executed independently). The best workflow is to run your tests (outside/independently of TestRail) and then afterwards a) create a new test run and b) add the results (ideally with the bulk API methods in one step). It can also make sense to reuse the same test run multiple times over time and this will result in a smaller TestRail database and can make sense if you have many automated tests that are executed regularly.

Regarding the IDs: we would recommend using the IDs as this would be much faster than looking up the IDs every time you run your tests. The typical workflow would be to store the test case IDs together with your automated tests (e.g. as code attribute or similar). The title/name of the test can change but the IDs would be static so using the IDs would be the preferred option. The IDs can be used directly for the case selection or adding test results via add_result_for_case or add_results_for_cases (bulk version):

http://docs.gurock.com/testrail-api2/reference-results

Cheers,
Tobias


#12

Excellent - thank you! We already group the tests by functional areas/product type and do some post-processing on the results. Sounds like this is the preferred method rather than adding results individually.


#13

Hi John,

Great to hear this helps. Yes, we can recommend the bulk methods and this would be more efficient and faster (on both sides) when adding results (so your automated tests would also finish faster).

Cheers,
Tobias


#14

Hi Tobias,

We’ve taken a different approach with our TestRail integrations: we store comprehensive data on our tests in TestRail. In fact, we use it as the “one source of truth” for BDD. We do that by syncing each BDD scenario with a Testrail test case, storing the up-to-date Cucumber steps in TestRail each time the test suite is run.

Projects have their own subheadings in TestRail, and there is a strict 1:1 correspondence between a test case and a Cucumber scenario (or JUnit test.) It is a test error to have a test for a project without a TestRail case id under the same project’s subheading, a case id under a project subheading without a corresponding (automation) test, two tests referencing the same caseId etc.

The TestRail integration is new, so we had not yet had the experience of using it on a large project before my contract finished.

Cheers,

Nick


#15

Hi Nick,

Thanks for sharing this, that’s appreciated!

Cheers,
Tobias