Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Re: API to update test results


#1

We have written automation test suites in robotium(junit based java framework for android applications). We’re planning to add the test results programmatically in testrail after running each test case.

In our setup, we have weekly releases and therefore will be running the tests weekly.For each week we create a test plan.

The API, add_result_for_case uses <run_id>/<case_id> - Needs Test run id and test case id.

However, looking around on testrail, we see that Test case id in “Test suites & cases”, and the test case id under test plan(same test case included in test plan) is different.

If i have Test case id T1. A test plan created including this test case has id C1 for the same test case.

So each time a new test plan is created a different id is generated for the same test case.

Now we are trying to automatically update the results in test plan.Do i need to manually give that test id(listed in test run) in code? Or is it sufficient to provide the fixed test case id (independent of what is listed in the test plan ).

Thanks
Sandhya


#2

Hello Sandhya,

Thanks for your posting. TestRail supports two methods on how to submit test results via the API:

  • add_result
  • add_result_for_case

The add_result method would indeed require that you have the list of test IDs (T123…) for every new test run that you start. However, the add_result_for_case method allows you to add results to tests by specifying just the test case + test run ID. TestRail would automatically look up the relevant test for this.

So you could store just the test case ID for each of your automated tests, and when you execute your tests pass the relevant test run ID, for example.

I hope this helps.

Thanks,
Dennis


#3

Thanks for the response Dennis.

However, looking up the API documentation at

http://docs.gurock.com/testrail-api/reference-results

We see that add_result API only requires test ID. There is no run id parameter available.

Please let me know if i am looking at the wrong reference . If not please point me to the API that you mentioned.

Thanks
Sandhya


#4

Hello Sandhya,

you need to use the add_result_for_case method instead, which takes the case ID + run ID as arguments:

http://docs.gurock.com/testrail-api/reference-results#add_result_for_case

I hope this helps.

Regards,
Dennis


#5

Thanks Dennis Got it.


#6

is there sample (maybe Java) code available to parse junit test results and submit them to TestRail?
don’t want to reinvent the wheel…


#7

Hello Asko,

I believe JUnit outputs XML files for the results and you would usually be able to use a simple XML library to load the file and you wouldn’t have to parse that yourself. Alternatively you can just submit the results directly from your tests.


#8

right, my point is that both these steps -

  • load XML results produced by JUnit
  • submit JUnit-model objects to TestRail

were probably implemented hundreds of times already, and maybe someone has sample code to share.
I see this code https://github.com/achikin/testrail-jenkins-plugin/tree/master/src/main/java/testrail/testrail
which can be partially reused.


#9

on submitting results directly from tests - does not sound like a good idea. this creates unnecessary coupling between tests and a 3rd party system. tests should not know or care where results are stored. it is not their level of responsibility.
the system that initiates test runs, though, can watch progress and submit results wherever it wants ether on-the-fly or when all finished.


#10

I don’t necessarily mean directly from the respective test method. But many systems allow you to add listeners for test results and you can easily submit test result from a central place for all your tests then. This can often be a good alternative to parsing/submitting results outside of the framework.


#11

right, makes sense. I created a simple Jenkins Plugin for this. thank you!


#12

is this the right way ?

JSONObject r = (JSONObject)client.sendPost(“add_result_for_case/R123/T123”, data);


#13

Hello,

Thanks for your reply! With the TestRail API, you wouldn’t need to include any prefix to the entity ID within a request. So in this case you wouldn’t need the R and T, and can just include the ID number without the prefix. Hope this helps!

Regards,
Marco