Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Wanting to add test cases and test results simultaneously to a test run using API?


#1

Hi,

I would like to add both a set of test cases and associated test results simultaneously to a test run using the Testrail API. Is there a way of doing this without doing? It seems I need to first add all the test cases to the test run, and then run another loop through to add the results afterwards?

Basic flow is:

  1. Establish connection with testrail (fine)
  2. Create a regression run place holder (fine)
  3. Obtain the regression run ID programatically (fine)
  4. Loop through the log and extract testrail testcase ID and result (fine)
  5. Add testrail testcase using ID and result to Testrail (where things go wrong as is expecting testcase already there)

I’m using JAVA and the APIClient library. Appreciate any assistance thanks.

Mo


#2

Hi Mo,

Thanks for your posting. If you use the include_all = true option for runs, TestRail would automatically add new cases to the runs and you wouldn’t need to do this manually. You can then simply add the case to the repository (step 1) and then submit new test results using the case and run IDs (step 2).

Cheers,
Tobias


#3

Hi Tobias,

Apologise for the confusion in my question, I just realised it didn’t make that much sense :stuck_out_tongue: (typing on a Friday afternoon doesn’t help!). What I wanted was to add both the testcase+result of that test case in one simultaneous call to the api, rather than two separate calls.

I’ll describe what i’m doing and paste the code below. I have an AutomationRun.log file which will contain the testrail ID + result of the test, marked as ‘PASS’, ‘FAIL’. I loop through this file and add those two fields to hashmap ‘testCaseResults’ , while simultaneously adding the testrail ID to another List ‘testCases’. You might ask why am I doing this:

  1. When I call sendPost “add_run”, I’m adding the list of testcases (i.e. testrailIDs) as a second parameter to hashmap ‘data’ and calling data.put(“case_ids”,testCases).
  2. I will be using testCases list as a reference to get the result value later on when referencing hashmap testCaseResults and calling “add_result_for_case”.

What would be ideal is:

  1. Rather than call add_run and add_result_for_case, just have one API call so that I can create+add both run+testcaseID+results
  2. alternatively if that isn’t possible, i’d like to eliminate the use of 2 hashmaps and just use the one?

So far i’ve had great success with this code, but i feel it isn’t good coding practice.

Here is the script:

String testRailURL = “{our testrail URL}”;
_ String testRailUser = “{our testrail user}”;_
_ String testRailUserPassword = “{our testrail password}”;_

_ int i;_
_ int testID; //this will be updated dynamically when reading through logs_
_ int suite_id = 133;// Suite ID of Automation Regression Suite_
_ int milestone_id = 15; // ID of Regression Milestone in Testrail_
_ int assignedto_id = 12; // ID of TestRail User_
_ int project_id = 17; // ID of WCMS Testrail project_

_ List testCases = new ArrayList();_
_ HashMap<Integer,Integer> testCasesResults = new HashMap<Integer,Integer>();_
_ Map data = new HashMap();_

_ APIClient client = new APIClient(testRailURL);_
_ client.setUser(testRailUser);_
_ client.setPassword(testRailUserPassword);_

_ data.put(“suite_id”, suite_id);_
_ data.put(“milestone_id”, milestone_id);_
_ data.put(“name”, "Regression Run " + TestCaseBaseSauceLabs.buildDate);_
_ data.put(“assignedto_id”, assignedto_id);_
_ data.put(“include_all”, false);_
_ data.put(“description”, "Regression run build : " + TestCaseBaseSauceLabs.buildDate); // Description for Regression run._

_ CSVReader reader = new CSVReader(new FileReader(“src/test/resources/data/AutomationRun.log”),’,’);_
_ // First loop add all tests to test run_
_ String [] nextLine;_
_ while ((nextLine = reader.readNext()) != null) {_

_ testCases.add(Integer.parseInt(nextLine[2])); //Add testcase ID to testCases[] (Testrail tcID is at position [2] in logs)_

_ switch (nextLine[4]) { //Test results are at position [4] in logs. Add ID and results to testCasesResults[]_
_ case “PASS”:_
_ testCasesResults.put(Integer.parseInt(nextLine[2]),1);_
_ break;_
_ case “FAIL”:_
_ testCasesResults.put(Integer.parseInt(nextLine[2]),5);_
_ break;_
_ default:_
_ testCasesResults.put(Integer.parseInt(nextLine[2]),4);_
_ break;_
_ }_

_ }_

_ data.put(“case_ids”,testCases);_
_ JSONObject c = (JSONObject) client.sendPost(“add_run/” + project_id, data);_

_ data.clear();_
_ JSONObject r;_

_ for(i = 0; i < testCasesResults.size() ; i++) {_

_ testID = testCases.get(i);_
_ data.put(“status_id”,testCasesResults.get(testID));_
_ r = (JSONObject) client.sendPost(“add_result_for_case/” + c.get(“id”) + “/” + testID, data);_
_ }_

Here is what an entry in AutomationRun.log looks like. 21420 is the testrail testcase ID.

[2016:07:209 17:03:58],SmokeTest,21420,homePage_SmokeTest_ArticleLandingPage,PASS

Hope this clarifies things a little more?

Thanks and regards,
Mo


#4

Hi Mo,

Thanks for the additional details. Instead of looping through your cases and calling add_result_for_case for each of them, you can also look into using the bulk-add API method add_results_for_cases instead:

http://docs.gurock.com/testrail-api2/reference-results

This would reduce the number of API calls to just two (add_run and add_result_for_case) and would be much more efficient than calling add_result_for_case separately per case.

I hope this helps!

Cheers,
Tobias


#5

Hi Tobias,

Thanks for pointing me to this. I’ve nearly finalised a solution and copied below the solution i’m basing it on.

The output is in the correct format I want it to be, but as is obvious I have a bug where the last ‘put’ (or ‘add’?) is overwriting the previous entries, resulting in duplicated entries (see below). Once I sort this issue I can call 'add_results_for_cases; fine :slight_smile:. Thanks!

  JSONObject obj = new JSONObject();
  JSONObject js = new JSONObject();
  JSONArray results = new JSONArray();

  js.put("case_id","12345");
  js.put("status_id","1");
  results.add(js);

  js.put("case_id","78910");
  js.put("status_id","5");
  results.add(js);

  js.put("case_id","1569");
  js.put("status_id","4");
  results.add(js);

  obj.put("results",results);
  System.out.print(obj);

Output:

{“results”:[{“status_id”:“4”,“case_id”:“1569”},{“status_id”:“4”,“case_id”:“1569”},{“status_id”:“4”,“case_id”:“1569”}]}


#6

Hi Mohammed,

Yes, you would need to add three independent results (otherwise, js always points to the same object), for example:

JSONObject js1 = new JSONObject();
js1.put("case_id","12345");
js1.put("status_id","1");
results.add(js1);

JSONObject js2 = new JSONObject();
js2.put("case_id","78910");
js2.put("status_id","5");
results.add(js2);

JSONObject js3 = new JSONObject();
js3.put("case_id","1569");
js3.put("status_id","4");
results.add(js3);

I hope this helps!

Cheers,
Tobias


#7

Works like a charm! Managed to simplify the code a lot more :smiley:

Thanks Tobias!


#8

That’s great to hear and you are welcome, Mohammed :slight_smile:

Cheers,
Tobias