Join 34,000+ subscribers and receive articles from our
blog about software quality, testing, QA and security.

TestRail API: add_plan_entry creates run with all tests from the suite


#1

Hi there,

I am integrating my automation tests with TestRail. What I do is I create a test run in a specified test plan by add_plan_entry api call. It creates a run with all the tests from the input test suite. However, I do not want all of them to be a part of run (rather only the desired ones). Is there a way I can just create a blank run initially and throw tests later with their results?

My flow is: check the annotation of a test. The annotation will have test case id from TestRail. However, if I have only 2 tests annotated and suite has 6, then the run has all six tests with 4 untested and 2 of them will have their results. This is confusing and I only want 2 tests in the test run.

Is there a quick solution for this? Something you guys have that I can use. Thanks for the help in advance.

Pratik


#2

Hi Pratik,

Thanks for your posting! Yes, you can change this with the include_all and case_ids attributes. This requires at least one test case initially but wouldn’t include all cases of a suite. That said, we usually recommend setting the case selection upfront when creating the test plan/entry if possible and this avoids lots of API calls just for changing the case selection.

I hope this helps!

Cheers,
Tobias


#3

Hi Tobias,

Thanks for your quick response. Let me share my addTestRun() method snippet:

Map<String, String> data = new HashMap<String, String>();
		data.put("suite_id", getTestSuiteId());
		data.put("name", testRunName);
		data.put("description", "This run is created by Automation Job");
		setAuthCredits();

		JSONObject obj = (JSONObject) client.sendPost(String.format(String.valueOf(uri.ADD_PlAN_ENTRY), getTestPlanId()), data);
	JSONArray arr = (JSONArray) obj.get("runs");
	for (int i = 0; i < arr.size(); i++) {
		JSONObject jo = (JSONObject) arr.get(i);
		testRunId = jo.get("id").toString();
	}

	return testRunId;

Here enum ADD_PLAN_ENTRY holds value of “add_plan_entry/%s”. Can you help me out on how do I modify this method so that I can use those include_all and case_ids attributes. Also, here getTestPlanId() just returns the test plan id that is given as a part of config constant.

Thanks, Pratik!


Using include_all, false and specifying testcases with add_plan_entry
#4

So, I understand you:

Here is my modified method:

Map<String, String> data = new HashMap<String, String>();
		data.put("suite_id", getTestSuiteId());
		data.put("name", testRunName);
		data.put("case_ids", "[31527]");
		data.put("description", "This run is created by Automation Job");
		setAuthCredits();

		JSONObject obj = (JSONObject) client.sendPost(String.format(String.valueOf(uri.ADD_PlAN_ENTRY), getTestPlanId()), data);
		JSONArray arr = (JSONArray) obj.get("runs");
		for (int i = 0; i < arr.size(); i++) {
			JSONObject jo = (JSONObject) arr.get(i);
			testRunId = jo.get("id").toString();
		}

		return testRunId;

Am I missing something like include_all set to false / data.put(“include_all”, false); …also I wish to create an empty run initially (by faking something for my purpose) and then feed all the desired/annotated tests with their results. Worst case scenario would be to have atleast one id as an attribute and then create run with it and use my same logic. Help is appreciated.

Pratik


#5

Hello Pratik,

Thanks for the update. The case_ids attribute would need to be passed as Java array or List object and the binding would serialize this to the expected format of the API. You would also need to pass include_all:

data.put(“case_ids”, new int[] { 31527 });
data.put(“include_all”, new Boolean(false));

If this doesn’t work, you might need to use a List object instead of an array instead.

An empty run is not supported but you can create one with just a single case. But as mentioned in my previous post, we would recommend setting the case selection upfront when creating the test plan/entry if possible. If you plan to change the case selection as you go through your tests, you would need to send an extra API call (update_plan_entry) in addition to adding the result (e.g. with add_result_for_case).

Cheers,
Tobias


#6

Hi Tobias,

Thanks for the information. Here is what I am getting:

TestRailApiException: TestRail API returned HTTP 400("Invalid or incomplete JSON string in API request.")

Here are my attributes:

 data.put("suite_id", getTestSuiteId());
 data.put("name", testRunName);
 data.put("case_ids", new int[] { 31527 });
 data.put("include_all", new Boolean(false));
 data.put("description", "This run is created by Automation Job");

Is my request incomplete? I believe its something else. You can find my whole method code in the comments above if needed.

Pratik


#7

Hi Tobias,

You are right,

data.put("suite_id", getTestSuiteId());
data.put("name", testRunName);
data.put("case_ids", Arrays.asList(31527));
data.put("include_all", new Boolean(false));
data.put("description", "This run is created by Automation Job");

Using of List fixed the issue but looks like I won't be able to add the result for my other tests since my run has only 1 test. It is very difficult for me to decide case selection upfront and so, I do not want to go in that direction unfortunately. What would be the order of making the API call to update_plan_entry? I do make a call to add_result_for_case for each of my tests that are annotated with test case ids from TestRail.

EDITED: Lets say I change my mind and will try to store test case ids upfront in an array and give that as an input to the case_ids attribute.

Here, data.put(“case_ids”, Arrays.asList(31527)); works. Also if I give more than one id like for eg. data.put(“case_ids”, Arrays.asList(31527, 31528)); my request works. There are cases where I want to give 100 test case ids and I would rather create a Constants.java class and store ids like:

public static final int[] CASES = { 31527, 31528, 33333,..... };

For above array case, when I do:

data.put("case_ids", Arrays.asList(Constants.CASES)); API throws TestRail API returned HTTP 400(“Invalid or incomplete JSON string in API request.”)

I believe its not the Java error and something with the Api. Other words, manually putting comma separated numbers worked but as global array (CASES in my example) fails to work.

Sorry for so many edits but I want to let you know whatever I tried.

Thanks!
Pratik


#8

Hi Pratik,

I would recommend the following approach and this has more control over the objects/classed used and how this is serialized:

List cases = new ArrayList();
cases.add(new Integer(31527));
cases.add(new Integer(31528));
..
data.put("case_ids", cases);

You could write a small routine which builds a List from your static CASES list and this should result in the correct JSON format the API would expect.

Cheers,
Tobias


#9

Hi Tobias,

That worked, I create a routine that creates a list for me. Now I am able to create a run with case_ids attribute.

You said that it is not recommended to create a run with single test and then post results for different tests in it. However, I guess its not possible? I got TestRail API exception of “no active test found”, since the run did not have tests to which I could post the results (speaking for apart from that 1 single case with which I created a run).

How should I fix this scenario? We have 1200 test cases in a test suite out of which 300 test cases can be associated with our automated tests and we have those corresponding testrail case ids placed in our code as annotations.

Also these 300 automated test cases that are mapped to test cases in testrail (via test their ids as java test annotations) are under different subsections. So, if I query the head section, I get empty array via API and so, using API for each and every subsection for say depth=1 or depth=2 is kind of pain (imagine 60 different subsections)

If I just include all 1200 test cases in my run while making a call to add_plan_entry, then more than 500 test cases will be “untested” and this will impact statistics of passed percentage and other pie chart useful stats.

Advise and let me know if I have confused you.

Thanks in advance,
Pratik


#10

Hi Pratik,

You can simply create a test run with a case selection based on your 300 automated test cases. You would then go through your automated tests and add results via add_result_for_case (via the run ID and case ID). This assume that you store the case IDs as part of your automated tests (the C### IDs in TestRail). An advanced version of this is to store your results temporarily and to use the bulk version called add_results_for_cases to add your results after your automated tests are completed.

I hope this helps!

Cheers,
Tobias