Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

How to filter testcases in the API


#1

We are attempting to log results to Testrail via the API (though automation). Our testruns typically have over 5k testcases that get run.

The question is how to get a testcase-id from the list of testcases that are available (that we can then add to the testrun).

We first tried to use “get_case” and iterate through each testcase in a Suite, matching the returned values to a set of criteria; basically implementing Filters though on a post-processing basis. However, depending on when the testcase was found it would cause 1N requests to Testrail in a very short amount of time; something that Testrail didn’t appreciate. So while this worked ok when our Testrun was a few 100; it started failing when it got much bigger.

Next, we tried to use “get_cases” which essentially does the above except goes out and grabs the entire suite. We would then filter them down based on some set of criteria and create our testplan. Again, for the first 1000 or so no problem. But as our testruns got to be 5k it started taking a long time (as the testcase returns ALL the information about each testcase). And around 10k testrail again stopped responding (giving us an Internal Server Error no less).

So … that leaves us with few options from the standpoint of the API. And I went looking to see how I could request testcases based on a filter or some set of criteria that Testrail could process and return us testcases only applying to that filter. I did not find any such API.

My question then is; given a Testsuite of 1000’s of testcases, how can I use the API to only grab a subset of those and add them to a Testrun?

IE: create a testrun of all testcases marked as “automated” (via the API)

Thanks,
–jim


#2

Hi Jim,

Thanks for your posting! Starting with TestRail 4.0, get_cases includes request filters and you would be able to get all automated tests via the type_id filter:

http://docs.gurock.com/testrail-api2/reference-cases#get_cases

GET index.php?/api/v2/get_cases/1&suite_id=2&type_id=3

(assuming your Automation type used ID 3)

This would only return test cases that are set to Automated and you can then easily build your case selection for new test runs/plans based on this.

Regards,
Tobias


#3

Should have mentioned that I knew about the various “Request Filters” that are in place. But you are making a huge assumption that is where people keep whether a testcase is automated or not. We use that field to describe the Type of testcase it is in relation to System, Performance, UI, or whatnot. It would be a waste of the field to just use it for “manual” or “automated”.

So we have a custom field (“testcase_state”) that we use for this.

In addition, there are some other fields we would like to search on rather than have the entire suite returned to us – and none of them are those standard Request Filters.

Then this is compounded by the fact that the entire testcase record is returned to us – we can’t specify the information that we need. So lets take an example, a very simple testcase (no steps, no real informaton, just a title) returns 605 bytes. This is probably the smallest record possible (and many records are 2-3 or more times the size).

If I have 5000 testcases I am dealing with (we are) that means the json response has to return 3,025,000 bytes. I only need the information from a very small portion of that (specifically the ID and a few custom fields). But it has to download the entire thing.

And as I said before, this doesn’t work as we get those testcases up in the 30k range (30k on simple testcases would return an over 18m json response).

Basically, this doesn’t scale. We definitely need a way to either have the response return only certain fields (based on REST parameters I would guess) or give us access to more fields (such as custom ones) when using the Request Filter so that I can reduce the number of responses.

Of course, best of all would be both.


#4

So… any further comments?


#5

Sorry for the delayed response, I didn’t get an automatic notification for this thread.

It’s already planned to add support for custom fields as part of the request filters and I’m happy to add another vote to this feature request. There’s no currently no direct workaround for this if you don’t use the system fields and I would recommend reading the entire suite and then filtering the cases on your side. We would also like to look into limiting the results to just the fields you are interested in (so you could omit the large text fields from the response, for example) and this would also help a lot in this situation. We are happy to look into both features for a future API version.

Regards,
Tobias


#6

We would like to see the ability to search on arbitrary fields as well.


#7

Added your +1 to the feature request list, thanks for your feedback on this!

Regards,
Tobias


#8

Please count 1 more vote for me for this feature.
Thanks,
Vincent


#9

Added to the list as well, thanks Vincent!

Regards,
Tobias


#10

+1 more vote for being able to filter on custom fields for get_cases!


#11

Added to the list, thanks Kris!

Regards,
Tobias


#14

i like also to see the possibility to filter on customised fields, so please add another vote


#15

Added, thanks :slightly_smiling:

Cheers,
Tobias


#16

This would also be useful for our team.

+1 for me.


#17

Happy to add another vote, thanks Chris :slightly_smiling:

Cheers,
Tobias


#18

+1 for me.
It is also very important to be able to search with the indication of custom fields.


#19

Please add another vote!