Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

A lot of untested test cases after automation run

Hi all,

I’m having an issue with large test runs(over 60 test cases or so).
After some test cases are being marked as tested(Failed/Passed according to test itself), around one-third of the test cases are set as ‘untested’.
It happen despite the fact I send an api call with all the relevant test cases and all the test result as I expect them to be.

Is there any limitation on how many test cases can be passed in one ‘add_results_for_cases’?
If not, is there any way to check why I encounter this issue?

Thanks!
Yoav

Hi Yoav,

Thanks for the post! The add_results_for_cases API method does not currently have a limit on the number of results which can be added with a single request.

If you are experiencing issues with individual tests not receiving results, we would recommend debugging your API script and request contents to determine if there is a specific point on the request which is failing or if there is another underlying issue in the request.

We wouldn’t be able to directly debug this ourselves, however, if you are able to find a valid entry in your request which is being ignored by TestRail, you can send this to contact@gurock.com so we can review this in greater detail and go through some troubleshooting steps with you. In this case, please be sure to send the actual data being sent in JSON format.

Regards,
Jon

Out of concerned curiosity, how do we know if we’re hitting max API requests? The enterprise version boasts “More API calls resulting in faster execution” which suggests the non-enterprise version supports less. How much less and how do we know where we stand?

Hi Alex,

If you exceed the API limit for TestRail Cloud, you will receive a 429 response code and an error message indicating how many seconds you should wait before retrying the request. The current API limit for Standard TestRail Cloud accounts is 180 requests per minute per instance.

If you are automating your tests via TestRail’s API, we would typically recommend adding results in bulk (via the add_results or add_results_for_cases methods), and using other bulk methods where possible as this would reduce the number of requests as opposed to adding/retrieving each result individually with a separate requests. Bulk methods can help with rate limit issues on TestRail Cloud.

Hope this helps,
Jon

1 Like

Thanks for the response!
Apparently there was one TC in the results list I tried to send that doesn’t exist, so all the TCs that followed it were marked as untested because of the error.

This one is on me :slight_smile:

Regards,
Yoav

1 Like

@Yoav_serrouya have you thought about automatically creating the TestRail test case if it does not yet exist?