Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Getting successive 429 error codes since friday


#1

Hello,

We are getting progress reports via REST API and since friday our instance constantly replies with Error 429. We have tried long wait times but the issue persists. Even on the login request it sometimes is responding with 429.


#2

DITTO! Our group is suffering failed jenkins builds as well due to this. And we have 429 handling in our binding.


#3

Same here, we have multiple teams blocked with this issue, we are having to disable test rail on our test suites, is there any ETA of a fix?


#4

Having the same issue since Friday. All test results are randomly failing without any updates from us.


#5

Hi guys, we are having exactly the same issue. Jenkins builds are failing due to Rate Limit issues. Is there any update on this?


#6

Hi guys, I got a response from TestRail:

Hello Daniela,

Thank you for your email! We have been evaluating and monitoring API usage for all of our TestRail Cloud customers and established an API rate limit of 60 requests per minute per instance. The newly implemented rate limit and response information will allow teams to continue to use TestRail’s API effectively while allowing thousands of API requests per day.

This limit isn’t an issue for most customers in practice, however rate limiting is a widely used technique and we must implement rate limiting going forward to protect the larger customer community as TestRail continues to grow.

If your team plans to add thousands of test results per hour, it’s sometimes a better idea to use a local installation of TestRail instead. With TestRail Server, there wouldn’t be a limit imposed as this would just be managed by your web server/PHP configuration, so you can configure this to support your specific use case if needed.

If you are automating your tests via TestRail’s API, we would typically recommend adding results in bulk (via the add_results or add_results_for_cases methods), and using other bulk methods where possible as this would reduce the number of requests as opposed to adding each result individually with a separate add_result request. Bulk methods can help with rate limit issues on TestRail Cloud (and with general performance related issues on TestRail Server as well).

If you do run into any rate limiting with your requests, you should get the 429 along with the message “API Rate Limit Exceeded - 60 per minute maximum allowed” and a “Retry-After” header. You can then parse the “Retry-After” header to automatically retry the API request after the specified number of seconds.

If you’re receiving any other errors when submitting API requests, please reply back with the errors you’re seeing and if possible details about the volume of requests and the types of requests themselves so that we can better assist you.

Kind regards,
Jon Reynolds


#7

Basically TestRail team decided to implement a limitation on the number of requests via API to 60 per minute which I think is extremely low! So now because of this silent change we need to refactor all our tools that were using the REST API. Current mood in house is - When do we abandon TestRail?


#8

Was there any customer notification regarding these changes? It would seem irresponsible to those customers who do rely on result uploads to just make the change silently and wait to see who complains (and then recommend they pay more money for an on prem instance.) We’re presently restructuring our automation scripts so our CI flow is unblocked from pushing to production, thanks to this change. If this is the lack of communication we can expect from Gurock going forward, it’s safe to say we’ll be looking into other solutions.


#9

We are shutting off TestRail (sigh) until our implementation sends only 1 add_results transaction per build per parallel thread. It also means no more real time updates. With prioritization, we may not be saving results anymore for a few weeks.


429 - To many requestion- > Question how many requests before this procs?
#10

We are having the same issues, but we are nowhere near 60 requests per minute. We started getting 429 Errors after our Jenkins job ran approximately 50 tests over a period of 30 minutes.


#11

Hi all - thanks for your feedback on the recent API rate limit change.

Ideally there should have been some kind of advance notification of the change, to provide teams with the opportunity to refactor tooling ahead of time. I’m sorry there wasn’t, and we’ll definitely make sure that future changes get communicated so they don’t cause problems like you’ve all described.

Per the customer support response quoted above however, the limit is not an issue for most customers. Since we implemented the rate limit we have seen a sharp decline in the number of 429 responses across all our customer instances. Evidence for us that the change is having the desired effect of improving the experience of all our customers.

Clearly with a change like this, some teams will experience more challenges than others. If the increased rate limiting is still causing you problems, please do reach out directly to our support team via contact@gurock.com, where we’ll be happy to work with you directly (increasing the limit temporarily if necessary) until you’ve refactored your tooling successfully.

Cheers,
Simon


#12

Currently we are seeing multiple 429 responses even when I have implemented the wait times coming from the header. Can you imagine a team with more than 150 users and more than 20 active projects working with a limitation like this? It is inimaginable to me that you believe this is acceptable. I consider this breach of contract conditions because you are not maintaining quality for your customers and performed a change without any notification that severely impacts their work. This limitation should take into account the number of users per instance providing better quality of service to a client that pays you more for that service.


#13

This is ridiculous. We are now completely blocked form releasing, until we refactor our automation. I am very unhappy and will start looking into new test management tools. The lack of communication really does not sit well with me.


#14
  1. Communication issues
    Because of lack of communication, we have 2 devs working on it for a whole day to check where the failures are coming from for the nightly runs. And our nightly runs have less than 20% since Saturday and the business is putting vast amount of pressure on us.
  2. Api limit
    We have multiple browsers running hence it updates test cases roughly about the same time. We are using a third party gem BBC res (https://github.com/bbc/res) and it updates one case at a time (not bulk). Now we are dependent on them to update (which is not looking sooner) and Testrail to find a sensible solution
    Suggested solution is to look for other reporting tools quickly to display the nightly runs results. We liked Testrail. but :frowning:

Thanks


#15

Fortunately we embedded a Testrail enable/disable toggle within our solution by requiring a TESTRAIL_USERNAME environment variable to be set, and if the credential is null or blank, Testrail is skipped altogether.


#16

Is it just me, or is everyone having the same issue getting any communication established with Gurock?


#17

Not just you unfortnately…


#18

I have called multiple times, I have emailed multiple times, absolutely no response.


#19

I am torn between hosting TestRail in house on our servers, or leaving altogether. Given the lack of communication, I am inclined to do the latter. Any suggestions of a good Test Management System?


#20

Hi all,

We’ve heard your feedback and have now increased the API rate limit to 180 requests per minute for all TestRail Cloud customers. This will help customers experiencing challenges with the previous API rate limit. Please note that the API rate limit has been implemented to benefit the TestRail Cloud community as a whole, as this helps to protect the infrastructure from degradation due to excessive API burst usage. That said, we’ve built-in methods (e.g. via the retry-after header) that allows you to handle this within your API development. We will continue to monitor the API usage and listen to customer feedback to help guide our policies. If your team is still running into issues with the new limit, please reach out to us directly at contact@gurock.com so we can work with you to help reduce the impact this has on your team (in case it will take longer to address this in your API scripts).

I hope this helps!