Subscribe Join 180,000+ subscribers and receive articles from our blog about software quality, testing, QA and security. Subscribe

TestRail 6.7 Released

We’re pleased to announce that TestRail 6.7 has been released to our hosted TestRail Cloud customers, with various changes and enhancements designed to improve performance and scalability for all TestRail cloud customers.

In brief:

  • We’ve introduced a new Data Storage Policy for TestRail Cloud customers that will take effect on February 1st, 2021 , along with new admin features to help monitor and manage your storage usage.
  • We’ve made some small changes to some data export and reporting settings to reduce unintended impact on performance.
  • We’ve added pagination to certain pages of the web app and all bulk API methods to more gracefully serve customers with large amounts of test cases and historical testing artifacts (results, runs, milestones, etc). Customers who currently use any bulk TestRail endpoints must switch to the new paginated bulk API endpoints by February 26th, 2021 to avoid any interruptions in service.
  • We’ve added a 90-day password expiration policy for users that have a Gurock Customer Portal account for billing and license management. This does not affect any users of the TestRail app .

Data Storage Changes

The biggest change for this release is the introduction of some adjustments to how data storage is currently reported in hosted TestRail Cloud instances, and what reporting on data storage means for customers going forward.

Historically, we haven’t reported on data storage for our cloud instances at all! And generally, that hasn’t been too much of a problem, since for most customers our out-of-the-box storage plan has been perfectly adequate for both their (and our) needs.

To be completely clear:

  • Our hosted TestRail customers need to be able to store their testing data (test cases, tests, runs, plans, etc., along with any associated attachments and generated reports).
  • Our hosted TestRail customers also need to be able to export that data out of TestRail, should they wish to transfer it, or back it up locally.
    We (Gurock) should be able to back-up hosted instances on a daily basis so we can restore them to a prior state if needed.

Unfortunately, some (less than 1%) of hosted TestRail Cloud customer instances have grown too large for some of these operations to be carried out safely, successfully, and without impacting other customers hosted across our cloud infrastructure. While we continue to make performance improvements in the platform itself, we had to think of a way to change our current storage model and policy, such that it provides all our customers with the best possible TestRail experience.

As a result, in TestRail 6.7 we have introduced a data storage policy and a number of features in the TestRail app to help your team monitor and manage your storage.

Data Storage Policy

For more information about our Data Storage policy you can refer to the full policy posted here, but here are the highlights:

Starting February 1, 2021:

  • Professional TestRail customers will get 50GB data storage included in their cloud subscription.
  • Enterprise TestRail customers will get 500GB data storage included in their cloud subscription.

As part of the improvements included in this release, we have migrated TestRail Enterprise instances to a segregated, more resource-intensive cluster to ensure high performance even with larger storage amounts being used.

We recognize that for some of our Professional customers, 50GB may not provide enough data storage space for their needs. So we have introduced some further billable storage options, billed at $10 per month for 25GB increments up to a maximum of 200GB.

Basically what this means is that, if you go over your Professional allowance, we will charge you the additional $10 per month for each 25GB increment, based on the peak storage amount reached during the billing period – up to the maximum allowable 200GB. This charge will be added to your regular monthly bill. If you reach 200GB, you have the options to:

  • Reduce your data storage footprint by carrying out some remedial actions (refer to our TestRail Data Management documentation).
  • Upgrade your subscription to Enterprise, thereby bumping your allowance to 500GB.
  • Export your data to an on-premise installation of TestRail.

For both Professional and Enterprise tiers of TestRail, if you reach the maximum allowable storage capacity (200GB and 500GB respectively), we will notify you about the overage and help find a solution to reduce your storage footprint. If you do not take any actions to reduce your storage, some features may be restricted so as to prevent their impacting performance in our cloud infrastructure:

  • Data can’t be exported from the administration console (CSV & XML test case exports are allowed still).
  • Additional custom fields can’t be created.
  • Additional attachments can’t be uploaded.

Hosted customers will be able to see what their current data storage looks like in the TestRail Administration > Overview console:

Administrators will also be able to review their current data storage footprint in the new Data Management area:

You’ll also find the enhanced Data Export feature there too, so let’s talk about that.

Data Export Changes

We’ve made a few changes to the export function in TestRail. As of 6.7, when scheduling an export you’ll need to specify up-front which version of SQL you want to use: MySQL or MSSQL.

Additionally, unless you check the boxes to include either reports, attachments, or both, they’ll be excluded from the export by default.

We’ve made these changes for the same reasons as implementing the data storage policy. Carrying out those exports and including all of the information by default resulted in performance risk across our cloud infrastructure. Reducing the size of those exports by keeping them to a single SQL version and only including explicitly requested data helps us to reduce the risk of performance degradation for all our hosted customers.

Pagination & API Changes

Pagination is another big feature for our scalability release, affecting both the TestRail UI and the TestRail API.

Pagination in the UI

For the UI, we’ve implemented pagination on all of the pages that could be impacted by attempting to load a large number of test cases, tests, runs, and other TestRail entities by default. The full list of pages where pagination has been implemented is here:

  • /runs/view/
  • /tests/view/
  • /cases/results/
  • /runs/overview/
  • /suites/runs/
  • /suites/overview/
  • /plans/view/

TestRail administrators can control the pagination level across those pages using the Pagination Limit settings in the Administration > User Interface console, shown below.

Pagination in the API

We’ve also introduced pagination for bulk requests in the API. In the future, when you make a bulk request using a TestRail API endpoint such as get_cases, you’ll see that the response structure has changed so that it returns an object with additional pagination fields and an array of up to 250 entities.

All bulk API endpoints will contain the new structure and the pagination links. If your request results in a response that exceeds the maximum number of objects allowed, you will receive the first 250 of them and a link to the next set of results in the response body.

 {"offset": 0,
     "limit": 250,
     "size": 250,
     "_links":{
     "next": "/api/v2/get_cases/1&limit=250&offset=250",
     "prev": null
     },
     "cases":[
         { "id": 1, "title": "..", .. },
         { "id": 2, "title": "..", .. },
         ..
         ]
     }

You can apply an offset to your request, or further limit the number of entities returned by adding offset and limit parameters to the request URL:

GET index.php?/api/v2/get_cases/:project_id \ &suite_id=:suite_id limit=:limit &offset=:offset

The full list of bulk API endpoints that include this change are below:

  1. get_cases
  2. get_runs
  3. get_results
  4. get_tests
  5. get_results_for_case
  6. get_results_for_run
  7. get_plans
  8. get_projects
  9. get_sections
  10. get_milestones
  11. get_history_for_case
  12. get_attachments_for_case
  13. get_attachments_for_run
  14. get_attachments_for_plan

6.7 API Beta Period

We recognize that it may take some time to adjust to these changes. We will have therefore released them in beta, so that API users have sufficient opportunity to make any necessary changes to code and integrations which leverage the endpoints.

If you currently use one of the bulk API methods enumerated above, you must switch to using the new paginated API method described above before February 26, 2021 .

In our TestRail 6.7 release, the API will use the old API requests/responses for all the endpoints above.

If you want to try out the new API, you can do so by adding the following header and parameter to your API requests:

--header 'x-api-ident: beta'

After the beta period has finished, the updated endpoints will be used for all responses.

Further information about the changes can be found in the TestRail API documentation.

Report Changes

The 6.7 release also contains some minor changes and a fix to TestRail reports.

For performance reasons, we have removed the capability to select “all” for the specific report dimension. For example, when running the Comparison for Cases report, you won’t see the “All Test Runs” radio button any longer.

We’ve made this change for similar reasons as the other 6.7 features. On a large hosted instance of TestRail with a great number of runs, generating a report for all runs by default was a somewhat risky approach. Executing that report could potentially take a long time, and have a performance impact on the hosted instance while it was being generated. We have therefore removed “all” as a default option for any reports which used it.

Of course, if you really do want to report on “all” for a specific dimension, you can certainly do so still, by setting an appropriate filter.

Oh, and we also fixed a bug to prevent reports being endlessly rerun in the event of a failure. Because, you know, performance. :yum:

Updates to the TestRail Customer Portal

Finally, to improve security for all TestRail customers, we have introduced a new password 90-day expiration policy for the Gurock Customer Portal accessible at https://www.gurock.com/portal/. This password expiration policy will not affect any app users; it only applies to users named as primary, technical, or billing contacts for TestRail cloud subscriptions or server licenses in reference to their customer portal account.

Getting TestRail 6.7

You can start a 14-day free trial of TestRail here (cloud or server): https://www.gurock.com/testrail/trial/

If you want to create a subscription for TestRail Cloud, you can do so from within TestRail via Administration > Subscription . Or, if you want to order TestRail Server licenses you can do so from our website here: https://secure.gurock.com/customers/shop/annual/purchase/

Registered customers can download the full version from our customer portal: https://www.gurock.com/portal/

TestRail Enterprise

If you’d like to learn more about increasing your storage limit or enabling project-level administration, single sign-on, or any of the other features on TestRail Enterprise, please email us for a trial or quote via contact@gurock.com, or you can use the contact form here: https://secure.gurock.com/customers/support/.

Not sure which TestRail plan you’re on? Reach out and we’d be happy to help.

Updating to TestRail 6.7

TestRail Cloud

TestRail Cloud instances are automatically updated to the latest version. You can check your version via the TestRail Help > About TestRail menu item.

TestRail Server

TestRail Server & TestRail Docker versions will be available for download soon.

We are aware of 2x issues with the release currently:

  • API responses currently limited to 250 max objects
  • Some pagination bugginess resulting in some data not appearing on expected pages

We are working on fixes for both issues, and expect to release an update very shortly.

Thanks for your patience in the meantime,
Simon

My pagination issue is slightly different to that mentioned in this post.

I have one Test Run listed under my Test Run and Results. I expect to see no pagination as there are no other pages.

The pagination displayed is:

« Prev 1 2 3 4 5 6 Next »

The additional pages are blank.

It seems that there are more changes for TestRail Cloud version. Besides API changes of v6.7, is there any changes that TestRail Server customers need to be aware of?

Hi,

It states that the option to choose all Test Runs in the comparison for references report is GONE. but its still there on the screen. The issue is that if you choose that then there is an error.
Why have radio buttons which dont work???

image

Oh, and we also fixed a bug to prevent reports being endlessly rerun in the event of a failure. Because, you know, performance. :yum:

do you know if this bug fix will also fix the issue of where I still have some reports that are still running?

So after reviewing this a little deeper there are no more “BULK APIs” in reality since the bulk APIs are subject to the same 250 limit as the other APIs?

Also, this now increases the need to make more API calls. If I need info on 1000 test cases I went from having to make 1 call to having to make 4 calls.

Will you be increasing the number of call we can make in a minute? This would go along way to smoothing things over.

I understand what you are doing here, trying to speed up performance but you are doing it at the cost of your more advanced uses.
Yes, we could go from Pro to Enterprise but if we are going to invest that kind of money we would consider inviting in engineering resources to bring something like TestLink online.

@jeffrey_zhang the pagination & API changes + report changes will be included in the server build when we release it. The data storage and export changes will not affect server installations.

Hi @vinodt - that sounds like a bug! Rest assured, we are dealing with it accordingly :slightly_smiling_face:

Hi @cah99 - our DevOps team recommends you get in touch with support, so we can take a closer look at this issue for you.

Please send an email with your instance URL or account details to contact@gurock.com, and we’ll be happy to help out.

@mmaliska - we can certainly review the rate limiting situation if the additional API calls are causing a headache for your team.

As you are no-doubt aware, the API changes have been introduced on a beta basis precisely so we can support customers during the adjustment period. As such, we’d be very happy to take on-board your feedback so we can make improvements where necessary.

Please feel free to reach out directly if you have some specific feedback you’d like to discuss in more detail.

Cheers,
Simon

@clairecoupland we’re still working on this one… :construction_worker_man:

Thank you, Simon, and I do apologize for being a little abrupt.
Just venting a little frustration.
Thank you for the quick turn around on the fix for the APIs, and we will start working on the changes on our end right away.

Kind Regards
Mike

No problem and no apologies necessary! :slight_smile:

Please do let us know if you have any feedback.

Hello all,

We have updated our GitHub repository with a set of API bindings which will utilize the x-api-ident header. This is a single line of code added to each binding and you can find the ‘beta’ set here:

Once the grace period ends in February 2021, either binding set should work, as we’ll simply ignore the x-api-ident header.

I hope this helps,
Jon

Hi, can you clarify if the changes to the API will affect the Server users too?
I take as a given that Server instances will have no limitation on storage capabilities (aside from the physical storage limit of the instance), but it is unclear if the API changes will also apply to us.
Thanks!

My issue is that tests under test runs are not even displayed. Page just says “Loading tests …” where the test cases should be. This only happens in a certain project. No issues on other projects.

Hi all, just a ping here, would be great to have some extra clarification. We are fast approaching the date for the API switch to pagination and I would like to make sure that nothing breaks at the next update cycle if this is to apply also to server instances.

Hi @a_ws - we plan on applying the same changes, with some additional configuration giving server administrators the capability to enable or disable paginated responses.

We are currently working on a bugfix release (6.7.1.x) which we will also make available for server customers. The server build will contain the same API functionality deployed in 6.7; the standard API response or the beta API response if the x-api-ident header is passed with the bulk endpoint request.

Server customers will benefit from an additional API config setting to our hosted instances in the 6.7.1.x release, enabling them to turn pagination on or off. With the current beta controls, what that looks like from a behaviour perspective is below:

  1. pagination disabled in server & NO x-api-ident header passed = standard API response body
  2. pagination disabled in server & x-api-ident header passed = beta API response body with ALL objects returned
  3. pagination enabled in server & NO x-api-ident header passed = standard API response body
  4. pagination enabled in server & x-api-ident header passed = beta API response body with max 250 objects returned

After the February 26th API cut-off point, any future versions of TestRail Server will return the new API response body. I.E., options 1 & 3 will no longer be available. However, server administrators will still have the capability to enable or disable the paginated response - either returning ALL objects in a bulk endpoint response, or enabling the paginated limit of 250 objects.

Hopefully that’s all clear. My apologies for the delayed response! :pray:

Simon

Hi Gurok team,

can you explain the new password 90-day expiration policy for the Gurock Customer Portal?
Does it mean, we have to log in every 90 days to change the password?

We actually have no reason to log into there on a regular basis as invoices are delivered by mail anyway. This only means an extra manual step we need to set a reminder for…

Thx
Detlef