Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Baselines


#1

Thanks for this new release. You put a lot of effort into making TestRail even better, with lots of changes and improvements.

One of the features I was most curious about was the baseline support. Unfortunately, this feature does not work as I expected it. As it is implemented now, a baseline doesn’t seem to be different from a normal test suite. If a baseline is created, all test cases are duplicated and are independent from the master. Changes to the master are not reflected in the baselines. I expected that baselines are just as links to the master. As soon as a test case is changed for a baseline, it becomes a new test case.

What is the suggested work flow for using a baseline? Shouldn’t be duplicating test cases be prevented if it’s not necessary?

Greetings,
Chris


#2

Hello Chris,

Thanks for your posting and great to hear that you like the new version!

Baselines are basically test suite copies and form a different “branch”. This is useful to have separate test case repositories for different versions of your products to test.

For example, if you finished working on version 3.0 of your product and then start working on a 3.1, you usually add new test cases for the 3.1. In order to have a set of test cases that applies to 3.0 and 3.0 only, you would create a new, separate baseline for version 3.0 (from Master) before adding new cases to Master. This is useful if you plan to maintain and test 3.0 and 3.1 in parallel and may need to release updates for the 3.0 branch (e.g. 3.0.1, 3.0.2). The Master branch/suite would always refer to the latest/current version.

I’m happy to provide additional details on this as needed.

Regards,
Tobias


#3

Tobais,

How do you envision this feature being used with automated tests? We were planning on associating our Java test cases with case ids using Java annotations. The same java test cases would be executed against master and baselines in most cases.

When we create a new baseline from master, all cases get new ids in testrail. This would mean we’ll have to associated multiple IDs with each java test case as well (which isn’t practical and won’t scale). So, how do you suggest associating cases in testrail with test cases in Java?

Or, is there a way to filter cases using a custom field? This would allow us to create a new custom field and use that as a unique key for test cases and then use that to look up the case id in test rail. Based on the current test rail api documentation, I don’t think filtering cases by custom fields is possible, but it’ll be great if you can confirm that and/or give some direction on how we can get this working.


#4

Hello,

Thanks for your posting. You could indeed look into using a (static) custom field for this. While filtering for custom fields on the API level is (not yet) available, you can simply use get_cases to read the full list of test cases for a baseline/master suite. You can the filter the response on your side based on your custom field and to find out the mapping between your custom field and the test case IDs (which you need for subsequent API calls, e.g. to add a test result or to create a test run with a custom case selection). Would this work for you?

Regards,
Tobias


#5

Thanks Tobias. After looking into this further, this is indeed the approach I am prototyping right now. My concern is if this will scale with the ~5000 automated tests that we have. The response for get_cases is going to be quite large as I didn’t see any pagination in API docs. I will test that out soon as well. Do you have a white paper on how best to use TestRail at scale? e.g. when we are dealing with ~10000 tests with half or more automated.

Thanks again.


#6

Hello,

Thanks for your reply. There shouldn’t be any scalability issues for the numbers you mentioned. You would just need to make sure to read the list of cases just once and not for every test result, for example. Instead of calling get_cases for the entire repository/test suite, you can also use get_sections to get the list of valid sections first and then call get_cases section wise. This would definitely scale better for very large repositories/test suite.

While we currently don’t have a performance/scalability guide for the API, we are happy to help in case you have any further questions. We usually recommend using bulk API methods if available (e.g., add_results_for_cases vs. add_result_for_case) as well as caching results where possible and where it make sense (to limit the load and improve the performance on your side).

I hope this helps and I look forward to your reply.

Regards,
Tobias


#7

That’s great to know. I am planning on calling get_cases once in the beginning (might use sections eventually if needed), though I have to use add_result_for_case so that we can see a realtime status of test execution.

Right now, for ~10000 tests, the get_cases call returns in about in about a minute, so we definitely have to do some caching. :slight_smile: Eventually having more APIs (or options within existing APIs) would definitely help.

I have put together some ideas around this on https://github.com/nullin/testrail-integration (very much a work-in-progress). It might come in handy for some folks though.

Thanks again Tobias.


#8

Good to hear that this helps and thanks for sharing your code, this looks really useful! For 10000 test results, I would highly recommend using the mass-add add_results_for_cases API method as this would be a lot more efficient and faster. If you don’t want to submit all test results at once, you could also send this in chunks (e.g. section-wise or 100 results per API call). I’m happy to provide additional details as needed.

Regards,
Tobias


#9

I know this is a really old post but we are having the same issue with baseline’s and automation. Have to go back through hundreds of test cases and manually add a unique number for automation to be scalable is quite a hassle and not really worth all the effort. Is there a better way for automation and baselines to use a unique ID and not have to create a custom field and update every test case?


#10

Hi!

Do you need baselines/different branches for your automated tests or do you just target the latest master repository/branch? If you also use baselines, one way to make this more flexible is to use a separate configuration file with the automated tests -> case ID mappings and you can have different configuration files per TestRail baselines/branch (which you can simply download as a CSV export). I know this is approach is used by other customers and we would be less work than manually updating the case IDs in your code. You would just need to implement a way to read the case ID from the configuration file instead of managing them as part of your code. Would this work for you?

Cheers,
Tobias


#11

Tobias - This does not work for our API automation. Are your other customers only doing UI automation?
Currently we use a tool called Runscope for our API tests.


#12

Hi!

I don’t have experience with the implementation details of Runscope, but do you necessarily need baselines? TestRail also has advanced versioning features even without using baselines, mainly through closing & archiving test plans & runs. If you don’t need to track different branches of your case repository (and run automated tests for those branches), then the default single-suite mode should also work fine and the case IDs would be static.

Cheers,
Tobias