Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Milestones and Baselines


#1

Hi there,

I am planning the structure of our tests within TestRail, and have a quick question. I want to arrange my test cases by product, and have it broken down by release and sprint which I was planning on doing via Milestones and sub milestones. We also have a number of automated tests, and development would like to be able to specify which branch these are run on. Is that best to be done by a custom field?

Although I’ve read the notes on baselining, I’m at a loss currently to see how I’d use that feature in the set up I’ve outlined above.

Thanks and Regards,
Ellie


#2

Hi Ellie,

Thanks for your post! You can use the built-in Type field to mark your tests as an automated test (or other types of tests as well), and this would also support adding custom types as well under Administration > Customizations. In regards to testing a specific branch/version of the test case, this would be difficult to manage with a custom field as any change you make to the test case would be reflected in all active test runs that use the test case. So changing the version for one test via a custom field would change it on any other active test run. If you won’t be testing these simultaneously in different branches/version then this could still work, but usually if you plan to test multiple branches/versions in parallel (especially if you do this for an extended time and often) then we would recommend using the baselines project mode.

Baselines allow you to maintain multiple versions/branches of your test case repository and so you wouldn’t need to worry about a change to a test case effecting all others (as you would just execute the test in its baseline). The master baseline would act as the most recent version, and all other baselines would act as older branches/versions of your test cases. When creating your test runs you would specify which baseline(version/branch) you want to execute and add your cases from that baseline. This might be ideal if your team will be testing this way often.

Hope this helps!

Regards,
Marco


#3

Which API do we use to create a baseline under a particular project ?


#4

Hi Madhura,

Thanks for your reply! I answered this in a previous thread here:

Regards,
Marco


#5

Hi Marco,

Thank you for your reply and apologies for the delay in getting back to you. I don’t think that I would be changing the test case each time, I think it would be more specifying it in the results. I’ve looked at adding a custom field to the results, which I think is going to be the best way to do this, but have another question. I would like very much that the build is available as a drop down, and that when a build is triggered in our build system (we use Jenkins) that the list is updated via the api. Is this something which would be possible? Or give the way the field is managed, will it cope with an ever growing list, or would it be better to have it as a plain text field? I’m thinking that it probably wouldn’t be possible to keep the list maintained, as would it affect the results if we removed older builds periodically to make it easier to use for manual testers? I would have to have a different field per project to ensure only the correct build options are presented.

With regards to baselines, I understand now that this is to ensure integrity between test case and test runs, so we know exactly what version of the test case was used for that particular run. Is that correct? The problem with that it only seems to be available for single repositories? I like the ability to have different suites within a project. For example, I am structuring TestRail by having a project per product, but one of our products is made up of two distinctly separate applications, which are governed by the same release, so I want to be able to report on both at the same time. I’d also like a different suite for our automated tests.

Kind Regards,
Ellie