Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Do you have a suggestion for how to use baselines with automated tests?


#1

Do you have a suggestion for how to use baselines with automated tests?

A little background… Each of our automated tests is linked to a case id so that we can write results. We also discover and execute the tests (using a custom Python nose plugin) that should be run in a session using the case ids.

The problem is that the case id is updated when creating a baseline. I also did not see a field in the baseline case that indicates what the master case is.

Thanks!


#2

i think the suggestion once was to create a custom field to have an unique id for this, which is not really applicable (i think) for bigger suites.

It would be great if the baseline cases could have a link (the id) of the original test case or so, as you will always have different baselines in different runs and therefore the match of test-case-id and test-run-id could still be used


#3

Exactly. I would like to be able to call an API method that takes a baseline id + a case id and get the corresponding baseline case id.


#4

Hi all,

Thanks for your feedback! We are happy to look into this and I agree that this makes sense. I would currently recommend keeping track of the (maybe multiple) case IDs as part of your test automation but being able to use the master ID is certainly easier and we will make sure to review this for a future TestRail version.

Cheers,
Tobias


#5

Adding additional feedback here to agree. Having the test ID change with each version makes linking automated tests messy. With every new release the automated tests reference outdated test IDs. Finding the correct test from an automated run is difficult when you have thousands of tests.

Maybe there could be another identify that is shared between tests across versions…


#6

Hi @ezraw,

Thanks for your feedback. An alternative would be to use a custom field (with a UUID, for example) which wouldn’t change when you create a new baseline. If you are not using baselines (or copies of test suites), you can use the test case IDs (C###) as they will stay the same even if you create multiple test runs over time (only the test ID T### would change/be dynamic).

I hope this helps!

Cheers,
Tobias


#7

I understand that workaround for baselines. I would object to it being acceptable however. :slight_smile:

It diminishes the usability of a test case management system to have to manually manage test IDs. To do this, we’d need to 1) manually manage the ID creation to avoid clashes, or 2) build something custom to generate unique IDs or 3) use something generally human-unfriendly like a UUID.

Even if those were acceptable workarounds, it also introduces an problem in that if we have a duplicate test case for every release, we’ll eventually end up with tens of thousands of cases. So, let’s say I have a failed test and need to look it up. If I search by UUID, I might get 10 matching results (one for each release) with no way to tell which version I need to look at. I have to manually click into each one to find the right release :rage: This happens now and we are only tracking 3 releases in Testrail. See screenshot below. Which test case is which? It’s not a scalable solution.

I would suggest that versioning of a test case across releases is critical feature for a test case management tool. The baseline feature was nice try, but honestly it seems like a workaround to meet a marketing/sales need rather than something that a software team could use with thousands of test cases and an automated test runner.

As a customer who truly enjoys Testrail and uses it nearly every day, I would ask that you refactor how you handle test case versions and look to source control system for design patterns. For example, I should be able to just switch to a version of a suite/projects and see/search all the cases as it existed at that time without having to click and open each one to see the version.


#8

Thanks for your feedback! In most cases, you wouldn’t need to use baselines and the standard versioning via closing test runs & plans would be sufficient. Baselines would only be needed if you constantly work on multiple versions/branches in parallel and closing test runs/plans would usually be sufficient for everything else. The advantage would be that the case IDs wouldn’t change and you would always use the same case repository. A good overview of versioning with TestRail can also be found on our blog:

https://blog.gurock.com/test-management-test-case-versioning/

Cheers,
Tobias


#9

Thanks for listening! We’ll consider dropping baselines and moving to test runs.

But here is another problem with the design of baselines for your consideration anyway! :slight_smile: Every time we create a new baseline, each ticket in Jira gets another linked test from the new baseline. So eventually we would have a list of tests in each Jira ticket.

This is the exact same issue we have with baselines with managing our automated test IDs. There is no versioned, “master” test.

While this is unlikely to happen when working on an active release, when I went back to look at an older ticket I noticed this issue. Any suggestions for managing this?


#10

Hi @ezraw,

This would be expected as each baseline has its own set of test cases and each test case is a separate link to the JIRA issue. While this may look confusing at first in JIRA (with the duplicate cases), it allows you to quickly jump to the different baselines (versions) and not just to the master case/repository. TestRail will also show all related test results in the TestRail: Results panel for all baselines. The Results panel shows the most recent results first so you will always see the most relevant results (and can use pagination to look for previous results).

I hope this helps!

Cheers,
Tobias