Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

How to keep different versions of test cases


#1

It seems that when you edit a test case - those changes filter down to previous test runs and results. Is that expected?

e.g.

  1. You have a Test Suite with a Test Case.
  2. You then create and run various Test Runs with that Test Case.
  3. Then something changes so you need to make a slight change to Test Case.
  4. However, that change should only apply from now on - for future Test Runs.
  5. But, it seems the old tests from 2) are also updated with the change at 3).

So it seems there is no concept of “versions” of Test Cases. How would you suggest we manage Test Suites which have Test Cases that will naturally evolve over time, but still keep historical executed tests at their original state?
I hope that makes sense:)

Many thanks.

BG


#2

Hello BG,

Thanks for your message. Your observations are correct, except for the fact that you can close test runs and ‘freeze’ the associated test case details for completed runs. I understand that this feature might not be that easy to discover and we might need to make this more prominent. Here’s how it works:

  • You can close a test run once it’s completed. To close a run, click the lock icon in TestRail’s toolbar on a run page.
  • When a run is closed, TestRail copies all associated test case details internally and thus ‘freezes’ the test cases.
  • When you now update a test case, the test case data of a closed run is not affected by the change.

I hope this helps!

Thanks,
Dennis


#3

That solution is exactly what I was after:)

Many thanks.

(I did see the lock icon - but didn’t know exactly what it did. A full user guide (when available) would be very useful if it covers stuff like this.)


#4

Great! I will make that it’s added to the online help (likely Tips and Tricks section) in one of the next updates.

Regards,
Dennis


#6

Hello,

I have a quite similar problem like BG. But it goes further. We do the following to create Baselinse:

  1. We have a test Suite with a lot of Cases (C#).
  2. We create a Test Run (R#)
  3. We create Test Cases (T#) using only a subset of our Cases.
  4. We are running the Test Cases several times.
  5. At the end we close the Test Run to “‘freeze’ the associated test case details for completed runs”.

Following these step, we can Baseline our Test Cases respectively our Cases, since they are related to each other.

Now, to run the same(!) Test Run again, I tried to open the Test Run and click onto “Rerun” to create the new Test Run again.

My Problem: The Test Cases of the newly created Test Run references to the current version of the related Cases and not, how we need it, to the ‘frozen’ version.

Is there a possibility to handle this problem?
We already tried two workarounds, which would fit our needs. In both of them we tried to create a new Test Suite with the ‘frozen’ Test Cases, but neither worked:

  1. Export Test Run XML File and import it back into a new Suite. The Import of the Sections works, but the Cases are not imported.

  2. Creating a “Baseline” Suite only with the Cases used in the Test Run. This attempt failed due to the effort needed copying the related (Test) Cases. We didn’t find a way to select and copy the Test Cases out of the Test Run nor did we find a possibility to filter for Cases related to a specific Test Run.

Is there an easy solution for our need (Creating a Test Run with the “frozen” Test Cases)?

Thanks,

cac


#7

Hi cac,

Thanks for your reply! The Rerun feature would always use the current version of the test case that sits in your test case repository, so if you’ve made changes to this since the run was closed, then the test cases wouldn’t be the same for the new run. That said, you could export the closed test run via XML, and format this for the import by converting the following tags in the XML file:

<run>  ->  <suite>
</run>  ->  </suite>
<test>  ->  <case>
</test>  ->  </case>
<tests>  ->  <cases>
</tests>  ->  </cases>

You can leave the rest of the XML file as-is, and the importer should recognize this for import. If you run into any issues with this, feel free to email us the XML export file directly to contact@gurock.com and we’d be happy to help. Going forward, it would be recommended to keep your baselines for as long as you need them (and you can even freeze your baselines to ensure that test cases aren’t changed when editing the baseline). Hope this helps!

Regards,
Marco


#8

Going forward, it would be recommended to keep your baselines for as
long as you need them (and you can even freeze your baselines to ensure
that test cases aren’t changed when editing the baseline). Hope this
helps!

Hello Marco

Thank you for your answer, it helped. Now I have a question about the second part of the answer. What do you mean with “baseline” in the two sentences above? Are you taking about closed Test Runs, or are you taking about copies of cases inside a separate Suite?

Regards,
cac


#9

Hi cac,

Thanks for your reply! You had mentioned baselines in your previous response above:

So I had assumed you were using the baseline project suite mode for your project. With the baseline suite mode, you would have multiple baselines per project that act as separate test suites which would be copies of your master baseline (for use when testing older versions in parallel to your current version). You can learn more about baselines on our website here (Suite Modes and Baselines section):

https://blog.gurock.com/introducing-testrail-4-0/

Hope this helps!

Regards,
Marco


#10

Hello Marco,

This is an interesting workaround.

I tried it, but got an error saying “No tag found.” when attempting to import the XML file.
Here’s what I did. Could you kindly suggest me if there is anything wrong?

  1. Exported a closed test run via XML
  2. Edited the XML file by converting the tags as you suggest
  3. Saved it leaving the rest of the file as-is
  4. Created a new test suite
  5. Attempted “Import from XML” by specifying the XML file and choosing “Add new test cases” option

Thanks,
Yoshi


#11

Hi Yoshi,

Thanks for your reply! This would be the correct process, however this wouldn’t work when exporting an entire test plan (as this would cause an error similar to the one you mentioned). You would need to be sure to export just the test run itself. If you were doing this for a test run export and still received the same error, I would recommend sending a copy of the original XML export file over to contact@gurock.com so that we can have a look and attempt to convert it with this method. Hope this helps!

Regards,
Marco